Sharing a recent project. Today during lunch, I suddenly realized an issue: my home garage door doesn't have a magnetic contact sensor installed. We've had a few instances where we forgot to close the garage door at night. Because there was no sensor, our security system couldn't trigger an alarm, leaving the garage door open all night—which is extremely unsafe.
I happened to have a thumb-sized microcontroller on hand, an ESP32-CAM (Figure 1), equipped with Wi-Fi, a camera, and a decent amount of compute (dual-core 240 MHz). So I wondered: could I turn this microcontroller into a sensor itself? By using photo capture paired with an on-chip deep neural network, it could classify whether the garage door is open or closed. Whenever I send it an HTTP request, it tells me right away if the garage door is shut.
This idea might sound crazy at first glance, but the actual difficulty is lower than it seems. The reason is that we aren't trying to build a general-purpose classifier (like "hotdog or not hotdog") that needs to handle endless edge cases. What we want here is a hyper-overfitted classifier. It only needs to solve the problem for my specific garage; it doesn't matter how weird anyone else's garage looks. Once the device position is fixed, it never moves, and the garage door remains in the exact same spot. It doesn't even need high generalization intelligence—it just needs to memorize what the door looks like open vs. closed under different lighting conditions. Overfitting is precisely what I want.
So I explained this idea to AI and assigned the task to GPT-5.6 SOL, feeding it past security camera footage recorded from nearly the same angle as training data. Every hour or two, I checked in on its progress for two to five minutes, giving next-step guidance via voice recognition. By evening, I discovered it had accomplished the following workflow:
What AI Did
First, it annotated a batch of initial data on its own. Soon after, it identified and confirmed a core problem: the garage door is closed the vast majority of the time. To train an effective classifier, the key was finding enough positive examples where the garage door was open. So it annotated some initial samples (using a local Vision LLM, then double-checking them itself using a contact sheet, see Figure 2. On a side note: I wonder if many photographers today even know what a contact sheet is!). Next, it trained an initial Vision Transformer, then ran this neural network across hundreds of thousands of video frames from my past few days of security footage to mine more instances of open garage doors. It then retrained the model on the newly augmented data. Repeating this loop a few times, it gathered sufficient and balanced data to build a complete training and testing dataset.
Building on this dataset, it moved on to training and deploying to the microcontroller. Given the extremely constrained compute on the chip, it selected MobileNetV4—a pure convolutional neural network with roughly 900,000 parameters. (Considering today's LLMs routinely reach tens or hundreds of billions of parameters, a model with a few hundred thousand parameters feels delightfully compact.) It first ran an opportunity sizing test using open weights, discovering that end-to-end inference took only ~200 milliseconds (compared to 400 milliseconds just to capture a photo). Thermal rise was minimal, making it completely viable for sporadic inference. It then fine-tuned MobileNetV4 on the newly collected training data to produce a custom model.
The fine-tuned model performed remarkably well, but it couldn't be deployed directly onto the microcontroller yet. For performance and memory efficiency, it needed to be quantized to int8. After quantizing, the AI noticed a major problem: model accuracy degraded significantly, with over 30% of classification outputs flipping incorrectly. Seeing this as unacceptable, it adopted Quantization-Aware Training (QAT) at my suggestion to retrain the model (specifically choosing LSQ-based QAT). This dramatically reduced quantization error, bringing the AP drop down to just 8 percentage points (Figure 3).
Next, it built a dedicated testing firmware, flashed it to the microcontroller, and ran an HTTP server alongside the quantized model on-chip. Whenever we sent an image via HTTP POST, the microcontroller handled preprocessing, inference, and postprocessing, returning the final verdict. Using this setup, we further validated hundreds of test images, confirming that accuracy, RAM usage, latency, and thermal limits all met our requirements (see Figure 4 for sample classification outputs). Finally, the AI wrote the production firmware: capturing a photo every 10 minutes for inference, then enabling Wi-Fi to push the result back to my Mac.
It also gave me a helpful heads-up: because the microcontroller spends most of its time in Deep Sleep where current draw is microscopic, some power banks with smart auto-off protection might misinterpret this low load and cut off power, bricking the device. So it suggested I test across different power banks. I eventually found one that doesn't auto-shut off and used it for a temporary deployment, which is currently collecting real-world data for long-term precision evaluation.
Looking back, what did AI actually accomplish during those 10 hours from noon to 10 PM? This was no textbook toy demo. Its impressive feat was identifying a set of distinct engineering challenges unique to this project and actually solving them—such as severe data imbalance, scarce negative samples, and severe quantization loss. Of course, not to be overly modest, leadership direction was also critical—I contributed a modest share in keeping the overall trajectory on track. But any single one of these hurdles would present a real bump for even a senior scientist. Honestly, if a senior scientist could complete this entire pipeline from scratch in a month—getting a neural network actually running on a chip—I'd consider that quite impressive. Yet today, I hold a camera-equipped, inference-capable microcontroller in my hand after just 10 hours, having spent virtually no manual labor beyond checking in every hour or two to offer high-level direction.
What We Did
In sharing this, I'm certainly not implying that AI is about to replace everyone's job, or that anyone can trivially achieve this result with AI. Let me highlight the two most important contributions I made in this process:
First, aggressively building an environment for AI that enables autonomous self-iteration. A conventional hardware workflow often looks like this: the AI writes firmware and flashes it, a human manually pushes a button to trigger capture and inference, the microcontroller outputs results to a display, and the human feeds observations back to the AI for the next iteration. In that model, the human remains tethered to the computer. That's why in my microcontroller development Skill, I repeatedly emphasize to the AI that it must close the loop for automated self-development. For instance, after flashing firmware, the AI sends a serial command; the chip automatically takes a picture and computes inference, returning complete telemetry—including temperature, latency, and prediction results—via serial. The AI ingests this telemetry, diagnoses issues, refines its code, and re-flashes autonomously. Going through the trouble to write a special version of firmware that humans don't need and only AI uses would be a total waste of time in traditional development. But for autonomous AI development, this is the single most critical point, allowing me to leverage five minutes of guidance to unlock hours of high-intensity AI execution.
Second, providing core technical guidance at critical decision nodes. I anticipated where the major pitfalls lay and gave the AI explicit strategies up front. For example, knowing that extreme data imbalance was the primary bottleneck for this garage classifier, I highlighted it right from the start so the AI didn't waste time going down dead ends. The solution strategy also came directly from me: use a local Vision LLM for initial labeling, then review the results itself. When quantization loss proved too severe, I immediately suggested switching to QAT. Crucially, 80% to 90% of my cognitive effort was focused purely on these high-leverage technical decisions—without spending days on "influence without authority," holding endless meetings as an IC, promising deliverables to leadership, or managing team alignment. Being able to spend my time entirely on technical work I'm both good at and enjoy is what brought me the most joy.
That's why over the past few years, my view on how humans should adapt in the AI era hasn't changed: programmers used to be like rowers on a human-powered boat—whether in university training or corporate promotions, the metric was always how fast you could paddle. But AI came in like a steam engine and completely flipped the table: no matter how fast you paddle or how little you eat, you can never outrun a machine operating 24x7. But this doesn't mean humans are obsolete. The gap between individuals has shifted from "who paddles faster" to "who can keep AI working autonomously for the longest stretches without constantly stopping or being slowed down by human intervention." This is an entirely new skill set, and one that often conflicts with old paddling muscle memory. Only by shedding ingrained habits and embracing new AI-native paradigms can we achieve order-of-magnitude leaps in delivery speed. In this new paradigm, "where to steer" matters far more than "how fast to paddle"—though how to build the intuition to know where to steer is a story for another day.