In October 2024, Anthropic announced Computer Use alongside Claude 3.5 Sonnet: Claude could see your screen, identify interface elements, and operate your mouse and keyboard to complete tasks. Fill in expense reports in Salesforce, export design drafts from Figma, schedule meetings in your browser. It started as a developer API beta, opened up to Pro and Max subscribers via Claude Cowork in March 2026, and is now available as an API on Amazon Bedrock, Google Vertex AI, and Azure Foundry.
To train an agent like this, you need to feed it enormous volumes of paired data — “screenshot → what to click next.” Intuitively this seems straightforward: just screen recording, right? But where the data comes from, what format it takes, and whether there’s enough of it — these three things together determine whether an agent graduates from a demo to a product.
An Anthropic patent (U.S. 12,437,238), granted in October 2025 (filed October 2024), answers this question from the side. What it protects is not how the AI operates a computer — that’s called inference, and everyone is doing it. What it protects is the training data collection and generation pipeline.
Before Anthropic started building, training a computer-operating agent wasn’t without data. Academia accumulated a substantial body of UI grounding datasets over the past two years:
Structurally, these datasets share the same format as what training a Computer Use agent requires: a screenshot paired with an action target. The problem isn’t data structure. It’s three dimensions.
First, application coverage breadth. GUI-360 and OSWorld cover software types in the tens. An agent product faces hundreds or thousands — from modern web apps to legacy enterprise systems. Every additional software interface type adds a new dimension of layout patterns, interaction modes, and visual features the model needs to learn. Academic datasets can’t fill this long tail.
Second, the floor on trajectory quality. 1.2 million trajectories sounds like a lot, but most come from a small set of operation types: opening menus, filling forms, saving files. The interruptions that characterize real work — popups blocking flows, loading delays, form validation errors — appear far less frequently. Academic data is collected by researchers for specific benchmark tasks, carrying an inherent sampling bias toward clean, controlled scenarios.
Third, temporal continuity. Real operations are complete trajectories — “open software → click this → fill that → wait for load → see popup → close popup → continue filling.” While academic datasets offer semantic labels (“click the submit button” rather than just coordinates), their samples are predominantly single-step or short trajectories, lacking the sequential dependencies and interruption recovery contexts of long-horizon operations.
Stack these three gaps together: academic datasets can train a prototype that passes a benchmark. They can’t train a product that works reliably across arbitrary software.
Anthropic’s patent (“Generation of agentic trajectories for training artificial intelligence agents to automate multimodal interface task workflows”) describes a pipeline that transforms human operations into training data. The real dividing line from academic datasets isn’t data volume — it’s that every step in this pipeline attaches reasoning information to raw actions. Academic dataset samples are “see this interface → do this action,” a set of static mappings. The samples this pipeline produces are “see this interface → understand the current state → decide what to do next → do this action,” a reasoning chain.
The pipeline has three stages. Each contributes a layer of reasoning.
Stage one: interception. An intermediary layer sits between the user and the software interface. The user operates normally — clicking buttons, filling forms, scrolling pages — and this layer transparently records every step. But it’s not just a screen recorder. Before each action, it captures the interface state (screenshot + accessibility metadata + text content). It records what the user did. Then it captures the new interface state after the action.
The most interesting capability of this interception layer is in Claim 5: users can attach thought annotations — “I’m clicking this button because it’s usually in the bottom-right corner,” or “I should pick the third option here because the first two are grayed out.” These annotations capture the human reasoning process behind interface decisions, and they are directly encoded into the training data. For the model, this kind of data isn’t about “imitating this click.” It’s about “understanding why you click here in this state.”
Stage two: translation. The raw intercepted actions — “(342, 157) click” — are fed into a multimodal transformer model to understand. The model combines the interface screenshot and the action context to infer the user’s true intent, producing a semantic command: “identified a button element with text ‘Submit’ within bounding box (330, 150, 400, 170), execute click.” The key here isn’t coordinate conversion — it’s having the model reason about the intent behind the action. The user didn’t randomly click a pixel; the user wanted to submit a form. The translation model uses its own reasoning capability to fill in the semantic layer that raw actions lack.
Stage three: synthetic expansion. After interception and translation, a real operation trajectory becomes a training sample with a reasoning chain. Anthropic then uses a stronger model to perform synthetic expansion on this sample — given the same pre-action screenshot, the stronger model reasons through and generates multiple plausible variants of “what you could do upon seeing this interface.” One real trajectory expands into dozens of training samples, each containing a complete “see interface → reason → act” chain, covering different action choices, different error-handling paths, and different multi-step combinations.
Stacked together, the three stages do the same thing: turn raw operations into reasoning data. Stage one gets reasoning annotations from humans. Stage two uses a model to fill in the inferred intent behind actions. Stage three uses an even stronger model to generate more reasoning variants. RPA-style screen recording can only tell you “what the user did.” This pipeline tells you “why the user did it” — and that’s exactly what training an agent that can independently operate software requires.
Back to the opening question: where does Computer Use’s training data come from? The answer isn’t just “we recorded lots of users” — academia has been doing that too. Anthropic’s difference is that it engineered every stage of a three-stage pipeline and protected the combination with a patent.
When everyone can build a “see screen, click button” agent — Computer Use, OpenAI Operator, Codex desktop plugin all converging on the same capability — who can cover more software and complete more complex tasks more reliably comes down to training data quality and scale, not model benchmark scores. And training data quality and scale come down to whether you’ve built a pipeline that can continuously, cheaply, and at scale collect and expand real operation data.
The patent protects the combination of steps in this pipeline. Competitors can route around it — skip the interception layer and use screen recordings with manual labeling, skip synthetic expansion and scale up real data collection. But the time cost and scale constraints of those alternatives are Anthropic’s first-mover window.