Computer use: how AI agents operate computers

Understand computer use, computer-use agents and models, their observe-decide-act loop, and how they differ from APIs, RPA, and browser automation.

Computer use is the ability of an AI agent to operate a real computer and complete work across applications. A computer-use agent can inspect the computer's current state, choose an action, observe the result, and continue until it reaches a goal. It can work through code, structured tools, or the same graphical interfaces people use.

The term originally described agents that interpreted screenshots and controlled a graphical user interface (GUI) with mouse and keyboard actions. Cua uses Computer-Use 2.0 for a broader model: an agent chooses among writing and running code, calling structured tools and APIs, and driving the same graphical interface a person would use. The screenshot-and-click loop remains important as one action surface inside the larger system.

Computer use is therefore a capability rather than a single product, model, or interaction method. The defining property is that the agent can act on computer state, observe what changed, and continue across the boundaries between applications and interfaces.

Computer-use agents and computer-use models#

A computer-use model interprets an observation and proposes an action. Depending on the model, the observation may include screenshots, accessibility information, text, or structured application state. The proposed action may be a mouse movement, a keystroke, a tool call, or a program to run.

A computer-use agent is the running system around that model. It supplies the computer, tools, memory, instructions, permissions, and feedback loop needed to turn proposed actions into completed work. It also decides when to stop, retry, ask for confirmation, or move to a different action surface.

The distinction matters because a capable model alone does not provide a reliable agent. The surrounding system still has to deliver input to the intended application, preserve state between steps, recover from changed interfaces, and constrain consequential actions.

Three action surfaces#

On the coding surface, the agent writes and runs code as the action itself. This works especially well when the task is text-native, when the same operation must run more than once, or when a small program can cover a larger body of files and state than a human would want to touch by hand.

On the tool-use surface, the agent makes structured calls to tools, functions, APIs, and MCP servers. The action has a typed shape, with named inputs and defined outputs, so the agent can ask an external system to perform a specific operation without inventing a script or trying to reach the same control through the screen.

On the UI automation surface, the agent clicks, types, scrolls, and presses keys on the interface a person would use. This surface reaches controls in native applications, web applications, and legacy software that expose no useful API. In those systems, the GUI is often the only contract available to the user.

These surfaces are complementary. Computer use does not mean clicking through every task: the agent should use the most direct interface that preserves the required behavior and state.

Examples of computer use#

Computer use is useful when work spans interfaces that were designed for people rather than for one programmatic integration. Examples include:

testing a desktop application by installing it, changing settings, and verifying the visible result;
collecting information from a signed-in web application that has no suitable API;
moving between a terminal, code editor, browser, and operating-system dialog during one development task;
completing a business workflow across several applications while preserving the user's existing session;
reproducing a support issue on a real operating system and retaining the visual evidence needed to diagnose it.

In each case, the agent must connect actions across changing application state. A script or API may still cover part of the work, while computer use covers the boundaries that those interfaces cannot reach.

Choosing a surface#

A capable agent moves between these surfaces during a single task, choosing the one that fits the next piece of work. Code and structured tool calls are usually the right fit when the job is repeatable or text-heavy, especially when the system already exposes a clear interface. UI automation becomes the better choice when the work depends on visual state or when the application is unfamiliar; it is also the fallback when there is no useful API and the path exists only inside the interface.

The skill is the judgment behind that choice. An agent that edits a file with code, checks account state through an API, and then changes a setting in a desktop app is still doing one computer task; it is simply using different action surfaces as the task moves through different kinds of state.

Computer use is most valuable when a task crosses those boundaries. Browser automation can be enough for a stable web workflow, and an API is usually best for a well-defined integration. A computer-use agent becomes useful when the task also depends on visual state, native applications, signed-in sessions, or software that exposes no suitable API.

How computer use differs from other automation#

Computer use overlaps with APIs, browser automation, and robotic process automation (RPA), but each approach has a different interface and tradeoff.

Approach	Best fit	Main limitation
API or structured tool	Stable, well-defined operations with typed inputs and outputs	Cannot reach behavior the API does not expose
Browser automation	Repeatable workflows contained within web pages	Does not cover native applications or operating-system UI
Traditional RPA	Known, predefined business processes	Often depends on fixed selectors, coordinates, and workflow branches
Computer-use agent	Adaptive work across applications and interface types	Requires careful observation, recovery, permissions, and verification

The approaches can be combined. An agent may query an API for structured data, use code to transform it, and then enter the result into a native application. Computer use describes the whole task even when only one part requires direct GUI control.

The UI automation loop#

The observe, decide, act loop belongs specifically to the UI automation surface. Observation comes from a screenshot, an accessibility tree, or both, which gives the model either the pixels a person would see or the structured information exposed by the operating system. From there, grounding turns a target such as a button, field, menu item, or selected region into on-screen coordinates that input events can hit.

Planning carries the task across changing interface states. A click may open a dialog, a page may reflow after loading, or an application may show an error that changes the next useful action, so the model has to keep track of the goal while the computer responds. Frontier models such as Claude can handle understanding, grounding, and planning together in one call, while grounding-specialist models such as UI-TARS and Moondream can help when coordinate accuracy is the limiting factor.

The current wave of screenshot-driven computer use accelerated in October 2024, when Anthropic introduced an agent that operated a GUI through screenshots and input events. Through 2025, coding agents were increasingly recognized as computer-use agents too, with CoAct-1 making the connection explicit. Cua uses Computer-Use 2.0 as shorthand for that wider model. Francesco Bonacci traces that arc in A Story of Computer-Use.

Where Cua fits#

You bring the agent, which already handles coding and tool-use on its own. Cua gives that agent the UI automation surface and a real computer to act in, so the same task can move from code to tools to the graphical interface when the application requires it. SDKs, MCP, and process hosting are different ways to connect the agent to that surface.

Cua Driver drives the GUI of a real machine you already have, whether that machine runs macOS, Windows, or Linux. It is the right shape when the agent needs to work with local applications, signed-in accounts, existing files, or machine state that already lives on that computer.

Cua Sandbox is a fresh isolated computer running on local Docker or VM infrastructure where the agent can run code and drive the GUI together. It is a full computer with all three action surfaces available inside the same environment rather than a remote desktop, which matters when the task needs both programmatic work and visible interaction without changing the host desktop. The agent brings the model and its reasoning; Cua provides the computer where that reasoning can turn into action.

Safety and isolation#

Computer-use agents can affect the same files, accounts, and applications as a person, so the environment and permissions are part of the system design. Prefer the least privilege needed for the task, require confirmation for consequential actions, and keep sensitive or untrusted work isolated from a personal computer.

Cua Driver exposes permission policies for restricting what an agent can do on an existing machine. Cua Sandbox provides an isolated computer whose lifecycle and credentials can be scoped to the task.