How Cua Driver works

Observation, targets, the background-first action ladder, verification, choosing an integration, who owns the runtime, and the optional perception extension.

Cua Driver is the UI layer of a computer-use agent. The harness runs the model and the loop; Cua Driver turns each proposed step into an observation or an action on a real desktop, and reports what happened.

model -> agent harness -> Cua Driver (MCP, CLI, or SDK) -> OS accessibility + input -> apps

Observe: tree and pixels together#

get_window_state(pid, window_id) returns the window's accessibility tree and a screenshot in one call. The tree says what is actionable (roles, labels, an element_token per element); the screenshot says which one, and shows what the tree omits or gets wrong. Pass include_screenshot: false to skip the image when you only need to re-index. get_desktop_state captures the whole primary display.

When neither the tree nor typed browser state exposes the target, the optional perception extension can parse that same screenshot into labelled regions.

Act: one exact target per call#

Every input tool (click, type_text, press_key, hotkey, scroll, drag, move_cursor) names its target:

The target is chosen per call; nothing is locked into a session. A malformed or ambiguous target fails before anything is sent.

The action ladder#

Background delivery means the driver does not raise the window, move the real pointer, or switch the frontmost app. The agent gets its own cursor overlay.

1. Element, background. Act by element_token: UI Automation Invoke on Windows, AXPerformAction on macOS, AT-SPI actions on Linux. This is the only rung the driver can verify itself.
2. Pixel, background. Act by x, y read from the same screenshot. For keyboard tools, x, y first clicks to give the field renderer focus, which is how to type into Chromium and Electron inputs.
3. Page. For browser tabs, switch to the browser tools, which act on the DOM over CDP without focus.
4. Foreground. Retry only that action with delivery_mode: "foreground": the window is raised, the input lands, and the previous app is restored. Needed for games, canvas apps such as Blender, and focus-polling apps.

Each response says whether to climb:

A delivered event is not an applied change: Electron, Catalyst, and web content can echo a write they did not apply, so the driver reports those as unverifiable and recommends the next rung. After anything short of confirmed, observe again and check. Refusals carry a code and a recovery hint; stale_element_token means snapshot again, not that the element route is broken.

Platform mechanisms#

On Wayland, knowing where a window is does not change which window the compositor's seat sends input to, so the driver refuses rather than risk typing into the wrong app. A structured background_unavailable is part of the contract, not a failure to hide.

Choose an integration#

Pick by who owns the runtime: the process that holds the OS permissions, sessions and browser connections.

Inside a Cua Sandbox you do not install Cua Driver: the sandbox's guest service bundles its tools.

Processes and sessions#

The runtime owns permissions, sessions, element caches, recordings, and browser connections. Where it lives depends on how you connect:

The daemon exists for identity. On macOS, Accessibility and Screen Recording grants belong to CuaDriver.app (com.trycua.driver), not to whatever terminal spawned a cua-driver process. On Windows, an SSH process in Session 0 cannot see the desktop, so a daemon in the user's session does the work.

Each MCP connection or SDK runtime gets one implicit session, reused by unnamed calls and cleaned up when the connection closes, on end_session, or after five idle minutes. After idle expiry, the next unnamed call starts a new implicit session, and element tokens and snapshots from the ended one stay invalid, so observe again before acting. An ended named session refuses calls with session_ended until start_session restarts it. Pass a session label on each call for a readable name and a cursor badge; the label is not a credential. One-shot cua-driver call commands each get a disposable session. All sessions still share one screen, keyboard, and focus, and native input is admitted one action at a time.

The optional HTTP MCP listener is off unless CUA_DRIVER_RS_MCP_HTTP_PORT is set, and then requires CUA_DRIVER_RS_MCP_HTTP_TOKEN (32 to 4096 characters, sent as a bearer token). Windows named pipes accept only the daemon owner's user.

Permissions sit below every transport#

CLI, MCP, SDK, and daemon calls all pass the same authorization check inside the runtime. The launcher fixes the mode, manifest, and grants at startup; an agent cannot widen them. See Set permissions.

The optional perception extension#

Canvas apps, custom-drawn views, games, remote desktops, and some web pages have no useful accessibility tree or page structure, so an agent has nothing to address by role or label. The optional cua-perception extension gives those screens a structure: it parses one screenshot into labelled regions (text, icons, controls) on your own machine, without sending the screenshot to a remote model. Cua Driver then acts on exactly the region the agent chose.

It never replaces the tree and never captures on its own. Without it installed, parse_visual_regions returns not_installed, and agents continue with the accessibility tree or page structure. To run the loop step by step, see Parse visual regions.

Every action is bound to one screenshot#

A region only means something on the screen it was found on, so Cua Driver ties each perception-based action to its capture:

Agents receive region bounds and text, never screenshot bytes, and Cua Driver keeps the authority to capture and to act. A stale or reused screen is refused instead of guessed at.

What it contains#

Each release packs local models and a CPU runtime, so parsing needs no GPU and no network access.

Releases are published as cua-perception-v<version> GitHub releases for macOS arm64, Linux x64, and Windows x64. Each target has a signed catalog, the archive, an SBOM, provenance, a runtime contract, and an install check, plus one SHA256SUMS. Cua Driver verifies the signed catalog before it installs anything, and a verified install reports publisher-verified.

Install and speed#

Capture, parsing, Cua Driver startup, and update checks never install the extension. You inspect a catalog first, then install, check, update, or remove the extension with cua-driver extension commands. A running Driver picks up an install or removal without a restart.

Parsing runs on the CPU. Measured per screenshot on the canonical harnesses: about 2 to 2.5 seconds on macOS, 3.5 to 4 seconds on Linux, and 8 to 9 seconds on Windows. Prefer accessibility or page targets when they exist, and keep perception for surfaces that expose nothing addressable.

How it is validated#

A tool returning ok is not evidence. Each supported behavior is a cell in one Rust catalog (action × element/pixel × background/foreground × window/desktop × surface), run against source-built fixture apps in a real desktop session. A cell passes only when state the app or desktop owns changed, and, for background cells, when focus, z-order, the real cursor, and the foreground app stayed untouched. An exact refusal also passes; a silent success does not. Runs keep per-cell video and logs tied to the source commit. Current results are in Platform support.