Batch, recording and settings

Run several actions in one call, record and replay trajectories, and configuration and maintenance.

Served by cua-driver mcp; see MCP tools for every tool.

Batch and script tools#

Run several action tools, or a script that drives them, in one call.

Use run_steps for a fixed list of actions you have already decided. Use run_script when the flow needs logic between actions (a loop over rows, a branch on what the app shows, a retry): it runs a JavaScript program over the driver's own tools in one call and returns only its result. Every driver call either tool makes passes the same session, permission and approval checks as a direct call.

run_script runs in a QuickJS sandbox with no filesystem, network, process, timers or environment. Limits: timeout_ms (default 30 s, max 120 s), max_calls (default 100, max 500), a 32 MiB heap and a 64 KiB script. It is on by default. Whoever runs the driver turns it off with CUA_DRIVER_DISABLE_RUN_SCRIPT=1 or "disable_run_script": true in ~/.cua-driver/config.json (CUA_DRIVER_EXPERIMENTAL_SCRIPT=0 and "experimental_script": false also work); set_config cannot change it.

run_steps#

The default way to act: run one or more action tools in ONE call, in order, stop at the first failure, and (with observe) get what changed in the same response. Use it whenever you know the next action(s) and would otherwise re-read the window after them: fill three fields, click, and see the result in one call instead of five. Even a single step plus observe:{} replaces an action call followed by a get_window_state call. Each step is {tool, args} where tool is one of click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, move_cursor and args are exactly that tool's arguments. pid and window_id (or app/window) on run_steps itself are the default window of every step that names none. Reads (get_window_state, zoom) do not run between steps: one there is skipped; a get_window_state last is the observe, a zoom last runs after it. Every step goes through the same session, permission and approval checks as a direct call; a batch grants nothing extra. All steps are validated before the first one runs, so a malformed step 4 changes nothing. Element tokens and element_index values come from a get_window_state read before the batch; a step that changes the UI can invalidate later element targets, so put element-targeted actions before the actions that reshuffle the window, or target later steps by role/name. A token that went stale is retried once on the element with the same role and label, if exactly one matches.

Returns per-step status (ok or the error message) and, when observe is given, ONE bounded get_window_state read after the last executed step (after a failure caused by a dialog holding focus, that dialog is read, even without observe). A step refused because its app ignores background input says how to re-send it; foreground_fallback:true retries such steps in the foreground once (the window is activated). That read is a since:"latest" diff by default: only the rows that changed since your last read of the window, plus a new snapshot_id. Without observe nothing is read. The batch uses one session: steps may omit session or repeat the batch's. delay_ms pauses between steps (max 2000). At most 64 steps; delivery_mode on run_steps is the default for its steps.

TARGET BY NAME (no earlier read needed, so one batch can cross screens): a step can be written {"click": {...args}} as well as {tool, args}, and its args may name the target instead of carrying a token: role (button, textfield, link, checkbox, menuitem, ...; AX/UIA/AT-SPI names also work), name (the element's label; exact match wins over substring, case-insensitive), nth (0-based, when several match), and app (app name or bundle id) and/or window (title substring) instead of pid/window_id. The batch reads the window fresh, waits up to 3 s (step timeout_ms, max 10000) for the element to appear, and acts on it; with only app it uses that app's frontmost window, so a dialog that opened on top is found. Later steps inherit the window, so name app once. The first lookup in a window also reads it once for observe, so the end diff shows what the batch changed even in a window you never read. Several matches fail with the candidates listed; add nth or a fuller name.

WAITS AND CHECKS: a step may add wait_for (checked before its action) and expect (one check or up to 4, checked after it); a step may also be only a check. A check is {role?, name?, text?, gone?, value?, value_contains?, enabled?, selected?, app?, window?, timeout_ms?}: text matches any row of the tree, static text included; gone:true waits for it to disappear. Checks poll until they hold or time out (wait_for 5 s, expect 2 s, max 10 s), and a check that does not hold fails the step with what was found instead. The report says which step failed, in which phase (wait_for, find, action, expect) and why. Whole batch: at most 60 s.

Example (fill a form, submit, confirm the dialog, check the result): {"steps":[{"set_value":{"app":"Safari","role":"textfield","name":"Email","value":"ada@example.com"}},{"click":{"role":"button","name":"Submit"},"expect":[{"role":"button","name":"Confirm"}]},{"click":{"role":"button","name":"Confirm"},"expect":[{"name":"Confirm","gone":true},{"text":"Thanks"}]}],"observe":{}} Batching rules: get_guide({topic:"batching"}).

Effect: destructive. Platforms: macOS, Linux, Windows.

Example arguments

{"steps":[{}]}

run_script#

Run a JavaScript program that drives apps through Cua Driver, in ONE call: loops, conditions, reads and checks between actions, and only the result comes back. Use it when the flow needs logic (a loop over rows, a branch on what the app shows, a retry); use run_steps for a fixed list of steps.

The script is the body of an async function: use await, and return a JSON value. It runs in a sandbox with no filesystem, network, process, timers or environment; the only API is cua:

• await cua.getApp("TextEdit" | "com.apple.TextEdit" | {pid, windowId}) → app (follows the app's frontmost window unless a windowId is given)
• await cua.launch(nameOrBundleId) → app; cua.listApps(), cua.listWindows({pid?}), cua.sleep(ms), cua.computer.target
• app.getState(opts?) → get_window_state result (tree_markdown or tree_diff, snapshot_id; pass get_window_state options such as {since:"latest"})
• app.query({role?, name?, text?}) → matching elements across the app's windows {element_token, role, label, value, enabled, selected, frame}, then matching static-text rows {role, label, value, text, display_only} (read text such as a status line from their value)
• app.waitFor({role?, name?, text?, gone?, value?, timeout_ms?}) and app.verify(predicate | predicates) (verify_state)
• app.click(t), doubleClick(t), rightClick(t), typeText(text, t?), pressKey("return" | "cmd+s", t?), hotkey(["cmd","s"]), scroll(t, direction, amount?), setValue(t, value), drag([x,y], [x,y]), zoom([x1,y1,x2,y2]) where a target t is {role, name, nth?} (found fresh, waits up to 3 s), an element_token string, a row number from this app's last getState(), or [x, y] window pixels.
• cua.call(tool, args) for any other allowed tool: click, double_click, right_click, set_value, type_text, press_key, hotkey, scroll, drag, launch_app, list_apps, list_windows, get_window_state, verify_state, zoom. A failed driver call throws a CuaError (catch it to recover). Every call goes through the same session, permission and approval checks as a direct call.

Limits: timeout_ms wall time (default 30000, max 120000), max_calls driver calls (default 100, max 500), 32 MiB heap. Returns the return value, console output, and a per-call log; an error names the script line, the call and the reason.

Example: const app = await cua.getApp("Calculator"); for (const key of ["7", "multiply", "6", "equals"]) await app.click({role: "button", name: key}); const [display] = await app.query({role: "statictext"}); return display.value; Batching rules: get_guide({topic:"batching"}).

Effect: destructive. Platforms: macOS, Linux, Windows.

Example arguments

{"script":"<script>"}

Recording tools#

Record and replay trajectories.

start_recording#

Start trajectory recording for the calling session. Each action-tool invocation (click, right_click, scroll, type_text, press_key, hotkey, set_value) from that session writes a turn folder under output_dir. Without session, every call on the same connection is recorded, including named-session calls. Other connections and session lifecycle calls (start_session / end_session) are not recorded. CLI recordings started with cua-driver recording start are daemon-wide.

Each turn folder holds:

• before_state.json / after_state.json: application AX/UIA/AT-SPI state immediately before and after the action.
• before.png / after.png: target-window screenshots immediately before and after the action.
• evidence.json: capture status and a stable classification when an expected artifact could not be captured.
• app_state.json: post-action AX/UIA snapshot for the target pid.
• screenshot.png: compatibility alias of after.png.
• action.json: tool name, full input arguments, result summary, result-error flag, pid, click point (when applicable), ISO-8601 timestamp.
• click.png: for dispatched click-family actions only, before.png with a red marker at the click point. A call refused before target resolution is explicitly not applicable instead.

The per-turn accessibility walk is bounded like get_window_state: state_timeout_ms (default 1000) caps each before/after walk, a walk that runs out of budget records the PARTIAL tree, and evidence.json carries truncated, truncation_reason, nodes_visited, nodes_pending and timeout_ms for that phase. A provider that stops answering is abandoned after the budget plus a short grace and the phase is classified state_capture_timeout. Actions refused before dispatch (for example an unknown or expired capture_id) skip the state walk; their state is classified not_applicable / action_refused_before_dispatch. Pass include_accessibility_tree: false to record screenshots and actions without state.

Turn folders are named turn-00001/, turn-00002/, etc. Turn numbering restarts at 1 each time recording is (re-)started.

Video is off by default. Pass record_video: true to also capture the main display to <output_dir>/recording.mp4 (H.264 / 30 fps) for the lifetime of the session. The recording is torn down automatically when the MCP client disconnects.

macOS uses native ScreenCaptureKit (daemon-owned SCStream + SCRecordingOutput) under the daemon's Screen Recording grant, with no ffmpeg subprocess. Requires macOS 15.0+. On macOS 26 (Tahoe), the first direct capture can also show a one-time consent asking to let Cua Driver bypass the system private window picker and directly access your screen and audio; choose Allow, or run cua-driver permissions grant beforehand to answer it up front. The recorder captures screen video only and does not enable system-audio capture.

Windows + Linux use an ffmpeg subprocess (gdigrab / x11grab + libx264). Requires ffmpeg on PATH (winget install Gyan.FFmpeg / apt install ffmpeg); when ffmpeg is missing or fails on startup the per-turn capture (screenshots + action.json) still runs and the session's last_error field carries the diagnostic.

State persists for the life of the daemon; a restart resets to disabled with no on-disk state. Call stop_recording to disable + finalize the mp4.

Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.

Example arguments

{"output_dir":"<output_dir>"}

stop_recording#

Stop trajectory recording. Disables further per-turn capture and, when video was enabled, gracefully terminates the ffmpeg subprocess so the mp4's moov atom is finalized (the file is playable). Calling stop on an already-stopped session is a no-op. The response carries last_video_path pointing at the finalized mp4 (when video was on).

A manual stop_recording is unconditional: it stops whatever recording is active regardless of which session started it. Ownership-scoped teardown (so one client disconnecting can't stop a recording a later client started) is handled by the registry's session_end lifecycle hook, not by this tool.

Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.

get_recording_state#

Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled.

Pure read-only.

Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.

replay_trajectory#

Replay a recorded trajectory by re-invoking every turn's tool call in lexical order. dir must point at a directory previously written by start_recording. Each turn-NNNNN/ is parsed for action.json, and the recorded tool is called with its recorded arguments via the same dispatch path an MCP / CLI call uses.

Caveats:

• Element-token actions (click({pid, element_token}) etc.) will fail because element tokens are per-snapshot and don't survive across sessions. Pixel clicks (click({pid, x, y})) and all keyboard tools replay cleanly. Failures are reported but don't stop replay unless stop_on_error is true.
• get_window_state and other read-only tools are NOT currently recorded, so replays do not re-populate the per-(pid, window_id) element cache.
• If recording is ENABLED while replay runs, the replay itself is recorded into the currently configured output directory. That's deliberate: recording a replay against a new build and diffing the two trajectories is the regression-test workflow.

Effect: destructive. Platforms: macOS, Linux, Windows.

Example arguments

{"dir":"<dir>"}

install_ffmpeg#

Install the ffmpeg binary used by start_recording's video capture (Linux/Windows; macOS records natively and needs no ffmpeg). Two-step and confirmed: called without confirm it only REPORTS the exact install command for this platform's package manager; pass confirm: true to actually run it. No-op if ffmpeg is already on PATH. ffmpeg is run as a separate process, never linked into the driver.

Effect: destructive. Platforms: macOS, Linux, Windows.

Configuration and maintenance tools#

Configuration, permissions, health, updates, extensions and on-demand guides.

get_config#

Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.

Parameters on every platform

set_config#

Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.

Parameters on every platform

check_permissions#

Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.

Parameters on every platform

health_report#

Single-call end-to-end driver diagnostics. Designed to let downstream consumers ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and platform capability status. On macOS, prompt-capable direct capture is deliberately skipped. For standalone CuaDriver, use cua-driver permissions grant to verify it explicitly. For an embedded or linked driver, use the host's permission flow for the app or executable macOS attributes this process's permission request to; the standalone command does not verify the host's capture readiness. cua-driver owns the health model; consumers stay thin.

Input: all optional: { "include": ["<check_name>", ...], // run only these "skip": ["<check_name>", ...] // skip these } If both are given, include wins.

Canonical check names: macOS : binary_version, platform_supported, session_active, bundle_identity, tcc_accessibility, tcc_screen_recording, ax_capability, screen_capture_capability, windowserver_connection Windows: binary_version, platform_supported, session_active, ax_capability (via UIA), screen_capture_capability (via DXGI) Linux : binary_version, platform_supported, session_active, ax_capability (via AT-SPI), screen_capture_capability (via X11)

Output: stable contract, schema_version="1": { "schema_version": "1", "platform": "darwin" | "win32" | "linux", "driver_version": "<semver>", "overall": "ok" | "degraded" | "failed", "checks": [ { "name": "<one of the canonical names above>", "status": "pass" | "fail" | "skip", "message": "<one-line summary, always present>", "hint": "<remediation step, present when status=fail>", "data": { /* check-specific structured fields */ } }, ... ] }

overall rules:

• ok : every non-skipped check passes
• degraded: at least one non-core check fails (binary is still usable)
• failed : any core check fails (binary_version, platform_supported, session_active)

Stability: schema_version="1" is the contract. Future breaking changes will be "2". Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names.

Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.

check_for_update#

Check the saved stable/nightly Cua Driver channel for a release on GitHub. Returns current and selected channels, current and latest versions, an update_available boolean, the install one-liner, and the release notes URL. Read-only: never installs. Pacman-owned Linux executables return package-manager guidance without checking GitHub. Mirror of cua-driver check-update --json.

Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.

install_extension#

Preview or install one Driver-managed optional extension. The first call without confirm returns the exact signed artifact, destination, license, source, and trust plan without mutation. Re-call with confirm=true to perform that exact verified installation.

Effect: destructive. Platforms: macOS, Linux, Windows.

Example arguments

{"name":"perception"}

get_guide#

Read Cua Driver guidance on demand, taken from the guide bundled with this driver (no skill install needed). Topics: overview, drag, browser, permissions, batching, menus. Call with no topic to list them. Read-only.

Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.