Recording tools
Record and replay trajectories.
Record and replay trajectories.
| Tool | Description | Platforms |
|---|---|---|
start_recording | Start trajectory recording for the calling session. | macOS, Linux, Windows |
stop_recording | Stop trajectory recording. | macOS, Linux, Windows |
get_recording_state | Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. | macOS, Linux, Windows |
replay_trajectory | Replay a recorded trajectory by re-invoking every turn's tool call in lexical order. | macOS, Linux, Windows |
install_ffmpeg | Install the ffmpeg binary used by start_recording's video capture (Linux/Windows; macOS records natively and needs no ffmpeg). | macOS, Linux, Windows |
Served by cua-driver mcp; see MCP tools for every tool.
start_recording#Start trajectory recording for the calling session. Each action-tool invocation (click, right_click, scroll, type_text, press_key, hotkey, set_value) from that session writes a turn folder under output_dir. Without session, every call on the same connection is recorded, including named-session calls. Other connections and session lifecycle calls (start_session / end_session) are not recorded. CLI recordings started with cua-driver recording start are daemon-wide.
Each turn folder holds:
before_state.json / after_state.json: application AX/UIA/AT-SPI state immediately before and after the action.before.png / after.png: target-window screenshots immediately before and after the action.evidence.json: capture status and a stable classification when an expected artifact could not be captured.app_state.json: post-action AX/UIA snapshot for the target pid.screenshot.png: compatibility alias of after.png.action.json: tool name, full input arguments, result summary, result-error flag, pid, click point (when applicable), ISO-8601 timestamp.click.png: for dispatched click-family actions only, before.png with a red marker at the click point. A call refused before target resolution is explicitly not applicable instead.The per-turn accessibility walk is bounded like get_window_state: state_timeout_ms (default 1000) caps each before/after walk, a walk that runs out of budget records the PARTIAL tree, and evidence.json carries truncated, truncation_reason, nodes_visited, nodes_pending and timeout_ms for that phase. A provider that stops answering is abandoned after the budget plus a short grace and the phase is classified state_capture_timeout. Actions refused before dispatch (for example an unknown or expired capture_id) skip the state walk; their state is classified not_applicable / action_refused_before_dispatch. Pass include_accessibility_tree: false to record screenshots and actions without state.
Turn folders are named turn-00001/, turn-00002/, etc. Turn numbering restarts at 1 each time recording is (re-)started.
Video is off by default. Pass record_video: true to also capture the main display to <output_dir>/recording.mp4 (H.264 / 30 fps) for the lifetime of the session. The recording is torn down automatically when the MCP client disconnects.
macOS uses native ScreenCaptureKit (daemon-owned SCStream + SCRecordingOutput) under the daemon's Screen Recording grant, with no ffmpeg subprocess. Requires macOS 15.0+. On macOS 26 (Tahoe), the first direct capture can also show a one-time consent asking to let Cua Driver bypass the system private window picker and directly access your screen and audio; choose Allow, or run cua-driver permissions grant beforehand to answer it up front. The recorder captures screen video only and does not enable system-audio capture.
Windows + Linux use an ffmpeg subprocess (gdigrab / x11grab + libx264). Requires ffmpeg on PATH (winget install Gyan.FFmpeg / apt install ffmpeg); when ffmpeg is missing or fails on startup the per-turn capture (screenshots + action.json) still runs and the session's last_error field carries the diagnostic.
State persists for the life of the daemon; a restart resets to disabled with no on-disk state. Call stop_recording to disable + finalize the mp4.
Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.
| Parameter | Type | Default | Description |
|---|---|---|---|
include_accessibility_tree | boolean | true | Default true. Set false to skip the per-turn before/after accessibility walks entirely; screenshots, click markers and action.json are still recorded and state is classified state_capture_disabled. |
output_dir | string | required | Absolute or ~-rooted directory where turn folders and (when enabled) the video file are written. |
record_video | boolean | Capture the main display to <output_dir>/recording.mp4. Default: false. Set to true to also capture the main display to recording.mp4 (otherwise only the per-turn screenshots + JSON are recorded). On macOS this uses native ScreenCaptureKit (macOS 15.0+); macOS 26 can show a one-time direct screen-capture consent on first use (see cua-driver permissions grant). On Windows + Linux it requires ffmpeg on PATH. | |
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. | |
state_timeout_ms | integer | 1000 | Wall-clock budget in milliseconds for EACH per-turn before/after accessibility walk (same default and bounds as get_window_state's timeout_ms). A walk that runs out records the partial tree and marks it truncated in evidence.json. Range: 100 to 120000. |
Example arguments
{"output_dir":"<output_dir>"}stop_recording#Effect: mutating, idempotent. Platforms: macOS, Linux, Windows.
Stop trajectory recording. Disables further per-turn capture and, when video was enabled, gracefully terminates the ffmpeg subprocess so the mp4's moov atom is finalized (the file is playable). Calling stop on an already-stopped session is a no-op. The response carries last_video_path pointing at the finalized mp4 (when video was on).
A manual stop_recording is unconditional: it stops whatever recording is active regardless of which session started it. Ownership-scoped teardown (so one client disconnecting can't stop a recording a later client started) is handled by the registry's session_end lifecycle hook, not by this tool.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
get_recording_state#Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled.
Pure read-only.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
replay_trajectory#Effect: destructive. Platforms: macOS, Linux, Windows.
Replay a recorded trajectory by re-invoking every turn's tool call in lexical order. dir must point at a directory previously written by start_recording. Each turn-NNNNN/ is parsed for action.json, and the recorded tool is called with its recorded arguments via the same dispatch path an MCP / CLI call uses.
Caveats:
click({pid, element_token}) etc.) will fail because element tokens are per-snapshot and don't survive across sessions. Pixel clicks (click({pid, x, y})) and all keyboard tools replay cleanly. Failures are reported but don't stop replay unless stop_on_error is true.get_window_state and other read-only tools are NOT currently recorded, so replays do not re-populate the per-(pid, window_id) element cache.macOS parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
Parameters on every platform
| Parameter | Type | Default | Description |
|---|---|---|---|
delay_ms | integer | Milliseconds to sleep between turns, for human-observable pacing. Default 500. Range: 0 to 10000. | |
dir | string | required | Trajectory directory previously written by start_recording. Absolute or ~-rooted. |
stop_on_error | boolean | Stop replay on the first tool-call error. Default true: set false to best-effort through the full trajectory. |
Example arguments
{"dir":"<dir>"}install_ffmpeg#Effect: destructive. Platforms: macOS, Linux, Windows.
Install the ffmpeg binary used by start_recording's video capture (Linux/Windows; macOS records natively and needs no ffmpeg). Two-step and confirmed: called without confirm it only REPORTS the exact install command for this platform's package manager; pass confirm: true to actually run it. No-op if ffmpeg is already on PATH. ffmpeg is run as a separate process, never linked into the driver.
macOS parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
Parameters on every platform
| Parameter | Type | Default | Description |
|---|---|---|---|
confirm | boolean | Run the install command. Without it, only the planned command is reported. |