MCP Tools

Specification for every MCP tool exposed by Cua Driver

Cua Driver exposes 36 MCP tools through a single stdio server (cua-driver mcp). Every tool is also callable from the shell as cua-driver call <tool-name> '<JSON-args>'.

Tool names are snake_case. Responses are MCP CallTool.Result envelopes: a text content block prefixed with a ✅ summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them.

Inspection tools

list_apps

List macOS apps, both currently running and installed-but-not-running, with per-app state flags.

Per-record fields:

Only apps with NSApplicationActivationPolicyRegular are included. Installed apps are scanned from /Applications, /Applications/Utilities, ~/Applications, /System/Applications, and /System/Applications/Utilities.

Arguments: none.

list_windows

List all layer-0 top-level windows currently known to WindowServer, including off-screen windows (minimized, on another Space, hidden-launched).

Per-record fields: window_id, pid, app_name, title, bounds (x/y/width/height, top-left origin), z_index (higher = frontmost), is_on_screen, on_current_space.

get_window_state

Walk a running app's AX tree and return a Markdown rendering of its UI, tagging every actionable element with [element_index N]. Also captures a PNG screenshot of the specified window.

Invariant: call get_window_state once per turn per (pid, window_id) before any element-indexed action. The index map is replaced by the next snapshot.

{"pid": 844, "window_id": 10725}

get_accessibility_tree

Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. No TCC grants required.

Arguments: none.

get_screen_size

Return the logical size of the main display in points plus its backing scale factor. Agents click in points; Retina displays have scale_factor 2.0. Requires no TCC permissions.

Arguments: none.

get_cursor_position

Return the current mouse cursor position in screen points (origin top-left).

Arguments: none.

get_config

Return the current cua-driver-rs configuration.

Arguments: none.

get_recording_state

Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled. Pure read-only.

Arguments: none.

get_agent_cursor_state

Return the current state of this session's agent cursor: position, config (color, icon, label, size, opacity), enabled flag. Scoped to the cursor the call resolves to (precedence: explicit cursor_id > session identity > "default").

Action tools

launch_app

Launch a macOS app in the background. The target does NOT come to the foreground.

Returns the launched app's pid, bundle_id, name, and a windows array (same shape as list_windows). When the focus-steal demotion check ran, the response also includes self_activation_suppressed: bool.

kill_app

Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). Unsaved state is lost.

{"pid": 844}

bring_to_front

Activate a window so subsequent input tools with dispatch:"foreground" land on it without a per-call SetForegroundWindow flash. Windows-only. On macOS this tool returns an error. On Linux this tool stubs out.

{"pid": 844}

click

Left-click against a target pid.

Two addressing modes:

• Element index (element_index + window_id): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. Cache is scoped per (pid, window_id) and is replaced by the next snapshot.
• Pixel (x, y): CGEvent path. Synthesizes mouse events posted to pid. Needs a visible on-screen window.

{"pid": 844}

double_click

Double-click at (x, y) or on an AX element identified by element_index + window_id.

AX path: performs AXOpen when the element advertises it; otherwise resolves the element's on-screen center and falls back to a pixel double-click.

Pixel path: two down/up pairs ~80 ms apart.

{"pid": 844}

right_click

Right-click against a target pid.

Two addressing modes:

• Element index (element_index + window_id): performs AXShowMenu. Pure AX RPC, works on backgrounded/hidden windows.
• Pixel (x, y): synthesizes rightMouseDown/rightMouseUp CGEvent pair posted to the pid.

Exactly one of element_index or (x AND y) must be provided. pid always required.

{"pid": 844}

drag

Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels.

{"from_x": 100, "from_y": 200, "pid": 844, "to_x": 300, "to_y": 400}

type_text

Insert text into the target pid via AXSetAttribute(kAXSelectedText). Works for standard Cocoa text fields and text views. No keystrokes are synthesized. For Chromium/Electron inputs that don't implement kAXSelectedText, the tool falls back to CGEvent character synthesis automatically.

{"pid": 844, "text": "hello"}

press_key

Press and release a single key, delivered to the target pid via CGEventPostToPid. No focus steal.

Three delivery paths:

• window_id + element_index: focuses the AX element first, then posts via the auth-message path (Chromium-safe).
• window_id only (no element_index): NSMenu path. Briefly activates the window, posts without the auth envelope so IOHIDPostEvent fires and NSApplication.sendEvent: dispatches NSMenu key equivalents. Restores prior frontmost immediately.
• No window_id: standard auth-message path.

Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1–f12, plus any letter or digit.

{"key": "return", "pid": 844}

hotkey

Press a combination of keys simultaneously. The combo is posted directly to the target pid's event queue; the target does NOT need to be frontmost.

Two delivery paths:

• Default (no window_id): auth-message envelope. Chromium/Electron apps accept the keystrokes as trusted live input on macOS 14+.
• With window_id: NSMenu path. Briefly activates the target WindowServer-frontmost, posts without the auth envelope so IOHIDPostEvent fires and NSMenu key equivalents dispatch (e.g. Cmd+Z undo, Cmd+W close). Restores prior frontmost immediately.

Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Order: modifiers first, one non-modifier last.

{"keys": ["cmd", "c"], "pid": 844}

set_value

Set a value on a UI element.

Two modes:

• AXPopUpButton / select dropdown: finds the child option whose title or value matches value (case-insensitive) and AXPresses it directly. No native popup menu is opened.
• All other elements: writes AXValue directly (sliders, steppers, date pickers, native text fields that expose a settable AXValue).

{"element_index": 14, "pid": 844, "value": "42", "window_id": 10725}

scroll

Scroll the target pid's focused region by synthesized keystrokes.

Mapping: by='page' → PageDown/PageUp × amount; by='line' → DownArrow/UpArrow × amount. Horizontal variants use Left/Right arrow keys.

{"direction": "down", "pid": 844}

move_cursor

Move the agent cursor overlay to (x, y). Does NOT move the real mouse cursor.

{"x": 100, "y": 200}

zoom

Capture a cropped JPEG of a window region (x1, y1)–(x2, y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide.

After a zoom, pass from_zoom=true to click/type_text to auto-translate coordinates back to full-window space.

{"window_id": 10725, "x1": 100, "y1": 200, "x2": 400, "y2": 500}

Browser tools

page

Interact with the browser page loaded in a running app. Supports Chrome, Brave, Edge, Safari (via AppleScript on macOS), Electron apps (via CDP), Chromium/Firefox on Windows (via UIA for read; CDP for execute_javascript when --remote-debugging-port is set), and WKWebView/Tauri/AT-SPI fallbacks.

Actions:

{"action": "get_text"}

Recording tools

start_recording

Start trajectory recording. Every subsequent action-tool invocation writes a turn folder under output_dir.

Turn folder contents:

Turn folders are named turn-00001/, turn-00002/, etc. Numbering restarts at 1 each time recording is (re-)started.

Video recording:

• Default: off. Pass record_video: true to also capture the main display to <output_dir>/recording.mp4 (H.264, 30 fps).
• On macOS: uses native ScreenCaptureKit (requires macOS 15.0+, no extra TCC prompt).
• On Windows + Linux: uses an ffmpeg subprocess (gdigrab / x11grab + libx264). Requires ffmpeg on PATH.
• Recording stops automatically when the session ends on the daemon-proxy path.

{"output_dir": "~/cua-trajectories/demo1"}

stop_recording

Stop trajectory recording. When video was enabled, gracefully terminates the ffmpeg subprocess so the mp4's moov atom is finalized (file is playable). Calling stop on an already-stopped session is a no-op. The response carries last_video_path pointing at the finalized mp4 when video was on.

A manual stop_recording is unconditional. It stops whatever recording is active regardless of which session started it.

Arguments: none.

replay_trajectory

Replay a recorded trajectory by re-invoking every turn's tool call in lexical order. dir must point at a directory previously written by start_recording.

Caveats:

• Element-indexed actions fail because element indices are per-snapshot. Pixel clicks and keyboard tools replay cleanly.
• get_window_state and other read-only tools are not recorded, so replays do not re-populate the element cache.
• When recording is enabled while replay runs, the replay itself is recorded into the currently configured output directory.

{"dir": "~/cua-trajectories/demo1"}

Configuration tools

set_config

Update cua-driver-rs configuration. Changes to capture_mode and max_image_dimension take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart.

Per-session isolation (daemon-proxy path). On the daemon-proxy path, set_config writes an in-memory, session-scoped override that does not touch the global DriverConfig or persist to disk. get_config and capture tools resolve effective values as: call-arg > session override > global default. The override is dropped automatically when the client disconnects. Only the anonymous path (cua-driver config set CLI, one-shot cua-driver call) writes the persisted global default.

start_session

Declare a session: a named, color-coded identity for the current agent run. The agent cursor, per-session config, and recording all key on the session id, and it follows the run across apps/windows. Idempotent (re-calling refreshes the idle-TTL). End it with end_session or let the idle-TTL reclaim it.

end_session

End a session declared with start_session: removes its agent cursor, stops any recording it owns, and clears its per-session config. Idempotent.

set_agent_cursor_enabled

Show or hide the agent cursor overlay for a cursor instance. The overlay is on by default for each MCP session.

{"enabled": false}

set_agent_cursor_motion

Configure the visual appearance and motion curve of an agent cursor instance.

Appearance parameters:

Motion curve parameters:

set_agent_cursor_style

Update the visual style of the agent cursor overlay.

Maintenance tools

check_permissions

Report TCC permission status for Accessibility and Screen Recording. By default also raises the system permission dialogs for any missing grants (Apple's request APIs are no-ops when the grant is already active). Pass {"prompt": false} for a read-only status check.

check_for_update

Check whether a newer cua-driver-rs release is available on GitHub. Returns the current and latest versions, an update_available boolean, the install one-liner, and the release notes URL. Read-only. Mirror of cua-driver check-update --json.

Arguments: none.

install_ffmpeg

Install ffmpeg, which the recording video backend shells out to. Two-step and confirmed: called without confirm it only reports the command it would run (a read-only preview); pass confirm: true to actually run it. No-op if ffmpeg is already on PATH.

Arguments: