# Cua Documentation — full text > Cua is an open-source platform for computer-use automation: AI agents that operate real computers by clicking, typing, and reading the screen and accessibility tree. Drive a machine you already have with Cua Driver, provision hosted desktops with Cloud Fleets and the Sandbox SDK, or use Lume as local Apple Silicon/macOS VM infrastructure underneath your own workflows. Source: https://cua.ai/docs # Cua documentation Open-source computer-use automation for real machines and isolated desktops. Cua is an open-source MIT-licensed platform ([github.com/trycua/cua](https://github.com/trycua/cua)) for **computer-use automation**: letting AI agents operate computers by clicking, typing, and reading the screen and accessibility tree. What we refer to as **Computer-Use 2.0** treats the GUI as one tool surface in an agent loop, alongside code, shell, files, and APIs. Use Cloud Fleets to provision isolated cloud desktops, Cua Driver to operate a machine you already have, Lume to manage local macOS VMs, or Cua-Bench to evaluate computer-use agents. For live service availability and incident updates, see [Cua Status](https://status.cua.ai). ## Choose your path [Your first Cloud Fleet]() Provision a Cloud Fleet, claim an isolated Linux desktop, run a command, save a screenshot, and delete the cloud resources when you finish. [Drive a real app — Cua Driver]() Run a background driver on macOS, Windows, or Linux so an agent can operate native GUI apps you already have without bringing them to the foreground or moving the real mouse. It speaks MCP over stdio and is also a plain CLI. [Create a local macOS VM with Lume]() Install Lume on an Apple Silicon Mac, create a vanilla Tahoe guest, and connect over SSH. [Build your first Cua-Bench task]() Create and verify a simulated computer-use task before evaluating agents. Local sandboxes and Cloud Fleets share the [Sandbox SDK](), but their credentials, images, connections, and cleanup behavior differ. Start with [How sandboxes work]() for the shared model and [Runtime support]() for the differences. The Lume tutorial uses the Lume CLI directly, not the Sandbox SDK. Building Cua Driver into your own product? Start with [Choose a Cua Driver integration]() to decide whether MCP, the SDK, or an app-hosted service should own the runtime. ## Cua Driver in action These recordings show agents operating desktop apps while the user's active window stays in place. Each demo was recorded by the Cua team on the named platform. [Build and check a WPF app on Windows]() Claude Code builds a WPF CRM, runs it, patches the code, and checks the app again. [Fill a Linux spreadsheet over SSH]() An agent enters a monthly budget in Gnumeric on a remote machine with no attached display or GPU. ## How these docs are organised Tutorials: learn by doing. [Choose a first tutorial](). Concepts: understand how and why Cua works. Start with [Concepts](). Use Cua with: choose a model provider, agent harness, or local runtime. Start with [Use Cua with](). How-to guides: recipes for a goal. Start with [How-to guides](). Reference: precise technical description. Start with [Reference](). ## Find a tool or API - [Cloud Fleets overview](): pools, claims, and cloud capacity. - [Sandbox SDK reference](): the shared Python API and its runtime-specific contracts. - [Cua CLI reference](): the general command-line interface, with separate [authentication]() and [CLI MCP server]() references. - [Lume reference](): local VM commands. - [Connect your agent to Cua docs](): documentation lookup, distinct from controlling a desktop with Driver. --- # Cloud Fleets Find tutorials, guides, explanations, and reference for managed cloud sandboxes. Cloud Fleets provisions and manages sandbox capacity for your workloads. The Sandbox SDK represents a Fleet as a `Pool`; your code claims a sandbox from the pool and uses that guest computer. Start with [Your first Cloud Fleet]() to provision a pool, run a command, save a screenshot, and clean up the cloud resources. ## Find the right page Choose a page based on what you want to do: | Your goal | Start here | | ------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Learn through a first working example | [Your first Cloud Fleet]() | | Create reusable capacity | [Create a sandbox pool with Python]() or [Configure a sandbox pool with Terraform]() | | Control resource lifetime | [Expire pools and claims automatically]() | | Prepare a guest environment | [Prepare and reference a Fleet image]() | | Understand the model | [How sandboxes work]() and [How Fleet images work]() | | Look up APIs and limits | [Sandbox SDK reference](), [Pool reference](), and [Sandbox runtime support]() | ## Which computer runs the work? With Fleet, the SDK connects to a claimed cloud guest. Shell commands and GUI actions through that sandbox act inside the guest. With `local=True`, the SDK uses a sandbox runtime on your own hardware. See [How sandboxes work]() for the distinction between the host, guest, and control connection. Fleet manages capacity; the Sandbox SDK acquires and uses a guest. The SDK's default guest endpoint is computer-server. Cua Driver can control the desktop where it is installed and targeted, but using it inside a Fleet guest requires a compatible image and a validated connection path. Choosing a Fleet image does not by itself establish Driver integration. Check the [image and runtime limits]() before selecting your environment. ## Claims and capacity have separate lifetimes Exiting a `pool.claim()` context releases the claim. The pool can still retain warm sandboxes, and cloud resources can incur usage charges after your workload finishes. The hosted tutorial includes pool deletion; reusable pools need an explicit cleanup or expiry policy. See [Pool lifecycle]() for connection and release semantics, and [Expire pools and claims automatically]() for deadlines and their effect on active work. --- # Start here Learn Cua by following end-to-end builds. Choose the tutorial that matches where you want to work. Each tutorial follows one path from setup to a verifiable result. - [Your first Cloud Fleet]() - provision an isolated Linux desktop in the cloud and clean it up. - [Drive your first app with Cua Driver]() - operate a native app on a machine you already have. - [Create a local macOS VM with Lume]() - install Lume, boot a vanilla Tahoe guest, and connect over SSH. - [Build your first Cua-Bench task]() - create and verify a simulated computer-use task. The Fleet tutorial needs Fleet credentials and account access. The Driver tutorial acts on your existing desktop. The Lume tutorial uses its CLI on an Apple Silicon Mac; it is not a local Sandbox SDK tutorial. Each page lists its own prerequisites. Building a Python or TypeScript application with Driver? [Use Cua Driver in process]() after completing the Driver tutorial. Once you are comfortable, [How-to guides]() cover specific goals and [Reference]() has the exact APIs. --- # Your first Cloud Fleet Provision a Cloud Fleet, claim an isolated Linux desktop, run a command, save a screenshot, and clean up the cloud resources. In this tutorial, you provision a one-sandbox **Cloud Fleet** on `run.cua.ai`, claim its Linux desktop, run `uname -a`, save a screenshot, and delete the Fleet. The Cua Sandbox SDK represents a Fleet as a `Pool` and an active desktop as a claim on that pool. **Note** You need Python `>=3.11,<3.14`, [uv](https://docs.astral.sh/uv/), and Fleet credentials with permission to manage sandbox pools. **Warning** A pool with `replicas=1` keeps one cloud sandbox warm until you delete the pool. Cloud resources can incur usage charges. The script requests deletion in a `finally` block and sets a one-hour pool TTL as a fallback. Keep the local run record until cleanup is confirmed. ## 1. Authenticate with Fleet First, [create a Fleet user API key and check access](). That setup explains account access, the **API keys** page, payment-method requirements, and a read-only check that does not provision cloud resources. Complete it before continuing. The SDK connects to `https://run.cua.ai` by default. In the same shell, use the **Client ID** and **Client Secret** from your Fleet key: ```bash export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" export CUA_TOKEN_URL="https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token" unset FLEETS_TOKEN ``` `cua auth login` does not supply credentials to the Fleet SDK. If you already have a valid Fleet access token, you can set `FLEETS_TOKEN` instead; it takes precedence over the client credentials and must remain valid through cleanup. ## 2. Prepare a directory for this run Use a separate directory for the script and its output: ```bash mkdir first-cloud-fleet cd first-cloud-fleet ``` The script reserves a new, randomly named namespace before creating its pool. Reservation refuses an existing name instead of modifying an existing pool. It saves the namespace name and creation timestamp in `cloud-fleet-run.json`. Do not edit that record or reuse the namespace for other work: cleanup deletes the namespace and its resources. ## 3. Create the Fleet script Save the following script as `first_cloud_fleet.py`: ```python title="first_cloud_fleet.py" # /// script # requires-python = ">=3.11,<3.14" # dependencies = [ # "cua-sandbox==0.4.3", # "cua-fleet==0.1.14", # ] # /// import asyncio import json import os import sys from pathlib import Path from uuid import uuid4 from cua_sandbox import Image, Pool from fleet_sdk import ( CyclopsClient, CyclopsConfiguration, CyclopsCredentials, CyclopsTokenProviderConfiguration, ) IMAGE = ( "public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04" "@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57" ) RECORD = Path("cloud-fleet-run.json") def fleet_client(): configuration = dict( base_url=os.environ.get("CUA_FLEET_BASE_URL", "https://run.cua.ai"), pool_poll_interval_ms=2000, pool_poll_limit=300, claim_poll_interval_ms=2000, claim_poll_limit=300, ) token = os.environ.get("FLEETS_TOKEN") if token: return CyclopsClient.connect_with_access_token_and_native_http_client( CyclopsTokenProviderConfiguration(**configuration), token ) return CyclopsClient.connect_with_native_http_client(CyclopsConfiguration( **configuration, token_url=os.environ["CUA_TOKEN_URL"], credentials=CyclopsCredentials( os.environ["CUA_CLIENT_ID"], os.environ["CUA_CLIENT_SECRET"] ), )) async def find_namespace(client, name): # A direct lookup outside an owned namespace can return 403, even when # the resource does not exist. Use a successful account inventory instead. namespaces = await client.list_namespaces() return next((item for item in namespaces if item.name == name), None) async def delete_namespace(client, name, created_at): current = await find_namespace(client, name) if current is None: print(f"Namespace no longer in account inventory: {name}") return if not created_at or current.created_at != created_at: raise RuntimeError("Namespace identity changed; refusing cleanup.") await client.delete_namespace(name) for _ in range(60): if await find_namespace(client, name) is None: print(f"Namespace no longer in account inventory: {name}") return await asyncio.sleep(2) raise TimeoutError("Namespace is still visible. Retry cleanup after it terminates.") async def cleanup(): record = json.loads(RECORD.read_text()) if not record.get("created_at"): raise RuntimeError( "Creation was not confirmed. Check this run's name in Fleet before deleting anything." ) await delete_namespace(fleet_client(), record["name"], record["created_at"]) async def run(): pool_name = f"first-fleet-{uuid4().hex}" # Exclusive creation prevents overwriting an earlier run's cleanup record. with RECORD.open("x") as output: json.dump({"name": pool_name, "created_at": None}, output) client = fleet_client() # Only HTTP 201 confirms reservation. A collision (409) or denied request # stops here, without reconciling or deleting another namespace's resources. namespace = await client.create_namespace(pool_name) try: RECORD.write_text(json.dumps({ "name": pool_name, "created_at": namespace.created_at, })) if not namespace.created_at: raise RuntimeError("Namespace creation timestamp missing; inspect Fleet before cleanup.") print(f"Provisioning Cloud Fleet: {pool_name}") pool = await Pool.apply( Image.from_registry(IMAGE), name=pool_name, replicas=1, cpu=4, memory_mb=4096, services={"server": 8000}, ttl_seconds_after_created=3600, ) async with pool.claim( name="first-claim", service="server", time_to_start=900, ) as sandbox: print(f"Connected to sandbox: {sandbox.name}") result = await sandbox.shell.run("uname -a") if not result.success: raise RuntimeError(result.stderr) print(result.stdout.strip()) screenshot = Path("cloud-fleet.png") screenshot.write_bytes(await sandbox.screenshot()) print(f"Screenshot saved to {screenshot.resolve()}") finally: print(f"Deleting Cloud Fleet: {pool_name}") await delete_namespace(client, pool_name, namespace.created_at) if __name__ == "__main__": if sys.argv[1:] == ["run"]: asyncio.run(run()) elif sys.argv[1:] == ["cleanup"]: asyncio.run(cleanup()) else: raise SystemExit("Usage: first_cloud_fleet.py run|cleanup") ``` ## 4. Run it Run the script from the same directory. `uv` reads the inline metadata and installs `cua-sandbox==0.4.3` and its required Fleet SDK, `cua-fleet==0.1.14`, in an isolated environment: ```bash uv run first_cloud_fleet.py run ``` The first run can take several minutes while Fleet provisions the Linux sandbox. The script prints the sandbox name and Linux kernel information, then writes `cloud-fleet.png` in the current directory. When the claim block exits, the SDK releases the claimed sandbox. The `finally` block requests deletion of the reserved namespace, which includes the pool, template, and sandbox resources. It polls successful account-inventory responses until that namespace is absent. It does not interpret a `403` as absence. If cleanup fails or times out, follow the next section and check the Fleet UI for this run's resources. A deletion request alone does not prove that all resources have finished terminating. ## Clean up an interrupted run If the process is killed or loses connectivity after provisioning, the `finally` block might not reach Fleet. Restore connectivity and valid Fleet credentials for the same account, then run the deletion-only command from the same directory: ```bash uv run first_cloud_fleet.py cleanup ``` This command reads `cloud-fleet-run.json`, checks that the recorded namespace still has the same creation timestamp, and deletes it. It does not call `Pool.apply()`, claim a sandbox, or repeat the workload. It also handles a run that reserved a namespace but failed before creating a pool. Keep using the same account: absence from another account's inventory is not cleanup proof. The script confirms account-inventory absence, not an independent inspection of the underlying guest's termination. Save any remote files you need before cleanup; deletion destroys the sandboxes and their state. If the record has no creation timestamp, namespace reservation was not confirmed. This can happen after a name collision, denied request, or interrupted response. The command refuses deletion. Find the exact recorded name in the Fleet UI and verify whether this run created any resources before removing them. Do not delete an existing pool to resolve an access or billing error. Keep `cloud-fleet.png` as the local result. For another run, use a new directory so the earlier cleanup record remains available. ## What you built The Fleet SDK reserved a namespace exclusively for this run. `Pool.apply()` created a named Fleet pool and its Linux sandbox template inside it. `pool.claim()` reserved one sandbox and connected the SDK to its `server` service on port `8000`. This service is the sandbox computer server; it is not a Cua Driver MCP endpoint. The shell and screenshot calls target the claimed cloud desktop. This script checks the connection and saves an image; it does not run an AI agent or connect a local Cua Driver session. Exiting the claim releases the sandbox, and namespace deletion requests resource removal. ## Next steps - [Create a reusable sandbox pool with Python]() - [Configure a sandbox pool with Terraform]() - [Expire pools and claims automatically]() - [Choose a sandbox image]() - [Sandbox SDK API reference]() --- # Drive your first app with Cua Driver Install Cua Driver, connect your agent, and verify a result in a desktop app on macOS, Windows, or Linux. By the end of this tutorial, your agent will open Calculator, compute 6 × 7, and report 42 through **Cua Driver**. You will install the driver, connect your agent, and run a prompt that matches its integration on macOS, Windows, or Linux. Have an agent installed and a desktop calculator available before you begin. On Linux, use your distribution's calculator, such as GNOME Calculator. New to agents that operate applications? Start with [What is computer use?]() for the model behind the workflow, then return here to build one. This tutorial connects an existing agent to Cua Driver. If you are embedding Cua Driver in a product, start with [Choose a Cua Driver integration]() instead. [Drive a legacy postal app with Cua Driver]() Cua Driver fills shipment details and prints receipts in a Windows desktop app with no API. You will start with a smaller Calculator task below. ## 1. Install Cua Driver Use the same one-line installer on every platform; it picks the right path for the host and needs no administrator access. **macOS** Requires macOS 14 (Sonoma) or later. ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" ``` Start the daemon through the app bundle so macOS attributes permission prompts to CuaDriver.app. This is what makes the TCC grant stick to the driver: ```bash open -n -g -a CuaDriver --args serve ``` Then grant Accessibility and Screen Recording: ```bash cua-driver permissions grant ``` macOS prompts once for each permission. Neither prompt grants anything on its own — both offer **Open System Settings**, not **Allow**: ![The macOS Accessibility Access prompt for CuaDriver, offering Open System Settings or Deny](https://github.com/user-attachments/assets/b7a5a0c0-f297-4249-81f9-ca42c6b6f8ba) ![The macOS Screen Recording prompt for CuaDriver, offering Open System Settings or Deny](https://github.com/user-attachments/assets/69efc964-a375-495e-8a27-400c9dd08e17) Clicking **Open System Settings** adds CuaDriver to that permission's list for you — switched off. Toggling it on is what actually grants access: ![CuaDriver listed under Accessibility in System Settings with its toggle switched off](https://github.com/user-attachments/assets/d149fe5a-28cb-460c-8868-4eda80d540a4) ![CuaDriver listed under Screen & System Audio Recording in System Settings with its toggle switched off](https://github.com/user-attachments/assets/469308b5-84d7-4fd6-a580-e62925120337) **Warning** Do this for both **Accessibility** and **Screen & System Audio Recording**; choosing **Deny** leaves the driver unable to see or drive your desktop. macOS may offer to quit and reopen CuaDriver when you flip a toggle — accept, because the driver only picks up a changed grant after a full relaunch. If the daemon does not come back, start it again with `open -n -g -a CuaDriver --args serve`. macOS does not always raise both prompts in one pass. Check what actually landed: ```bash cua-driver permissions status ``` If one is still missing, run the pair again to prompt for it: ```bash open -n -g -a CuaDriver --args serve cua-driver permissions grant ``` **Windows** Requires Windows 10/11 with an interactive desktop session and PowerShell. ```powershell irm https://cua.ai/driver/install.ps1 | iex cua-driver autostart kick ``` The installer attempts to register Cua Driver to start automatically at sign-in. `kick` starts that task immediately. If registration was skipped or failed, run `cua-driver serve` in a terminal in the interactive desktop session and leave it open. Use a second terminal for the checks below. **Linux** Requires an x86_64 Linux desktop session with X11 or XWayland, plus AT-SPI 2. On a minimal Debian or Ubuntu image, install `libxi6` and `at-spi2-core` first; see [Linux requirements](). Run this tutorial inside the desktop session, where the driver can reach its display and accessibility bus. ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" ``` Start the daemon from a terminal in the same desktop session: ```bash cua-driver serve ``` Leave that terminal open. Run the verification commands below in a second terminal in the same desktop session. **Note** Full install steps, PATH setup, and permission details live in the install guide. See [Install Cua Driver](). The daemon you just started runs in the default `standard` authorization mode, which is the right choice for this tutorial; once you move past Calculator, read [Permission modes]() to decide whether a real workload should be `bounded` instead. ## 2. Verify it is working First confirm that the daemon is reachable: ```bash cua-driver status ``` Expect `Cua Driver daemon is running`. If it is not running, repeat your platform's startup step above. Then check the environment: ```bash cua-driver doctor ``` Then list the running GUI apps the driver can reach: ```bash cua-driver call list_apps ``` Read the `doctor` report, including warnings: a zero exit code does not prove desktop readiness. On macOS, also confirm both grants with `cua-driver permissions status`. Continue when the required permissions and desktop services are available, and `list_apps` includes a GUI app you recognize. An empty list is not a successful desktop check: open Calculator in that session and retry. Resolve permission or display errors before connecting the agent. ## 3. Connect your agent Choose one integration for your agent. A skill teaches the agent to use the Cua Driver CLI; MCP registration exposes Cua Driver as a tool server. `mcp-config` only prints setup instructions: you must run the registration command it prints. Use the output from your machine, not the illustrative paths below. **Claude Code** Install the Cua Driver skill for Claude Code: ```bash cua-driver skills install cua-driver skills status ``` Confirm that the status includes a Claude Code skill link, then start a fresh Claude Code session. If the link is missing, follow [Install the agent skill](). Prefer plain MCP instead of the skill? Print the Claude Code registration command: ```bash cua-driver mcp-config --client claude ``` It prints a command you run to register the server (paths will be specific to your install): ```bash claude mcp add-json --scope user cua-computer-use '{"args":["mcp"],"command":"/Users/you/.local/bin/cua-driver"}' ``` **Codex** Print the Codex registration command: ```bash cua-driver mcp-config --client codex ``` It prints a command you run to register the server. The emitted command uses the absolute installed binary path: ```bash codex mcp add cua-driver -- /Users/you/.local/bin/cua-driver mcp ``` Run `codex mcp list` to confirm the `cua-driver` registration. Start a fresh Codex session and use `/mcp` to check that the server is active. See the [official Codex MCP instructions](https://learn.chatgpt.com/docs/extend/mcp). **Prime Agent** Install the Cua Driver skill pack. Prime Agent discovers it from its native skill directory and calls Cua Driver from the persistent IPython environment; no MCP registration is required. ```bash cua-driver skills install cua-driver skills status ``` Run `/reload` in Prime Agent or start a new session after installation so the skill appears. Use `/skill:cua-driver` to load it explicitly. **OpenClaw** Print the OpenClaw registration command: ```bash cua-driver mcp-config --client openclaw ``` It prints a command you run to register the server, using the absolute installed binary path: ```bash openclaw mcp set cua-driver '{"command":"/Users/you/.local/bin/cua-driver","args":["mcp"]}' ``` **Warning** On macOS, a server spawned this way by the gateway does not inherit OpenClaw.app's own Accessibility and Screen Recording grants. If you are embedding Cua Driver inside OpenClaw rather than registering it as a separate MCP server, follow [Embedding]() so the host owns the grants. **Hermes** Hermes includes a built-in `computer_use` toolset. Verify it and the available Cua Driver instead of adding a second raw MCP server: ```bash hermes computer-use status hermes computer-use doctor hermes tools list ``` If `computer_use` is disabled, enable it for the CLI: ```bash hermes tools enable computer_use --platform cli ``` Using Cursor, Gemini/Antigravity, OpenCode, or Pi instead? See [Connect your agent](). For Grok Bot, see [Grok Bot](). ## 4. Ask your agent Use the prompt that matches the setup you chose. Ask the agent to read the calculator's displayed result after acting, so it verifies the GUI instead of answering the arithmetic from memory. **Claude Code** ```text Using the Cua Driver skill, open the installed calculator app, compute 6 × 7, and read the displayed result back from a fresh snapshot. ``` If you chose MCP registration instead, replace “Cua Driver skill” with “cua-computer-use MCP server.” **Codex** ```text Using the cua-driver MCP server, open the installed calculator app, compute 6 × 7, and read the displayed result back from a fresh snapshot. ``` **Prime Agent** ```text Using the Cua Driver skill, open the installed calculator app, compute 6 × 7, and read the displayed result back from a fresh snapshot. ``` **OpenClaw** ```text Using the cua-driver MCP server, open the installed calculator app, compute 6 × 7, and read the displayed result back from a fresh snapshot. ``` **Hermes** ```text Using the computer_use toolset, open the installed calculator app, compute 6 × 7, and read the displayed result back from a fresh snapshot. ``` The app name, controls, and action sequence vary by platform and calculator. The agent must discover them from fresh state; element indices from another run are not reusable. If no calculator is installed, install one through your platform's normal app or package manager, then repeat the prompt. To understand capture and delivery modes, see [Capture and delivery modalities](). ## 5. Confirm what happened Confirm that the calculator displays **42** and that the agent's final observation reports that value. A correct answer alone does not prove that the agent operated the app. Cua Driver uses best-effort background delivery. Observe whether your active app, keyboard focus, and real pointer stay unchanged during the interaction; do not infer this from the answer. The visible agent cursor is a separate overlay. macOS and Windows accessibility actions can operate supported controls in the background, but some apps require explicit foreground delivery. Linux AT-SPI actions and X11 routes can also work in the background; native Wayland depends on compositor support and has raw-keyboard limits. Launching an app can itself raise a window. See [Best-effort background]() for these platform differences. ## What you did You installed Cua Driver, checked its daemon and desktop access, connected your agent, and verified a result computed inside a desktop app. ## Next steps - [Connect your agent](): register Cua Driver with Cursor, Antigravity, OpenCode, OpenClaw, Pi, and more. - [Use Cua Driver in process](): embed the typed SDK in a Python or TypeScript application. - [Verify a desktop action](): check an action against an independent postcondition. - [Best-effort background](): how Cua Driver avoids focus and cursor disruption when the target app supports it. - [How-to guides](): keep the driver running, update it, and more. --- # Create a local macOS VM with Lume Install Lume and create a vanilla macOS Tahoe VM from an Apple restore image. This tutorial walks through one complete local VM setup on an Apple Silicon Mac. You will install Lume, download a Tahoe restore image, create a vanilla VM, and connect to it over SSH. This path uses the Lume CLI directly. It does not create a Cloud Fleet or connect the Sandbox SDK to the VM. ## Before you start You need an Apple Silicon Mac with macOS 13 or later, 8GB of available memory, and at least 50GB of free disk space. The restore image requires additional download and VM disk space. ## Install Lume Run the installer: ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/lume/install.sh)" ``` Check the installed version: ```bash lume --version ``` ## Download the Tahoe restore image Ask Lume for the latest supported restore image URL and download it locally: ```bash IPSW_URL="$(lume ipsw | tail -n 1)" curl -L "$IPSW_URL" -o ~/Downloads/macos-tahoe.ipsw ``` ## Create the VM The Tahoe preset prepares the installed guest offline. It creates the `lume` user, enables SSH, configures autologin, and disables sleep and screen locking. ```bash lume create macos-tahoe \ --ipsw ~/Downloads/macos-tahoe.ipsw \ --unattended tahoe ``` Creation includes a temporary boot, offline disk setup, and an SSH health check. The VM is stopped when creation finishes. ## Run the VM Start the VM with its display: ```bash lume run macos-tahoe ``` The default guest credentials are `lume` / `lume`. In another terminal, verify the guest account over SSH: ```bash lume ssh macos-tahoe 'id -un' ``` The command prints `lume`. Change the password before using the VM for anything sensitive. ## Next steps - [Create a vanilla Tahoe VM]() for lifecycle commands and troubleshooting. - [Serve the Lume API]() for local tools and scripts. - [Read the CLI reference]() for every command and option. --- # Build your first Cua-Bench task Create, inspect, solve, and evaluate a simulated computer-use task. In this tutorial, you will create a small computer-use task and verify it with its oracle solution. The task runs in a simulated desktop, so it does not need Docker, a VM, or an API key. ## Before you start Install: - Python 3.12 or 3.13; - [uv](https://docs.astral.sh/uv/). ## Install Cua-Bench Install the CLI and the browser support used by simulated tasks: ```bash uv tool install 'cua-bench[browser]' uv tool run --from 'cua-bench[browser]' playwright install chromium ``` Confirm that the CLI is available: ```bash cb --help ``` You should see command groups including `run`, `interact`, `task`, `trace`, and `dataset`. ## Create a task Create a working directory, enter it, and start the task scaffolder: ```bash mkdir cua-bench-tutorial cd cua-bench-tutorial cb task create first-task ``` Use these values when prompted: ```text Author name: Your name Author email: you@example.com License [MIT]: Task description: Click the Submit button Task difficulty (easy|medium|hard) [easy]: Task category (e.g., grounding, software-engineering) [grounding]: Tags (comma-separated): button,simulated ``` The scaffold contains the task definition and a small HTML interface: ```text first-task/ ├── main.py ├── pyproject.toml └── gui/ └── index.html ``` ## Inspect the task Ask Cua-Bench to load the definition: ```bash cb task info first-task ``` The output reports a simulated provider, one macOS-themed variant, and check marks for the setup, solve, and evaluate functions. This confirms that the CLI can discover the complete task lifecycle. ## Run the oracle Run the task with its reference solution: ```bash cb interact first-task --variant-id 0 --oracle --no-wait ``` A desktop window opens while Cua-Bench sets up the interface, clicks the button, and evaluates the final state. The final lines should include: ```text ✓ Solution complete ✓ Evaluation result: [1.0] ✓ Task completed successfully! ``` The reward of `1.0` means the evaluator observed the state produced by the oracle. ## Try the task yourself Run the same variant without the oracle: ```bash cb interact first-task --variant-id 0 ``` Click **Submit** in the task window. Return to the terminal and press Enter to evaluate and close the task. ## What you built You created a task with a prompt, simulated desktop, setup function, oracle solution, and evaluator. You then used the same evaluator to verify both a reference solution and a manual attempt. ## Next steps - Understand [the Cua-Bench task lifecycle](). - Learn how to [run and validate an existing task](). - Consult the [task definition reference](). --- # Concepts Understand the design ideas behind Cua and how its main pieces fit together. Use these pages when you want the model behind Cua rather than a step-by-step guide or API table. They explain what computer use means in Cua, how Cua Driver keeps the desktop usable while it acts, and how Cua Sandbox gives an agent an isolated computer on local infrastructure or Fleet. For sandbox compatibility lookup, use [Sandbox runtime support](). For the relationship between an SDK image and a pool's boot artifact, read [How Fleet images work](). For Linux, read [Linux desktops and computer use]() to understand X11, Wayland, XWayland, and the relationship between Omarchy and Hyprland before choosing a support-matrix row. Start with [What is computer use?]() for the basic model. If you are adding Cua Driver to an agent or product, use [Choose a Cua Driver integration]() before reading the deeper [SDK, MCP, and process hosting]() model. Read [Best-effort background]() to understand Cua Driver's default behavior on a shared machine, then [Capture and delivery modalities]() for the action axes. [How we continuously validate Cua Driver]() explains why desktop support requires application-owned E2E evidence in addition to unit tests. Read [How sandboxes work]() when you need the model for disposable computers, then [Sandbox lifecycle]() for the lifetime and connection patterns. Read [How Lume creates local macOS sandboxes]() for the local VM setup model, and [How SIP works in Lume VMs]() for signed boot-policy changes. For evaluation, start with [What is Cua-Bench?](), then read [The Cua-Bench task lifecycle](). --- # Computer use: how AI agents operate computers Understand computer use, computer-use agents and models, their observe-decide-act loop, and how they differ from APIs, RPA, and browser automation. **Computer use** is the ability of an AI agent to operate a real computer and complete work across applications. A computer-use agent can inspect the computer's current state, choose an action, observe the result, and continue until it reaches a goal. It can work through code, structured tools, or the same graphical interfaces people use. The term originally described agents that interpreted screenshots and controlled a graphical user interface (GUI) with mouse and keyboard actions. Cua uses **Computer-Use 2.0** for a broader model: an agent chooses among writing and running code, calling structured tools and APIs, and driving the same graphical interface a person would use. The screenshot-and-click loop remains important as one action surface inside the larger system. Computer use is therefore a capability rather than a single product, model, or interaction method. The defining property is that the agent can act on computer state, observe what changed, and continue across the boundaries between applications and interfaces. ## Computer-use agents and computer-use models A **computer-use model** interprets an observation and proposes an action. Depending on the model, the observation may include screenshots, accessibility information, text, or structured application state. The proposed action may be a mouse movement, a keystroke, a tool call, or a program to run. A **computer-use agent** is the running system around that model. It supplies the computer, tools, memory, instructions, permissions, and feedback loop needed to turn proposed actions into completed work. It also decides when to stop, retry, ask for confirmation, or move to a different action surface. The distinction matters because a capable model alone does not provide a reliable agent. The surrounding system still has to deliver input to the intended application, preserve state between steps, recover from changed interfaces, and constrain consequential actions. An **agent harness** is the runtime that hosts this loop. A coding agent such as Claude Code or Codex can supply instructions, memory, and MCP connections while using either a hosted model or a model served on the same machine. Cua Driver is the UI tool layer connected to that harness; it does not select or run the model. ```text model -> agent harness -> Cua Driver -> operating system and applications ``` This separation lets the same driver work with different models and harnesses. It also makes the cost of each layer visible. Local models often have less inference throughput and a tighter usable context budget, so tool schemas, screenshots, and accessibility trees can materially affect how long the agent can continue. A smaller tool surface and bounded state reads leave more context for the task itself. See [Run a local model with Cua Driver]() for one tested configuration. ## Three action surfaces On the **coding surface**, the agent writes and runs code as the action itself. This works especially well when the task is text-native, when the same operation must run more than once, or when a small program can cover a larger body of files and state than a human would want to touch by hand. On the **tool-use surface**, the agent makes structured calls to tools, functions, APIs, and MCP servers. The action has a typed shape, with named inputs and defined outputs, so the agent can ask an external system to perform a specific operation without inventing a script or trying to reach the same control through the screen. On the **UI automation surface**, the agent clicks, types, scrolls, and presses keys on the interface a person would use. This surface reaches controls in native applications, web applications, and legacy software that expose no useful API. In those systems, the GUI is often the only contract available to the user. These surfaces are complementary. Computer use does not mean clicking through every task: the agent should use the most direct interface that preserves the required behavior and state. ## Examples of computer use Computer use is useful when work spans interfaces that were designed for people rather than for one programmatic integration. Examples include: - testing a desktop application by installing it, changing settings, and verifying the visible result; - collecting information from a signed-in web application that has no suitable API; - moving between a terminal, code editor, browser, and operating-system dialog during one development task; - completing a business workflow across several applications while preserving the user's existing session; - reproducing a support issue on a real operating system and retaining the visual evidence needed to diagnose it. In each case, the agent must connect actions across changing application state. A script or API may still cover part of the work, while computer use covers the boundaries that those interfaces cannot reach. ## Choosing a surface A capable agent moves between these surfaces during a single task, choosing the one that fits the next piece of work. Code and structured tool calls are usually the right fit when the job is repeatable or text-heavy, especially when the system already exposes a clear interface. UI automation becomes the better choice when the work depends on visual state or when the application is unfamiliar; it is also the fallback when there is no useful API and the path exists only inside the interface. The skill is the judgment behind that choice. An agent that edits a file with code, checks account state through an API, and then changes a setting in a desktop app is still doing one computer task; it is simply using different action surfaces as the task moves through different kinds of state. Computer use is most valuable when a task crosses those boundaries. Browser automation can be enough for a stable web workflow, and an API is usually best for a well-defined integration. A computer-use agent becomes useful when the task also depends on visual state, native applications, signed-in sessions, or software that exposes no suitable API. ## How computer use differs from other automation Computer use overlaps with APIs, browser automation, and robotic process automation (RPA), but each approach has a different interface and tradeoff. | Approach | Best fit | Main limitation | | ---------------------- | ------------------------------------------------------------- | --------------------------------------------------------------------- | | API or structured tool | Stable, well-defined operations with typed inputs and outputs | Cannot reach behavior the API does not expose | | Browser automation | Repeatable workflows contained within web pages | Does not cover native applications or operating-system UI | | Traditional RPA | Known, predefined business processes | Often depends on fixed selectors, coordinates, and workflow branches | | Computer-use agent | Adaptive work across applications and interface types | Requires careful observation, recovery, permissions, and verification | The approaches can be combined. An agent may query an API for structured data, use code to transform it, and then enter the result into a native application. Computer use describes the whole task even when only one part requires direct GUI control. ## The UI automation loop The observe, decide, act loop belongs specifically to the UI automation surface. Observation comes from a screenshot, an accessibility tree, or both, which gives the model either the pixels a person would see or the structured information exposed by the operating system. From there, grounding turns a target such as a button, field, menu item, or selected region into on-screen coordinates that input events can hit. Planning carries the task across changing interface states. A click may open a dialog, a page may reflow after loading, or an application may show an error that changes the next useful action, so the model has to keep track of the goal while the computer responds. Frontier models such as Claude can handle understanding, grounding, and planning together in one call, while grounding-specialist models such as UI-TARS and Moondream can help when coordinate accuracy is the limiting factor. The current wave of screenshot-driven computer use accelerated in October 2024, when Anthropic introduced an agent that operated a GUI through screenshots and input events. Through 2025, coding agents were increasingly recognized as computer-use agents too, with CoAct-1 making the connection explicit. Cua uses **Computer-Use 2.0** as shorthand for that wider model. Francesco Bonacci traces that arc in [A Story of Computer-Use](https://github.com/trycua/cua/blob/main/blog/clawdbot-computer-use-history.md). ## Where Cua fits You bring the agent, which already handles coding and tool-use on its own. Cua gives that agent the UI automation surface and a real computer to act in, so the same task can move from code to tools to the graphical interface when the application requires it. [SDKs, MCP, and process hosting]() are different ways to connect the agent to that surface. **Cua Driver** drives the GUI of a real machine you already have, whether that machine runs macOS, Windows, or Linux. It is the right shape when the agent needs to work with local applications, signed-in accounts, existing files, or machine state that already lives on that computer. **Cua Sandbox** is a fresh isolated computer running on local infrastructure or Fleet where the agent can run code and drive the GUI together. It is a full computer with all three action surfaces available inside the same environment rather than a remote desktop, which matters when the task needs both programmatic work and visible interaction without changing the host desktop. The agent brings the model and its reasoning; Cua provides the computer where that reasoning can turn into action. Available guest operating systems and operations depend on the [runtime and image](). ## Safety and isolation Computer-use agents can affect the same files, accounts, and applications as a person, so the environment and permissions are part of the system design. Prefer the least privilege needed for the task, require confirmation for consequential actions, and keep sensitive or untrusted work isolated from a personal computer. Cua Driver exposes [permission policies]() for restricting what an agent can do on an existing machine. Cua Sandbox provides an [isolated computer]() whose lifecycle and credentials can be scoped to the task. ## Further reading - **Learn it:** [Drive your first app with Cua Driver]() on an existing macOS, Windows, or Linux computer. - **Use it:** [Install Cua Driver]() and connect it to an agent you already use. - **Guide the agent:** [Install the Cua Driver agent skill]() to add cross-platform tool-selection and verification instructions. - **Isolate it:** [Create your first Cloud Fleet]() for an isolated Linux computer. - **Understand delivery:** [Browser targeting and background delivery]() explains how browser tabs map to native windows. - **Evaluate it:** [How we continuously validate Cua Driver]() describes the evidence used to test desktop behavior. --- # Choose a Cua Driver integration Choose between MCP, the SDK, a private worker, and an app-hosted service based on who owns the runtime. Most agent harnesses should connect to Cua Driver through MCP. Embed the SDK when computer use is part of your product, and use an app-hosted service when a desktop application must own the operating-system permissions. ## Start with who owns the runtime The runtime is the process that owns Cua Driver's permissions, browser connections, recordings, and lifecycle state. Choose its owner before choosing the transport: | You are building | Start with | Runtime owner | | -------------------------------------------------- | ----------------------------------------------------------------------- | ------------------------------------------------- | | An agent or harness | [MCP]() | The MCP process, or an explicitly selected daemon | | An application with built-in computer use | [Same-process SDK]() | Your application process | | A signed desktop app that serves an external agent | [App-hosted service]() | The permission-owning desktop app | These are starting points. Add a private worker or shared daemon only when you need the process boundary it provides. ## Connect an agent through MCP Use MCP when an existing agent or harness needs Cua Driver's tool catalog. ```text Agent or harness -> cua-driver mcp -> Cua runtime -> desktop ``` The MCP client starts `cua-driver mcp` and keeps its standard input and output open. That authenticated transport receives a private implicit lifecycle session. Closing the transport releases its session state. This is the normal route for Codex, Claude Code, Cursor, and a standard OpenClaw gateway. Hermes's built-in `computer_use` toolset also uses MCP internally, but owns the connection and presents a smaller Hermes-native tool surface. Generate configuration for clients that require direct registration with `cua-driver mcp-config --client `. On Windows and Linux, bare `cua-driver mcp` owns its runtime. On macOS, it normally proxies to the installed `CuaDriver.app` daemon so Accessibility and Screen Recording remain attached to the app identity. ## Embed the SDK in an application Use the Python or TypeScript SDK when your product owns computer use as a feature and wants a typed API instead of an agent protocol. ```text Your application -> CuaDriver.create() -> desktop ``` The application owns runtime startup, permissions, sessions, cancellation, and shutdown. It can expose Cua operations directly or map them into a smaller product-owned tool. Use a private worker when the application needs the same SDK contract in a supervised child process: ```text Your application -> inherited pipes -> private Cua worker -> desktop ``` The worker has no listener or reconnect path. Closing its private channel ends the runtime. This gives one host process isolation without creating a shared desktop service. ## Host Cua Driver from a desktop app Use an app-hosted service when a signed desktop app owns Accessibility and Screen Recording permissions while a backend or external agent needs MCP. ```text Desktop app -> private Cua service -> generated MCP connection -> agent ``` The desktop app starts `EmbeddedCuaDriverHost` and passes its returned MCP connection to the backend. The backend launches that connection unchanged. It must not start a second host. This distinction matters on macOS. The permission-owning app must start the private service directly so the child stays in the app's TCC responsibility chain. Starting it from a gateway or backend gives the service that process's identity instead. ## Less common variants Use one-shot CLI calls when an agent can run shell commands but cannot keep an MCP transport open: ```text Agent -> cua-driver call -> Cua Driver daemon -> desktop ``` Use an explicit shared daemon when several clients must reconnect to one long-lived desktop service or when the MCP process cannot reach the interactive desktop: ```text Trusted launcher -> cua-driver serve --socket Agent A ---------> cua-driver mcp --socket Agent B ---------> cua-driver mcp --socket ``` The daemon owns the runtime. Each MCP transport still receives its own private lifecycle session. ## Permissions belong to the runtime **Note** The trusted host fixes the permission mode, capability manifest, launch grants, and authorization callbacks when it starts the runtime. An agent can request actions, but it cannot widen the runtime's permissions. The optional public session name is a label for display and explicit lifecycle control. It is not a credential and does not carry permission. Two MCP transports that use the same public label still have separate private lifecycle sessions. ## Existing browser sessions A driver-owned isolated browser profile is the default. Attaching to an existing Chrome or Edge profile is more sensitive because it exposes signed-in tabs, cookies, and storage through CDP. Browser integrations that depend on launching Chromium with a remote debugging flag cannot attach that way to a profile already running without the flag. Cua Driver can instead use the browser's per-instance remote debugging control in the exact approved window, complete the browser-owned consent flow, and bind the endpoint to that browser process and lifecycle session. It does not copy, edit, restart, or terminate the selected profile. Standard mode requires the trusted launcher to pass `--grant existing-profile`, or an embedding application to approve the exact request. The old `browser-approve` token is not part of this model, and an MCP tool argument cannot create the grant. See [Browser Profile Attachment]() for the supported browsers, setup effects, grant lifetime, and refusal codes. ## Go deeper - [Connect your agent](): configure an MCP client. - [Use Cua Driver in process](): embed the typed SDK. - [Expose MCP from a desktop app](): host a private service from a signed app. - [SDK, MCP, and process hosting](): understand the complete interface and topology grid. - [Process model](): check platform-specific runtime ownership. - [Permission modes](): check the authorization contract. --- # SDK, MCP, and process hosting How Cua Driver separates its typed application contract, agent protocol, and desktop process identity. Cua Driver has two caller interfaces and three local execution topologies. These solve different problems: - The **SDK** is a typed application API for Python, TypeScript, and Rust. - **MCP** is the runtime-neutral agent protocol exposed by the Cua Driver server. - A **same-process runtime** performs desktop work inside the importing app. - A **private worker** gives one SDK host a supervised child-process boundary over inherited pipes, without a listener or reconnect path. - A **daemon runtime** performs desktop work in a long-lived process with a stable operating-system identity. These dimensions form a grid: | Topology | Typed SDK | MCP / CLI | | --- | --- | --- | | Same process | **Primary for applications embedding Cua Driver** | Default stdio MCP owner on Windows and Linux; explicit `--direct` on macOS | | Private worker | `create_private_worker()` / `createPrivateWorker()` for per-host process isolation | Not reconnectable and not advertised as an MCP endpoint | | Daemon | Compatibility or app-shared runtime | Standalone macOS default and explicit `--socket` service mode | The SDK adds unique value in the same-process cell: typed records and direct calls across the generated native boundary without a socket. If application code talks to the same daemon as an agent, the SDK is mainly a typed compatibility adapter over that existing daemon. ## One typed contract The public typed `CuaDriver` SDK contract is canonical: ```text private platform implementations | v public typed CuaDriver SDK contract | | | v v v Rust apps UniFFI MCP server adapter / \ | Python TypeScript agents and CLI ``` The MCP server is downstream of the same public SDK contract as an application. It does not maintain a second implementation of desktop tools. MCP remains valuable because it standardizes discovery, calls, results, tasks, and transport for agent runtimes; the generated language SDKs are for applications embedding the native runtime. Authorization is part of the native runtime, below this topology choice. A same-process `CuaDriver.create()` call and a daemon-backed call both pass the same registry authorization boundary before platform dispatch. MCP, HTTP, CLI, and daemon adapters may reject a call earlier for defense in depth, but they do not replace or weaken that native check. Released calls resolve a process-owned compatibility context. A trusted same-process host can additionally construct immutable session-bound action objects beneath an immutable runtime ceiling. No model-facing tool, public session string, environment field, or reconnect label can select that authority. An embedded service may additionally bind a session only on its original authenticated host connection. A standalone shared daemon still refuses trusted-session creation. ## Why the daemon still exists Some external agents and CLI calls are short-lived or run outside the desktop app that owns permissions. A daemon gives them a stable execution identity, long-lived session state, and one controlled gateway to the physical desktop. On macOS, Accessibility and Screen Recording grants attach to an application identity and responsibility chain. The standalone `CuaDriver.app` daemon keeps that identity stable across CLI and agent reconnects. A signed app that needs to expose MCP can instead spawn a private daemon directly so the child reuses the app's TCC grants. On Windows, an explicit daemon can remain in the logged-in interactive session while an SSH or service-side client cannot access that desktop. Ordinary Windows and Linux stdio MCP processes now own their runtime directly and shut it down on stdin EOF. On every platform, service mode also owns cleanup for recordings, cursors, policy, and per-session state when a client disconnects. Bare `cua-driver mcp` therefore no longer discovers and joins an already running default service on Windows or Linux. Use `cua-driver mcp --socket ` when an agent must share that service's sessions and resources. The explicit connection path preserves the released daemon protocol. Standalone macOS keeps the opposite default because `CuaDriver.app` owns its stable TCC identity. `cua-driver mcp --direct` is the explicit opt-in for a host that wants the MCP process itself to own the runtime and accepts the spawning application's TCC attribution. It cannot be combined with `--socket`. Private workers use the same generated SDK contract but exchange versioned request/response envelopes over child stdin/stdout. The host supplies the authorization ceiling before readiness, owns the only channel, and terminates the child when that channel closes. A timed-out or broken action reports whether it was definitely not started, completed, or has unknown completion. The worker provides native cursor and main-thread facilities without opening a daemon socket. Remote carriers use the same transport-free Rust envelope seam. The carrier must authenticate a principal, bind a connection generation, preserve request IDs and deadlines, negotiate a compatible envelope version and cancellation support before dispatch, forward cancellation when an action future is dropped, and return a separately bound channel for trusted sessions. The Cua Driver core does not depend on gRPC, HTTP/2, or another carrier, and no generated Python/TypeScript remote constructor is shipped yet. ## Why embedded applications may choose either topology Apps such as signed Electron desktop clients may want their own permission row and onboarding. If only the app calls Cua Driver, `CuaDriver.create()` runs the runtime in that permission-owning process. If external agents must connect, the app can host a private daemon and publish its MCP connection. Both routes consume the same typed behavior; the difference is lifecycle and process identity, not a different action implementation. See [Process model]() for platform details and [Expose MCP from a desktop app]() for the hosted-daemon procedure. ## Current process facility contract A trusted application may own multiple direct runtimes in one process. This is lifecycle and resource coordination, not a security boundary or desktop virtualization: | Facility | Current ownership | Concurrency rule | | --- | --- | --- | | Runtime ceiling, effective session contexts, revocation | Runtime-owned | Isolated within each opaque runtime generation | | Tool registry, browser engine and grants, recording session | Runtime-owned | Released only by their owning session/runtime | | Permission/policy compatibility caches and bounded compatibility manifest | Immutable process configuration | Explicit configured constructors must agree with contradictory environment values or fail | | Session activity, modality telemetry, element tokens, browser refs, cursor keys | Runtime-namespaced | The same public session label in two runtimes remains independent; stale or cross-runtime handles fail closed | | Physical pointer, keyboard, focus, overlay, and platform event loops | Process/platform coordinated | Native input and focus action turns are admitted one at a time; all runtimes still operate the same physical desktop | | CDP listener claims and the download gate | Process-coordinated | A listener/profile claim or download lease cannot be silently taken over by another runtime | | Recording platform callbacks, video/PiP factories, policy, observers, ABI executor | Immutable process callbacks/coordinator | Shared only where the process identity and platform configuration are the same | Shutting down runtime A does not revoke runtime B's sessions or stop its recording. Public session labels and element-token wire formats remain unchanged; the runtime generation is private and never accepted from a tool argument. Each runtime transport receives a private implicit lifecycle session. Repeated unnamed actions on that transport reuse it, while concurrent transports remain isolated. An explicit public label is optional. The first enabled runtime initializes the process-global overlay template; later runtimes reuse it and can customize their own namespaced cursors through the cursor tools. Arbitrary code in the host process can inspect or interfere with every runtime. Use separate private workers or services for crash isolation, different trust domains, different OS identities, or independent desktops. On macOS, the cursor overlay additionally requires an AppKit main-thread UI owner. A direct runtime without a suitable host adapter returns structured `facility_unavailable` results for cursor-overlay operations; it does not report success or start hidden AppKit work on an unsafe thread. A private worker or service owns the required event loop. A headless service or worker without Window Server graphic-session access returns the same refusal instead of claiming an overlay that cannot render. ## Service transport hardening The optional loopback HTTP MCP listener is disabled unless `CUA_DRIVER_RS_MCP_HTTP_PORT` is set. When enabled, `CUA_DRIVER_RS_MCP_HTTP_TOKEN` is also required and must contain 32–4096 non-whitespace characters. Clients send it as `Authorization: Bearer `. A port-only configuration now fails daemon startup instead of silently exposing or disabling the endpoint. Windows daemon named pipes now grant access only to the daemon owner's user SID and verify the connected process SID before reading a request. Clients running as another user or security principal are rejected; run the client and daemon under the same intended interactive user. This is an intentional fail-closed change from the earlier broad local ACL. ## Agent SDK integration is intentionally asymmetric | Agent SDK | Native application callbacks | External MCP route | | --------- | ---------------------------- | ------------------ | | Claude Agent SDK | Supported in Python and TypeScript through its in-process custom-tool server | Supported in Python and TypeScript | | Codex SDK | No direct custom-tool callback boundary | Supported in Python and TypeScript | Claude's in-process custom-tool server is an adapter inside the host application; the callbacks still call the native Cua Driver SDK directly. Codex uses `cua-driver mcp` because MCP is its supported tool-extension boundary. MCP remains an adapter over the same SDK-owned runtime; using MCP does not imply that a daemon is present. See [Use Cua Driver with Claude Agent SDK]() and [Use Cua Driver with Codex SDK](). --- # Best-effort background How Cua Driver tries to operate apps without taking focus, moving the cursor, or raising windows, and when it must fall back to foreground. Best-effort background means Cua Driver tries to operate a target app while preserving the user's active desktop. The default paths do not move the real pointer, do not raise the target window, and do not switch the user's frontmost app. This is a best effort rather than an absolute promise. Most app automation can stay in the background through accessibility actions, routed input, and window-specific capture. A small set of apps and OS surfaces only accept real foreground input, so the driver reports that limit and lets the caller choose a foreground escalation for that specific action. ## Why it matters Traditional GUI automation assumes the automated app owns the desktop. It activates a window, moves the pointer, and repeats. That is fine for unattended jobs or disposable desktops, but it breaks down when a person is using the same machine. Cua Driver's default path lets the agent operate an app in the background while the developer keeps coding, reading logs, or using another app. The visible agent cursor is an overlay; the real mouse pointer stays where the user left it. ## Platform mechanisms Each OS splits accessibility, input delivery, capture, and focus policy differently. Cua Driver chooses the most background-capable path the platform and app expose. ### macOS The Accessibility API can press buttons, set values, and read semantic state even when the app is not frontmost. ScreenCaptureKit can capture a specific window without requiring that window to be raised or visible on the active Space. Cua Driver also uses scoped CoreGraphics and SkyLight delivery for routed input when an app responds better to pointer-like events than accessibility actions. Some macOS surfaces still need foreground. SwiftUI windows parked on another Space can lose their detailed accessibility tree, and game/canvas surfaces may reject routed input. Those cases are documented in [Known limits](). ### Windows UI Automation can inspect and operate controls by window handle and automation element while another app is active. For input-like behavior, Cua Driver can post messages to a target window or use foreground escalation when the app only listens to active device input. Windows also has session boundaries. A daemon running in the interactive user session can see and operate the desktop; a process launched from OpenSSH in Session 0 cannot. See [Process model]() and [Drive a Windows app over SSH](). ### Linux AT-SPI provides the semantic path on Linux. Element actions call the toolkit's own accessibility action (`Action.DoAction`) and do not need pointer injection or foreground focus. On X11, window-addressable input and capture can also route to a target window. On Wayland, synthetic input is intentionally constrained by the compositor, so the background path depends more heavily on AT-SPI and reconstructed element frames. The remaining Linux gap is raw keyboard injection into native Wayland apps. Typing into accessible fields can still work through AT-SPI, but shortcuts or raw key events may need XWayland or foreground/user-granted compositor paths. See [Known limits](). ## The agent cursor Cua Driver does not move the user's real pointer to show agent activity. It renders a synthetic cursor overlay for supervision. The user can see where the agent is acting while their own cursor and active app stay untouched. The Linux recordings below make that separation visible. Synthetic pointers act on background windows while the foreground window remains untouched. [Four pointers across two XFCE windows]() Four synthetic pointers draw in two paint windows while a small text window stays in front. [Sixteen background windows on Wayland]() Sixteen pointers draw in separate windows around a foreground window that Cua Driver does not operate. The next recording shows the same background behavior during a complete development loop on macOS. The agent tests and fixes an app while the terminal remains in front. [Fix and check an app without taking over the desktop]() The terminal stays active while Cua Driver operates Chrome and a native task app behind it. ## How fallback works The safest ladder is: 1. Act by `element_index` in the background. 2. If the element path is unavailable or unverifiable, act by `x, y` from the same window screenshot. 3. If the app still rejects the action, retry that one action with `delivery_mode: "foreground"`. Foreground escalation is explicit. It is the right answer for apps that only accept focused input, but callers should use it narrowly and only when interrupting the user's desktop is acceptable. For the agent-side action behavior, see [Agent action policy](). For the reference matrix, see [Interface contracts](). --- # Capture and Delivery Modalities How Cua Driver observes and acts on an app. Perception returns both the accessibility tree and a screenshot; each action chooses its target, ax or px rung, and delivery mode. Every Cua Driver action is shaped by four things: **what the agent observes**, **which rung delivers the action**, **how input is delivered**, and **what coordinate space the action targets**. Most callers use the defaults: background, per-window, accessibility-first automation. The key change from earlier versions is that perception is no longer a mode you pick. `get_window_state` returns *both* the accessibility tree and a screenshot in one call. The action call chooses `ax` or `px` by how it addresses the target. ## The Axes ### 1. Perception: what the agent observes `get_window_state(pid, window_id)` returns **both the accessibility tree and a screenshot by default**, in one call. There is no capture mode to pick: you ground on the tree and the screenshot together and cross-check one against the other. This matters because the tree *lies* on some surfaces: it can expose useful structure while still echoing a write the app did not apply, omitting the rendered value, or reporting geometry that disagrees with the pixels. A grounding screenshot is always present, so when the tree looks wrong you check the pixels in the *same* response. The accessibility tree is the ground truth for *what is clickable*: roles, labels, advertised actions, and an `element_index` handle on every actionable element. The screenshot tells you *which one*. It disambiguates repeated or empty labels and shows captions, colors, and layout the tree omits, which is common in Chromium and Electron. They come back together because each catches what the other misses. > **Performance opt-out: `include_screenshot`.** `include_screenshot` (boolean, default `true`) is the one performance knob. The default returns both. Pass `include_screenshot: false` to skip the screen grab and get the tree only when you are re-indexing before an element ax action and do not need fresh pixels. The `ax`-versus-`px` decision still lives at action time. > **`capture_mode` is deprecated and ignored.** `get_window_state` still accepts it so old callers do not error, but both the tree and the screenshot come back regardless of what you pass. The old `ax` / `vision` / `som` / `screenshot` values all decode (`som` mapped to `ax`, `screenshot` to `vision`) but none changes what is captured. Perception is always both. ### 2. Action rung: how the target is addressed You don't pick a capture mode; you pick **how you address the target** on the action call, and that one choice selects the rung: | Rung | Address with | Delivered through | Properties | |---|---|---|---| | **element ax action** | `element_index` / `element_token` | the accessibility rung: UIA Invoke (Windows), `AXPerformAction` (macOS), AT-SPI `doAction` (Linux) | Backgroundable, z-order-independent, and the only **driver-verifiable** rung. | | **element px action** | `x, y` | the pixel rung, reading the coordinate straight off the screenshot already in the `get_window_state` response | Best-effort; the caller confirms the effect off the screenshot. | Default to the element ax action because the driver can verify it and often keep it in the background. Drop to an element px action when the tree cannot disambiguate repeated or empty labels, when it is empty (`degraded`, a non-AX surface), when an action came back `suspected_noop`, or when the tree disagrees with the pixels. You never re-capture to switch rungs. The screenshot is already in the snapshot, so you only change *how you address* the target. Both rungs apply to the **keyboard family** (`type_text`, `press_key`, `hotkey`) as well as the pointer tools. Address by `element_index` (ax) to target a field with no pre-click. Address by `x, y` (px) to pixel-click at `(x, y)`, establish real renderer focus, and deliver the keystroke(s) to the now-focused element. The px form is the one-call path for Chromium/Electron inputs the AX layer cannot focus: `type_text({ pid, window_id, x, y, text })` focuses and types in a single call. The two forms are mutually exclusive. `set_value` is the exception: it stays ax-only because it sets the value of a non-text control like a dropdown, checkbox, or slider. ### 3. Delivery: how input is delivered Set `delivery_mode` per call on the input family (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`). The same two values work on Windows, macOS, and Linux. | `delivery_mode` | Behavior | |---|---| | `background` (default) | Input is routed to the target process/window/element directly. The user's frontmost app, real cursor, and window z-order are untouched when the target surface supports background delivery. See [Best-effort background](). | | `foreground` | The target is briefly fronted for that action, input lands on the now-active window, then the prior frontmost is restored. Use this when a background attempt did not land, or when the app only accepts events while foregrounded (DirectInput games, raw-input canvases). | Only `background` and `foreground` are valid; the historical `auto` heuristic is removed. At runtime, omitted or unknown values fall back to `background` for safety. Element ax actions (`element_index`) address an element instead of the focused window, so they hold the background path without any `delivery_mode` flag. The delivery axis matters most for the pixel rung (`x, y`), where `background` routes the event to the target and `foreground` raises the window first. `bring_to_front` is not a normal escalation step. Reserve it for focus-proxy surfaces that must stay foreground across multiple calls, such as a remote desktop session. For an ordinary `background_unavailable` response, retry only the refused action with `delivery_mode:"foreground"`. ### 4. Action target: what coordinate space the action uses Select the target on each action: | `target` | Coordinate space | Capture surface | |---|---|---| | `{kind:"window", pid, window_id}` | coordinates relative to the exact window, or an element from that window's snapshot | `get_window_state` | | `{kind:"desktop", display_id:"primary"}` | screen coordinates on the primary display | `get_desktop_state` | The per-call target keeps window and desktop work independent. A desktop action does not change session state or disable window tools for the next call. Legacy `capture_scope` configuration remains available during migration. ## Response signals: verification, effect, and escalation Delivery success differs from application state change. The driver can verify an effect only when it can read the changed state back through the accessibility layer. That is why `verified: true` is reserved for AX read-back: it means the driver observed the effect after sending the event. Pixel input, foreground input, and echo-prone AX surfaces can be routed correctly while still leaving confirmation to the caller. `effect` is the confidence signal that separates those cases. `"confirmed"` means the driver verified the result through AX read-back. `"unverifiable"` means the delivery path ran, but the driver cannot prove the application applied it. `"suspected_noop"` means an AX action ran but almost certainly did not change the target. Treat `effect` as the action outcome. `escalation` is the machine-readable climb-the-ladder hint. When present, it tells the caller which surface to try next: `"px"` for acting off the screenshot, `"foreground"` for explicitly fronting the target, or `"page"` for the browser-tab DOM path through the `page` tool. See [Agent action policy]() for the agent behavior and [MCP tool notes]() for the field table. ### When the tree lies Some accessibility layers echo writes they did not apply. Electron can report an AX value change through its shim while the renderer stays unchanged. Catalyst controls can expose null `AXValue`s. Chromium/WebKit web content can reflect a write through the accessibility bridge without proving the DOM or rendered view changed. The driver treats those as surface-aware verification cases. It probes at the element level for a web-content surface, including an `AXWebArea` ancestor, so native chrome such as a browser address bar stays trusted while browser-tab content does not get a false confirmation. On those surfaces the driver refuses false `verified: true` responses and returns `verified: false`, `effect: "unverifiable"`, and an `escalation` object instead. Electron app surfaces recommend `"px"` so the caller can act by pixel off the screenshot in the same response; browser-tab web content recommends `"page"` so the caller can switch to DOM/CDP via the `page` tool. ## Where the exact matrix lives Perception is no longer an axis: every `get_window_state` returns both the tree and a screenshot by default. The enforceable constraints are the combination of **action target**, **action rung**, and **delivery mode**. The exact validity matrix and platform-support table live in [Interface contracts](). Keep this page as the mental model: observe both tree and pixels, start with the accessibility rung, use the screenshot-backed pixel rung when the tree is insufficient, and escalate to foreground only when the target app requires it. --- # Linux desktops and computer use How X11, Wayland, desktop environments, and compositors affect Cua Driver's capture and input routes. X11, Wayland, Hyprland, and Omarchy describe different parts of a Linux desktop, not interchangeable operating systems. For Cua Driver, the important questions are which display session is running, how the target application connects to it, and which capture and input routes that combination exposes. This page explains those layers. For accepted capabilities and limitations, use the [Linux support matrix](). ## Where the names fit A **distribution**, such as Ubuntu, Fedora, or Arch Linux, supplies the operating system and packages. A **desktop environment**, such as GNOME, KDE Plasma, or Xfce, supplies the shell and desktop applications. The same distribution can run different desktop environments and display sessions. **X11** and **Wayland** are display protocols. They define how applications communicate with the software that displays their windows and delivers input. - In an X11 session, an X server such as **Xorg** manages the display and input. A window manager handles placement and focus. A compositing manager may add visual effects. **Xvfb** is an X server backed by a virtual framebuffer, useful for automated desktops; it does not reproduce every physical Xorg input path. - In a Wayland session, the **compositor** combines display-server, window management, and composition responsibilities. **Sway** and **Hyprland** are different Wayland compositors. GNOME uses **Mutter**, and KDE Plasma uses **KWin** for its Wayland session. **Omarchy** is an Arch-based desktop setup built around Hyprland and Wayland. An Omarchy machine therefore needs Hyprland-specific validation, not just an "Arch Linux works" or "Wayland works" result. A plugin for Hyprland is not limited to Omarchy, but it must match the Hyprland build that loads it. **wlroots** is a library for building Wayland compositors, including Sway. It is not another display protocol. Modern Hyprland is not based on wlroots, although it implements some protocols with `wlr` names. Shared protocol names do not establish identical behavior or transfer Sway's test results to Hyprland. ## XWayland is an application compatibility path **XWayland** runs X11 applications inside a Wayland session. One desktop can contain both native Wayland applications and X11 applications using XWayland. Some toolkits can use either backend, so an application's name alone does not identify its input path. Cua Driver can use an X11 route only when the target exposes a real X11 window. XWayland does not make a native Wayland window addressable through X11, and it does not guarantee that every X11 background-input technique works unchanged inside a Wayland compositor. ## Capture, accessibility, and input are separate capabilities Seeing an application's pixels does not imply permission or a safe route to control it. Cua Driver combines several facilities: - **AT-SPI** exposes application accessibility trees and semantic actions, such as invoking a button or editing an accessible text field. Availability depends on the application and its accessibility bridge as well as the desktop session. - **Capture and window-discovery protocols** provide images, window identity, and geometry. Wayland compositors expose different combinations, sometimes requiring a compositor-specific adapter or helper. - **Portals and libei** provide permission-mediated capture or emulated-input routes where the desktop implements them. An input grant does not by itself provide an arbitrary background-window target. - **Compositor integrations** can expose capabilities absent from ordinary client protocols. They need their own compatibility, authorization, and behavioral tests. This is why a successful `cua-driver doctor` check establishes prerequisites, not proof that every action reaches every application. ## Wayland has coordinates, but controls input routing Wayland compositors have output layouts and surface coordinates. What an ordinary application generally lacks is unrestricted access to other windows' global geometry and an API for sending raw input to any chosen window. A compositor adapter can tell Cua Driver where a window is. That still does not change which window receives input from the compositor's active **seat**, the logical grouping of pointer, keyboard, and other input devices. Moving a virtual pointer to a screen coordinate can target an occluding window instead of the intended background window. Isolated background input therefore requires more than coordinate conversion or drawing a second cursor. It must keep the user's pointer focus, held buttons, keyboard state, and active gestures independent from agent input. Client applications must also accept the resulting events. ## Background actions are not all raw input An AT-SPI action can invoke an accessible button without synthesizing a mouse click. That can work while the window stays in the background. A pixel-addressed request can also resolve to an accessible control and use a semantic action; pixel addressing does not necessarily mean raw event injection. A canvas drag or game keypress may require raw pointer or keyboard events instead. When Cua Driver has no safe target-addressed route, the correct outcome is a structured refusal, not a hidden focus change. Foreground delivery is a separate, explicitly authorized operation. See [Capture and delivery modalities]() for the action axes and [Best-effort background]() for the no-foreground contract. ## A private desktop is different from a shared desktop A Fleet VM gives the agent a separate desktop. Foreground input inside that VM does not take over the user's host desktop, but it can still interfere with another actor inside the same VM. VM isolation does not prove isolated background delivery within one desktop session. Likewise, the experimental nested `cua-compositor` controls its own Wayland session. Its input capabilities do not establish support on stock Sway, Hyprland, GNOME, or KDE. For an existing Linux host, use the support row for its compositor and the target application's backend. For a separate Omarchy desktop, see [Run Omarchy on Fleet]() or the experimental [Apple Silicon VM guide](). Those guides describe environments, not a broader input-support guarantee. ## Find the applicable support evidence Use [Platform Support]() for accepted behavior and [Platform Roadmap]() for engineering work and missing evidence. In particular, the [Hyprland and Omarchy status]() separates driver validation from the experimental plugin foundation. A source-branch test result is not a released capability. Record the driver version, compositor version, application backend, and tested source revision when comparing results. [How Cua Driver is validated]() explains why a passing test must observe application state and desktop side effects, not only a successful tool response. --- # Browser Targeting and Background Delivery Why browser automation needs an exact native-window-to-tab binding, and how CDP adds a full-background rung to Cua Driver. A browser presents two identities at once. The desktop knows a native process and window; the browser runtime knows DevTools targets and tabs. Acting safely in the background requires proof that both identities describe the same surface. ## The connection point is the native window Agents still begin with `list_apps` and `list_windows`. They select a concrete `(pid, window_id)`, just as they do for accessibility and pixel actions. `get_browser_state` correlates that native window with a browser target and mints opaque target and tab capabilities. Raw DevTools target identifiers are not part of the public contract. Selected-tab state is deliberately tri-state. `active: true` or `false` is reported only when the native window title uniquely identifies one DevTools tab. Duplicate titles, empty titles, and other ambiguity produce `active: null` for every candidate instead of treating DevTools list order as native selection evidence. A caller can still target an explicit returned tab id. This preserves one targeting model across native and web content. It also prevents a browser helper from choosing the first tab, first window, or first process match when several candidates exist. ## Exact or refused Mutation is allowed only after independent evidence agrees: - the DevTools endpoint is loopback-only and owned by the requested process; - the native window belongs to that process; - native and DevTools window geometry identify the same surface; and - the binding still holds immediately before each mutation. Electron versions that do not expose a DevTools window identifier have a narrow fallback: the process must own exactly one native window and the endpoint must expose exactly one page. If either count changes, mutation is refused. A heuristic discovery result may be useful for inspection, but it is never promoted into an action route. ## A full-background rung Accessibility actions and targeted pixel injection depend on what the OS and application surface accept. Chromium can reject some background keyboard or pointer routes even when the operating system delivered them correctly. The browser tools add a higher, page-aware rung through the Chrome DevTools Protocol. Page navigation, the default ref-bound text insertion route, and an explicit synthetic DOM click can address an occluded tab without moving the real cursor or borrowing keyboard focus. This is a full-background route for an exactly bound Chromium page, rather than best-effort OS input. Trusted CDP pointer input remains distinct from synthetic DOM events. The default click uses `Input.dispatchMouseEvent`, but Chromium's standalone window is known to activate on that route on macOS and Linux. Cua Driver returns `browser_input_trust_unavailable` there before dispatch. Standalone Chrome and Edge on Windows, and the bounded embedded Electron route, have passing trusted background evidence. The same trust distinction applies to hover, right-click, double-click, scroll, and drag through `browser_pointer`. A caller must explicitly request `input_route: "dom_event"` to invoke an element's DOM click behavior. That route is synthetic even when it preserves full-background posture. A completed JavaScript dispatch does not prove the control activated: applications may ignore events whose `isTrusted` value is false. The result therefore remains `effect: "unverifiable"` and recommends a fresh page-state check. The driver does not silently change trust models or foreground the browser to make a call appear successful. Page-owned JavaScript dialogs are modeled as short-lived capabilities rather than native-window guesses. Inspection returns the kind and an opaque dialog generation; accept or dismiss succeeds only while that exact dialog remains current. The page's initial creation of a Chromium native modal may activate the browser; after occlusion is re-established, inspection and resolution do not require another activation. File assignment bypasses native pickers through an exact live file input ref. Downloads additionally cross an open-world filesystem boundary, so they require host approval, a canonical destination directory, exact event correlation, and path-free output. ## Capabilities have a lifetime Target ids, tab ids, and page refs belong to a named driver session. Page refs also belong to one snapshot. A newer snapshot or navigation invalidates older refs, and ending the session revokes all of its browser capabilities. This makes stale state visible. An agent re-snapshots and retries with current evidence instead of accidentally acting on a node that moved, disappeared, or now belongs to another document. ## Why setup is explicit Browser inspection never enables remote debugging or restarts a browser as a side effect. `browser_prepare` is a separate explicit authorization boundary. It may launch another browser with a driver-owned `isolated_new` or `isolated_named` profile. For an existing supported Chromium profile, a stronger operation-bound grant may instead authorize one exact-window AX, UIA, or AT-SPI setup of the browser's per-instance remote-debugging switch. The route is proven for Chrome and Edge on macOS and Windows and for Chrome in the validated Linux X11 and Sway lanes. It never copies profile data, edits profile files, restarts, or terminates the selected process, and it reports its temporary-tab and setting effects. Remote-debugging arguments passed through `launch_app` are refused. Current macOS Chrome can expose the native address field and selected internal tab while withholding that page's web AX subtree. In that case, the macOS adapter uses only the temporary tab that it created and navigated to the fixed internal URL, requires the committed address value and expected selected-tab title with no active omnibox edit, then requires one unique checkbox-shaped control inside a bounded setup-page region. The click is PID-routed to the revalidated unchanged browser window and the same control's state transition is verified. Because macOS delivers that bounded pixel action through global input, the driver may briefly foreground the exact approved window, then restore the previous frontmost app. The result reports both the foreground and global-input effects. Unsupported appearance, scale, zoom, window-size, or toolbar geometry is refused without a click. It does not generalize that fallback to web pages or arbitrary dialogs. The setup transition and protocol attachment are separate proofs. A listener must be loopback-only, attributed to the approved pid, and either discoverable as DevTools or correlated with the exact approved checkbox transition. The driver then requires a successful DevTools WebSocket claim before reporting an attachment; a bare browser-owned loopback listener is not sufficient. For an existing authenticated profile, standard mode requires either the trusted launch option `--grant existing-profile` or an authorization callback supplied by an embedding host. Bounded mode requires a matching reviewed manifest. Unrestricted mode requires launch-time risk acceptance. A Boolean supplied by the model and ordinary MCP destructive-tool approval do not authorize this boundary. An existing authenticated profile has a stronger boundary than a driver-created isolated profile. Attaching exposes the profile's live pages, cookies, and storage to the browser protocol, so ordinary MCP transport approval is not enough. The attachment grant is bound to one runtime, process fingerprint, native window, and named driver session. ## The CDP trust boundary CDP is used because it is the only supported Chromium interface that can address an exact inactive tab, inspect its document, and perform declared background operations without borrowing the person's keyboard or pointer. That power is also the reason Cua Driver treats attachment as a security boundary rather than a connection detail. CDP exposes broad browser authority, including page runtime, DOM, network, storage, and cookie domains; it is not a least-privilege API for one button or one tab. A loopback listener prevents remote hosts from connecting directly, but loopback is not authentication. Another process running as the same operating system user may be able to discover and connect to an exposed endpoint. Cua Driver's PID ownership proof, native-window correlation, scoped capabilities, and approval grant prevent the driver from attaching to the wrong endpoint; they cannot turn the Chromium endpoint itself into an authenticated service or protect it from unrelated local software. Use a driver-owned isolated profile by default. Attach an existing profile only when the task genuinely needs its authenticated session, only on a trusted machine, and only for the duration of that task. An endpoint that the browser already exposes can use the ordinary exact binding route without another setup transition because Cua Driver did not create or widen the listener. That does not reduce the endpoint's authority: the operator is responsible for how that browser was started and for closing remote debugging when it is no longer needed. Chromium applies its own protections to remote debugging on default profiles. Cua Driver does not bypass those protections, copy a profile, or weaken the browser's data encryption. See Chromium's [remote-debugging security guidance](https://developer.chrome.com/blog/remote-debugging-port) and the [Chrome DevTools Protocol domains](https://chromedevtools.github.io/devtools-protocol/) for the underlying browser boundary. Chrome can ask for consent whenever a genuinely new browser-level socket is opened. Cua Driver therefore keeps one socket for each approved connection generation and multiplexes tabs over it. If the socket drops, one reconnect leader re-proves process, endpoint, and native-window identity before it may press the exact browser-owned consent action. A successful reconnect changes the generation and makes every old target, tab, snapshot, and ref stale. This is deliberate: silently remapping an old capability could act on a different tab after a browser lifecycle change. ## Platform meaning Page-aware routes are cross-platform, but exact native correlation and input trust remain platform and surface capabilities: - macOS uses process ownership and native window geometry. - Windows uses HWND ownership and DPI-aware geometry. - Linux X11 and Sway expose sufficient ownership and geometry for exact routes in the validated configurations. - generic Wayland discovery without compositor-provided exact geometry remains read-only; it cannot authorize mutation. The same rule applies to embedded webviews. Electron's bounded single-page, single-window shape can be exact. A native host and renderer split across different processes, as in common WebView2 deployments, is refused until the driver can prove that relationship independently. See [Drive a web page]() for the workflow and [Known limits]() for current scope. --- # How permission policies work How the Cua Driver permission policy engine evaluates YAML and Rego policies, the trust model, and the guarantees the engine makes. Cua Driver's permission policy engine sits at the native runtime dispatch boundary, between every public caller and the tool implementation. Before a direct SDK runtime, private worker, MCP process, or daemon executes a tool call, it asks the same policy engine whether the call is allowed. ## The enforcement point Every public path reaches the authorization coordinator before platform dispatch. It evaluates the built-in risk map, managed policy, user policy, and optional capability manifest in order. Adapters may repeat a check earlier as defense in depth, but they cannot authorize a request the runtime denies. A denial returns an error to the client and the tool implementation is never reached. ## Deny-by-default The engine is deny-by-default. A tool that is not explicitly mentioned in the policy is blocked. This means adding a new tool to the driver does not automatically expose it to agents; each tool must be explicitly permitted. The deny-by-default behavior applies within each configured policy. When `CUA_DRIVER_POLICY_FILE` is unset, that layer is absent for compatibility. The reviewed built-in tool and risk map still rejects unknown tools, and the default permission mode remains `standard`. An explicitly configured policy path is an operator assertion that the layer must exist. If the path is missing, unreadable, empty, or invalid, runtime construction fails before tools are registered or a service binds its action endpoint. ## Policy composition and modes `CUA_DRIVER_MANAGED_POLICY_FILE` loads an administrator ceiling in the same YAML or Rego formats. A call must pass both the managed and user layers. The runtime hashes each immutable policy snapshot and includes those hashes in authorization-host requests and status output. Permission mode is separate from capability policy. Policy answers whether a call is inside the allowed ceiling. Mode supplies the default autonomy model: `standard` admits the reviewed built-in operation set, `bounded` narrows calls to a reviewed manifest, and `unrestricted` removes Cua's runtime restrictions after explicit launch-time risk acceptance. No mode can widen a managed or user policy ceiling. See [Permission modes and bounded autonomy](). ## Runtime-lifetime snapshot The policy file is loaded once when the runtime starts. All subsequent calls through that runtime generation share the same immutable policy object. There is no reload endpoint and no hot-swap path. Changing the policy takes effect only after a direct runtime, private worker, MCP process, or daemon is restarted. This makes the policy a reliable static contract: the same rule that was in effect when the runtime started remains in effect through its last admitted call. ## YAML evaluation A YAML policy encodes three data structures at load time: - A set of denied tool names (`deny.tools`). These are checked first, before allow rules. - A set of unconditionally allowed tool names (`allow.tools`). - A list of compiled rules (`allow.rules`), each pairing a tool name with a set of compiled constraints. At evaluation time, for a given `(tool, arguments)` pair: 1. If the tool name is in the deny set, return **Deny**. 2. If the tool name is in the allow set, return **Allow**. 3. Find all rules whose `tool` field matches. If there are none, return **Deny**. 4. For each matching rule, test all constraints against the argument object. If all pass, return **Allow**. If any fail, collect the failure reason and try the next rule. 5. If no rule passed, return **Deny** with the collected reasons. Regular expression patterns in YAML constraints are compiled once at load time using the RE2-compatible Rust regex engine. There is no backtracking and no lookahead, which bounds evaluation time regardless of input length. ## Rego evaluation Rego policies are loaded into a [Regorus](https://github.com/microsoft/regorus) engine — a Rust-native OPA evaluator — at process startup. The engine validates that `data.cua.policy.allow` exists and evaluates to a boolean on a synthetic input before accepting the policy. A policy whose rule returns the wrong type is rejected at load time, not at call time. At evaluation time, the driver: 1. Constructs the input object: `{ "server": "cua-driver", "tool": "", "arguments": { … } }`. 2. Clones the engine (the clone carries the compiled policy but gets a fresh input). 3. Sets the input and evaluates `data.cua.policy.allow`. 4. Maps the result: `true` → Allow, `false` or `undefined` → Deny, error → Error. Because Regorus runs inside the runtime owner and does not spawn a policy subprocess, there is no additional policy IPC per call. ## Argument sanitization before evaluation Two transformations happen before the arguments reach the policy engine: - **Internal session-field removal.** Runtime and transport adapters may inject reserved session fields for lifecycle tracking. These are stripped before evaluation so a caller-controlled label cannot change policy authority. - **Tool name canonicalization.** The deprecated `type_text_chars` alias is normalized to `type_text` before any rule is consulted, so policies written against the canonical name cover both forms automatically. ## What the engine does not cover The policy engine controls **which tool calls are executed**. It does not: - Inspect or modify tool responses. - Limit screenshot output, file paths read, or network traffic. - Enforce rate limits or per-session quotas. - Authenticate or identify the caller. A policy that allows `screenshot` permits an agent to take an unlimited number of screenshots. A policy that allows `type_text` with a length constraint still permits the agent to call `type_text` up to the character limit on each call. Use the policy to define an allowed set of operations; combine it with OS-level sandboxing and process isolation if you need stronger guarantees. ## Trust model The policy is evaluated in the same process as the tool implementation. An agent that can replace or inject code into the runtime-owning process can bypass it. Cua Driver does not render authorization UI. Use a trusted host to construct direct runtimes, a trusted launcher to supply explicit launch grants, or OS isolation for workers and services. For remote agents connecting through an authenticated service, the policy provides a meaningful boundary: the service runtime will not execute a tool that the policy blocks, regardless of what the agent sends. ## Related - [Restrict tool access with permission policies](): step-by-step setup guide - [Permission policies](): YAML schema and Rego input interface - [Permission modes and bounded autonomy](): how modes, launch grants, manifests, and policy layers compose - [Process model](): direct, worker, MCP, and service ownership --- # How we continuously validate Cua Driver How source-built desktop harnesses, independent application oracles, and retained evidence validate Cua Driver between releases. Desktop automation crosses boundaries that ordinary unit tests cannot observe. A request can be valid, reach an operating-system API, and return success while the target application receives nothing. Cua Driver therefore treats protocol correctness and observed desktop behavior as two different kinds of evidence. ## Two layers answer different questions Unit and protocol tests answer whether the driver made a deterministic decision correctly. They cover schemas, transport, sessions, element identity, route selection, capture helpers, coordinate conversion, and structured errors. They run without a target GUI application and catch inexpensive regressions early. Harness end-to-end tests answer whether an action actually reached a real desktop surface. They build a small application from source, launch it in a real graphical user session, drive it through the Rust driver, and inspect state owned by the application or desktop rather than trusting the response. Neither layer replaces the other. An E2E result does not exhaustively test protocol edge cases, and a unit test cannot prove that a click changed an application. ## The harnesses model representative surfaces The shared harness presents the same deterministic web behavior through representative renderer hosts on each operating system. Native harnesses then exercise accessibility, controls, windowing, capture, and input APIs that a shared renderer cannot represent. The current surface inventory belongs in [Platform support](); this explanation describes the validation model that remains stable as surfaces are added. These applications are fixtures, not mocks. Each is compiled and launched as a real process. Their purpose is to expose deterministic state for clicks, text, keys, scrolling, dragging, child windows, controls, and editor behavior. This makes a failure reproducible without depending on the changing state of an installed third-party application. The optional standalone-browser lane launches installed Chrome, Edge, or Chromium with a fresh repo-owned profile and one exact source-built driver. Its rows cover binding and ambiguous active-tab evidence, semantic snapshots, frames, inactive tabs, JavaScript dialogs, file assignment, extended pointer actions, and approval-scoped downloads. The target stays fully occluded behind the foreground sentinel while the fixture journal proves page state. This lane is the release evidence for typed browser behavior; CDP acknowledgements alone do not count as delivery. ## One catalog describes each behavior cell The Rust catalog records the dimensions that affect delivery: | Dimension | Examples | | --------------- | ------------------------------------------------------------------------------------------------------ | | Action | left click, right click, double click, type text, key, hotkey, scroll, drag, child window, editor save | | Addressing | AX element or PX coordinate | | Delivery | foreground or background | | Scope | target window or full desktop | | Surface | shared renderer, native toolkit, embedded web view, or compositor | | Expected result | delivered or one exact structured refusal | Foreground and background are dimensions of an action, not separate test families. When a surface supports both delivery modes, both belong in the same catalog. A platform-specific runner establishes the desktop session and collects evidence, but it does not redefine the expected behavior. ## Independent oracles define success A successful driver response is not a passing E2E result. A delivered action must change state that the fixture or desktop independently owns. | Oracle | What it establishes | | -------------------- | --------------------------------------------------------------------------------------- | | Fixture state | The application recorded the click, text, key, selection, scroll, drag, or saved state. | | Accessibility state | UIA, AX, or AT-SPI reports the expected control value or structure. | | Pixel state | Before and after images contain the required visible change. | | Focus and z-order | A background action did not activate or raise the target. | | Cursor state | The physical user cursor did not move when background delivery promised that property. | | Leaked-input journal | The foreground sentinel did not receive input intended for the background target. | | Protocol state | An unsupported route returned the exact declared refusal. | Background checks combine target-state and side-effect oracles. This matters because input delivered to the wrong foreground application is worse than an honest refusal. A refusal passes only when its exact code is expected and the desktop remains unchanged. ## Evidence makes a result auditable Canonical GUI runs retain the typed result for every declared cell, the source commit, per-cell desktop video, before and after state, the tool trajectory, fixture journals, driver logs, and environment preflight. The reporter writes a Markdown summary that links each matrix row to its retained evidence. The reporter rejects missing or duplicate rows, undeclared outcomes, contradictory results, and incomplete required evidence. A failed environment cannot silently turn into a smaller green matrix, and an `ok` response without an observed effect remains a failure. ## Release validation balances cost and fidelity Deterministic unit, protocol, compile, and packaging checks run automatically where configured. Interactive E2E suites require a real graphical session and are maintainer-triggered because they are slower and more environment-sensitive. Windows and Linux have dispatchable GitHub Actions workflows. A maintainer runs macOS in a disposable clone of a stopped Lume seed with a logged-in Aqua session, stable source-build signature, and inherited macOS consent. See [Run Cua Driver macOS tests in a Lume VM]() for that process. Each accepted run builds the driver and fixtures from one exact source commit. Support claims change only after an unchanged behavior cell produces new application-owned evidence or an explicit structured refusal. Skips, weaker oracles, and command-return-only checks do not establish support. ## Related reference - [Platform support]() records currently proven environments and limitations. - [Platform roadmap]() records remaining engineering work and hard platform boundaries. - [Interface contracts]() defines public driver behavior and refusal semantics. - [Run Cua Driver macOS tests in a Lume VM]() describes the maintainer-owned macOS acceptance gate. --- # How sandboxes work How a Cua Sandbox gives an agent one isolated computer it can both run code in and drive through the GUI. A Cua Sandbox is a **full, isolated computer**, not a remote desktop session. In that one machine, an agent can run code through Python, shell commands, or a PTY, and it can also drive the graphical interface through screenshots, the accessibility tree, clicks, and typing. Those are two complementary halves of the same computer. The code half and the GUI half share one filesystem, one set of processes, and one OS state. The value is that they live in the same place. An agent can click through an app and then run a Python function against the files or process state that app produced. It can also set up state in code and then automate the UI over that state. For the broader model that combines code, structured tools, and graphical interfaces, read [What is computer use?](). ## Sandboxes and real machines Cua Driver observes and controls the desktop where it is installed and targeted. That can be your existing computer or a guest with a compatible installation and connection path. Running Driver on your host does not automatically target a sandbox guest. A sandbox is an isolated computer created for a task. It starts from an image and accumulates its own state. Deleting it discards that state, so save any results you need first. Code and GUI actions through the sandbox connection target the guest. Explicitly shared files and exposed services remain connections to the outside world. This isolation is what makes a sandbox useful for repeatable agent work. The agent gets a whole computer, but the effects are contained within that computer. ## The code half and the GUI half The _code half_ is how an agent runs programs inside the sandbox. It includes shell commands through `shell.run`, an interactive PTY or terminal through `computer.pty` and `cua do shell`, and sandboxed Python that runs a function inside the sandbox's own virtualenv through `venv_install`, `venv_exec`, and the `@sandboxed` decorator. The _GUI half_ is how an agent uses the sandbox like a desktop computer. It can observe the screen through screenshots and the accessibility tree, then act through clicks, typing, scrolling, keypresses, and other input events. The Sandbox SDK's default Fleet connection uses the guest's computer-server endpoint. Fleet provisions and manages the capacity; the SDK acquires a guest and connects to its service. Using Cua Driver in that guest requires its own compatible installation and validated integration; it does not follow merely from creating a pool. See [How Fleet images work]() for the default guest service contract. Because both halves are interfaces to the same machine, they can be mixed within one task. A shell command can create a file that is opened in the GUI. A GUI workflow can download data that is then inspected with Python. A PTY session can start a server, and the GUI half can open a browser against it. The important point is that all of these actions share the same filesystem and OS state. ## Containers and full VMs A Linux container sandbox starts quickly because it shares the host kernel. It layers a Linux userspace, a desktop such as XFCE, and a remote display stack such as KasmWeb. This makes startup fast, but it is not identical to a physical Linux box. Kernel behavior, device access, and isolation come from the container host. A full VM emulates hardware and boots its own kernel. macOS sandboxes use Apple Virtualization. Windows sandboxes use QEMU or Hyper-V. Android sandboxes use QEMU. Full VMs are slower to start than containers, but they provide higher OS fidelity because the guest OS owns its kernel and hardware model. The trade-off is startup latency versus OS fidelity. Containers are suited to fast Linux environments. Full VMs are suited to work that depends on the behavior of a complete guest operating system. ## Images as starting-state contracts An `Image` is the **immutable** description of the sandbox's starting environment. It is not the running sandbox. An `Image` defines the OS type, distro or OS version, packages, environment variables, copied files, and setup commands that should exist when a sandbox starts. The image builder composes layers such as `apt_install`, `pip_install`, `run`, `copy`, and `env`. Local execution applies those layers at launch to produce the sandbox's initial state. Fleet instead boots a prebuilt registry artifact and rejects these builder layers. Installing another package later changes that sandbox, not the Image. [How Fleet images work]() explains the published-artifact model. This separation makes environments reproducible. The `Image` describes what a fresh sandbox should look like. The sandbox is the live machine created from that description. ## Lifecycle patterns The patterns below describe local sandbox ownership. Fleet pools and claims add a separate capacity lifetime: releasing a claim does not delete its pool or remove the pool's desired warm capacity. [Cloud Fleets]() links the hosted workflow and resource cleanup guides. Sandbox lifetime is separate from agent connection lifetime. An ephemeral sandbox exists for one block of work and is destroyed at the end. This fits CI jobs, tests, and one-shot tasks where the state has no value afterward. A persistent or named sandbox is created once, identified by name, and can survive process exits. A later process can reconnect to the same sandbox and continue from the state it already has. Connect mode attaches to an already-running sandbox. It does not create a sandbox and it does not delete one. Disconnecting drops the control connection. Deleting destroys the machine and its state. Those are different operations, and the distinction matters when the sandbox contains work that should survive the current process. [Sandbox lifecycle]() explains local ownership and Fleet capacity; [Manage local sandbox lifecycle]() shows the local SDK patterns. ## Snapshots and forks A _snapshot_ records a machine's state for later reuse. A _fork_ starts a separate machine from that captured state. These mechanisms can avoid repeating expensive setup, but they require support from the provisioning and storage system. In `cua-sandbox` 0.4.3, `Sandbox.snapshot()` is not implemented for local sandboxes or the Fleet creation paths. Fleet also rejects snapshot-derived image inputs. For these paths, prepare a reusable boot artifact instead. See [Sandbox runtime support]() for the versioned limits and [Prepare and reference a Fleet image]() for the published-artifact workflow. ## Local execution Local mode runs on the developer's hardware. Linux containers use Docker Desktop, macOS VMs use Lume, and VM backends can use QEMU or Hyper-V. Local execution depends on the machine's available CPU, memory, disk, and virtualization support. Select it with `local=True`. The sandbox still starts from an `Image`, exposes a code half and a GUI half, and can be ephemeral or persistent. See [Choose and build a sandbox image]() for local setup. ## Fleet execution Fleet runs sandboxes as managed capacity. A pool defines the boot artifact and capacity; a claim reserves a sandbox for a workload. Preparing the image and acquiring a running sandbox are separate operations, so the guest's dependencies and services must already be part of the published artifact. Choosing local execution or Fleet changes the provisioning and connection path, not the idea of an isolated computer. It can also change which image inputs, customization methods, snapshots, and port connections are available. Use [Sandbox runtime support]() for the versioned contract, or [Your first Cloud Fleet]() for a guided hosted path. --- # How Fleet images work How Fleet boot artifacts define the operating system, compatibility boundary, and service contract for managed sandbox capacity. A Fleet image is a prebuilt guest boot artifact that a pool uses to start its sandbox replicas. The Sandbox SDK's `Image` specification selects that artifact; the pool boots it before a workload claims a replica. ## Fleet images and Sandbox SDK Images are separate layers The public `Image` type is the sandbox API's immutable environment specification. It lets you describe a sandbox's operating system, packages, copied files, setup commands, and ports before you start that sandbox. A Fleet image is the guest boot artifact referenced by a pool definition. A pool can keep multiple replicas running from the same artifact, and later workloads claim those replicas. The Fleet image is therefore an input to capacity management and replica compatibility rather than a step in one sandbox's SDK setup sequence. The public SDK model is documented in [How sandboxes work](), [Choose and build a sandbox image](), and the [Image reference](). ## Fleet uses a constrained subset of the SDK image surface Fleet uses a prebuilt registry artifact as its starting point. The Sandbox SDK can reference that artifact directly with `Image.from_registry(...)` or resolve a built-in image with a registry mapping. In `cua-sandbox` 0.4.3, the built-in Ubuntu 24.04 and Windows Server 2022 VM images have registry mappings. Fleet does **not** build an environment from SDK layers such as `apt_install`, `pip_install`, `uv_install`, `run`, `copy`, `env`, `from_file`, or snapshot-derived images. The Fleet registry input must reference a guest artifact published before sandbox creation. The versioned acceptance rules and their verification limits are in [Sandbox runtime support](). The publication and reference workflow is in [Prepare and reference a Fleet image](). ## Fleet references an OCI artifact that carries the boot image At the pool layer, Fleet stores the registry reference as the deployed boot artifact for the pool template. The referenced OCI artifact must carry the bootable guest image expected by Fleet. The SDK contract is the registry reference itself; the build pipeline determines how the bootable guest is packaged inside that artifact. `Image.from_registry(...)` does not assemble the artifact. It points Fleet at a prebuilt OCI image that already carries the bootable environment. Artifact publication occurs before the claim path. ## Image identity defines compatibility A pool's image reference is part of what makes one replica compatible with another. Two pools with different boot artifacts can differ in operating system, installed software, startup behavior, readiness checks, and service endpoints even if they request the same CPU and memory. A pinned image reference identifies the artifact used by a pool. With a mutable tag, registry content can change while the pool specification remains unchanged, so replicas created at different times can boot different artifact contents. A digest identifies specific registry content. It does not establish that Fleet can pull the artifact, that its guest architecture matches the runtime, or that its services work. Those require separate evidence. The [OS and image catalog]() records image identities and their evidence boundaries. ## Boot configuration must match the image The image and the pool's boot configuration must agree. Firmware expectations, disk layout, guest drivers, and startup services are properties of the boot artifact and its runtime configuration. Linux-oriented and Windows-oriented artifacts can require different firmware, devices, and service startup contracts before their configured services become ready. ## Service behavior starts at the image boundary A Fleet image does not only determine what operating system boots. It also determines what software is already present when a replica starts and which services can become ready inside that guest. Readiness depends on the services started by the image. A running replica whose configured readiness checks have not passed remains unready for claims. Ports you mark with `.expose(...)` become named Fleet services. A service URL identifies an endpoint in Fleet's proxy; it does not create a localhost listener or grant access by itself. Access requires Fleet authentication. In 0.4.3, pool-backed sandboxes use `sb.services.request()` for authenticated HTTP requests; their transport does not implement `sb.tunnel.forward()`. See [ports and transports](). With the Sandbox SDK's default service configuration, the published image must start the computer-server endpoint on port `8000`. The SDK creates the `server` service and a TCP readiness probe on that port, then waits for `/status` before returning a connected sandbox. ## Image contents are present before claims Dependencies and startup behavior included in the image are present when a replica boots. The claim path does not apply SDK image-builder layers. Changing the referenced image version requires publishing the corresponding artifact and updating the pool reference. Replicas created from the same immutable artifact receive the same image contents. ## Related pages - [How sandboxes work]() - [Choose and build a sandbox image]() - [Image reference]() --- # Sandbox lifecycle How connection lifetime, local sandbox lifetime, and Fleet pool capacity differ. A connection, a local sandbox, and a Fleet pool have separate lifetimes. Choose who owns each resource before deciding when to disconnect or clean up. ## Local machine and connection lifetime An ephemeral local sandbox belongs to one block of work. Its context manager calls cleanup when the block exits. A persistent local sandbox outlives the creating script until you delete it. Give it a name so another process can reconnect later. `Sandbox.connect` attaches to an existing running local sandbox. Closing that connection leaves the sandbox running; deleting the sandbox discards its state. Save results before deleting it. The [local lifecycle guide]() shows creation, reconnection, and cleanup patterns. ## Fleet claims and pool capacity On Fleet, `disconnect()` drops the connection without releasing the claim. Exiting a `pool.claim()` context releases the claim, but the pool can keep warm capacity available and incur cloud usage charges. Deleting the pool is a separate operation that removes its capacity and state. The [Pool reference]() defines ownership and release behavior. [Expire pools and claims automatically]() explains cleanup deadlines. Keep claim cleanup inside each workload and delete a pool only when you own its lifecycle; a shared pool can serve other callers. ## Runtime limits Local execution and Fleet share SDK concepts, but that does not make their lifecycle operations interchangeable. See [Sandbox runtime support]() for versioned image and transport limits. In `cua-sandbox` 0.4.3, `Sandbox.snapshot()` is not implemented for local sandboxes or Fleet creation paths, and Fleet rejects snapshot-derived images. ## Create and auto-destroy a sandbox See [Create and auto-destroy a sandbox]() for the local SDK pattern. ## Create a persistent sandbox See [Create a persistent sandbox]() for the local SDK pattern. ## Reconnect to a running sandbox See [Reconnect to a running sandbox]() for the local SDK pattern. ## List running sandboxes See [List running sandboxes]() for the local SDK pattern. ## Choose a local runtime See [Choose a local runtime]() for the local SDK pattern. --- # How Lume creates local macOS sandboxes How Lume turns Apple restore images into unattended local macOS sandboxes. Lume creates vanilla macOS VMs from Apple restore images. The unattended setup prepares the installed guest on disk, so the workflow does not depend on GUI clicks or a particular Setup Assistant layout. ## The setup sequence When you run `lume create` with `--unattended tahoe` or `--unattended sequoia`, Lume: 1. Installs macOS from the IPSW into a new VM disk. 2. Boots the VM once so macOS materializes its first-boot state. 3. Stops the VM and mounts its Data volume on the host. 4. Patches the guest account and login configuration offline. 5. Enables SSH, autologin, and the no-sleep settings. 6. Boots the VM again and verifies SSH before finalizing creation. The default account is `lume` with the initial password `lume`. Change that password before using the VM for sensitive work. ## Why the setup is offline GUI automation depends on display timing, window state, and the controls that a particular macOS release exposes. Disk setup writes the desired guest state directly and keeps the verification step small: Lume checks that the guest boots and accepts SSH. ## Boot policy changes use Recovery Offline setup can change guest files, but it cannot sign Apple silicon boot policy. SIP changes therefore use the VM's paired Recovery environment and a normal boot verification. Read [How SIP works in Lume VMs]() for the policy model. ## Presets and images The `tahoe` and `sequoia` presets use the same offline patcher. Tahoe is the verified preset for the current vanilla workflow. Sequoia can still show the Accessibility step of Setup Assistant on its first display boot; see [issue #2155](https://github.com/trycua/cua/issues/2155). The IPSW flow creates a fresh vanilla guest each time. Registry snapshots remain available for workflows that need a previously prepared disk, but they are not required for unattended setup. ## Related pages - [Create your first local macOS VM]() - [Create a vanilla Tahoe VM]() - [Change SIP on a macOS VM]() - [Lume CLI reference]() --- # How SIP works in Lume VMs Understand signed LocalPolicy state, paired Recovery, and why Lume verifies SIP after reboot. System Integrity Protection (SIP) on an Apple silicon VM is part of a signed boot policy. Changing files on the macOS Data volume cannot change that policy. ## SIP state is a signed LocalPolicy Apple silicon stores SIP configuration in a LocalPolicy signed by the Secure Enclave. Virtualization.framework provides the corresponding security service for a macOS VM. The policy belongs to a larger paired state: - `disk.img` contains macOS, personalized boot files, and Recovery; - `nvram.bin` contains auxiliary security and anti-replay state; and - the VM configuration identifies the virtual hardware that boots them. Copying or editing one part does not mint a valid policy for another VM. ## Offline setup and SIP use different paths Lume's unattended setup mounts the Data volume and writes account, SSH, autologin, and power settings. Those settings are ordinary guest files, so an offline patcher can prepare them without driving Setup Assistant. SIP is different because the policy must be signed from the VM's paired Recovery environment. Editing `disk.img` or patching bytes in `nvram.bin` cannot produce a valid signature. ## Why `lume sip` uses Recovery `lume sip` coordinates three VM sessions: 1. A normal boot checks that the requested administrator account works over SSH. 2. A Recovery boot opens Terminal over a temporary VNC session and runs `csrutil enable` or `csrutil disable`. 3. A final normal boot runs `csrutil status` over SSH. During Recovery, Lume checks the named account prompt and reads the terminal result with OCR before it stops the VM. Tahoe Recovery does not always halt after its shutdown command, so Lume can stop that exact `lume run` process after a grace period. It waits for the VM to report `stopped` before the final boot. ## Verification happens after reboot The Recovery message confirms that `csrutil` accepted the policy change. The normal boot proves that the VM can use the new policy. Lume accepts only the canonical top-level status: ```text System Integrity Protection status: enabled. ``` or: ```text System Integrity Protection status: disabled. ``` A customized policy can list enabled and disabled components at the same time. Lume rejects that output instead of inferring a state from one component line. ## Clones keep paired files together `lume clone` copies the VM disk and auxiliary storage as a pair. This is the supported way to reproduce a prepared seed. Copying only `disk.img` loses the matching security state; copying only `nvram.bin` loses the matching boot files. ## Related pages - [Change SIP on a macOS VM]() - [How Lume creates local macOS sandboxes]() - [Lume limits]() --- # How Lume expands macOS disks Understand RecoveryOS relocation, APFS growth, and rollback during a Lume disk resize. A macOS virtual disk contains more than the main APFS container. Apple silicon macOS installations also include iBoot System Container (ISC) and paired RecoveryOS partitions. Their order determines where new capacity appears when the disk image grows. ## Why growing the image is insufficient A standard Lume macOS disk has this layout: ```text [ ISC ][ main APFS ][ RecoveryOS ] ``` Increasing the length of `disk.img` appends free sectors after RecoveryOS: ```text [ ISC ][ main APFS ][ RecoveryOS ][ free space ] ``` APFS can grow only into adjacent free space. RecoveryOS therefore prevents the main container from using the appended capacity. Increasing the image length alone changes the virtual device size without increasing the guest filesystem. ## RecoveryOS moves with its identity intact Lume resizes a stopped macOS VM offline. It copies RecoveryOS to the new end of the disk, preserving the partition type, unique identifier, attributes, size, and contents. It then rewrites both GUID Partition Table copies and asks `diskutil` to grow the main APFS container into the adjacent space. ```text [ ISC ][ expanded main APFS ][ RecoveryOS ] ``` Keeping RecoveryOS preserves the paired recovery environment used by macOS updates and startup-security operations. Deleting the partition can make the space available, but leaves the VM without that paired environment. ## Resizing is a transaction Before modifying the disk, Lume validates the partition table, confirms that the VM is stopped, and creates a copy-on-write backup when the storage volume supports it. A persistent marker records the transaction phase. Run, clone, and push operations refuse to use the VM while that marker exists. Lume verifies the relocated RecoveryOS bytes and the expanded APFS capacity before it updates the configured disk size. If a step fails, it restores the disk and configuration from the backup. A later resize invocation also detects and restores an interrupted transaction. Skipping the backup removes automatic rollback. It does not relax partition, VM-state, or post-resize verification checks. ## Supported layouts The resizer accepts the standard three-partition Lume macOS layout and fails closed when it finds unfamiliar geometry. It supports increases only. FileVault guests are excluded because offline host tools cannot safely validate and grow their encrypted APFS state through this flow. Linux disk images do not contain paired RecoveryOS. Lume can increase their image size, but the guest remains responsible for growing its partition and filesystem. ## Related pages - [Manage local Lume VMs]() - [How SIP works in Lume VMs]() - [Lume CLI reference]() --- # What is Cua-Bench? Understand how Cua-Bench defines, runs, and scores verifiable computer-use tasks. Cua-Bench is an MIT-licensed framework for computer-use benchmarks and reinforcement-learning environments. It packages the task, starting state, agent interface, and evaluator needed to run a repeatable experiment across Linux, Windows, Android, browser, or simulated environments. New to Cua-Bench? Follow [Build your first Cua-Bench task]() to create and verify a small simulated task. [An open-source registry and runner for computer-use tasks]() The overview follows a task from the registry through variations, agent adapters, the CLI runner, and self-hosted execution. ## Why tasks are environments A computer-use score is meaningful only when the starting state and success condition are repeatable. A Cua-Bench task therefore defines more than a prompt. It prepares an environment, exposes observations and actions to an agent, and evaluates the final state with task-specific checks. Task variations test whether an agent can solve the same objective from different inputs or starting conditions. Datasets group related tasks so a run can use one agent configuration and produce comparable traces and scores. ## How the pieces fit - **Tasks** define the prompt, setup, variations, and evaluator. - **Providers** supply the browser, simulated surface, container, VM, or hosted computer where the task runs. - **Agent adapters** connect a model or agent implementation to the environment interface. - **The runner** executes one task or a dataset, records traces, and supports parallel workers. This separation lets the same task definition run against different agents and infrastructure. It also keeps evaluation logic independent from the model being tested. ## Relationship to the rest of Cua Cua-Bench supplies the task and evaluation layer. Cua computer and sandbox components supply machine environments, while agent adapters translate model output into computer actions. The resulting trajectory can be inspected, scored, or used as training data. [Browse the source]() Read the package, task examples, CLI implementation, and test suite. [Browse the task registry]() Explore published tasks and the applications they cover. --- # The Cua-Bench task lifecycle Understand how task variants, setup, solutions, and evaluators form a repeatable experiment. A Cua-Bench task is a repeatable experiment, not only a natural-language instruction. It combines the state an agent starts from, the interface it can operate, and the evidence used to score the result. ## One definition, many variants The task configuration returns one or more variants. Every variant has its own prompt and may carry different metadata, provider settings, operating-system theme, or initial state. Variants let a benchmark change inputs without duplicating its setup and evaluation code. An agent can therefore face the same underlying skill across different labels, layouts, or operating systems. ## Four lifecycle functions A task module describes four distinct phases: 1. **Configuration** returns the task variants available for a dataset split. 2. **Setup** prepares the selected variant and opens the applications or interface the agent will use. 3. **Solve** provides an optional oracle: a known procedure that exercises the task and checks whether the environment can produce a successful result. 4. **Evaluate** inspects final state and returns one or more rewards. Configuration is shared while setup, solving, and evaluation operate on one selected variant and its session. ## Why the oracle and evaluator are separate The oracle describes one way to reach the goal. The evaluator defines what counts as reaching it. Keeping them separate prevents a benchmark from scoring an agent by whether it copied one prescribed sequence of clicks. The same evaluator can score an oracle, a human attempt, or an agent trajectory. An oracle reward below the expected score usually indicates a task or environment problem rather than an agent-quality result. ## Providers and sessions The provider in a variant's computer configuration determines where the task runs: - a **simulated** provider renders a lightweight desktop through Playwright; - a **native** provider supplies a real operating-system environment through the configured container or virtual-machine platform. Setup, solve, and evaluate functions receive a session interface. The task uses that interface to open windows, execute input actions, capture screenshots, or read state needed for evaluation. This keeps the lifecycle consistent while the underlying environment changes. ## From one task to a benchmark A dataset groups task definitions. The runner expands their variants, assigns them to environments, records their results, and aggregates rewards. Traces preserve the observations and actions from individual sessions so failures can be inspected after the run. The benchmark score is therefore the end of a chain: ```text dataset → task → variant → environment session → trajectory → evaluator → reward ``` The [task definition reference]() describes the exact Python contract. To see the lifecycle operate end to end, build [your first Cua-Bench task](). --- # Use Cua with Choose a model provider, agent harness, or local runtime for Cua Driver. Cua Driver is the computer-use layer. An agent harness connects to it over MCP or through the Cua Driver CLI, then gives a model screenshots, accessibility state, and input tools. Pick the part of the stack you are starting from. ## Models and providers These pages show the verified harnesses and local runtimes that can put each model family in front of Cua Driver. Cua Driver does not connect directly to a model API. [Anthropic / Claude]() Use Claude through Claude Code or another MCP-capable harness. [OpenAI]() Connect Codex to Cua Driver through its native MCP support. [Google / Gemini]() Use Antigravity CLI as the MCP-capable harness. [xAI / Grok]() Connect Grok Build or Grok Bot to Cua Driver. [Meta]() Run Muse Glimmer locally with a harness such as Claude Code. [Qwen]() Use Qwen Code or serve a Qwen model through a local runtime. [Kimi]() Connect Kimi Code CLI through its MCP support. [MiniMax]() Use MiniMax M-series models through Codex or Claude Code. ## Agent harnesses Harnesses own the agent loop and invoke Cua Driver. Some are tied to a model provider; others let you choose the model separately. [Claude Code]() [OpenAI Codex]() [Antigravity CLI]() [Grok Build]() [Grok Bot]() [Hermes]() [OpenClaw]() [T3 Code]() [Qwen Code]() [Kimi Code]() ## Local runtimes [Ollama]() Launch a compatible harness against a locally served model. [llama.cpp and Unsloth]() Use the tested Muse Glimmer GGUF path on Apple Silicon. If you already know which agent you want to use, go straight to [Connect your agent to Cua Driver](). --- # Anthropic / Claude Use Claude models with Cua Driver through an agent harness. Anthropic provides the Claude model family. To give Claude access to a desktop, run it in an agent harness that can call Cua Driver. The most direct path is [Claude Code](), which supports local MCP servers. | Start here when… | Recommended path | | -------------------------------------- | ---------------------------------------------------------------------------------------------- | | You use Claude Code | [Connect Claude Code to Cua Driver]() | | You use another MCP client with Claude | [Use the generic MCP configuration]() | | You want an isolated desktop | [Start a Cloud Fleet]() | Claude Code can also act as the harness for a model served locally. That is a harness feature, not a direct connection between Cua Driver and Anthropic's API. See Anthropic's [MCP documentation](https://docs.anthropic.com/en/docs/mcp) and [Claude Code CLI reference](https://docs.anthropic.com/en/docs/claude-code/cli-usage) for the upstream client behavior. [Claude Code builds and checks a Windows app]() Claude Code uses Cua Driver to operate a WPF app without taking over the user's active window. --- # OpenAI Use OpenAI models with Cua Driver through Codex. OpenAI Codex supports local MCP servers in its CLI, IDE extension, and desktop app. Register Cua Driver once, then use its tools from a Codex session. | Start here when… | Recommended path | | ------------------------------------------ | ---------------------------------------------------------------------------------------------- | | You use Codex | [Connect Codex to Cua Driver]() | | You use another MCP-capable OpenAI harness | [Use the generic MCP configuration]() | | You are building your own agent | [Read the MCP tool reference]() | Cua Driver supplies desktop state and actions. Codex owns the model interaction, reasoning loop, and tool selection. See OpenAI's current [Codex MCP documentation](https://learn.chatgpt.com/docs/extend/mcp?surface=cli) for supported transports and configuration. --- # Google / Gemini Use Gemini models with Cua Driver through Antigravity CLI. Antigravity CLI can run local stdio MCP servers, so it can use Cua Driver as its computer-use layer. | Start here when… | Recommended path | | --------------------------------- | ---------------------------------------------------------------------------------------------- | | You use Antigravity CLI | [Connect Antigravity CLI to Cua Driver]() | | You are migrating from Gemini CLI | Follow Google's [migration guide](https://antigravity.google/docs/cli/gcli-migration) | | You configure MCP with JSON | [Use the generic MCP configuration]() | Google ended Gemini CLI service for individual Google AI Pro, Ultra, and free-tier accounts on June 18, 2026. Enterprise licenses and paid API-key access remain supported, but Cua recommends Antigravity CLI for new terminal setups. Cua Driver keeps `--client gemini` as a legacy alias for the Antigravity configuration. See Google's [transition announcement](https://github.com/google-gemini/gemini-cli/discussions/28017) and [Antigravity MCP documentation](https://antigravity.google/docs/mcp) for the current client behavior. --- # xAI / Grok Use Grok Build or Grok Bot with Cua Driver. Use Grok Build when the agent and Cua Driver run on the same computer. Use Grok Bot when a persistent cloud agent should operate apps on your local computer, or when it should drive Grok Bot's own cloud computer. | Start here when… | Recommended path | | ----------------------------------------- | -------------------------------------------------------------------------------- | | You use Grok Build | [Connect Grok Build to Cua Driver]() | | You use Grok Bot | [Let Grok Bot drive local or cloud-computer apps]() | | You are building a custom Grok agent | Call Cua Driver through [MCP]() or the CLI | | You want to manage the agent from T3 Code | Configure Grok Build first, then [use it through T3 Code]() | Cua Driver does not call the Grok API directly. Grok Build can launch Cua Driver as a local tool server. Grok Bot can invoke the local CLI under its local-computer policy, install Cua Driver on its own cloud computer, or reach an authenticated Streamable HTTP MCP endpoint through a gateway. See xAI's [Grok Build overview](https://docs.x.ai/build/overview), [MCP server guide](https://docs.x.ai/build/features/mcp-servers), and [Grok Bot overview](https://docs.x.ai/grok-bot/overview). --- # Meta Run Meta computer-use models locally with Cua Driver. Meta's Muse Glimmer 30B is an open-weight vision-language model designed for local agentic computer use. A runtime serves the model, an agent harness owns the loop, and Cua Driver supplies screenshots, accessibility state, and input actions. The tested Cua path uses: - Meta's Muse Glimmer 30B model - Unsloth's `UD-Q4_K_XL` GGUF - `llama.cpp` on Apple Silicon - Claude Code as the harness - Cua Driver running inside a macOS Lume guest Follow [Run a local model with Cua Driver]() for the complete setup, including the smaller tool surface used to control context growth. See Meta's [Muse Glimmer model card](https://huggingface.co/meta-models/Muse-Glimmer-30B) and Unsloth's [Muse Glimmer guide](https://unsloth.ai/docs/models/muse-glimmer) for the upstream model and quantization details. [Create a checklist in Apple Notes]() Muse Glimmer runs locally and completes the task through Cua Driver. [Schedule an Apple Reminder]() Muse Glimmer sets the reminder and verifies the final state. --- # Qwen Use Qwen models with Cua Driver through Qwen Code or a local runtime. Qwen models can reach Cua Driver through an MCP-capable harness or a local serving stack. | Start here when… | Recommended path | | ------------------------------ | ---------------------------------------------------------------------------------------------------------------- | | You use Qwen Code | [Connect Qwen Code to Cua Driver]() | | You serve a Qwen model locally | Choose [Ollama]() or [llama.cpp](), then connect a compatible harness | | You use another MCP client | [Use the generic MCP configuration]() | The model and computer-use layer remain separate. Qwen Code or another harness selects tools; Cua Driver observes and acts on the desktop. See Qwen Code's [MCP documentation](https://qwenlm.github.io/qwen-code-docs/en/users/features/mcp/). --- # Kimi Use Kimi models with Cua Driver through Kimi Code CLI. Kimi Code CLI supports local stdio MCP servers. Configure Cua Driver in the CLI, then let Kimi Code invoke its desktop tools as part of the agent loop. | Start here when… | Recommended path | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- | | You use Kimi Code CLI | [Connect Kimi Code to Cua Driver]() | | You manage MCP with a project file | Add Cua Driver to `.kimi-code/mcp.json` using the [generic MCP shape]() | | You are building another Kimi agent | Use the [MCP tools]() or Cua Driver CLI | See Kimi Code CLI's [MCP customization guide](https://www.kimi.com/code/docs/en/kimi-code-cli/customization/mcp.html) and [`kimi mcp` reference](https://moonshotai.github.io/kimi-cli/en/reference/kimi-mcp.html). --- # MiniMax Use MiniMax M-series models with Cua Driver through Codex or Claude Code. MiniMax documents M-series model configurations for both Codex and Claude Code. Configure the harness to use MiniMax, then connect that same harness to Cua Driver over MCP. Choose the path for your harness: | Start here when… | Recommended path | | ----------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | You use Codex | Follow MiniMax's [Codex setup](https://platform.minimax.io/docs/token-plan/codex), then [connect Codex to Cua Driver]() | | You use Claude Code | Follow MiniMax's [Claude Code setup](https://platform.minimax.io/docs/token-plan/claude-code), then [connect Claude Code to Cua Driver]() | | You use another MCP-capable harness | Configure MiniMax using its [model invocation guide](https://platform.minimax.io/docs/guides/text-generation), then [add Cua Driver over MCP]() | Cua Driver supplies desktop state and actions. The harness owns the MiniMax model connection, reasoning loop, and tool selection. MiniMax recommends its Anthropic-compatible API for direct model calls and also provides an OpenAI-compatible API. Use the upstream guides above for the supported endpoints, model names, and authentication settings. --- # Claude Code Connect Claude Code to Cua Driver over MCP. Claude Code can launch Cua Driver as a local stdio MCP server. Install Cua Driver first, then register the server: ```bash claude mcp add --transport stdio cua-driver -- cua-driver mcp claude mcp list ``` For an absolute-path command generated by the installed Cua Driver version, run: ```bash cua-driver mcp-config --client claude ``` Restart Claude Code or open a fresh session after adding the server. On macOS, the Cua Driver app identity that owns the runtime needs Accessibility and Screen Recording permission. Continue with [Connect your agent to Cua Driver]() for permission modes, the computer-use compatibility profile, and local-model setup. [Claude Code checks its work in a Windows app]() The agent builds a WPF app, operates it through Cua Driver, patches the code, and checks it again. Upstream: [Claude Code MCP documentation](https://docs.anthropic.com/en/docs/mcp). --- # OpenAI Codex Connect Codex to Cua Driver over MCP. Codex supports local MCP servers in the CLI, IDE extension, and desktop app. Generate the command with the installed Cua Driver version: ```bash cua-driver mcp-config --client codex ``` The generated command uses the absolute executable path. Its shape is: ```bash codex mcp add cua-driver -- /absolute/path/to/cua-driver mcp codex mcp list ``` Open a fresh Codex session after registration. You can also install the Cua Driver agent skill for action-selection guidance: ```bash cua-driver skills install cua-driver skills status ``` Continue with [Connect your agent to Cua Driver]() and [Install the Cua Driver agent skill](). Upstream: [Codex MCP documentation](https://learn.chatgpt.com/docs/extend/mcp?surface=cli). --- # Antigravity CLI Connect Google's Antigravity CLI to Cua Driver over MCP. Antigravity CLI supports local stdio MCP servers. Generate the current Cua Driver configuration: ```bash cua-driver mcp-config --client antigravity ``` Merge the generated `cua-driver` entry into the top-level `mcpServers` object in the global configuration: ```text ~/.gemini/config/mcp_config.json ``` For project scope, use `.agents/mcp_config.json` in the workspace. The configuration has this shape: ```json { "mcpServers": { "cua-driver": { "command": "/absolute/path/to/cua-driver", "args": ["mcp"] } } } ``` Restart `agy` after changing the file, then open `/mcp` to inspect the server status. Cua Driver's legacy `--client gemini` alias prints the same Antigravity configuration. Continue with [Connect your agent to Cua Driver]() for runtime and permission guidance. Upstream: [Antigravity MCP documentation](https://antigravity.google/docs/mcp) and Google's [Gemini CLI migration guide](https://antigravity.google/docs/cli/gcli-migration). --- # Grok Build Connect xAI's Grok Build coding agent to Cua Driver over MCP. Grok Build is xAI's coding CLI over stdio MCP. It is not [Grok Bot](). Grok Build supports local stdio MCP servers. Find the installed Cua Driver path, then register it: ```bash command -v cua-driver grok mcp add cua-driver -- /absolute/path/to/cua-driver mcp grok mcp list grok mcp doctor cua-driver ``` The `--` separates Grok Build's options from the Cua Driver server command. Grok Build stores user configuration in `~/.grok/config.toml`; add `--scope project` if the configuration belongs to one repository. Cua Driver currently has no dedicated Grok Build preset, so this page follows Grok Build's native MCP command and the standard `cua-driver mcp` entry point. Continue with [Connect your agent to Cua Driver]() for installation, permissions, and runtime modes. Upstream: [Grok Build MCP server documentation](https://docs.x.ai/build/features/mcp-servers). --- # Grok Bot Let Grok Bot operate desktop apps through Cua Driver on your local computer or on its own cloud computer. **Note** This page is Grok Bot, the persistent cloud agent. It is not [Grok Build](), xAI's coding CLI over stdio MCP. [Grok Bot](https://docs.x.ai/grok-bot/overview) runs on a persistent cloud Linux computer, separate from your laptop. Cua Driver can drive either desktop: - **Your local Mac or Windows computer.** Install Cua Driver there. Grok Bot uses local-command execution (`cua-driver call …`) with approval. - **Grok Bot's own cloud computer.** If that is the GUI being driven, install Cua Driver on that machine and call `cua-driver` there. Use command execution for the first setup on either path. This keeps Cua Driver on the computer it controls. ## Drive your local computer Install Cua Driver on the computer with the apps you want to control, grant the required operating-system permissions, and start its daemon. Verify the local CLI before involving Grok Bot: ```bash cua-driver --version cua-driver doctor cua-driver call list_apps '{}' ``` See [Install Cua Driver]() and [Keep Cua Driver running]() if these commands do not succeed. ## Allow local commands In Grok Bot, open **Settings → General → Agent → Execution on Local Computer** and choose **Ask every time**. This is xAI's default and lets you review each local command before it runs. The Bot's cloud computer and your local computer are separate. Make this boundary explicit in the task: ```text Use command execution on my local computer for every cua-driver command. Do not install software, change Cua Driver permissions, or start an unrestricted daemon. Before each UI action, get fresh window state and use an element token from that response. Get fresh state after the action and verify the result. Stop for approval before sending, publishing, purchasing, deleting, changing permissions, or modifying a production system. ``` Start with a read-only request: ```text Run `cua-driver call list_apps '{}'` on my local computer and tell me which apps have visible windows. Do not interact with them yet. ``` After that succeeds, give the Bot a narrow UI task. Cua Driver actions should follow this loop: 1. Find the target app and window. 2. Call `get_window_state` for that exact window. 3. Act through an `element_token` from the latest state when one is available. 4. Call `get_window_state` again and verify the expected result. Read [Agent action policy]() for the full selection and escalation rules. **Warning** Grok Bot's local-command approval controls whether a command may run. Cua Driver's permission mode controls what that command may do. Keep both boundaries enabled. Do not set Grok Bot to **Always allowed** or run Cua Driver in unrestricted mode for routine use. ## Drive Grok Bot's cloud computer If the desktop being driven is Grok Bot's own computer, install Cua Driver there — not on your laptop. That machine is persistent cloud Linux, separate from your Mac or Windows computer. Stable is the default: ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" ``` To follow nightly instead: ```bash curl -fsSL https://cua.ai/driver/install.sh | bash -s -- --channel nightly ``` Linux requirements: x86_64, with X11 or XWayland. `get_window_state` needs AT-SPI 2. After install, verify: ```bash cua-driver --version cua-driver doctor cua-driver call list_apps '{}' ``` If `doctor` warns that AT-SPI is unreachable, `get_window_state` will not work until the accessibility bus is available. See [Install Cua Driver]() for Linux libraries and display-server details. `cua-driver skills install` writes the pack to `~/.cua-driver/skills/cua-driver`. It auto-links only Claude Code, Codex, Prime Agent, OpenClaw, OpenCode, Antigravity, and Hermes. Grok Bot is not in that list. On Grok Bot, save a private skill or workflow that points at that pack and keeps snapshot-before-action, element tokens, and post-action verification. Then give the Bot a narrow UI task on that computer. Cua Driver actions should follow the same loop as the local path: find the window, call `get_window_state`, act through an `element_token` from the latest state, then verify. ## Save the workflow as a Grok Bot skill Once a safe task succeeds, ask Grok Bot to save the method as a private skill: ```text Save the Cua Driver process we just used as a skill. Preserve the fresh-state, snapshot-bound element-token, and post-action verification rules. Require my approval for consequential actions. Keep Cua Driver on the computer it is driving — my local computer, or Grok Bot's cloud computer when that is the desktop — and report any refusal instead of bypassing it. ``` Test the saved skill on a disposable example before attaching it to a routine. Grok Bot routines can run while your laptop is closed, but a routine that depends on local commands also depends on your computer being reachable and Cua Driver running. A routine that drives Grok Bot's own computer does not depend on your laptop. ## Custom MCP is an advanced route Grok Bot can use connectors and MCP servers. Cua Driver exposes MCP over local stdio and can also enable an authenticated Streamable HTTP endpoint. xAI requires a custom MCP server to be reachable from the public internet, so connecting the two requires an authenticated TLS tunnel or gateway. This route adds a public network boundary to desktop control. Use it only when you can provide all of the following: - A host-generated bearer token of at least 32 characters - A Cua Driver bounded capability manifest for the required tools and apps - A TLS endpoint that forwards only to Cua Driver's loopback MCP listener - A separate MCP connection for each Bot that needs an independent session - A way to revoke the endpoint and token when the task ends Cua Driver does not provide a managed public gateway or a Grok Bot connector preset. Review [SDK, MCP, and hosting]() and [Restrict tool access]() before building this route. xAI documents the other side in [Connectors](https://docs.x.ai/grok/connectors) and [Custom MCP Server Tunneling](https://docs.x.ai/grok/connectors/custom-mcp-tunneling). ## Shared-state boundaries All Bots on one Grok Bot account share its cloud computer, browser sessions, files, and command-line credentials. Each screen is a separate work surface within the same security boundary. Grok Bot also installs connectors account-wide. Cua Driver sessions isolate transient state such as element cursors. They do not isolate the underlying desktop account or the apps running in it. Use separate operating-system accounts or machines when tasks require separate trust boundaries. Upstream: [Grok Bot overview](https://docs.x.ai/grok-bot/overview), [computer and apps](https://docs.x.ai/grok-bot/computer-and-apps), and [approvals, security, and privacy](https://docs.x.ai/grok-bot/approvals-security-and-privacy). --- # Hermes Use Cua Driver through Hermes Agent's built-in Computer Use toolset. Hermes Agent includes a `computer_use` toolset that wraps Cua Driver with Hermes-native actions, approvals, image handling, diagnostics, and session cleanup. Use this built-in integration instead of adding Cua Driver as a second, raw MCP server. Hermes setup normally attempts to install Cua Driver, but that step is best-effort. Install or repair the driver when needed, then verify the complete path: ```bash hermes computer-use install hermes computer-use status hermes computer-use doctor hermes tools list ``` If `computer_use` is disabled, enable it for the Hermes CLI: ```bash hermes tools enable computer_use --platform cli ``` Start a session with the Computer Use toolset: ```bash hermes -t computer_use chat ``` For deeper platform guidance, install Cua Driver's maintained skill pack. Cua Driver links it into the standard Hermes skill directory: ```bash cua-driver skills install cua-driver skills status ``` Raw `cua-driver mcp` registration remains available for driver development and low-level MCP debugging, but it exposes an overlapping interface without Hermes's Computer Use wrapper. Do not enable both interfaces by default. The [`mcp-config` reference]() records that advanced path. [Four Hermes sessions share a Windows desktop]() Independent Hermes agents operate separate apps through the same Cua Driver runtime. Upstream: [Hermes Computer Use documentation](https://hermes-agent.nousresearch.com/docs/user-guide/features/computer-use). --- # OpenClaw Connect OpenClaw to Cua Driver over MCP. OpenClaw can register Cua Driver as a local MCP server. Generate the current command: ```bash cua-driver mcp-config --client openclaw ``` The generated command uses the absolute Cua Driver executable path: ```bash openclaw mcp set cua-driver '{"command":"/absolute/path/to/cua-driver","args":["mcp"]}' ``` **Warning** On macOS, a gateway-spawned MCP process does not inherit OpenClaw.app's Accessibility and Screen Recording grants. Grant permissions to the identity that actually runs Cua Driver, or use the documented embedding path. Continue with [Connect your agent to Cua Driver]() for the separate MCP architecture. To run the bundled provider inside an isolated Linux Fleet VM, use [Run OpenClaw on Cloud Fleet](). For a local macOS VM, use [Run OpenClaw in a Lume VM](). Upstream: [OpenClaw MCP CLI documentation](https://docs.openclaw.ai/cli/mcp). --- # T3 Code Use Cua Driver from an agent harness managed in T3 Code. T3 Code is a control plane for coding agents. It runs and organizes harnesses such as Claude Code, Codex, Qwen Code, and Grok Build; it is not itself the MCP client that launches Cua Driver. To use Cua Driver from T3 Code: 1. Choose one of T3 Code's supported agent harnesses. 2. Register Cua Driver in that harness using its page in this section. 3. Start or restart the harness from T3 Code and confirm the Cua Driver tools are available. This separation keeps the MCP configuration portable. The same harness can use Cua Driver whether it is launched from T3 Code or directly from a terminal. Start with [Claude Code](), [OpenAI Codex](), [Grok Build](), or [Qwen Code](). Upstream: [T3 Code](https://t3.codes/) and the [T3 Code repository](https://github.com/pingdotgg/t3code). --- # Qwen Code Connect Qwen Code to Cua Driver over MCP. Qwen Code supports local stdio MCP servers. Generate the current command with Cua Driver: ```bash cua-driver mcp-config --client qwen ``` The generated command uses the absolute executable path: ```bash qwen mcp add cua-driver /absolute/path/to/cua-driver mcp qwen mcp list ``` You can also add the generic `mcpServers` object to `~/.qwen/settings.json`. Restart Qwen Code after changing the configuration. Continue with [Connect your agent to Cua Driver]() for runtime and permission guidance. Upstream: [Qwen Code MCP documentation](https://qwenlm.github.io/qwen-code-docs/en/users/features/mcp/). --- # Kimi Code Connect Kimi Code CLI to Cua Driver over MCP. Kimi Code CLI supports local stdio MCP servers. Find the installed Cua Driver path, then register and test it: ```bash command -v cua-driver kimi mcp add cua-driver -- /absolute/path/to/cua-driver mcp kimi mcp test cua-driver ``` The `--` marks the start of the stdio command. Kimi Code can also load MCP configuration from `~/.kimi-code/mcp.json` or a project-level `.kimi-code/mcp.json`. Cua Driver currently has no dedicated Kimi Code preset, so this page follows Kimi Code's native MCP command and the standard `cua-driver mcp` entry point. Continue with the [generic MCP configuration]() for Cua Driver runtime and permission guidance. Upstream: [Kimi Code CLI MCP guide](https://www.kimi.com/code/docs/en/kimi-code-cli/customization/mcp.html) and [`kimi mcp` reference](https://moonshotai.github.io/kimi-cli/en/reference/kimi-mcp.html). --- # Ollama Serve a local model with Ollama and connect it to Cua Driver through an agent harness. Ollama serves local models. It does not replace the agent harness or connect to Cua Driver by itself. Use an Ollama-supported harness for the agent loop, then register Cua Driver in that harness. For Muse Glimmer on Apple Silicon, the tested stack is: ```text Muse Glimmer in Ollama → Claude Code → filtered Cua Driver MCP → desktop ``` Follow [Run a local model with Cua Driver]() for the exact model, context, and Claude Code launch command. Local computer use consumes context quickly. Start with a model that supports images and tool calling, expose only the Cua Driver tools the task needs, and use a context window of at least 64K. The tested guide uses 128K. Upstream: [Ollama's Claude Code integration](https://docs.ollama.com/integrations/claude-code) and [tool-calling documentation](https://docs.ollama.com/capabilities/tool-calling). --- # llama.cpp and Unsloth Serve a local GGUF model with llama.cpp and connect it to Cua Driver. `llama.cpp` gives you direct control over local model serving. Unsloth publishes quantized GGUF variants that can reduce the memory required to run supported models. The recorded Muse Glimmer demos use: ```text Unsloth UD-Q4_K_XL GGUF → llama.cpp → Claude Code → filtered Cua Driver MCP → macOS Lume guest ``` Follow [Run a local model with Cua Driver]() for the tested `llama-server` flags and harness configuration. The local-model guide also covers the main optimization: reduce the MCP tool catalog before the model sees it. Tool schemas, screenshots, and accessibility trees all consume context. Keep state reads bounded and expose only the actions required by the task. Upstream: [`llama.cpp`](https://github.com/ggml-org/llama.cpp), its [server documentation](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md), [function-calling guide](https://github.com/ggml-org/llama.cpp/blob/master/docs/function-calling.md), and Unsloth's [Muse Glimmer guide](https://unsloth.ai/docs/models/muse-glimmer). --- # How-to guides Follow recipes for specific Cua goals. How-to guides are recipes for **a specific goal**. They assume you already know the basics, so do a tutorial first if not. Unlike tutorials, they are *not a lesson*; they get you to a result and stop. They are grouped by product: Cua Driver for operating apps on a physical machine, Sandbox for isolated computers on local infrastructure or Fleet, Lume for local Apple silicon VMs, and Cua-Bench for verifiable computer-use tasks. Recipes and agent context cover workflows that span those products. Check [Sandbox runtime support]() for the image and connection requirements of a sandbox recipe. --- # Install Cua Driver Install Cua Driver on macOS, Windows, or Linux with a one-line script. Cua Driver supports macOS, Windows, and Linux. Use the same **one-line installer** on each platform; the script selects the right install path for the host and _does not require administrator access_. New to Cua Driver? Follow [Drive your first app]() for a guided walkthrough from installation to your first app interaction. **Note** Cua Driver sends content-free product telemetry by default. The installer shows this notice before the first event. Run `cua-driver telemetry disable` at any time to stop telemetry; the preference persists across upgrades. See [Telemetry and privacy]() for the exact event schema and identity controls. **macOS** **Requirements:** macOS 14 (Sonoma) or later on Apple Silicon or Intel. ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" ``` The installer places `CuaDriver.app` in `/Applications` and creates the `~/.local/bin/cua-driver` symlink. The app bundle uses the `com.trycua.driver` signing identity, so macOS TCC permissions for Accessibility and Screen Recording remain attached across upgrades. If `~/.local/bin` is missing from your PATH, the installer detects your shell (`zsh`, `bash`, or `fish`) and adds the matching `export PATH=…` line to your rc file. Reload the shell with `source ~/.zshrc` or open a new terminal window. **Windows** **Requirements:** Windows 10/11 or Windows Server with an interactive desktop session and PowerShell. ```powershell irm https://cua.ai/driver/install.ps1 | iex cua-driver autostart kick ``` The installer downloads the release under `%USERPROFILE%\.cua-driver\packages\releases\` and exposes `cua-driver.exe` from `%LOCALAPPDATA%\Programs\Cua\cua-driver\bin`. It runs without administrator privileges, detects whether the host is x64 or arm64, and appends the install directory to your User-scope `Path` so new PowerShell windows can resolve `cua-driver.exe`. It also attempts to register the `cua-driver-serve` autostart task; `kick` starts that task immediately, so you do not need to sign out or restart Windows. Registration needs an interactive session, so on a non-interactive or SSH install it is skipped — set it up later with [`cua-driver autostart enable`](). To skip the PATH change, pass the no-update flag: ```powershell & ([scriptblock]::Create((irm https://cua.ai/driver/install.ps1))) -NoPathUpdate ``` **Linux** **Requirements:** an x86_64 Linux desktop session with X11/XWayland or the opt-in native Wayland backend described below, plus AT-SPI 2 for accessibility-tree tools. The window-driving tools (`click`, `type_text`, `get_window_state`) need a live display server and a running app, so a headless server has no windows to drive until you start a desktop session (for example `xfce4` under `Xvfb`). On a minimal or server image, install the runtime libraries the binary links against first. Unlike the installer itself, this step needs `sudo`; without `libXi` the binary fails to launch with `libXi.so.6: cannot open shared object file`, before `doctor` can even run: ```bash sudo apt install libxi6 at-spi2-core ``` ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" ``` The installer downloads the binary to `~/.cua-driver/packages/releases/`, points `~/.cua-driver/packages/current` at that release, and creates the `~/.local/bin/cua-driver` symlink. It does not use sudo. If `cua-driver` is not found, open a new terminal to pick up the installer's PATH change. Installing the binary does not start the Linux daemon. Follow [Start the daemon on Linux](#start-the-daemon-on-linux) below before calling desktop tools. X11 support has toolkit-specific limits. Native Wayland remains opt-in with `CUA_DRIVER_RS_ENABLE_WAYLAND=1` and has compositor-specific limits. Sway is supported with limits, GNOME requires the maintained WinRects helper, and KDE and Hyprland/Omarchy have separate experimental support entries. Without the opt-in, Wayland sessions use XWayland routes where the target exposes a real X11 window. See [Platform support]() for the exact accepted surfaces. After installation, run `cua-driver doctor` to check distro-specific requirements such as the AT-SPI bus and display server. If these names are unfamiliar, start with [Linux desktops and computer use](). The experimental Hyprland plugin is separate from this installer; installing the driver does not enable isolated raw background input on Omarchy. ## Verify the install The version and paths in the following output are illustrative; your installed release and machine determine the actual values. ```bash cua-driver --version # cua-driver 0.13.0 ``` For a full environment and install report: ```bash cua-driver doctor # [ok ] binary: cua-driver 0.13.0 (aarch64-macos) # [ok ] install dir: /Users/you/.local/bin/cua-driver # [ok ] home dir: /Users/you/.cua-driver (3 release dirs cached) # ... ``` `doctor` checks the version, install layout, and telemetry setup on every platform. It also probes the interactive session on Windows and AT-SPI plus the display server on Linux. On macOS, use `cua-driver permissions status` after starting the daemon to check its Accessibility and Screen Recording grants. ## Follow the nightly channel Stable is the default. To install the newest nightly and save that preference: **macOS and Linux** ```bash curl -fsSL https://cua.ai/driver/install.sh | bash -s -- --channel nightly ``` **Windows** ```powershell & ([scriptblock]::Create((irm https://cua.ai/driver/install.ps1))) -Channel nightly ``` Later `cua-driver check-update` and `cua-driver update --apply` stay on nightly. Switch back without reinstalling immediately with: ```bash cua-driver channel set stable cua-driver update --apply ``` `channel set` saves intent only; `update --apply` performs the binary change. ## Install an exact nightly Nightlies are immutable builds of exact `main` commits. Copy the full `nightly-cua-driver-rs-v…` tag from the component's GitHub release, then pin that tag during installation. The tag below illustrates the format and is not a verified release: replace the entire tag with one that exists and has an asset for your platform before running either command. **macOS and Linux** ```bash CUA_DRIVER_RS_VERSION=nightly-cua-driver-rs-v0.19.4-nightly.20260812.123456789 \ /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" ``` **Windows** ```powershell $env:CUA_DRIVER_RS_VERSION = "nightly-cua-driver-rs-v0.19.4-nightly.20260812.123456789" irm https://cua.ai/driver/install.ps1 | iex Remove-Item Env:CUA_DRIVER_RS_VERSION ``` The pin is one-shot: it does not change the saved update channel. A machine without a saved preference therefore remains on stable; a machine following nightly keeps following nightly. Exact pins never fall back to another release when an asset is missing, and cannot be combined with `--channel`. ## Choose a permission mode Cua Driver authorizes every agent action inside the native runtime, and the mode is fixed when that runtime starts. An agent cannot change it, and neither can you change it on a running daemon — you restart the daemon with different flags. Decide which mode you want before starting the daemon below. | Mode | Use it when | | -------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `standard` | Normal local CLI and MCP use. Observation, input, isolated browser use, recording, and validated file transfer run without prompts. Residual boundaries, such as attaching to an existing logged-in Chromium profile, still need an explicit grant. | | `bounded` | An unattended agent, gateway, or embedded application must stay inside a reviewed manifest of tools, applications, browser origins, and directories. Anything outside the manifest is denied. | | `unrestricted` | The machine is disposable or fully trusted and you accept every capability the built-in, managed, and user policy ceilings still allow. | `standard` is the default and is what the rest of this guide assumes, so plain `cua-driver serve` needs no flags. For `bounded`, pass a manifest plus the review acknowledgement: ```bash cua-driver serve \ --permission-mode bounded \ --capability-manifest ~/cua-session.yaml \ --approve-capability-manifest ``` For `unrestricted`, pass the dangerous acknowledgement. `--permission-mode unrestricted` on its own fails closed: ```bash cua-driver serve --dangerously-bypass-approvals ``` On macOS, put those flags after `serve` in the app-bundle launch so TCC attribution stays with `CuaDriver.app`. The manifest path must resolve on its own: your shell expands `~` before `open` sees it, but a relative path fails, because an app started by `open` does not inherit the shell's working directory. ```bash open -n -g -a CuaDriver --args serve \ --permission-mode bounded \ --capability-manifest ~/cua-session.yaml \ --approve-capability-manifest ``` See [Permission modes]() for exactly what each profile allows, and [Write a capability manifest]() for a tested manifest. To keep a non-default profile across reboots, its flags belong in the autostart entry — see [Pin a permission mode](). ## Start the daemon on Linux In a terminal inside your graphical desktop session, start the daemon: ```bash cua-driver serve ``` Leave that terminal open. `serve` stays in the foreground; it must have access to the same display and accessibility bus as the apps you want to drive. Run the readiness checks below from a second terminal in that session. To start the daemon automatically at sign-in, see [Keep Cua Driver running](). ## Grant TCC permissions (macOS only) **Start the daemon first** so macOS attributes the TCC request to `CuaDriver.app` instead of your terminal: ```bash open -n -g -a CuaDriver --args serve ``` Then grant both permissions: ```bash cua-driver permissions grant ``` That command launches CuaDriver through LaunchServices so macOS attributes the prompts to the app, then waits. macOS prompts once for Accessibility. Note that it offers **Open System Settings**, not **Allow** — the dialog alone grants nothing: ![The macOS Accessibility Access prompt: “CuaDriver” would like to control this computer using accessibility features, with Open System Settings and Deny buttons](https://github.com/user-attachments/assets/b7a5a0c0-f297-4249-81f9-ca42c6b6f8ba) Clicking **Open System Settings** adds CuaDriver to the Accessibility list, still switched off. Toggle it on: ![CuaDriver listed under Accessibility in System Settings with its toggle switched off](https://github.com/user-attachments/assets/d149fe5a-28cb-460c-8868-4eda80d540a4) Screen Recording prompts separately, in the same shape: ![The macOS Screen Recording prompt: “CuaDriver” would like to record this computer’s screen and audio, with Open System Settings and Deny buttons](https://github.com/user-attachments/assets/69efc964-a375-495e-8a27-400c9dd08e17) Toggle CuaDriver on under **Screen & System Audio Recording** too: ![CuaDriver listed under Screen & System Audio Recording in System Settings with its toggle switched off](https://github.com/user-attachments/assets/469308b5-84d7-4fd6-a580-e62925120337) The prompt only registers the app; the toggle is what grants access. macOS may offer to quit and reopen CuaDriver when you flip one — accept, because a changed grant takes effect only after the responsible app fully relaunches. If the daemon does not come back, rerun `open -n -g -a CuaDriver --args serve`. macOS does not always raise both prompts in one pass. Confirm what landed with `cua-driver permissions status`, and if only one of the two appeared, run the pair again to prompt for the other: ```bash open -n -g -a CuaDriver --args serve cua-driver permissions grant ``` If CuaDriver is missing from either list, add it with **+** and pick `/Applications/CuaDriver.app` — see [macOS permissions]() for recovering a stale registration. You can also trigger the prompts yourself: ```bash cua-driver check_permissions ``` macOS opens the Accessibility and Screen Recording prompts. Grant both permissions, then run the check again: ```bash cua-driver permissions status # ✅ Accessibility: granted. # ✅ Screen Recording: granted. ``` **Note** `cua-driver permissions status` reads the driver's actual grant state through the daemon. When no daemon is running, it reports `❓ unknown` instead of reporting your terminal's grants, and it does not claim `granted` unless the driver has that permission. ## Verify desktop readiness After starting the daemon and granting any required platform permissions, run: ```bash cua-driver status cua-driver doctor cua-driver call list_apps ``` `status` should report `Cua Driver daemon is running`. Read the `doctor` report for interactive-session, display, or accessibility-bus warnings and errors; warnings can still produce a zero exit code. On macOS, confirm both grants with `cua-driver permissions status`. Then confirm that `list_apps` includes a GUI app you recognize. If the list is empty, open an app in the same desktop session and retry. A version string alone verifies the binary, not desktop access. If the Windows autostart task was not registered, run `cua-driver serve` in an interactive desktop terminal and use a second terminal for these checks. ## Next steps - [Choose a Cua Driver integration](): decide whether your agent or product should use MCP, the SDK, or an app-hosted service. - [Permission modes](): what standard allows, and how bounded and unrestricted differ. - [Install the Cua Driver agent skill](): add the cross-platform instructions from ClawHub or install them directly for another agent. - [Keep Cua Driver running](): configure autostart so the daemon comes back after reboots. - [Connect Cua Driver to an MCP client](): register it with Claude Code, Cursor, Codex, and other clients. - [Update Cua Driver](): check for new releases and apply them. --- # Use Cua Driver in process Call the typed Cua Driver SDK directly from Python or TypeScript without a daemon. Use the same-process SDK when your application owns the desktop-control lifecycle. `CuaDriver.create()` loads the Rust runtime into the importing process; it does not launch `cua-driver serve` or use IPC. The importing application owns OS permissions and permission UX. On macOS, direct `check_permissions` calls are read-only even when `prompt` is requested; relaunch the responsible host after changing TCC grants. Direct macOS runtimes also return `facility_unavailable` for agent-cursor overlay operations unless the host installs a suitable AppKit main-thread adapter. Use a private worker or explicit service when the overlay is required. ## Install the SDK **Python** ```bash python -m pip install cua-driver ``` **TypeScript** ```bash npm install @trycua/cua-driver ``` ## Capture the desktop with an implicit session The first stateful call creates one implicit session for the SDK transport. `start_session` is optional and no capture scope is stored on the session. Actions select an exact window or desktop target per call. **Python** ```python import asyncio from cua_driver import ( CuaDriver, EndSessionInput, GetDesktopStateInput, ) async def main() -> None: driver = CuaDriver.create() try: result = await driver.get_desktop_state( GetDesktopStateInput( session=None, screenshot_out_file=None, ) ) if result.is_error: raise RuntimeError(result.text) print(result.images[0].mime_type) finally: await driver.end_session(EndSessionInput(session=None)) await driver.shutdown() asyncio.run(main()) ``` **TypeScript** ```ts import { CuaDriver, EndSessionInput, GetDesktopStateInput, } from '@trycua/cua-driver'; const driver = CuaDriver.create(undefined); try { const result = await driver.getDesktopState(GetDesktopStateInput.new({})); if (result.isError) throw new Error(result.text); console.log(result.images[0]?.mimeType); } finally { await driver.endSession(EndSessionInput.new({})); await driver.shutdown(); driver.uniffiDestroy(); } ``` Keep one `CuaDriver` object for the application lifetime. Repeated unnamed calls reuse its implicit session. End it explicitly when useful; `shutdown()` also runs cleanup. `shutdown()` is idempotent and rejects new work after shutdown while allowing already-admitted operations to finish. If an external agent must also connect to your signed desktop application, use [Expose MCP from a desktop app]() instead. --- # Verify a desktop action Check a Python or TypeScript desktop action against an independent postcondition. Use this guide to capture the desktop, enter a unique value in a local browser fixture, and prove that the page received it. A loopback state endpoint provides the proof independently from the action response. ## Get the example Clone Cua and enter the executable example directory: ```bash git clone https://github.com/trycua/cua.git cd cua/libs/cua-driver/examples/agent-sdks ``` The fixture listens only on `127.0.0.1`. Start it in one terminal: ```bash python3 fixture_server.py ``` Your browser opens a page containing one autofocus input. Keep that window focused. On macOS, grant Screen Recording and Accessibility permission to the Python or Node process that imports Cua Driver. ## Run the native SDK loop **Python** ```bash python3.12 -m venv .venv .venv/bin/pip install -r requirements.txt .venv/bin/python native_driver.py ``` The complete executable is `libs/cua-driver/examples/agent-sdks/native_driver.py`. **TypeScript** ```bash npm install npm run native ``` The complete executable is `libs/cua-driver/examples/agent-sdks/native_driver.ts`. The program performs four observable steps: 1. `CuaDriver.create()` loads the Rust runtime in the importing process. 2. `get_desktop_state` returns the initial whole-desktop image. 3. `type_text` and `press_key` submit a randomly generated value. 4. The program polls the fixture's `/state` endpoint and captures the desktop again only after the exact value appears. Successful output resembles: ```text perceived desktop: image/png verified submitted value: cua-2ca7d7cc81 ``` ## See how uncertain actions are handled An input action can reach the OS even if the caller times out before receiving the response. The examples catch that condition but do not repeat the action. They check the independent fixture first. If the value is present, the postcondition resolves the unknown response; if it is absent, the run fails without silently replaying a mutation. ## Shut down the fixture Press `Ctrl+C` in the fixture terminal. The SDK examples already end their session in `finally` and await runtime shutdown. Next, connect the same driver to [Claude Agent SDK]() or [Codex SDK](). --- # Use Cua Driver with Claude Agent SDK Choose same-process native callbacks or the MCP agent boundary in Python and TypeScript. Claude Agent SDK supports both Cua Driver integration routes: - Use **native callbacks** when your application already owns the agent loop, driver runtime, permissions, and session lifecycle. - Use **MCP** when Claude is an external agent or should discover the complete Cua Driver tool catalog. The native callbacks are packaged by Claude Agent SDK as an in-process SDK MCP server. Despite that helper's name, it does not start a subprocess or use a socket; each callback calls `CuaDriver.create()` in the application process. ## Install the executable examples From `libs/cua-driver/examples/agent-sdks`: **Python** ```bash python3.12 -m venv .venv .venv/bin/pip install -r requirements.txt ``` **TypeScript** ```bash npm install ``` Authenticate Claude Agent SDK using its normal local login or `ANTHROPIC_API_KEY`. Set `CLAUDE_MODEL` only when you need to override its default model. ## Run the native callback route **Python** ```bash .venv/bin/python claude_agent.py --route native \ "Inspect the fixture, enter a short value, submit it, and verify the result" ``` **TypeScript** ```bash npm run claude -- --route native \ "Inspect the fixture, enter a short value, submit it, and verify the result" ``` The examples expose only `observe_desktop`, `click_desktop`, `type_text`, and `press_key`. Their handlers call the same-process Python or TypeScript SDK. Every mutation returns a fresh desktop observation. If a call times out, the handler labels its outcome unknown and tells the model to inspect before any retry. The complete executables live at `libs/cua-driver/examples/agent-sdks/claude_agent.py` and `libs/cua-driver/examples/agent-sdks/claude_agent.ts` in the Cua repository. ## Run the MCP route Install Cua Driver so `cua-driver` is on `PATH`, then run: **Python** ```bash .venv/bin/python claude_agent.py --route mcp \ "Inspect the active app and summarize what is visible" ``` **TypeScript** ```bash npm run claude -- --route mcp \ "Inspect the active app and summarize what is visible" ``` The host supplies one long-lived `cua-driver mcp` transport to Claude. The first admitted stateful call creates its implicit session, later unnamed calls reuse it, and transport shutdown runs lifecycle cleanup. Claude discovers the full live MCP surface; the generated Cua language packages are not acting as MCP clients. Use only trusted tasks with these examples. They remove interactive approval prompts and deliberately exclude Claude Code's built-in shell and file tools. --- # Use Cua Driver with Codex SDK Configure Cua Driver MCP for the Python and TypeScript Codex SDKs. Codex SDK supports MCP servers but does not expose application-owned custom tool callbacks. Use `cua-driver mcp` as its Cua Driver boundary. Do not build a second tool loop that translates the native SDK back into an ad hoc protocol. ## Install the executable examples From `libs/cua-driver/examples/agent-sdks`: **Python** ```bash python3.12 -m venv .venv .venv/bin/pip install -r requirements.txt ``` This installs `openai-codex` and imports it as `openai_codex`. **TypeScript** ```bash npm install ``` This installs `@openai/codex-sdk`. Install Cua Driver so `cua-driver` is on `PATH`, or set `CUA_DRIVER_BIN` to its absolute path. ## Run a desktop task **Python** ```bash .venv/bin/python codex_agent.py \ "Inspect the active app and summarize what is visible without changing it" ``` **TypeScript** ```bash npm run codex -- \ "Inspect the active app and summarize what is visible without changing it" ``` The complete scripts live at `libs/cua-driver/examples/agent-sdks/codex_agent.py` and `libs/cua-driver/examples/agent-sdks/codex_agent.ts` in the Cua repository. Both scripts: 1. create one required, long-lived Cua MCP transport; 2. let the first admitted stateful call create its implicit session; 3. run Codex in a read-only filesystem sandbox; 4. instruct Codex to use only Cua tools for desktop work; and 5. rely on transport shutdown to run session cleanup. The filesystem sandbox does not sandbox the separate desktop MCP server. Restrict the task itself, choose an exact window or desktop target for each Cua action, apply the appropriate authorization policy, and never blindly retry a mutation after a timeout or disconnect. --- # Expose MCP from a desktop app Host a private Cua Driver daemon when an external agent must reuse your app's desktop permissions. Use a daemon-backed host only when a signed desktop application must expose Cua Driver to an external MCP client. If only your application calls the driver, use [`CuaDriver.create()`]() instead. ## Bundle the executable Ship a compatible `cua-driver` executable as an application resource. In an Electron app, keep it outside the ASAR archive, preserve its executable bit, and sign the nested executable before signing and notarizing the enclosing app. The npm package supplies the native SDK library but does not bundle the executable. The Python wheel does bundle it. ## Start the host from the permission-owning process ```ts import { CuaDriver, EmbeddedCuaDriverHost } from '@trycua/cua-driver'; const host = new EmbeddedCuaDriverHost( '/path/inside/YourApp.app/Contents/Resources/cua-driver', 'com.example.your-app' ); const connection = await host.start(); const driver = CuaDriver.connect(connection.socketPath); // Application code calls the same typed SDK methods on `driver`. // The external agent launches `connection.mcp.command` with // `connection.mcp.args` and `connection.mcp.environment`. ``` On macOS, call `start()` from the app process that owns Accessibility and Screen Recording permission. A gateway, terminal, `open`, or `NSWorkspace` changes the TCC responsibility chain and cannot lend the app's grants to the child. ## Pass MCP configuration to a backend If the desktop app has a separate Node backend, keep `EmbeddedCuaDriverHost` in the Electron main process. Pass the returned MCP configuration through the backend's existing bootstrap or IPC channel: ```ts const connection = await host.start(); backend.send({ type: 'bootstrap', cuaMcp: connection.mcp, }); ``` The backend must launch the exact `cuaMcp.command` with `cuaMcp.args` and `cuaMcp.environment`. It may pass those values to an agent runtime such as Codex. It must not start another embedded host. After `host.restart()`, send the replacement `connection.mcp` before accepting new agent work. Stop old MCP proxies because their private endpoint belongs to the previous generation. ## Shut down in dependency order Stop accepting new work, end active sessions, close MCP clients, then tear down the SDK client and host: ```ts await driver.shutdown(); driver.uniffiDestroy(); await host.stop(); host.uniffiDestroy(); ``` `start()` coalesces concurrent callers. `stop()` cancels startup and is idempotent. After `restart()`, discard the old connection: the generation, PID, and endpoint change. Use `waitForExit(connection.generation)` to observe an unexpected child exit, and never automatically replay an action whose outcome is unknown. For the complete lifecycle and low-level launch contract, see [Embedding reference](). --- # Install the Cua Driver agent skill Install Cua Driver instructions for OpenClaw from ClawHub, or install them directly for another supported agent. The Cua Driver agent skill teaches an agent how to choose tools, address windows and elements, preserve focus, and verify each action on macOS, Windows, or Linux. **Note** The skill contains agent instructions. It does not install the `cua-driver` executable. [Install Cua Driver]() on the same machine first. ## Install from ClawHub Run the command from your OpenClaw workspace: ```bash clawhub install @cua/driver ``` ClawHub installs the skill under the workspace's `skills` directory. The bundle contains the shared instructions and the macOS, Windows, and Linux guides, so the agent can read the guide for its current host. Confirm that ClawHub tracks the install: ```bash clawhub list ``` Restart OpenClaw or start a new agent session so it discovers the installed skill. ## Verify Cua Driver Check the executable and host requirements separately: ```bash cua-driver --version cua-driver doctor ``` On macOS, also confirm that Cua Driver has Accessibility and Screen Recording permission: ```bash cua-driver permissions status ``` ## Update or remove the ClawHub skill Update the installed skill: ```bash clawhub update driver ``` Remove it from the current workspace: ```bash clawhub uninstall driver ``` These commands change the agent instructions only. Use [`cua-driver update --apply`]() to update the executable. ## Install directly for another agent Cua Driver can install its bundled skill into supported agent directories without ClawHub: ```bash cua-driver skills install cua-driver skills status ``` Direct installation keeps only the current host's OS guide by default. Pass `--all-platforms` when the same skill directory must cover macOS, Windows, and Linux: ```bash cua-driver skills install --all-platforms ``` Prime Agent is supported directly. When `~/.prime/agent/skills/` exists, `cua-driver skills install` links the pack there; Prime Agent also discovers the shared `~/.agents/skills/` location used by Codex. Run `/reload` in Prime Agent or start a new session after installing or updating the skill so it reloads the guidance. `cua-driver skills install` auto-links Claude Code, Codex, Prime Agent, OpenClaw, OpenCode, Antigravity, and Hermes. Grok Bot is not in that list. On Grok Bot, save a private skill that follows the installed pack at `~/.cua-driver/skills/cua-driver`. See [Grok Bot](). ## Next steps - [Connect your agent to Cua Driver](): register the MCP server when the agent uses MCP. - [Keep Cua Driver running](): configure the daemon to return after restarts. - [Agent action policy](): review the behavior an agent wrapper should follow. --- # Run Cua Driver in a macOS Lume VM Pull the published Tahoe image, install Cua Driver, grant macOS consent, and connect an agent to the VM desktop. This guide shows you how to run Cua Driver in a local macOS Tahoe VM on an Apple Silicon host. It uses the published `macos-tahoe-cua:26.5.2` Lume image so the operating system, Command Line Tools, login session, and SSH setup match the maintainer test environment. **Note** The published image has SIP disabled, but SIP does not grant Accessibility, Screen Recording, Automation, or direct-capture consent. Approve those requests through macOS in your private VM. Cua Driver also works with SIP enabled; this guide uses the SIP-disabled image to match the maintainer test base. ## Before you start You need: - an Apple Silicon Mac with [Lume installed]() and `jq` available on the host; - enough host space for a 150 GB sparse VM disk; and - access to the VM display for the first-run macOS prompts. Use the versioned image tag for repeatable work. Reserve `latest` for experimentation. ## Pull and boot the VM On the host, run: ```bash IMAGE=macos-tahoe-cua:26.5.2 VM=cua-driver-dev-26.5.2 lume pull "$IMAGE" "$VM" lume run "$VM" ``` Keep `lume run` open. The VM display should log in as `lume`; the initial username and password are both `lume`. Open Terminal in the VM and inspect the base: ```bash sw_vers csrutil status xcode-select -p xcrun swiftc --version ``` The published `26.5.2` image reports: - macOS 26.5.2, build 25F84; - `System Integrity Protection status: disabled.`; and - `/Library/Developer/CommandLineTools` as the selected developer directory. Stop if those values differ. Pull the versioned image again instead of repairing an unknown base. ## Install Cua Driver in the VM Run the release installer from Terminal in the VM display: ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" ``` Verify the installed app and CLI: ```bash /Users/lume/.local/bin/cua-driver --version /Users/lume/.local/bin/cua-driver doctor ``` The installer puts the app at `/Applications/CuaDriver.app` and the CLI at `/Users/lume/.local/bin/cua-driver`. ## Grant every macOS consent path Request the app-owned Accessibility and Screen Recording grants: ```bash /Users/lume/.local/bin/cua-driver permissions grant ``` Approve both requests for CuaDriver. Then trigger app enumeration: ```bash /Users/lume/.local/bin/cua-driver list_apps '{}' ``` When macOS asks whether CuaDriver may control System Events, choose **Allow**. Tahoe also asks separately whether CuaDriver may capture the screen and audio without the system picker. Trigger that request with an actual desktop capture: ```bash DRIVER=/Users/lume/.local/bin/cua-driver "$DRIVER" call get_desktop_state '{}' \ > /tmp/cua-driver-desktop-state.json ``` Choose **Allow** on the direct-capture prompt. Check that the command returned a PNG: ```bash /usr/bin/python3 - <<'PY' import json with open('/tmp/cua-driver-desktop-state.json') as source: state = json.load(source) assert state['screenshot_mime_type'] == 'image/png' assert state['screenshot_width'] > 0 assert state['screenshot_height'] > 0 assert state['screenshot_png_b64'] PY ``` Rerun `list_apps` and the capture commands. They must finish without another prompt. ## Verify the driver-owned grants Query the running daemon rather than Terminal's permission state: ```bash /Users/lume/.local/bin/cua-driver permissions status --json \ > /tmp/cua-driver-permissions.json /usr/bin/python3 - <<'PY' import json with open('/tmp/cua-driver-permissions.json') as source: status = json.load(source) assert status['accessibility'] is True assert status['screen_recording'] is True assert status['screen_recording_capturable'] is None assert status['direct_capture_status'] == 'not_checked' assert status['source']['attribution'] == 'driver-daemon' PY ``` Run the human-owned permission setup command from the VM's interactive Terminal. Agent and daemon tool calls are deliberately read-only and cannot raise macOS permission UI. Then exercise both the accessibility and capture paths: ```bash /Users/lume/.local/bin/cua-driver permissions grant /Users/lume/.local/bin/cua-driver permissions status --json \ > /tmp/cua-driver-permissions-live.json /usr/bin/python3 - <<'PY' import json with open('/tmp/cua-driver-permissions-live.json') as source: status = json.load(source) assert status['screen_recording_capturable'] is None assert status['direct_capture_status'] == 'not_checked' assert status['direct_capture_verification']['source'] == 'permissions_grant' assert status['direct_capture_verification']['verified_at'].endswith('Z') assert status['direct_capture_verification']['bundle_id'] == 'com.trycua.driver' PY /Users/lume/.local/bin/cua-driver call get_accessibility_tree '{}' /Users/lume/.local/bin/cua-driver call get_desktop_state '{}' \ > /tmp/cua-driver-desktop-state-second.json ``` The commands must succeed without opening System Settings or another consent dialog. ## Connect an agent ### Run the agent inside the VM Run the agent inside the VM when it needs to operate the VM desktop. Generate the current registration command for your client: ```bash /Users/lume/.local/bin/cua-driver mcp-config --client codex /Users/lume/.local/bin/cua-driver mcp-config --client claude ``` Follow the printed command, then restart the client. See [Connect your agent to Cua Driver]() for other supported clients. To use a locally served model, keep Cua Driver inside the guest and connect it to the model's harness through the generated MCP command. [Run a local model with Cua Driver]() shows a tested Muse Glimmer and Claude Code configuration. Running the exact driver in the guest keeps screenshots, input, and recording attached to the guest's macOS identity. ### Run the agent on the host The host can expose the guest's MCP process over SSH. First, copy only your host public key into the guest: ```bash VM_IP="$(lume get "$VM" --format json | jq -r '.[0].ipAddress')" cat ~/.ssh/id_ed25519.pub | \ ssh "lume@${VM_IP}" \ 'umask 077; mkdir -p ~/.ssh; cat >> ~/.ssh/authorized_keys' ``` Enter the initial `lume` password when asked. Verify that noninteractive SSH reaches the app-owned daemon: ```bash ssh -T -o BatchMode=yes "lume@${VM_IP}" \ /Users/lume/.local/bin/cua-driver permissions status --json ``` Register that SSH command as a stdio MCP server in Codex: ```bash codex mcp add cua-driver-vm -- \ ssh -T -o BatchMode=yes "lume@${VM_IP}" \ /Users/lume/.local/bin/cua-driver mcp ``` The remote CLI proxies to the CuaDriver daemon in the logged-in guest session, so the daemon retains the app's macOS permission identity. ## Save a reusable private VM Close applications, stop the VM, and clone it while stopped: ```bash lume stop "$VM" lume clone "$VM" "${VM}-backup" lume ls ``` Keep private VMs local. Do not add registry tokens, source-control credentials, maintainer SSH private keys, or personal files before cloning. ## Troubleshooting ### A permission is `unknown` Start CuaDriver through LaunchServices and query it again: ```bash open -n -g -a CuaDriver --args serve cua-driver permissions status --json ``` `unknown` means no app-owned daemon answered the status request. ### A clone asks for permissions again Confirm that `/Applications/CuaDriver.app` has the release signature. A source build with an ad-hoc or changing signature gets a different TCC identity. Use the release installer for ordinary VM use. Maintainers testing source builds should follow [Run Cua Driver macOS tests in a Lume VM](), which creates a stable local signing identity. ### Commands work over SSH but GUI behavior differs Use Terminal in the VM display for initial consent and GUI tests. SSH is useful for source sync and artifact retrieval, but it is not the foreground Aqua session that owns the visible desktop. ## Related guides - [Run a local model with Cua Driver]() - [Manage local Lume VMs]() - [macOS permissions reference]() - [Run Cua Driver macOS tests in a Lume VM]() --- # Run Cua Driver macOS tests in a Lume VM Build a private Tahoe test seed, run the complete GUI E2E gate from a disposable Lume clone, and retain its evidence. This guide shows maintainers how to run the canonical Cua Driver macOS GUI E2E gate in Lume. Each accepted run installs one committed source revision in a disposable clone, executes the complete Rust behavior catalog from the logged-in VM display, and pulls the evidence back to the host. **Note** This is a maintainer-dispatched gate. It does not run in a GitHub-hosted macOS GUI job. The repository contains the runner and verifier; a private, host-local Lume seed supplies the macOS consent state. ## Before you start You need: - an Apple Silicon Mac with Lume and `jq` installed; - enough host space for the 150 GB sparse builder, seed, backups, and worker; - a clean, committed checkout of `trycua/cua`; and - access to the VM display for Keychain and macOS consent prompts. Record the host Lume version: ```bash lume --version ``` Use `macos-tahoe-cua:26.5.2` as the public base. It contains macOS 26.5.2 build 25F84, disabled SIP, Command Line Tools 26.6, autologin, and SSH. It has no repository source, TCC grants, or local signing identity. Keep the private builder and seed on host-local Lume storage. Never push them to a registry. ## Build the private seed once You only need to build a seed when the macOS version, public base, developer tools, signing identity, or consent state changes. Every acceptance run starts from a clone of the stopped seed. ### Pull and inspect the public base On the host, run: ```bash IMAGE=macos-tahoe-cua:26.5.2 BUILDER=cua-driver-macos-e2e-builder-26.5.2 lume pull "$IMAGE" "$BUILDER" lume run "$BUILDER" ``` Keep `lume run` open. In Terminal in the VM display, run: ```bash sw_vers csrutil status xcode-select -p xcrun swiftc --version ``` Require macOS 26.5.2 build 25F84, disabled SIP, and `/Library/Developer/CommandLineTools`. The Command Line Tools agreement must already be accepted. If macOS presents it during installation, accept it in the VM display and rerun the checks. ### Install the pinned guest toolchain Install Homebrew with the pinned upstream installer: ```bash HOMEBREW_INSTALL_COMMIT=4b0227cf8416504142d23893368c2e1d211d5191 HOMEBREW_INSTALL_SHA256=99287f194a8b3c9e6b0203a11a5fa54518be57209343e6bb954dec4635796d9d HOMEBREW_INSTALLER="/tmp/homebrew-install-${HOMEBREW_INSTALL_COMMIT}.sh" curl -fsSL \ "https://raw.githubusercontent.com/Homebrew/install/${HOMEBREW_INSTALL_COMMIT}/install.sh" \ -o "$HOMEBREW_INSTALLER" printf '%s %s\n' "$HOMEBREW_INSTALL_SHA256" "$HOMEBREW_INSTALLER" \ | shasum -a 256 -c - /bin/bash "$HOMEBREW_INSTALLER" eval "$(/opt/homebrew/bin/brew shellenv)" brew install node ffmpeg jq rust printf '\n%s\n' 'eval "$(/opt/homebrew/bin/brew shellenv)"' \ >> ~/.zprofile ``` Open a new Terminal window and verify the tools: ```bash xcrun swiftc --version cargo --version node --version npm --version ffmpeg -version | head -1 ffprobe -version | head -1 jq --version ``` Keep autologin, sleep prevention, and screen-lock prevention enabled. Add only the maintainer host's public SSH key to `/Users/lume/.ssh/authorized_keys`. Never copy a private key, registry credential, or source-control token into the VM. ### Sync one committed source revision From the repository root on the host, get the builder IP and push the checkout: ```bash VM_IP="$(lume get "$BUILDER" --format json | jq -r '.[0].ipAddress')" libs/cua-driver/scripts/sync-vm-worktree.sh push \ "lume@${VM_IP}" '~/cua' ``` The sync rejects a dirty host checkout. It omits `.git`, local credential files, build outputs, and host-side test evidence. It writes the exact commit to `/Users/lume/cua/.cua-e2e-source-sha`. ### Create a stable local signing identity Source builds need a certificate-backed identity so macOS keeps CuaDriver's TCC grants after the next install. Tahoe can unlock the login keychain while still denying the partition ACL used by noninteractive `codesign`, so create a dedicated keychain in the VM: ```bash SIGNING_KEYCHAIN="$HOME/Library/Keychains/cua-driver-signing.keychain-db" security create-keychain "$SIGNING_KEYCHAIN" security set-keychain-settings "$SIGNING_KEYCHAIN" security list-keychains -d user -s "$SIGNING_KEYCHAIN" security unlock-keychain "$SIGNING_KEYCHAIN" export CUA_DRIVER_LOCAL_SIGNING_KEYCHAIN="$SIGNING_KEYCHAIN" ``` Use the `lume` account password for this private test keychain. Back in the synced checkout, run the first install: ```bash cd /Users/lume/cua export CUA_DRIVER_SOURCE_SHA="$(cat .cua-e2e-source-sha)" bash libs/cua-driver/scripts/install-local.sh --release --autostart ``` Open Keychain Access in the VM display. Select the `cua-driver-signing` keychain, open `CuaDriver Local Signing (cua-driver-rs)`, and set Trust to **Always Trust**. The installer discovers this self-signed identity even when it is untrusted, so the **Always Trust** step is no longer required for signing — `codesign` uses the private key regardless. Setting trust remains useful if other Apple tooling in the VM validates the certificate chain. Return to Terminal and allow Apple tooling to use the private key: ```bash read -r -s -p 'Keychain password: ' KEYCHAIN_PASSWORD; echo security set-key-partition-list \ -S apple-tool:,apple:,codesign: -s -k "$KEYCHAIN_PASSWORD" \ "$SIGNING_KEYCHAIN" unset KEYCHAIN_PASSWORD ``` Run the installer again: ```bash bash libs/cua-driver/scripts/install-local.sh --release --autostart ``` Require the message `signed staged app with a stable local identity`. Do not approve macOS consent for an ad-hoc build. ### Grant the seed's consent paths Request Accessibility and Screen Recording through the app: ```bash /Users/lume/.local/bin/cua-driver permissions grant ``` Approve both CuaDriver requests. Then trigger the two Automation paths and Tahoe's direct-capture request: ```bash # The test sentinel runs this probe from Terminal. osascript -e \ 'tell application "System Events" to get name of first application process whose frontmost is true' # The installed app enumerates applications through System Events. /Users/lume/.local/bin/cua-driver list_apps '{}' # An actual desktop capture triggers Tahoe's separate direct-capture prompt. DRIVER=/Users/lume/.local/bin/cua-driver "$DRIVER" call get_desktop_state '{}' \ > /tmp/cua-driver-seed-desktop-state.json ``` Choose **Allow** for: - Terminal controlling System Events; - CuaDriver direct screen capture without the system picker. These are normal macOS consent flows. Do not hand-edit `TCC.db` while building a reusable seed. The checked-in SIP-off seed helper is only for disposable workers that are not cloned from a granted seed. Rerun all three probes and require them to finish without another prompt. Verify the app-owned grants, signature, capture result, and SIP state: ```bash "$DRIVER" permissions status --json | jq -e ' .accessibility == true and .screen_recording == true and .screen_recording_capturable == null and .direct_capture_status == "not_checked" and .source.attribution == "driver-daemon" ' "$DRIVER" permissions grant "$DRIVER" permissions status --json | jq -e ' .screen_recording_capturable == null and .direct_capture_status == "not_checked" and .direct_capture_verification.source == "permissions_grant" and (.direct_capture_verification.verified_at | endswith("Z")) and .direct_capture_verification.bundle_id == "com.trycua.driver" ' jq -e ' .screenshot_mime_type == "image/png" and .screenshot_width > 0 and .screenshot_height > 0 and (.screenshot_png_b64 | length) > 0 ' /tmp/cua-driver-seed-desktop-state.json >/dev/null codesign -d -r- /Applications/CuaDriver.app 2>&1 \ | grep 'certificate leaf' csrutil status ``` ### Freeze the seed and backups Before cloning, inspect the builder for credentials, personal files, temporary private keys, unrelated source, and old test evidence. Clean only the audited paths. Keep the dedicated signing keychain because it is part of the private seed contract. On the host, stop the builder and clone it while stopped: ```bash SEED=cua-driver-macos-e2e-seed-26.5.2-YYYYMMDD BACKUP_A="${SEED}-backup-a" BACKUP_B="${SEED}-backup-b" lume stop "$BUILDER" lume clone "$BUILDER" "$SEED" lume clone "$SEED" "$BACKUP_A" lume clone "$SEED" "$BACKUP_B" lume ls ``` Require the seed and both backups to show `stopped`. Record their names, the public image tag, macOS build, Lume version, Command Line Tools version, Rust version, Node version, and signing-certificate hash in the maintainer log. Treat these VMs as immutable and local-only. ## Run the acceptance gate from a fresh clone Create one worker per accepted run. Never run the matrix in the seed itself. ### Clone and boot a worker On the host: ```bash SEED=cua-driver-macos-e2e-seed-26.5.2-YYYYMMDD WORKER="cua-driver-macos-e2e-$(date -u +%Y%m%dT%H%M%SZ)" lume clone "$SEED" "$WORKER" lume run "$WORKER" ``` Keep the display open. In another host terminal, sync the exact clean commit: ```bash VM_IP="$(lume get "$WORKER" --format json | jq -r '.[0].ipAddress')" libs/cua-driver/scripts/sync-vm-worktree.sh push \ "lume@${VM_IP}" '~/cua' ``` ### Run the complete matrix in foreground Terminal Open Terminal in the VM display and run: ```bash cd /Users/lume/cua libs/cua-driver/tests/runners/macos-lume/run-all.sh ``` Enter the dedicated keychain password when asked. Keep Terminal in the logged-in Aqua session. Do not run this command over SSH. The runner verifies the source marker, console user, SIP state, toolchain, stable signature, app-owned TCC grants, Terminal Automation, CuaDriver Automation, and live capture before it builds the fixtures. It then executes the complete macOS catalog and requires typed result rows plus valid trajectory videos. An accepted run ends with: ```text macOS Rust E2E suite completed: all ``` Any setup failure is an environment failure. Do not report a smaller or filtered matrix as the canonical result. ### Pull and review the evidence Pull artifacts before stopping or deleting the worker, including after a failure: ```bash REMOTE_ARTIFACT_DIR=artifacts/cua-driver/macos \ libs/cua-driver/scripts/sync-vm-worktree.sh pull-artifacts \ "lume@${VM_IP}" '~/cua' ``` The command prints a host directory under `artifacts/cua-driver/vm/`. Review: - `summary.md` for the verified source SHA and pass/fail totals; - `cases.jsonl` and `results.jsonl` for the declared and observed rows; - `environment.jsonl`, `golden-environment.txt`, and `permissions.json` for the VM contract; - `environment-preflight.log` and per-lane logs; and - every `recordings/**/recording.mp4` referenced by a result row. The reporter rejects missing, duplicate, undeclared, skipped, or evidence-free rows. A zero exit status proves that the declarations, typed results, source identity, and required videos agree. After the artifacts are safely on the host: ```bash lume stop "$WORKER" lume delete "$WORKER" --force ``` Keep the seed and both backups stopped. ## Troubleshooting ### Codesign cannot find the local identity Unlock the dedicated keychain after each worker boot: ```bash security unlock-keychain \ "$HOME/Library/Keychains/cua-driver-signing.keychain-db" ``` If the identity is present but `codesign` still prompts or fails, rerun `security set-key-partition-list` in the builder and create a new seed. ### The run opens a System Events prompt The seed is missing an Automation grant. Stop the worker, repair the builder through the normal prompt flow, verify that the probe no longer prompts, and create a new seed. Do not add prompt handling to the test matrix. ### Permission status is green but capture blocks The ordinary Screen Recording toggle and Tahoe's direct-capture consent are separate. Run the desktop-capture probe in the builder, choose **Allow**, rerun it without a prompt, and create a new seed. ### The preflight works over SSH but the matrix fails SSH does not own the foreground Aqua session. Run `run-all.sh` from Terminal in the VM display so fixtures, focus checks, and macOS prompts use the console session. ### The runner rejects the source revision Commit or clean the host checkout, sync again, and confirm that `/Users/lume/cua/.cua-e2e-source-sha` equals `git rev-parse HEAD`. Diagnostic dirty syncs cannot produce an accepted result. ## Related documentation - [Run Cua Driver in a macOS Lume VM]() - [How Cua Driver is validated]() - [macOS permissions reference]() - [Change SIP on a macOS VM]() - [Runner source and seed reference](https://github.com/trycua/cua/tree/e743f8e4cce43f94c5718bc3cadf6024a63f4382/libs/cua-driver/tests/runners/macos-lume) --- # Connect your agent to Cua Driver Connect an agent to Cua Driver through MCP or the Cua Driver skill and CLI. Cua Driver lets a [computer-use agent]() drive the **host desktop**: installed apps, signed-in browser sessions, local files opened in apps, and the current OS user session. MCP-capable agents connect through `cua-driver mcp`; skill-based agents such as Prime Agent call the CLI directly. [Grok Bot]() is a separate path; it is not Grok Build and does not use stdio MCP. Building Cua Driver into an application instead? Start with [Choose a Cua Driver integration]() to compare MCP, the direct SDK, private workers, and app-hosted services. **Note** This connects an agent to Cua Driver on the current machine. To create an isolated cloud desktop, use [Cua Sandbox]() instead. ## Before you start Install Cua Driver and verify it can see the host desktop: ```bash cua-driver --version cua-driver call list_apps ``` On macOS, grant Accessibility and Screen Recording before connecting an agent: ```bash cua-driver permissions status ``` See [Install Cua Driver]() and [macOS permissions]() for setup details. ## Decide the permission mode first Registering the server does not choose an authorization [permission mode]() — the process that owns the driver runtime does, at launch. Every config below therefore runs in the default `standard` mode unless you change that process. - **macOS:** `cua-driver mcp` proxies to the `CuaDriver.app` daemon to keep TCC attribution with the app, so that daemon's launch flags decide the mode. Start it in the mode you want before the client connects, and use [autostart]() to make that stick. - **Windows and Linux:** bare `cua-driver mcp` owns its own runtime and has no `--permission-mode` flag. Either set `CUA_DRIVER_PERMISSION_MODE` — plus `CUA_DRIVER_CAPABILITY_MANIFEST_FILE` and `CUA_DRIVER_CAPABILITY_MANIFEST_APPROVED` for `bounded` — in the client's `env` block, or run a `cua-driver serve` daemon in that mode and point the client at it with `--socket`. `--grant existing-profile` is the one authorization flag `cua-driver mcp` accepts directly, and it applies only to a runtime this command launches. An agent cannot widen any of this from a tool call. **Warning** A `standard`-mode runtime allows input against every application on the desktop. If an agent should reach only a reviewed set of apps, origins, and directories, register it against a `bounded` runtime instead — see [Write a capability manifest](). ## Generate the client config Use `mcp-config` to print the registration command or JSON for a supported client: ```bash cua-driver mcp-config --client ``` The generated reference is the source of truth for the complete roster and exact command shapes: [`cua-driver mcp-config`](). ## Claude Code Register the plain stdio server: ```bash claude mcp add --transport stdio cua-driver -- cua-driver mcp claude mcp list ``` Claude Code can also use Cua Driver's computer-use compatibility profile. Generate the current command: ```bash cua-driver mcp-config --client claude ``` That profile exposes the same driver tools under the compatibility server name. It still runs Cua Driver over MCP; Anthropic's native computer-use API is separate. **Note: Using a local model** Claude Code can also act as the harness for a model served on the same machine. Follow [Run a local model with Cua Driver]() for a tested Muse Glimmer setup and a smaller MCP tool surface. ## Codex Print the Codex command: ```bash cua-driver mcp-config --client codex ``` It emits a registration using the absolute installed binary path, which avoids `PATH` issues in app-launched Codex sessions: ```bash codex mcp add cua-driver -- /Users/you/.local/bin/cua-driver mcp codex mcp list ``` Restart Codex or open a fresh session after adding the server. For richer agent guidance, install the Cua Driver skill: ```bash cua-driver skills install cua-driver skills status ``` See [Install the Cua Driver agent skill]() for the ClawHub and direct-install paths. ## Prime Agent Prime Agent uses Cua Driver through its persistent IPython environment and the Cua Driver agent skill. It does not need a local MCP registration. Install the skill pack after both tools are installed: ```bash cua-driver skills install cua-driver skills status ``` Cua Driver links the skill into `~/.prime/agent/skills/` when Prime Agent's skill directory exists. Prime Agent also scans the shared `~/.agents/skills/` directory, so an existing Codex-targeted link remains discoverable. Run `/reload` in Prime Agent or start a new session after installation, then ask it to use the Cua Driver skill. Use `/skill:cua-driver` to invoke it explicitly. See [Prime Agent's skill documentation](https://github.com/PrimeIntellect-ai/prime-agent/blob/main/packages/coding-agent/docs/skills.md) for its complete discovery and invocation behavior. Print the same current guidance from the installed binary: ```bash cua-driver mcp-config --client prime-agent ``` ## Cursor Generate the Cursor snippet: ```bash cua-driver mcp-config --client cursor ``` Paste the JSON into `~/.cursor/mcp.json`, or `.cursor/mcp.json` for project scope: ```json { "mcpServers": { "cua-driver": { "command": "cua-driver", "args": ["mcp"], "type": "stdio" } } } ``` Restart Cursor and confirm `cua-driver` appears in the MCP server list. ## Other supported clients `mcp-config` also prints the right shape for clients that use a config file or a different add command. | Client | Generate with | Notes | | ------------- | -------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | Antigravity | `cua-driver mcp-config --client antigravity` | Paste into `~/.gemini/config/mcp_config.json`; `--client gemini` is a legacy alias. | | OpenClaw | `cua-driver mcp-config --client openclaw` | Normal gateway-spawned MCP does not inherit OpenClaw.app's macOS permission grants; embedded hosts should use [Embedding](). | | OpenCode | `cua-driver mcp-config --client opencode` | Configure a real MCP server so screenshots are preserved in image blocks. | | Pi | `cua-driver mcp-config --client pi` | Pi does not support MCP natively; use one-shot `cua-driver call …` commands from its shell. | | Qwen Code | `cua-driver mcp-config --client qwen` | Supports both a CLI add command and `~/.qwen/settings.json`. | | Factory Droid | `cua-driver mcp-config --client droid` | Supports CLI and JSON config forms. | | ZCode | `cua-driver mcp-config --client zcode` | Configure MCP in the GUI, or use `zai mcp add` for Z.ai's separate CLI. | ## Generic MCP JSON For any client that accepts the standard `mcpServers` shape, print the generic config: ```bash cua-driver mcp-config ``` It returns: ```json { "mcpServers": { "cua-driver": { "command": "cua-driver", "args": ["mcp"] } } } ``` After saving the config, restart the client and confirm the `cua-driver` server is connected. ## Grok Bot Grok Bot is not Grok Build. It has no `mcp-config` preset and does not use stdio MCP. See [Grok Bot](). Two valid setups: 1. **Your local Mac or Windows computer.** Install Cua Driver there. Grok Bot uses local-command execution (`cua-driver call …`) behind its local-computer approval policy. 2. **Grok Bot's persistent cloud Linux computer.** If that machine is the desktop being driven, install Cua Driver there and call `cua-driver` on it. ## Clients without a dedicated preset Some MCP clients can run Cua Driver directly even though `mcp-config` does not have a named preset for them. After completing the installation and permission steps on this page, use the client-specific registration and verification steps: - [Grok Build](), xAI's coding CLI, uses stdio MCP. [Grok Bot]() is a separate local-command integration. - [Kimi Code]() uses stdio MCP through its native CLI or configuration file. T3 Code is a control plane for other coding agents rather than a separate MCP client. Configure Cua Driver in the underlying harness, then launch that harness from [T3 Code](). ## Next steps - [Keep Cua Driver running](): keep the daemon alive across reboots and sessions. - [Agent action policy](): behavior agent wrappers should follow for `element_index`, `x,y`, and foreground escalation. - [MCP tools](): inspect every exposed tool and parameter. --- # Run a local model with Cua Driver Connect Muse Glimmer to Cua Driver through Claude Code using Ollama or llama.cpp while keeping the model's tool context small. This guide shows you how to serve Muse Glimmer 30B locally, use Claude Code as the agent harness, and let the model operate desktop applications through Cua Driver. Choose Ollama for the shorter Apple Silicon setup. Choose llama.cpp with an Unsloth GGUF when you want the configuration used for the recorded runs or need direct control over inference settings. The configuration below reproduces the architecture used for these verified macOS runs: [Create a checklist in Apple Notes]() The model mixes accessibility and pixel-grounded actions, then verifies the checklist state. [Schedule an Apple Reminder]() The model uses native controls and a menu, then verifies the title, date, time, and completion state. **Note: Tested configuration** The recorded runs used Meta's Muse Glimmer 30B, Unsloth's `UD-Q4_K_XL` GGUF, llama.cpp's Anthropic-compatible server, Claude Code, and Cua Driver inside a macOS Lume guest. This is a tested configuration rather than a compatibility claim for every local model or inference server. ## Before you start You need: - [Cua Driver installed]() on the computer the model will operate; - Cua Driver's required operating-system permissions, confirmed with `cua-driver doctor`; - Ollama or a current `llama-server` build with multimodal and tool-calling support; - Claude Code; and - enough memory and storage for the selected quant and context size. This guide uses the host desktop. To keep the task in a disposable macOS VM, first [run Cua Driver inside a Lume guest](), then use the guest's MCP command as the filter's `--driver` value. ## Install the MCP schema filter Local models pay a context cost for every exposed tool schema. Install the small filter used by this guide: ```bash mkdir -p "$HOME/.local/bin" curl -fsSL \ https://cua.ai/docs/examples/local-models/cua-mcp-filter.py \ -o "$HOME/.local/bin/cua-mcp-filter" chmod +x "$HOME/.local/bin/cua-mcp-filter" ``` The filter changes the `tools/list` response so the model sees only the named tools. It does not authorize those tools or block a caller that already knows another tool name. Use [permission policies]() when you need an enforcement boundary. Create `muse-cua-mcp.json`: ```json { "mcpServers": { "cua-computer-use": { "command": "cua-mcp-filter", "args": [ "--allow", "start_session,end_session,launch_app,list_apps,list_windows,get_window_state,get_accessibility_tree,move_cursor,click,type_text,press_key,hotkey,invoke_menu" ] } } } ``` Keep the allowlist as small as the task permits. Add a tool only when the task needs it. ## Choose a serving path Both paths connect the same filtered Cua Driver MCP server to Claude Code. The Ollama path uses the official `muse-glimmer:30b-mlx` model. The llama.cpp path uses Unsloth's `UD-Q4_K_XL` GGUF and matches the recorded configuration. ### Path 1: Use Ollama on Apple Silicon Pull the official Muse Glimmer model: ```bash ollama pull muse-glimmer:30b-mlx ``` From the directory that contains `muse-cua-mcp.json`, launch Claude Code: ```bash CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \ ollama launch claude \ --model muse-glimmer:30b-mlx \ --yes \ -- \ --bare \ --strict-mcp-config \ --mcp-config ./muse-cua-mcp.json \ --tools "" ``` Ollama serves the model through its Anthropic-compatible API. The published model has a 128K context window and supports images and tools. ### Path 2: Use llama.cpp with an Unsloth GGUF Run llama.cpp's server in a separate terminal: ```bash llama-server \ --hf-repo "unsloth/Muse-Glimmer-30B-GGUF:UD-Q4_K_XL" \ --alias "muse-glimmer-local" \ --host 127.0.0.1 \ --port 8001 \ --ctx-size 131072 \ --parallel 1 \ --temp 1.0 \ --top-p 0.95 \ --top-k 64 \ --jinja \ --mmproj-auto \ --fit on \ --no-webui ``` The first start downloads the selected GGUF and vision projector. Keep the endpoint bound to `127.0.0.1` unless you have separately secured access to it. Wait for the server to finish loading, then check its health from another terminal: ```bash curl -fsS http://127.0.0.1:8001/health ``` From the directory that contains `muse-cua-mcp.json`, run: ```bash ANTHROPIC_BASE_URL=http://127.0.0.1:8001 \ ANTHROPIC_API_KEY=local-no-key-required \ CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \ claude \ --bare \ --strict-mcp-config \ --mcp-config ./muse-cua-mcp.json \ --tools "" \ --model muse-glimmer-local ``` For either path, `--bare` avoids loading unrelated project instructions and integrations. `--tools ""` removes Claude Code's built-in tools from this session, while the explicit MCP configuration keeps the Cua Driver tools available. ## Run a bounded smoke task Start with a short task that has an observable result: ```text Use only the cua-computer-use MCP tools. Launch Calculator, calculate 2 + 3, and verify from fresh state that the display shows 5. Take a fresh window state before every action and verify the result after every action. Keep get_window_state calls to max_elements=25 and max_depth=3 unless a deeper accessibility tree is required. ``` The model should launch Calculator, act through Cua Driver, read a fresh final state, and report the observed value. If it reports success without the final observation, ask it to verify again before accepting the result. ## Keep context use under control For local computer use, the largest avoidable costs are tool schemas and repeated desktop state. Use these rules: - expose only the tools required by the task; - request window state instead of the entire desktop when the target window is known; - bound accessibility reads with `max_elements` and `max_depth`; - reuse the `pid` and `window_id` returned by fresh state; and - batch deterministic text entry when one `type_text` action can replace several inference turns. In the tested Calculator runs, filtering tools, bounding state, and batching input reduced uncached input from 71,088 tokens to 12,251 and elapsed time from 660 seconds to 224 seconds. Both runs independently verified the displayed value. ## Run the model against a macOS Lume guest When the target is a Lume VM, run the exact Cua Driver executable inside the logged-in guest and grant permissions to that guest identity. Use SSH only to carry MCP stdio between the harness and the guest. This keeps screenshots, input, and recording attached to the macOS session being operated instead of relaying clicks through a VM viewer. Follow [Run Cua Driver in a macOS Lume VM]() for the guest setup and SSH command. Save that SSH command as an executable `guest-cua-mcp` wrapper in the current directory, then pass the wrapper to the filter: ```bash cua-mcp-filter \ --driver ./guest-cua-mcp \ --allow start_session,end_session,get_window_state,move_cursor,click,type_text,invoke_menu ``` ## Record the run Use [Record and render a Cua Driver trajectory]() to retain the raw display capture and per-action evidence. The two videos above add an editorial intro and summarized tool trace after Cua Driver finished the original recording. ## Troubleshooting ### Claude Code sees the full Cua Driver tool catalog Confirm that the session uses both `--strict-mcp-config` and the filtered `muse-cua-mcp.json`. A global Cua Driver registration can otherwise load alongside the filtered server. ### The model stops after a few actions Reduce `max_elements` and `max_depth`, remove unused tools, and shorten the task. With Ollama, confirm the selected model has at least a 64K context window. If llama.cpp lowers the usable context to fit memory, restart it with a context size the machine can hold reliably. ### Screenshots do not reach the model Use a real MCP connection. Shell wrappers that flatten MCP image blocks into text remove the visual input that a multimodal model needs for pixel grounding. ### macOS reports missing permissions Run `cua-driver permissions status` on the machine being operated. In a Lume VM, query the guest daemon and grant Accessibility and Screen Recording to the guest's Cua Driver identity. ## Related guides - [Muse Glimmer model and quant documentation](https://unsloth.ai/docs/models/muse-glimmer) - [Connect Ollama to Claude Code](https://docs.ollama.com/integrations/claude-code) - [Connect your agent to Cua Driver]() - [Run Cua Driver in a macOS Lume VM]() - [Record and render a Cua Driver trajectory]() - [Agent action policy]() --- # Drive a Web Page Bind an exact browser window, inspect its active tab, and navigate, click, or type without foregrounding it. Use the browser tools when an agent must act inside Chromium or an Electron page without raising the native window. The tools bind the `(pid, window_id)` you selected to a DevTools target, then issue page input through that exact binding. [Chrome remains behind the agent terminal]() Claude Code starts a YouTube video in Chrome without raising the browser window. ## Inspect the selected browser first **Warning** Prefer a driver-owned isolated profile. Existing-profile attachment gives CDP broad access to that profile's live pages, cookies, and storage. Use it only when the task needs an existing authenticated session and the machine and its local processes are trusted. First use `list_apps` and `list_windows` as usual, then call `get_browser_state` with the selected native `(pid, window_id)`. Driver-owned profiles and supported embedded applications can bind through an exact owned endpoint. A standalone Chrome or Edge profile still requires explicit existing-profile approval, even when the browser already exposes a loopback DevTools endpoint. In that case, `get_browser_state` returns `browser_consent_required` with `next_action: browser_prepare`. Do not pass remote-debugging flags or a user-data directory through `launch_app`. Cua Driver rejects those flags because they could expose a person's normal profile to DevTools. ## Prepare an isolated browser when required If inspection returns `browser_requires_setup`, call `browser_prepare` as a separate operation. Isolated-profile preparation follows the runtime's permission mode and optional capability manifest. In `standard` it is a routine operation, in `bounded` it must match the manifest, and in `unrestricted` it relies on the launcher's dangerous acknowledgement. It starts another Chromium process with a driver-owned profile; it never copies, modifies, restarts, or terminates the selected user profile. ```jsonc browser_prepare({ "pid": 844, "session": "research-1", "allow_launch": true, "profile": {"mode": "isolated_new"} }) ``` The response includes `prepared_pid`. Call `list_windows` for that new process, then bind its window with `get_browser_state`. An `isolated_new` profile is removed when its owning session ends. Use `profile: {"mode":"isolated_named","name":"research"}` only when the automation profile must survive across sessions. `allow_launch: true` states that this call may create the separate process; it does not widen runtime authorization. `get_browser_state` never performs setup itself. ## Attach to an existing Chrome or Edge profile Use this route only when the agent must work in a supported Chromium profile that is already running and authenticated. Cua Driver does not restart the browser, copy the profile, or modify profile files. The route supports Chrome and Edge on macOS and Windows, plus Chrome in the validated Linux X11 and Sway configurations. Chromium and Edge have descriptor-backed Linux routes but are not yet product-validated there. Start a named driver session and choose the permission mode before launching the daemon. Prefer `bounded` for repeatable browser automation: approve one short-lived manifest that names the browser profile kind, application identity, allowed origins, and typed browser tools. This avoids prompts without granting every Cua capability. For example: ```yaml version: 3 expires_after: 2h idle_timeout: 20m resources: apps: - bundle_id: com.google.Chrome launch: false windows: all terminate: deny browser: profiles: - kind: existing_profile origins: - https://app.example.com allow: tools: - start_session - end_session - get_browser_state - browser_prepare - browser_navigate - browser_click - browser_type ``` On Windows and Linux, use Chrome's canonical absolute executable path instead of `bundle_id`. The runtime still re-proves the live process fingerprint and exact native window before attaching. Launch Cua Driver with that reviewed scope: ```bash cua-driver serve \ --permission-mode bounded \ --capability-manifest ./browser-session.yaml \ --approve-capability-manifest ``` Bounded attachment does not display a Cua modal or banner. The manifest is the trusted launcher's authorization boundary, and calls outside it fail closed. For a disposable VM where you intentionally accept all Cua actions, launch unrestricted mode explicitly instead: ```bash cua-driver serve \ --permission-mode unrestricted \ --dangerously-bypass-approvals ``` Then request the exact process and native window: ```jsonc browser_prepare({ "pid": 844, "window_id": 10725, "session": "research-1", "strategy": {"kind": "existing_profile"} }) ``` On Chrome 144 and later, the approved preparation flow can attach through the running profile's agent auto-connect endpoint. It does not restart Chrome, so the profile keeps its current tabs, extensions, cookies, and sign-in state. The endpoint proves which browser owns the connection; it does not replace the existing-profile authorization. > **Warning:** Unrestricted mode has no runtime Cua approval prompts and makes > no prompt-injection safety claim. Prefer `isolated_new`, or a bounded > existing-profile manifest when authentication is required. Use unrestricted > existing-profile attachment only in an environment whose account and data > exposure you accept. A trusted embedding host can install the authorization callback described in [Permission modes and bounded autonomy](). In `standard`, the host may authorize the exact browser resource once. A standalone CLI or MCP launch can instead use `--grant existing-profile`. `bounded` uses the approved capability manifest. The browser may display its own remote-debugging consent prompt. Cua Driver will press only the exact browser-owned semantic allow action for this approved process and window. An absent, ambiguous, dismissed, or unrecognized prompt is refused; it is never treated as generic permission to click security dialogs. If the approved process has no DevTools endpoint, Cua Driver opens a temporary tab in the approved window and navigates to the product's fixed remote-debugging page (`chrome://inspect/#remote-debugging` or `edge://inspect/#remote-debugging`). It matches exactly one **Allow remote debugging for this browser instance** checkbox, verifies that it is off, and presses it once. It then proves that the new listener is loopback-only and owned by the approved process before closing the temporary tab. The result's `side_effects` reports whether the page was opened and closed, whether the address field was focused, whether the setting was enabled, and whether Chrome displayed a connection-consent prompt. If setup fails after a visible action, the structured refusal reports `detail.setup_side_effects`; a checkbox changed by that failed attempt is restored when its exact state can still be proven. Any ambiguous control or changed process/window identity is refused. On macOS, semantic AX matching remains the first route. If Chrome withholds the internal page's web AX subtree, Cua Driver opens and navigates its own temporary tab, waits for the fixed address to be committed with the expected selected-tab title and no omnibox edit in progress, then requires one unique checkbox-shaped control in a bounded setup-page region. It revalidates the unchanged target window, routes the click only to that browser PID, and verifies the resulting state on the same control. Because macOS delivers this bounded pixel action through global input, Cua Driver may briefly foreground the exact approved window, restore the previous frontmost app, and report those effects in `side_effects`. Unsupported appearance, scale, zoom, window-size, or toolbar geometry is refused without a click. This fallback never applies to ordinary web pages or generic security dialogs. On Linux, launch the browser with `--force-renderer-accessibility` unless a screen reader already enables its complete AT-SPI tree. Native Wayland also requires a validated compositor route that can prove the exact process, window, geometry, and temporary focus restoration. A missing prerequisite is reported as a refusal; Cua Driver does not fall back to an unscoped desktop click. After `attached_existing_profile`, list windows again if needed and call `get_browser_state({pid, window_id, session})`. Attachment invalidates older browser capabilities, so do not reuse a previous `target_id`, `tab_id`, or page ref. The grant lives only in runtime memory. It expires after inactivity, has an absolute lifetime, and is revoked when its owning session ends or the runtime shuts down. A dropped browser socket may reconnect up to three times under the same live grant; a successful reconnect invalidates capabilities again and requires another bind. Browser restart requires a new authorization. Existing-profile activity events record content-free action metadata and results. They omit screenshots, accessibility snapshots, URLs, and resource identities, so authenticated page content is not persisted as authorization evidence. ## Bind the native window Browser capabilities are scoped to a driver session. Start one and pass the same session value to every browser call: ```jsonc start_session({"session": "research-1"}) get_browser_state({ "pid": 844, "window_id": 10725, "session": "research-1" }) ``` Keep the returned opaque `target_id`. Use the `tab_id` whose `active` field is `true` only when selection is uniquely proven. `active: false` is proven unselected; `active: null` means the native window cannot distinguish tabs, such as two tabs with the same title. In that case, select an explicit tab by its returned metadata and never infer selection from list order. Mutation is available only when `binding_quality` is `exact`. A heuristic or ambiguous window match is refused. ## Snapshot the tab Snapshot the selected tab before using an element ref: ```jsonc get_browser_state({ "target_id": "", "tab_id": "", "session": "research-1", "snapshot_format": "semantic_v2" }) ``` Read `outline` for visible page content. Select actions only from `refs`, and check that the chosen entry declares `click` or `type` in `actions`. `content_refs` are read capabilities for `scope_ref`; they cannot be passed to an unsupported mutation. If `snapshot.continuation` is non-null, request the next ranked segment without changing the page: ```jsonc get_browser_state({ "target_id": "", "tab_id": "", "session": "research-1", "snapshot_format": "semantic_v2", "continuation": "" }) ``` Use `query` to find matching roles, accessible names, or visible text. Use a current `scope_ref` to inspect one semantic subtree when names repeat in different regions. A continuation is single-use; a newer snapshot invalidates it. Refs are valid only for that session, target, tab, document, frame, and latest snapshot. Take a fresh snapshot after navigation or whenever a call returns `browser_ref_stale`. `dom_refs_v1` remains available for compatibility when `snapshot_format` is omitted, but new workflows should request `semantic_v2`. ## Click and type The default click route requests Chromium's trusted input domain: ```jsonc browser_click({ "target_id": "", "tab_id": "", "ref": "p1:7", "session": "research-1" }) ``` Use viewport `x` and `y` instead of `ref` when the page snapshot cannot name the rendered target. Standalone Chrome and Edge on Windows have passing full-background evidence for this route. Standalone Chromium on macOS and Linux returns `browser_input_trust_unavailable` because dispatching the trusted pointer event activates the native browser window. To request a synthetic full-background DOM click deliberately, pass `input_route: "dom_event"` with a ref: ```jsonc browser_click({ "target_id": "", "tab_id": "", "ref": "p1:7", "input_route": "dom_event", "session": "research-1" }) ``` Use this only when the page's DOM click semantics are acceptable. The driver never silently falls back from trusted input to a DOM event. A successful `dom_event` dispatch returns `effect: "unverifiable"`: it does not mean the control activated, and trust-gated controls may ignore it. Take a fresh page snapshot and verify the expected state before continuing. Cua Driver does not automatically foreground the browser when that verification fails. Snapshot again, then type into the fresh input ref: ```jsonc browser_type({ "target_id": "", "tab_id": "", "ref": "p2:3", "text": "status: ready", "session": "research-1" }) ``` The default `insert_text` mode is efficient for ordinary text. Use `mode: "keystrokes"` when the page depends on per-character keyboard events. Both modes insert at the current selection. To replace a pre-filled value, pass `replace: true`. An empty `text` with `replace: true` clears the field through normal input events: ```jsonc browser_type({ "target_id": "", "tab_id": "", "ref": "p2:4", "text": "new value", "replace": true, "session": "research-1" }) ``` ## Use extended pointer actions `browser_pointer` supports `hover`, `right_click`, `double_click`, `scroll`, and `drag`. Use the trusted route when genuine browser input semantics are required and the platform reports that it can preserve posture. Use the explicit synthetic route with current refs when DOM event semantics are acceptable: ```jsonc browser_pointer({ "target_id": "", "tab_id": "", "ref": "p3:4", "action": "scroll", "input_route": "dom_event", "delta_y": 240, "session": "research-1" }) ``` Hover, right-click, double-click, and drag require a ref whose `actions` contains `pointer`. Scroll accepts either `scroll` or `pointer`, so a plain overflow container can be scrollable without gaining broader pointer authority. For drag, pass `destination_ref` from the same exact frame. The driver refuses mixed-frame or stale destinations instead of translating them approximately. ## Handle a page-owned dialog Prime and inspect the exact tab with `browser_dialog({action:"inspect"})`. When `present` is true, pass the returned `dialog_id` to `accept` or `dismiss`. Only an accepted prompt may include `prompt_text`. This tool does not handle browser permission prompts, extension UI, native sheets, or file pickers. Creating a Chromium native modal can activate its browser window; once you restore the intended occlusion, inspecting and resolving that exact dialog do not activate it again on Windows and macOS. Resolution defaults to `delivery_mode:"background"`. Linux Chromium cannot resolve the native modal while preserving background posture, so the driver returns `browser_input_trust_unavailable` before dispatch. Retry with `delivery_mode:"foreground"` only when foreground activation is acceptable. ## Assign files without a picker Take a semantic snapshot and choose a file-input ref whose `actions` contains `upload`. Call `browser_set_input_files` with absolute paths to direct regular files. Symlinks, directories, missing paths, and more than 32 files are refused. Verify the page's uploaded-file state afterward; the tool response contains only a count and never echoes the local paths. ## Download into an approved directory Choose a current ref that activates the download and call `browser_download`. An MCP host may still apply its own destructive-tool approval rules. Pass an existing canonical absolute `destination_root`. Cua's policy stack and bounded manifest, when active, must admit that directory. The tool temporarily scopes Chromium's browser-wide download behavior, correlates the exact frame and opaque download id, restores the default on every outcome, and returns only the final byte count and opaque id. It does not return a filename, URL, or local path. ## Navigate ```jsonc browser_navigate({ "target_id": "", "tab_id": "", "url": "https://example.com", "session": "research-1" }) ``` Navigation invalidates every prior ref for the tab. Wait for the destination, then call `get_browser_state` in snapshot mode again. ## End the session ```jsonc end_session({"session": "research-1"}) ``` Ending the session revokes its target, tab, and element capabilities. ## Embedded webviews Electron has an exact route only while one proven native window maps to one CDP page. Adding another page or window invalidates that route. Tauri, WKWebView, WebKitGTK, and the split-process WebView2 shape currently return `browser_route_unavailable` unless the driver can independently prove the native-host-to-engine relationship. Continue with native `get_window_state` and AX/PX actions for those surfaces. ## Legacy page actions The older `page` tool remains available for compatibility. Its read-only `get_text` and `query_dom` actions are available by default. Legacy mutation actions are disabled unless the daemon operator explicitly starts Cua Driver with `CUA_DRIVER_ENABLE_LEGACY_PAGE_MUTATIONS=1`. The flag is read when the daemon starts, so restart Cua Driver after changing it. That flag is a temporary compatibility escape hatch, not an equivalent browser grant: legacy CDP ports and URL hints do not receive the typed surface's exact native-window correlation, endpoint lifetime, or existing-profile consent checks. Do not enable it for untrusted agents or shared hosts. Migrate mutation workflows to `get_browser_state` and the typed `browser_*` tools. ## Related - [Browser targeting and background delivery]() - [Browser tool reference]() - [Existing-profile attachment reference]() - [Known limits]() - [Capture and delivery modalities]() --- # Record and render a Cua Driver trajectory Capture an agent's Cua Driver actions and turn the trajectory into a zoom-on-click MP4. Record an agent-driven task when you need both the action evidence and a short product demo. Cua Driver saves each action with before-and-after state, screenshots, and arguments. With video enabled, it also captures the display so you can render the trajectory with zoom effects around each action. [Turn an agent trajectory into a zoom-on-click demo]() The agent clicks cells in a numbered grid, then Cua Driver renders the recorded actions as a focused product demo. ## Before you start - [Install Cua Driver]() and [connect your agent](). - Install `ffmpeg`. Rendering requires it on every platform. Video capture also requires it on Windows and Linux. macOS 15 and later use ScreenCaptureKit for capture. ```bash tab="macOS" brew install ffmpeg ``` ```powershell tab="Windows" winget install Gyan.FFmpeg ``` ```bash tab="Linux" sudo apt install ffmpeg ``` ### Start recording with video enabled Ask your MCP client to call `start_recording` before the agent begins the task: ```json { "output_dir": "~/cua-trajectories/calendar-demo", "record_video": true } ``` The recording directory will contain one `turn-NNNNN` folder for each action and a display capture named `recording.mp4`. ### Let the agent complete the task Continue with the usual Cua Driver tools. Actions such as `click`, `type_text`, `press_key`, and `scroll` are added to the trajectory automatically. Keep the task focused. A short trajectory produces a clearer demo and makes the evidence easier to inspect. ### Stop and finalize the recording Ask the MCP client to call `stop_recording` with no arguments: ```json {} ``` Wait for this call to finish before opening or rendering `recording.mp4`. It finalizes the video file and returns its path. ### Render the demo Render the trajectory from the command line: ```bash cua-driver recording render \ ~/cua-trajectories/calendar-demo \ ~/cua-trajectories/calendar-demo.mp4 ``` The renderer uses the recorded action coordinates to add zoom effects around each interaction. Use `--scale` to adjust their strength, or `--no-zoom` for a plain render: ```bash cua-driver recording render \ ~/cua-trajectories/calendar-demo \ ~/cua-trajectories/calendar-demo.mp4 \ --scale 2.5 ``` ## Example: edit a local-model trajectory [Turn a local-model run into a short product demo]() Cua Driver captured the original display and action evidence while Muse Glimmer operated Reminders. An editorial pass added the opening prompt and summarized tool trace; those overlays are separate from the built-in trajectory renderer. For the complete recording schemas and render options, see [MCP recording tools]() and the [`cua-driver recording` CLI reference](). --- # Keep Cua Driver running Register Cua Driver as a persistent daemon that starts automatically and survives reboots. External MCP clients and one-shot CLI calls require a daemon. The daemon owns the per-pid element cache, permission policy, recording and configuration state, macOS TCC attribution, and the Windows interactive-session context. Keep it running when you use those interfaces. Applications that import `CuaDriver.create()` execute through the same-process SDK and do not need this setup. Every entry below starts the daemon in the default `standard` [permission mode](). The mode is fixed at launch, so a `bounded` or `unrestricted` daemon needs that configuration in the autostart entry itself — see [Pin a permission mode](#pin-a-permission-mode). **macOS** `cua-driver autostart` is not implemented for macOS yet, so write a LaunchAgent instead. If you have a checkout of the [cua repo](https://github.com/trycua/cua), the helper script writes it for you (run from the repo root): ```bash bash libs/cua-driver/scripts/install-local.sh --autostart ``` Local installers embed the checkout's full Git commit in `get_config.source_sha`. A build from a modified or untracked source tree uses `-dirty` so it cannot be mistaken for an exact commit build. Source snapshots without `.git` must provide `CUA_DRIVER_SOURCE_SHA` explicitly. If you installed via the one-line installer and have no checkout, create the plist yourself. Save this file at `~/Library/LaunchAgents/com.trycua.cua-driver.plist`: ```xml Label com.trycua.cua-driver ProgramArguments /Applications/CuaDriver.app/Contents/MacOS/cua-driver serve RunAtLoad KeepAlive ``` Load the LaunchAgent: ```bash launchctl load ~/Library/LaunchAgents/com.trycua.cua-driver.plist ``` **Note** A LaunchAgent daemon starts under `launchd` and is attributed to `com.trycua.driver`. Grant Accessibility and Screen Recording once, and those grants persist across reboots. If you start prompts from a terminal without a LaunchAgent, macOS attributes them to the terminal instead of the driver, so the grants do not apply to Cua Driver. **Windows** Run these commands once from an interactive session, either RDP or the local console: ```powershell cua-driver autostart enable cua-driver autostart kick ``` `enable` registers a Scheduled Task named `cua-driver-serve` with `LogonType: Interactive`. That keeps the daemon in Session 1+ instead of Session 0. The command is idempotent, so running it again after an upgrade updates the binary path automatically. Run it from a non-interactive context (over SSH, or in Session 0 as `SYSTEM`) and it fails with a confusing `No mapping between account names and security IDs was done` error — use RDP or the local console instead. `kick` starts the task immediately, so you do not need to wait for the next logon. Check the task state when you need it: ```powershell cua-driver autostart status # registered (running) ``` Only `not-registered` confirms that the Scheduled Task is absent. A `permission-denied` or `unknown` result means the current process could not inspect Task Scheduler; the command exits non-zero and preserves the original diagnostic instead of telling you to re-register blindly. Remove the autostart registration: ```powershell cua-driver autostart disable ``` **Note** `LogonType: Interactive` is required. Alternatives such as `S4U` or `Password` put the daemon in Session 0, where GUI tools return empty arrays. See [Drive a Windows app over SSH]() for the SSH proxy workflow that uses this setup. The Scheduled Task keeps running after an RDP disconnect. A disconnected session remains in `Disc` state and is still a live interactive session. The daemon stops only on explicit logoff or reboot; the next interactive logon triggers the task again. **Linux** `cua-driver autostart` is Windows-only today. Create a systemd user unit by hand. Save this unit at `~/.config/systemd/user/cua-driver.service`: ```ini [Unit] Description=cua-driver background daemon After=graphical-session.target PartOf=graphical-session.target [Service] Type=simple ExecStart=%h/.local/bin/cua-driver serve Restart=on-failure RestartSec=2 [Install] WantedBy=graphical-session.target ``` Reload systemd, enable the unit, and start it: ```bash systemctl --user daemon-reload systemctl --user enable --now cua-driver.service systemctl --user status cua-driver.service ``` On CI runners or headless machines where the daemon should survive user logout, enable linger: ```bash loginctl enable-linger $USER ``` A headless box has no `graphical-session.target`, so on those machines also change the unit's `[Install]` section to `WantedBy=default.target` — otherwise `enable` has nothing to hook into and the daemon won't start on boot. ## Pin a permission mode `--permission-mode`, `--capability-manifest`, `--approve-capability-manifest`, and `--dangerously-bypass-approvals` are read once, when the daemon starts. Put them in the autostart entry so an unattended restart comes back in the same mode instead of falling back to `standard`. **macOS** Add each flag as its own `` inside `ProgramArguments`, after `serve`: ```xml ProgramArguments /Applications/CuaDriver.app/Contents/MacOS/cua-driver serve --permission-mode bounded --capability-manifest /Users/you/cua-session.yaml --approve-capability-manifest ``` `launchd` does not expand `~` or run a shell, so the manifest path must be absolute. Reload the agent after editing: ```bash launchctl unload ~/Library/LaunchAgents/com.trycua.cua-driver.plist launchctl load ~/Library/LaunchAgents/com.trycua.cua-driver.plist ``` **Windows** `cua-driver autostart enable` always registers a bare `serve` command and has no mode flags, so configure the equivalent User-scope environment variables. The Scheduled Task inherits them at logon: ```powershell setx CUA_DRIVER_PERMISSION_MODE bounded setx CUA_DRIVER_CAPABILITY_MANIFEST_FILE "$env:USERPROFILE\cua-capabilities.yaml" setx CUA_DRIVER_CAPABILITY_MANIFEST_APPROVED 1 ``` `setx` affects new processes only, so restart the daemon for the change to take effect: ```powershell cua-driver stop cua-driver autostart kick ``` **Linux** Extend `ExecStart` in `~/.config/systemd/user/cua-driver.service`: ```ini ExecStart=%h/.local/bin/cua-driver serve --permission-mode bounded --capability-manifest %h/cua-session.yaml --approve-capability-manifest ``` Then reload and restart the unit: ```bash systemctl --user daemon-reload systemctl --user restart cua-driver.service ``` Every form is trusted launch configuration: anyone who can edit the plist, the Scheduled Task, the systemd unit, or those environment variables can change the mode the daemon comes back in. Bad configuration fails startup rather than silently downgrading to `standard` — a bounded daemon with no approved manifest, or `unrestricted` without its acknowledgement, does not bind its action socket at all. Confirm the daemon is up, then make one call your manifest does not allow and check that it is refused: ```bash cua-driver status cua-driver call '{}' # Error: ... ``` ## Verify it's running ```bash cua-driver status # Cua Driver daemon is running # socket: /Users/you/Library/Caches/cua-driver/cua-driver.sock # pid: 12345 ``` On Windows the socket path is `\\.\pipe\cua-driver` and `status` also reports the session number. ## Stop it cleanly ```bash cua-driver stop ``` This sends a shutdown signal to the daemon process. The autostart entry, whether LaunchAgent, Scheduled Task, or systemd unit, remains registered and starts it again on the next logon or trigger. --- # Restrict tool access with permission policies Use YAML or Rego policies to control which MCP tools an agent can call and what arguments it may pass. Cua Driver's permission policy engine lets you define exactly which tools an agent may call, and — for sensitive tools — restrict the argument values it can supply. The engine is deny-by-default: any tool not explicitly allowed is blocked. Two policy formats are supported — YAML for straightforward allow/deny lists and Rego for programmable logic. ## Before you start - Cua Driver installed and working (`cua-driver --version`) - A running `cua-driver serve` daemon (or a process about to be started) - For Rego policies: a build with the `rego` feature enabled (included in the default release binary) ## Create a YAML policy Create a file, for example `~/.cua-driver/policy.yaml`, that lists the tools you want to allow: ```yaml allow: tools: - screenshot - get_window_state - list_apps - list_windows - click - type_text - press_key - scroll - launch_app - wait deny: tools: - shell_execute ``` Every tool in `allow.tools` is unconditionally permitted. Every tool in `deny.tools` is rejected even if it also appears in an allow rule. Any tool not mentioned in either section is denied by the default policy. ## Add argument constraints to a YAML policy Use `allow.rules` to permit a tool only when its arguments satisfy bounds, length, pattern, or allowed-value checks: ```yaml allow: tools: - screenshot - get_window_state rules: - tool: type_text constraints: text: max_length: 500 pattern: "^[\\x20-\\x7E\\n\\t]*$" # printable ASCII only - tool: scroll constraints: amount: min: -20 max: 20 - tool: launch_app constraints: bundle_id: allowed: - "com.apple.Safari" - "com.google.Chrome" - "com.microsoft.VSCode" deny: tools: - shell_execute ``` Each `constraints` key matches a JSON argument name. A rule allows the call only when **all** constraints pass. If you list the same tool in more than one rule, the call is allowed when **any** rule matches. **Note** The `type_text_chars` tool name is automatically aliased to `type_text` before policy evaluation, so a rule on `type_text` covers both forms. ## Restrict a browser-only agent For an agent that should work only in a browser Cua Driver has already prepared, allow inspection and ordinary page interaction while denying profile setup, local-file transfer, downloads, and the legacy mutation surface: ```yaml allow: tools: - list_apps - list_windows - start_session - end_session - get_browser_state - browser_click - browser_type - browser_pointer - browser_dialog - wait rules: - tool: browser_navigate constraints: url: pattern: "^https://(docs\\.example\\.com|app\\.example\\.com)(/|$)" deny: tools: - browser_prepare - browser_set_input_files - browser_download - page - shell_execute ``` Replace the example hosts with the exact destinations required for the task. Run profile preparation separately under maintainer control before starting the restricted agent. **Warning** Tool policy cannot predict what a page will do after a click, form submit, script, or redirect. A host allowlist constrains explicit `browser_navigate` calls, not destinations reached through page behavior. Keep consequential accounts and existing authenticated profiles outside an untrusted agent's policy boundary. ## Create a Rego policy For more complex access control — for example, allowing different tool sets based on the active task or runtime conditions — write a Rego policy: ```rego package cua.policy import rego.v1 # Tools that are always allowed. safe_tools := { "screenshot", "get_window_state", "list_apps", "list_windows", "click", "press_key", "scroll", "wait", } allow if { input.tool in safe_tools } # Allow type_text only for short printable strings. allow if { input.tool == "type_text" count(input.arguments.text) <= 500 regex.match(`^[\x20-\x7E\n\t]*$`, input.arguments.text) } ``` Save this as `~/.cua-driver/policy.rego`. The policy engine evaluates `data.cua.policy.allow`; the rule must produce a boolean. `undefined` is treated as `false` (deny). ## Load a policy directory (Rego only) When you point the environment variable at a directory instead of a single file, the engine loads every `.rego` file in that directory in lexicographic order: ```bash mkdir -p ~/.cua-driver/policies # Place policy files in the directory: # 00_safe_tools.rego # 10_type_text.rego ``` Set the environment variable to the directory path: ```bash export CUA_DRIVER_POLICY_FILE=~/.cua-driver/policies ``` **Warning** A directory path only works with Rego. Pointing it at a directory when the `rego` feature is disabled returns an error on startup. ## Enable the policy Set `CUA_DRIVER_POLICY_FILE` to the path of your policy file before starting the driver: **bash/zsh** ```bash export CUA_DRIVER_POLICY_FILE=~/.cua-driver/policy.yaml cua-driver serve ``` **fish** ```fish set -x CUA_DRIVER_POLICY_FILE ~/.cua-driver/policy.yaml cua-driver serve ``` **env for a single process** ```bash CUA_DRIVER_POLICY_FILE=~/.cua-driver/policy.yaml cua-driver serve ``` The policy is loaded once when the process starts. Changing the file while the daemon is running has no effect until the daemon restarts. A missing or invalid configured file prevents the daemon from binding its action socket. ## Verify the policy is active Make a call that should be denied and confirm the error: ```bash cua-driver call shell_execute '{"command":"echo hello"}' # Error: tool 'shell_execute' is explicitly denied ``` Make a call that should be allowed: ```bash cua-driver call screenshot '{}' # ✅ ... ``` ## Add the policy to autostart (macOS launchd) If you use `cua-driver autostart enable`, add the environment variable to the launchd plist so it is set when the daemon starts automatically: 1. Find the plist file: ```bash cat ~/Library/LaunchAgents/com.trycua.cua-driver.plist ``` 2. Add an `EnvironmentVariables` key: ```xml EnvironmentVariables CUA_DRIVER_POLICY_FILE /Users/you/.cua-driver/policy.yaml ``` 3. Reload the agent: ```bash launchctl unload ~/Library/LaunchAgents/com.trycua.cua-driver.plist launchctl load ~/Library/LaunchAgents/com.trycua.cua-driver.plist ``` ## Troubleshooting **All calls are denied even though my policy allows them.** Check that `CUA_DRIVER_POLICY_FILE` points to the correct path and that the daemon has been restarted after setting the variable. A typo or unreadable file now fails daemon startup instead of disabling enforcement. **The daemon fails to start with a policy parse error.** YAML policies must not contain empty tool names, and Rego policies must define `data.cua.policy.allow` as a boolean rule. Run `cat $CUA_DRIVER_POLICY_FILE` to confirm the file is readable and syntactically valid. **A tool I allowed still returns a denial.** The rule might exist but the argument constraints are not satisfied. Check the denial message: `"argument 'text' must be at most 500 characters"` tells you which constraint failed. For Rego policies, add a `print` statement temporarily to trace which branch runs. **I want to allow all tools.** Do not set `CUA_DRIVER_POLICY_FILE`. The user-policy layer is then absent. The built-in risk map, permission mode, managed policy, and active resource grants still apply. ## Related - [Permission policies](): full YAML schema and Rego interface reference - [How permission policies work](): the engine internals, evaluation order, and trust model - [Permission modes and bounded autonomy](): standard, bounded, and unrestricted startup - [MCP tools](): complete list of tool names and their arguments --- # Write a capability manifest Narrow any Cua Driver permission profile to explicit tools and resources. Use a capability manifest when a Cua Driver runtime should stay inside a reviewed set of applications, browser origins, files, and tools. The manifest is optional in `standard` and `unrestricted` and required in `bounded`. ## 1. Create the manifest Save this as `cua-capabilities.yaml` and replace the example paths: ```yaml version: 3 expires_after: 8h idle_timeout: 30m allow: tools: - start_session - end_session - launch_app - get_window_state - click - type_text - press_key - kill_app resources: apps: - executable: /usr/bin/example-editor launch: true windows: all terminate: driver_launched files: read: - dir: /data/input recursive: true write: - dir: /data/output recursive: true desktop: display: false ``` This manifest drives a native application. Browser work uses a separate manifest, for the reason described in the next step. On macOS, replace `executable` with the application's `bundle_id`: ```yaml resources: apps: - bundle_id: com.example.Editor launch: true windows: all terminate: driver_launched ``` Windows and Linux application entries use a canonical absolute executable path. ## 2. Select browser access A capability manifest that drives a browser is a separate, typed-browser-only manifest. It cannot also allow generic input or window observation. Every navigation is checked against `resources.browser.origins`, so a browser manifest must list at least one origin. Declaring origins then excludes the tools that could reach a page without crossing the typed browser adapter: `click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`, the mouse primitives, `get_accessibility_tree`, `get_window_state`, `verify_state`, `get_desktop_state`, and `page`. Including any of them alongside origins is refused when the runtime starts: ```text authorization startup error: origin-scoped capability manifests cannot allow 'click' because it bypasses the typed browser origin adapter ``` The check reads `allow.tools` alone, so it applies even when the manifest's application scope contains no browser. Use `list_windows` rather than `get_window_state` to find the `window_id` a browser call needs. For a driver-owned browser profile: ```yaml version: 3 expires_after: 8h idle_timeout: 30m allow: tools: - start_session - end_session - launch_app - list_windows - browser_prepare - get_browser_state - browser_navigate - browser_click - browser_type - browser_download resources: browser: profiles: - kind: isolated origins: - https://app.example.com desktop: display: false ``` If the task must use a logged-in Chromium profile, make that choice explicit by replacing the profile kind: ```yaml resources: browser: profiles: - kind: existing_profile origins: - https://app.example.com ``` An existing-profile entry supplies resource scope. In `bounded`, the in-scope attachment is unattended. In `standard`, the normal launch grant or trusted host decision is still required. The typed browser tools validate the live top-level origin before input in every profile. An agent that needs both a browser and generic desktop input needs two runtimes: this manifest for the browser, and a separate application-scoped manifest for the desktop work. Scope that second runtime so it cannot reach a browser window. A runtime that can click the browser makes the first runtime's origin list decorative, because nothing checks an origin on the generic input path. Give it a capability manifest naming only the non-browser applications it needs, and keep `desktop.display: false`. A standard-mode runtime is not a substitute: standard allows input against any application, including the browser. ## 3. Authorize display access Application entries with `windows: all` authorize observation and input for windows belonging to that application. Keep `desktop.display: false` when the workflow does not need unfiltered desktop capture or desktop-coordinate input. Set it to `true` only when the agent needs full-display access: ```yaml resources: desktop: display: true ``` ## 4. Choose file roots Directory grants are checked by canonical path component: ```yaml resources: files: read: - dir: /data/input recursive: true write: - dir: /data/output recursive: true ``` `recursive: false` allows direct children only. Existing version 1 manifests may still list exact path strings, but version 2 and 3 directory roots are more practical for unattended output. ## 5. Start Cua Driver ```bash cua-driver serve \ --permission-mode bounded \ --capability-manifest ./cua-capabilities.yaml \ --approve-capability-manifest ``` The approval flag confirms that the trusted launcher reviewed this exact file. It is not accepted from a tool call. To narrow `standard`, use the same flags with `--permission-mode standard`. Routine standard behavior remains promptless, while residual boundaries such as existing-profile attachment still require their normal grant. To narrow `unrestricted`, add the manifest flags alongside `--dangerously-bypass-approvals`; the approval bypass then applies only inside the manifest. For MCP, point the client at the running daemon: ```json { "mcpServers": { "cua": { "command": "cua-driver", "args": ["mcp", "--socket", "/path/to/cua-driver.sock"] } } } ``` Use the platform's reported default endpoint unless your launcher selected a custom `--socket`. ## 6. Test denial before unattended use Confirm all three cases: 1. An allowed tool against an allowed resource succeeds without a Cua prompt. 2. An allowed tool against a different app, origin, or path returns `bounded_resource_outside_manifest`. 3. A tool omitted from `allow.tools` returns `permission_denied`. Also verify expiry and revocation: ```bash cua-driver revoke --session test-run cua-driver revoke --all ``` `revoke --all` suspends the complete runtime generation. Restart the daemon before starting a new run. ## Know what the manifest is worth A manifest bounds one runtime's tool surface. It is not a sandbox around your machine, your browser, or your logged-in accounts. Another Cua Driver runtime, or any other process running as the same user, is unaffected by it. A standard-mode runtime without its own capability manifest allows input against every application. Read [Permission modes]() before relying on a manifest as a security boundary rather than as a reviewed statement of what one agent run may do. ## Rules to remember - The manifest only narrows the built-in and configured policy ceilings. - Ownership never bypasses the manifest. - Unknown tools and missing resources fail closed. - Browser origins match exact scheme, host, and port. - Declaring browser origins excludes generic input and window observation from the same manifest. - `terminate: driver_launched` requires a fresh process fingerprint match. - Version 3 has no `ask.tools`; legacy version 1/2 `ask.tools` entries are deny. - Keep expiry and idle timeouts as short as the task permits. See [Permission modes]() for the complete mode matrix and [Permission policies]() for administrator and user policy layers. --- # Personalize the agent cursor Use Cua Driver's semantic default cursor or compile and install a safe custom dotLottie theme. Cua Driver ships one built-in theme, `cua.default`, on macOS, Windows, and Linux. It uses a session-colored pointer with a white outline and matching glow. Every state shares the same gentle floating motion, with semantic animations layered on top for actions such as observe, click, drag, scroll, type, and navigate. **Warning: Visual aid, not authorization** The cursor helps a person follow an agent. It is not a security indicator, consent prompt, or proof that an action succeeded. ![Interactive preview of the runtime cursor, session badge, delivery chip, and target chip](https://github.com/user-attachments/assets/cc315e23-ea8f-4564-b6c5-04a45bdbe2e8) ## Select a theme for a session Create a session and select an already-installed theme by ID: ```bash cua-driver start_session '{"session":"demo"}' cua-driver set_agent_cursor_theme \ '{"session":"demo","theme_id":"cua.default","reduced_motion":"auto"}' cua-driver get_agent_cursor_state '{"session":"demo"}' ``` Use `set_agent_cursor_enabled` to hide or show the session cursor. Use `set_agent_cursor_motion` for movement physics and visibility timing only. ## Session name badge The built-in overlay places the public session name in a compact badge below the pointer. The badge follows the cursor, uses the session color, and renders with the same display-scale-aware native pipeline as the theme. Choose a short, recognizable session name when starting the session: ```bash cua-driver start_session '{"session":"research"}' ``` The badge displays `research`. Cua Driver strips control characters, collapses whitespace, and shortens labels longer than 28 characters. The label is visual identity only. It does not grant authorization, select another session's capabilities, or expose Cua's private runtime identifier. The badge also shows execution context when it applies. A filled delivery chip identifies `background` or `foreground`, followed by an outlined target chip for `ax`, `pixel`, `browser`, or `desktop`. These chips stay visible while the action is active and fade independently from the session name. They are drawn by Cua Driver, so their meaning stays consistent across themes and operating systems. ![All session badge combinations for background and foreground delivery across accessibility, pixel, browser, and desktop targets](https://github.com/user-attachments/assets/071dbfd0-b0d4-442f-9654-994c84db3029) The old `set_agent_cursor_style` operation and its `cursor_id`, shape, color, label, size, opacity, image-path, gradient, and bloom styling fields have been removed. (`cursor_id` may still appear on input-delivery tools where it names a virtual pointer; it is no longer a cursor-theme selector.) ## Create a custom theme A source theme is a dotLottie archive with: - the standard `manifest.json`; - 128×128, 30 fps animations under `a/`, with at most 120 frames per animation and 1,000 frames across the complete compiled theme; - a `cua/theme.json` semantic manifest; - all twelve actions: `idle`, `observe`, `click`, `drag`, `scroll`, `text`, `key`, `navigate`, `app`, `transfer`, `record`, and `system`; - a representative `still_frame` for every animation; - an author and license. The compiler enforces the complete theme contract, including the semantic manifest, animation inventory, frame limits, bounded vector feature set, author, and license. It preserves paths and strokes as vectors, so the native renderer can draw them at the current display scale instead of enlarging a fixed-resolution image. Validate, compile, inspect, and preview the source: ```bash cua-driver cursor-theme validate theme.lottie cua-driver cursor-theme build theme.lottie --output theme.cua-theme cua-driver cursor-theme inspect theme.cua-theme cua-driver cursor-theme preview theme.cua-theme --output preview ``` Install it locally, then select its reverse-DNS ID: ```bash cua-driver cursor-theme install theme.cua-theme cua-driver cursor-theme list cua-driver set_agent_cursor_theme \ '{"session":"demo","theme_id":"com.example.cursor.studio"}' ``` Profile v2 does not compile dotLottie color/theme variants. Publish each visually distinct design under a separate theme ID. Profile v2 contains action animations only. The source manifest uses `"schema": "cua.cursor-theme/2"`, `"profile": "cua-driver-actions-v2"`, and `"semantics": 2`. V1 source and compiled artifacts are rejected. Remove the old `modifiers` section and rebuild the theme with the current compiler. ## Why installation is separate Theme authoring is a trusted local workflow. The short-lived compiler validates the ZIP and Lottie source, rejects unsupported or unbounded content, and emits bounded vector frames containing validated geometry, paints, and transforms. The privileged overlay loads only the compiled `.cua-theme` artifact. It never parses ZIP, JSON, Lottie, fonts, expressions, URLs, or an arbitrary source path. It rasterizes the bounded vector commands through Skia at the display's actual backing scale. MCP and SDK tools may select an installed ID, but they cannot install source or submit inline animation data. The built-in `cua.default` theme uses the same compiler and renderer. Cua ships its compiled artifact inside the driver, so users do not need to install it. The driver applies the stable session color and shared floating motion to the default theme's action layer. Installed custom themes keep their authored colors. The host renders delivery and target chips in the badge for both the default and custom themes. Remove a custom theme with: ```bash cua-driver cursor-theme uninstall com.example.cursor.studio ``` The built-in `cua.default` theme cannot be removed. --- # Update Cua Driver Check whether a newer Cua Driver release is available and install it. Cua Driver gives you three update paths: a passive banner at launch time, a CLI command that only checks, and a separate command that installs the update. Keep the **check and apply steps separate** so you can decide when to change the installed binary. Version numbers, timestamps, and release links in the sample output below are illustrative. Use your command's actual output to identify the installed and available releases; the 0.13.0 migration notes describe that historical upgrade. ## Before updating from 0.12.x to 0.13.0 The installer command, CLI/MCP connection flow, tool names, and Python and TypeScript SDK constructors remain the same. Two behavior changes require attention: 1. `standard` is the promptless practical default for normal observation, input, file, browser, and configuration tools. Attaching to an existing logged-in Chromium profile remains an explicit boundary. Start CLI or MCP with `--grant existing-profile`, use a matching capability manifest, or supply an authorization host from an embedding application. 2. The old cursor-style fields and `set_agent_cursor_style` operation are removed. Sessions use the built-in animated `cua.default` theme and display the public session name in a badge below the cursor unless an installed theme ID is selected. For unattended authenticated-browser work, create a short-lived `bounded` manifest instead of disabling every approval. Use `unrestricted` only when a trusted launcher explicitly supplies `--dangerously-bypass-approvals` and the environment can tolerate full Cua autonomy. See [Permission modes and bounded autonomy](), [Drive a web page](), and [Personalize the agent cursor]() before restarting a long-running daemon or worker. ## Check for updates Checks use the saved channel. Existing installations default to `stable`. Inspect it with: ```bash cua-driver channel status ``` To switch to nightly and install that channel's release: ```bash cua-driver channel set nightly cua-driver update --apply ``` Changing the channel does not install by itself. A stable/nightly transition is reported as available even when the target version would not compare as newer under ordinary SemVer ordering. ```bash cua-driver check-update ``` When you're behind: ``` Current: 0.12.6 Latest: 0.13.0 Update available. Run `cua-driver update --apply` to install. Release notes: https://github.com/trycua/cua/releases/tag/cua-driver-rs-v0.13.0 ``` When you're current: ``` Current: 0.13.0 Latest: 0.13.0 You're on the latest release. ``` Flags: - `--json`: return a machine-readable payload, shown below. - `--no-cache`: skip the 20-hour on-disk cache and force a fresh GitHub check. The command exits with `0` when the check succeeds. Read `update_available` in the JSON payload to learn whether a release is available. Non-zero exit codes mean the check failed, for example because the network was unavailable or parsing failed. ## Apply the update ```bash cua-driver update --apply ``` Output: ``` Current version: 0.12.6 Checking for updates… New version available: 0.13.0 Downloading and installing Cua Driver 0.13.0… Installed Cua Driver 0.13.0. ``` `update --apply` calls the canonical installer script, the same one-line installer used for the first install. If the daemon was running before the update, restart it so it uses the new binary. Stop it first: ```bash cua-driver stop ``` The installer may already have stopped the old daemon. If `stop` reports that it is not running, continue with startup. Then use your platform's startup command: on macOS, run `open -n -g -a CuaDriver --args serve` to preserve app-bundle permission attribution; on Windows with autostart enabled, run `cua-driver autostart kick`; on Linux or Windows without autostart, run `cua-driver serve` and leave that terminal open. Preserve any non-default permission-mode flags from your original launch. In a second terminal if `serve` occupies the first, run `cua-driver --version`, `cua-driver status`, and `cua-driver call list_apps`. Confirm the expected installed version, a running daemon, and access to a known GUI app. See [Verify desktop readiness]() if the daemon cannot reach your desktop. On Windows with autostart enabled, the Scheduled Task uses the new binary on its next invocation. `autostart enable` is idempotent, and the task path resolves through the `current` junction. ## Scripted check (CI / agents) ```bash cua-driver check-update --json ``` ```json { "current_version": "0.12.6", "current_channel": "stable", "selected_channel": "stable", "latest_version": "0.13.0", "update_available": true, "source": "github_releases", "checked_at": "2026-07-01T14:30:00Z", "cache_hit": false, "install_command": "curl -fsSL https://cua.ai/driver/install.sh | bash", "release_notes_url": "https://github.com/trycua/cua/releases/tag/cua-driver-rs-v0.13.0", "error": null } ``` When the installed version is current, `install_command` and `release_notes_url` are `null`. Use the JSON form from a bash script: ```bash if cua-driver check-update --json | jq -e '.update_available' > /dev/null; then echo "Cua Driver is outdated — applying update" cua-driver update --apply fi ``` The same payload is available over MCP through the `check_for_update` tool. MCP does not provide an `apply` variant because installing through MCP would replace the running server process. ## Passive banner Every `cua-driver mcp`, `serve`, and `doctor` invocation starts a background version check. If it finds a newer release, Cua Driver prints a two-line banner to stderr: ``` ✨ cua-driver v0.13.0 is available (you have v0.12.6). Update with: cua-driver update Release notes: https://github.com/trycua/cua/releases/tag/cua-driver-rs-v0.13.0 ``` The check uses a channel-keyed 20-hour cache and never blocks startup. A cached stable result can never satisfy a nightly check, or vice versa. One-shot commands such as `--version`, `call`, and `list-tools` skip it so piped output remains clean. Disable the check for one invocation: ```bash CUA_DRIVER_RS_UPDATE_CHECK=false cua-driver serve ``` Disable it permanently: ```bash cua-driver config set update_check_enabled false ``` --- # Use Computer History Enable, inspect, query, and delete the encrypted Computer History preview in Cua Driver nightly builds. Computer History keeps an encrypted local record of actions performed through Cua Driver. This guide covers the early preview in nightly macOS, Windows, and Linux builds. **Note** Computer History is off by default. It records metadata for Cua-mediated actions after you enable it. It does not watch unrelated desktop activity. ## Before you start - Use an interactive macOS, Windows, or Linux desktop session. - Grant Cua Driver the operating-system permissions needed for the actions you want it to perform. Computer History grants no new action permissions. - On Linux, run an unlocked Secret Service implementation such as GNOME Keyring. History fails closed when no unlocked Secret Service is available. ## Install the nightly build **macOS and Linux** ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)" -- --channel nightly ``` **Windows** ```powershell & ([scriptblock]::Create((irm https://cua.ai/driver/install.ps1))) -Channel nightly ``` To move an existing installation to nightly, run: ```bash cua-driver channel set nightly cua-driver update --apply cua-driver channel status ``` ## Enable history ```bash cua-driver history enable cua-driver history status ``` `history enable` initializes a device-local native credential, verifies an encrypted write and read, and starts capture. The default retention period is 7 days, and the encrypted-store quota is 100 MiB. The store uses the current user's macOS Keychain, Windows Credential Manager, or Linux Secret Service. There is no plaintext fallback, and capture performs no network I/O. ## Inspect recent events List the newest 50 events: ```bash cua-driver history list 50 ``` Show one event by sequence number: ```bash cua-driver history show 42 ``` Add `--json` to `status`, `list`, or `show` when another local program needs a structured response. ## Let an agent consult history An admitted Cua Driver runtime exposes two read-only tools: - `history_status`, authorized by `history.status`; - `history_query`, authorized by `history.query`. Connect the agent through the normal Cua Driver MCP or SDK path. The active permission mode, policy ceiling, and capability manifest authorize every call. A query returns at most 200 metadata events and appends an encrypted access-audit event when it returns data. Tool availability does not make an agent consult history automatically. For continuation and recent-work requests, give the agent a trusted history-first instruction or have the host perform the status and bounded-query preflight. See the [agent integration reference]() for the consultation flow and permission contract. The agent tools cannot enable, pause, resume, disable, delete, or export history. They cannot retrieve encryption keys. ## Pause, resume, or disable capture ```bash cua-driver history pause cua-driver history resume cua-driver history disable ``` Pause and disable preserve existing encrypted history. A permitted agent can still query that history while the runtime admits the preview. ## Delete history ```bash cua-driver history delete --yes ``` This destroys the exact namespace key and deletes the encrypted store. It does not claim physical erasure from backups, filesystem snapshots, copied ciphertext, SSD wear leveling, or memory that a process already decrypted. To delete history while uninstalling: **macOS and Linux** ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/driver/uninstall.sh)" -- --purge ``` Run this as the interactive login user, without `sudo`, so native credential cleanup stays in that user's context. **Windows** ```powershell $env:CUA_DRIVER_RS_UNINSTALL_FORCE = '1' $env:CUA_DRIVER_RS_UNINSTALL_PURGE = '1' irm https://cua.ai/driver/uninstall.ps1 | iex ``` ## Return to stable ```bash cua-driver history disable cua-driver channel set stable cua-driver update --apply ``` Changing channels preserves encrypted history unless you explicitly delete or purge it. ## Data boundaries History may contain timestamps, opaque session and action IDs, Cua capability names, fixed application identity fields, and fixed delivery, route, evidence, effect, lifecycle, access, and health categories. It never stores screenshots, video, audio, typed text, raw keystrokes, clipboard contents, raw tool arguments or results, accessibility trees, file paths, window titles, URLs, or free-form diagnostics. Default encrypted-store locations are: - macOS: `~/Library/Application Support/cua-driver/computer-history`; - Windows: `%LOCALAPPDATA%\cua-driver\computer-history`; - Linux: `$XDG_STATE_HOME/cua-driver/computer-history`, or `~/.local/state/cua-driver/computer-history` when `XDG_STATE_HOME` is unset. The filesystem can reveal that the directory exists, its total size, and file modification times. Copying only the encrypted files to another machine does not provide recovery because the namespace key stays in the current user's native credential store. ## Troubleshooting ### `history_preview_not_admitted` The running development daemon lacks preview admission. Restart it with: ```bash cua-driver serve --experimental-history ``` Installed nightly builds handle the verified relaunch during `history enable`. ### `history_key_locked` or `history_key_unavailable` Unlock the login session and its native credential store, then retry. On Linux, confirm that a Secret Service implementation is running. Capture stays disabled, and Cua Driver creates no plaintext history files. ### `history_quota_reached` Delete history if you no longer need it. Reaching the quota does not block the computer action that encountered the full store. ### `history_storage_corrupt` The reader found an invalid, incomplete, reordered, or unauthenticated record. It refuses the affected store. If you accept permanent data loss, run `cua-driver history delete --yes` and enable history again. ## Related documentation - [Computer History agent integration]() - [Permission modes]() - [Permission policies]() - [Install Cua Driver]() --- # Troubleshoot stale macOS permissions Reset stale app-scoped TCC grants when Cua Driver reports missing permissions that appear enabled in System Settings. This guide shows you how to repair stale macOS Transparency, Consent, and Control (TCC) grants for Cua Driver without resetting permissions for other apps. ## When to use this guide Use this guide when **System Settings → Privacy & Security** shows CuaDriver as enabled, but the `check_permissions` MCP tool or `cua-driver diagnose` reports `accessibility: false` or `screen_recording: false`. The current Cua Driver bundle ID is `com.trycua.driver`. Older releases also shipped as `com.trycua.cuadriver` and `com.trycua.cuadriverrs`. A grant attached to an older bundle ID or signing identity does not authorize the current app. ## Diagnose the installed app Run the read-only diagnostic before changing permissions: ```bash cua-driver diagnose ``` Confirm that `/Applications/CuaDriver.app` exists and that the daemon probes disagree with System Settings. If the app is missing, [install Cua Driver]() before continuing. **Warning** Sanitize `diagnose` output before sharing it. The report may contain usernames, local paths, process IDs, and other details about the local installation. ## Stop Cua Driver and MCP clients Quit every MCP client configured to start Cua Driver, then stop the daemon: ```bash cua-driver stop cua-driver status ``` Continue only when the status reports that the daemon is not running. Leaving a driver or MCP client running can relaunch the app and recreate TCC entries while you reset them. ## Reset only Cua Driver's TCC grants Reset Accessibility, Screen Recording, and Automation for the current bundle ID: ```bash tccutil reset Accessibility com.trycua.driver tccutil reset ScreenCapture com.trycua.driver tccutil reset AppleEvents com.trycua.driver ``` If this Mac previously ran an older Cua Driver release, also clear any entries for its historical bundle IDs: ```bash tccutil reset Accessibility com.trycua.cuadriver tccutil reset ScreenCapture com.trycua.cuadriver tccutil reset AppleEvents com.trycua.cuadriver tccutil reset Accessibility com.trycua.cuadriverrs tccutil reset ScreenCapture com.trycua.cuadriverrs tccutil reset AppleEvents com.trycua.cuadriverrs ``` These commands affect only the named app and service. Do not run an unscoped TCC reset, and do not open or edit `TCC.db`. ### If `tccutil` cannot find the current bundle ID If `tccutil` reports `No such bundle identifier` for `com.trycua.driver`, re-register the installed app with LaunchServices: ```bash /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/Support/lsregister \ -f /Applications/CuaDriver.app ``` Retry the three resets for `com.trycua.driver` after registration completes. A removed historical app may no longer be registered, so the historical bundle IDs can still produce `No such bundle identifier`. You can ignore that result: there is no installed historical bundle for `tccutil` to resolve. Do not create a temporary or synthetic app bundle to make an old identifier resolvable. ## Relaunch and grant permissions again Use the permissions command to launch `/Applications/CuaDriver.app` through LaunchServices under its current bundle identity: ```bash cua-driver permissions grant ``` In **System Settings → Privacy & Security**, enable CuaDriver under both **Accessibility** and **Screen Recording**. If macOS asks you to quit and reopen the app after granting Screen Recording, do so, then run the command above again. If CuaDriver is missing from **Screen & System Audio Recording**, click **+** and add `/Applications/CuaDriver.app` before enabling it. Automation consent is requested later when a driver action needs to control another app. ## Verify the repaired grants Check the running daemon without opening another permission prompt: ```bash cua-driver permissions status --json ``` The result should report `accessibility: true`, `screen_recording: true`, and `source.attribution: "driver-daemon"`. Because status is read-only, it should report `screen_recording_capturable: null` and `direct_capture_status: "not_checked"`. Run `cua-driver permissions grant` to request and verify direct ScreenCaptureKit access explicitly; do not use a read-only diagnostic as a prompt-capable probe. If either permission remains false, run `cua-driver diagnose` again and confirm that the reported executable and bundle are the installed `/Applications/CuaDriver.app`, not an older copy or a terminal-owned process. Remember to sanitize the report before sharing it. ## Related guides - [macOS permissions reference]() - [Install Cua Driver]() - [Keep Cua Driver running]() --- # Drive a Windows app over SSH Use Cua Driver to drive GUI apps on a remote Windows machine from an SSH session. ## The problem Windows OpenSSH server runs in **Session 0**, the non-interactive services session. Sessions 1, 2, ... are interactive logons, one for each console or RDP user. A process inherits its parent's session, so shells created by `sshd` also run in Session 0. The Win32 APIs used by Cua Driver window tools, including `EnumWindows`, `GetForegroundWindow`, `PrintWindow`, UIA, and `BitBlt`, are scoped to the caller's WindowStation and Desktop. Session 0 has no attached interactive desktop, so the tools cannot see the user's windows: ```powershell # Over SSH (Session 0), with no daemon helper: cua-driver call list_windows # [] ← empty, even though the user's RDP session has 12 windows open ``` **Note** Run `cua-driver doctor` to confirm this state. The Windows session probe reports it directly: ``` [warn] interactive session: running in Session 0 (services); window-driving tools (list_windows, click, type_text, get_window_state) will return empty results. These APIs need an attached interactive desktop. ``` ## The solution Run a `cua-driver serve` daemon in your **interactive session**, Session 1 or higher, through an autostart Scheduled Task. The SSH-side CLI or MCP adapter must use that daemon. MCP selects it explicitly with `--socket \\.\pipe\cua-driver`; `cua-driver call` already resolves that default pipe and accepts `--socket` when you want to make the endpoint explicit. The SSH process only moves protocol messages; the daemon performs the actual GUI work from a session with a desktop attached. ``` ┌───────────────────────────────────────────────────────────────┐ │ Session 1+ (RDP / console, has interactive desktop) │ │ │ │ cua-driver-serve (autostart Scheduled Task) │ │ ↑ │ │ │ named pipe: \\.\pipe\cua-driver │ │ │ │ └──────┼────────────────────────────────────────────────────────┘ │ ┌──────┼────────────────────────────────────────────────────────┐ │ Session 0 (services / SSH, no desktop) │ │ │ │ │ cua-driver mcp │ │ cua-driver call list_windows │ │ Claude Code → MCP stdio → cua-driver mcp │ └───────────────────────────────────────────────────────────────┘ ``` ## Set it up **1. From an interactive session, either RDP or the local console, run:** ```powershell cua-driver autostart enable cua-driver autostart kick ``` `enable` registers a Scheduled Task with `LogonType: Interactive`. That setting is required because the alternatives would start the daemon in Session 0. `kick` starts the task immediately instead of waiting for the next logon. **Warning** You need an active interactive session for `kick` to place the daemon in Session 1+. Confirm that state with `query session`; your row should show `Active` or `Disc`: ```powershell query session # SESSIONNAME USERNAME ID STATE TYPE # rdp-tcp#23 you 2 Active ``` If you have never opened RDP on this machine, connect once with RDP. The autostart trigger fires automatically on the next logon. **2. From your SSH session, verify that the daemon is reachable:** ```powershell cua-driver status # Cua Driver daemon is running # socket: \\.\pipe\cua-driver # pid: 12345 # session: 2 ← daemon is in your interactive session ``` **3. Call tools from SSH:** ```powershell cua-driver call list_apps # Equivalent explicit form: cua-driver call list_apps --socket \\.\pipe\cua-driver ``` ## Connect Claude Code over SSH After the daemon is running in the interactive session, register Claude Code the same way you would locally: ```powershell # From inside your SSH session: claude mcp add --transport stdio cua-driver -- cua-driver.exe mcp --socket \\.\pipe\cua-driver claude ``` Claude Code starts `cua-driver mcp` on the SSH side. The explicit socket keeps that process as a protocol proxy to the interactive daemon. Bare `cua-driver mcp` owns a direct runtime on Windows and therefore fails closed in Session 0; it never silently falls back to another session. ## Diagnose empty results Check these items before opening an issue: 1. Confirm that `cua-driver --version` on the SSH side reports the same current install you expect. Upgrade if needed with `irm https://cua.ai/driver/install.ps1 | iex`. 2. Run `cua-driver status` from SSH and confirm it reports a running daemon. If it does not, use `cua-driver autostart status` to see whether the Scheduled Task is registered. 3. Run `query session` and confirm your user has a row in `Active` or `Disc` state. 4. Run `cua-driver doctor` from RDP and confirm it reports `[ok] interactive session: session N has an attached interactive desktop`. 5. Confirm that the MCP command includes `--socket \\.\pipe\cua-driver`, or the exact endpoint reported by `cua-driver status`. If the explicitly selected interactive-session daemon is unavailable, MCP startup fails instead of attempting GUI work from the SSH session. --- # Set up Fleet credentials Create a Fleet user API key, configure the Sandbox SDK, and check access before provisioning. The Fleet SDK accepts an OAuth user API key or a Fleet bearer access token. `cua auth login` stores an interactive CLI session in the credential vault; it does not export either credential for the SDK. `cua sb launch --pool` also uses the SDK's separate credential configuration. Without it, even a logged-in CLI can fail with: ```console $ cua sb launch --pool my-pool --name my-sandbox Error: Fleet cloud sandboxes require CUA_CLIENT_ID and CUA_CLIENT_SECRET, or cua.configure(client_id=..., client_secret=...). ``` ## Create a Fleet user API key You need an account that can sign in to [Cua Fleet](https://run.cua.ai) and access **API keys**. Create the key while signed in as the account that will own the pools. User keys act on behalf of their owner; creating a key does not grant access to another account's pools or bypass pool admission rules. 1. Sign in to [Cua Fleet](https://run.cua.ai), then select **API keys** in the navigation, or open the [API keys page](https://run.cua.ai/user-keys). 2. Under **Create API key**, enter a descriptive **Name**, such as `first-fleet-tutorial`. For the first-Fleet tutorial, leave **Allowed Namespaces (optional)** at its default, **All namespaces (no restriction)**, because the tutorial creates a new pool namespace. 3. Select **Create key**. In **API key created**, copy **Client ID** and **Client Secret** into your secret manager before selecting **I have copied the credentials**. The secret is shown only once. If you cannot sign in, the page reports **API keys are unavailable**, or key creation is denied, contact [Cua support on Discord](https://discord.gg/mVnXXpdE85) to check account access before continuing. `cua auth login` is not a substitute for this key-creation step. Pool creation can also require a payment method. If Fleet shows **Payment method required**, open **Settings** and use **Add payment method** under **Payment method**. Review the applicable pricing and terms before adding a payment method or creating billable resources. The read-only access check below does not create a pool or claim. ## Configure the Sandbox SDK Set `CUA_CLIENT_ID` to **Client ID** and `CUA_CLIENT_SECRET` to **Client Secret**. The token endpoint below is the default for `run.cua.ai`; the dashboard's credential dialog does not display it: ```bash export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" export CUA_TOKEN_URL="https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token" unset FLEETS_TOKEN ``` `FLEETS_TOKEN` takes precedence over these client credentials, so unset it when switching to a user key. Inject secrets through your secret manager where possible; do not commit them or paste them into shared logs. ## Check access before provisioning With [uv](https://docs.astral.sh/uv/) installed, run this check in the same shell. It exchanges the user key for a short-lived access token and sends `GET /api/namespaces`. It prints only the number of namespaces returned: ```bash uv run --with 'httpx>=0.27,<1' python - <<'PY' import os import httpx with httpx.Client(timeout=30) as client: token_response = client.post( os.environ["CUA_TOKEN_URL"], auth=(os.environ["CUA_CLIENT_ID"], os.environ["CUA_CLIENT_SECRET"]), data={"grant_type": "client_credentials"}, ) token_response.raise_for_status() access_token = token_response.json()["access_token"] response = client.get( "https://run.cua.ai/api/namespaces", headers={"Authorization": f"Bearer {access_token}"}, ) response.raise_for_status() print(f"Fleet access verified: {len(response.json())} namespace(s).") PY ``` A successful response with zero namespaces is valid for an account with no pools. This verifies authentication and namespace listing; pool creation still depends on account permissions, admission rules, and resource availability. If the token request fails, check the client ID, secret, and token endpoint. If the Fleet request returns `401` or `403`, resolve the account access problem before provisioning; contact support if the credentials are correct. ## Credential lifetime and revocation A user API key is the client ID and secret used to request access tokens. Access-token lifetime comes from the issuer's `expires_in` response; do not assume a fixed duration. The key-creation form does not offer an expiration setting. When you finish using a key, return to **API keys**, select **Revoke** for that key, and confirm **Revoke** in **Revoke API key?**. Revocation removes the OAuth client so it cannot obtain new tokens. Already-issued access tokens can remain usable until they expire. `cua auth logout` revokes the CLI session's refresh token and clears its local vault entry. It does not revoke a Fleet user API key or unset environment variables. Remove your local secret references after revoking the key. ## Existing access tokens and GitHub Actions If you already have a valid Fleet bearer token, the Sandbox SDK also accepts: ```bash export FLEETS_TOKEN="" ``` A pasted token is not an OAuth client secret and cannot renew itself. Replace it when it expires and restart clients that hold it. For GitHub Actions, use the [workload identity flow]() instead of copying an interactive CLI session token. Terraform uses different variable names; see [Configure a sandbox pool with Terraform](). --- # Choose a sandbox image Choose between a published Fleet image, local image customization, and preparing a Fleet boot artifact. Choose an image workflow based on where the sandbox will run and whether a published artifact already contains the software you need. | Your task | Start here | What you get | | ---------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | | Run a Fleet workload using an existing image | [Choose a published Fleet image]() | A registry artifact reference for a pool. | | Add packages, files, or setup commands for local execution | [Choose a local base image]() | An image specification whose builder steps run locally. | | Run a Fleet workload that needs a custom guest | [Prepare and reference a Fleet image]() | A bootable guest disk packaged and published as a containerDisk artifact. | Before choosing an artifact, check the [OS and image catalog]() for architecture, guest services, and verification limits. A registry reference does not prove that the guest is accessible, bootable, or ready for your client. Declaring a service port does not install the service. In `cua-sandbox` 0.4.3, Fleet rejects SDK builder customization and snapshot-derived image inputs. Local builder methods do not publish an artifact or run during a Fleet claim. See [Sandbox runtime support]() for the versioned contract and [How Fleet images work]() for the published-artifact model. --- # Manage local sandbox lifecycle Create, reconnect to, list, and clean up local sandboxes with the Sandbox SDK. These patterns run on your own machine with `local=True`. Choose a [compatible host and runtime]() before running them. They are operation patterns, not a complete installation tutorial. For the ownership model, read [Sandbox lifecycle](). For Fleet, use [Create a sandbox pool with Python](): releasing a claim and deleting pool capacity are separate operations. ## Create and auto-destroy a sandbox Use `Sandbox.ephemeral` for scripts, CI, and one-off tasks. ```python import asyncio from cua import Sandbox, Image async def main(): async with Sandbox.ephemeral(Image.linux(), local=True) as sb: result = await sb.shell.run('echo hello') print(result.stdout) # 'hello\n' # sandbox deleted here asyncio.run(main()) ``` `Sandbox.ephemeral` is **auto-destroyed** when the `async with` block exits. Its context manager tears the sandbox down automatically. ## Create a persistent sandbox Use `Sandbox.create` when the sandbox **must outlive the script**. Call `await sb.disconnect()` to drop the connection while the sandbox keeps running. ```python sb = await Sandbox.create(Image.linux(), name='my-dev-sandbox', local=True) await sb.shell.run('apt-get install -y vim') await sb.disconnect() # sandbox keeps running ``` Delete the sandbox when you are done with it. ```python async with Sandbox.connect('my-dev-sandbox', local=True) as sb: await sb.destroy() ``` | Method | Effect | When | | ---------------------------- | ------------------------------------------- | --------------- | | `await sb.disconnect()` | keeps running | reconnect later | | `await sb.destroy()` | permanently destroyed | done with it | | `await Sandbox.delete(name)` | permanently destroyed (by name, no connect) | classmethod | ## Reconnect to a running sandbox Use `Sandbox.connect` to attach to an already-running sandbox. **Note** Sandbox.connect never starts or stops the sandbox. It only manages the network connection. Reconnect by name. ```python async with Sandbox.connect('my-dev-sandbox', local=True) as sb: result = await sb.shell.run('vim --version') print(result.stdout) ``` `Sandbox.connect` supports both `await` and `async with`. Its context manager calls `disconnect()` automatically, so the sandbox keeps running after the connection closes. ```python async with Sandbox.connect('my-sandbox', local=True) as sb: await sb.shell.run('echo reconnected') # connection dropped, sandbox keeps running ``` ## List running sandboxes List available sandboxes, then filter to local runtimes when needed. ```python sandboxes = await Sandbox.list(local=True) for info in sandboxes: print(info.name, info.os_type, info.status) ``` ## Choose a local runtime Pass `local=True` to use a runtime on your own machine. Cua selects a local runtime from the image type: Docker for Linux containers, QEMU for Linux VMs, Lume for macOS VMs, and QEMU or Hyper-V for Windows VMs. Selection still requires a compatible host and usable image; it is not a boot check. See the [runtime support reference]() before choosing a backend. ```python # local Docker desktop container async with Sandbox.ephemeral( Image.linux(kind='container'), local=True, ) as sb: ... ``` --- # Configure a sandbox pool with Terraform Configure reusable Linux and Windows sandbox pools on run.cua.ai with Terraform. Use this guide to configure a reusable sandbox pool on run.cua.ai with the public Cua Fleets provider. Terraform represents each pool as a `fleets_pool` resource. ## Prerequisites - A run.cua.ai user key or access token with permission to manage sandbox pools. Keep credentials in environment variables or a secret manager, not in source control. - Terraform or OpenTofu. Terraform installs `trycua/fleets` from the public Terraform Registry during `terraform init`. The examples pin provider version `0.2.0` so initialization and plans use the same release across environments. For OpenTofu, run the same commands with `tofu` in place of `terraform`. ## Authentication The provider accepts either `CYCLOPS_ACCESS_TOKEN` or all three OAuth user-key variables: `CYCLOPS_CLIENT_ID`, `CYCLOPS_CLIENT_SECRET`, and `CYCLOPS_TOKEN_URL`. First, [create a Fleet user API key and check access](). That setup covers account access, payment-method requirements, and revocation. Use the same key with the provider's variable names: ```bash export CYCLOPS_ENDPOINT="https://run.cua.ai" export CYCLOPS_CLIENT_ID="" export CYCLOPS_CLIENT_SECRET="" export CYCLOPS_TOKEN_URL="https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token" unset CYCLOPS_ACCESS_TOKEN ``` `CYCLOPS_CLIENT_ID` and `CYCLOPS_CLIENT_SECRET` are **Client ID** and **Client Secret** from Fleet's **API key created** dialog. The token endpoint above is for `run.cua.ai`. Unlike the Sandbox SDK, the provider requires an explicit token URL when using client credentials. For a short-lived workflow with an existing Fleet access token, use: ```bash export CYCLOPS_ENDPOINT="https://run.cua.ai" export CYCLOPS_ACCESS_TOKEN="" ``` `CYCLOPS_ACCESS_TOKEN` takes precedence over the client credentials. A pasted token cannot renew itself; replace it when it expires before running Terraform again. Provider arguments take precedence over their corresponding environment variables. `cua auth login`, `CUA_CLIENT_ID`, `CUA_CLIENT_SECRET`, and `FLEETS_TOKEN` do not configure the Terraform provider. ## Choose a sizing mode Every `fleets_pool` must configure exactly one sizing mode: - Set `replicas` for a static warm-pool target and omit `autoscaling`. - Set an `autoscaling` block for claim-driven scaling and omit `replicas`. In autoscaling mode, do not configure `replicas`; after apply and refresh it reports the current pool target. If you later switch to static mode, review the plan because the pool will use the `replicas` value you configure. ## Configure a Linux pool Create a new directory, save this as `main.tf`, and choose a unique lowercase DNS-label pool name. This example creates a KubeVirt Linux pool and exposes SSH. ```hcl title="main.tf" terraform { required_providers { fleets = { source = "trycua/fleets" version = "0.2.0" } } } provider "fleets" { endpoint = "https://run.cua.ai" } resource "fleets_pool" "linux" { name = "linux-pool" cpu_cores = 4 memory = "8Gi" container_disk_image = "public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04:main-809e3f81" runtime = "kubevirt" firmware = "bios" service { name = "ssh" target_port = 22 protocol = "TCP" } autoscaling { min_pool_size = 0 initial_pool_size = 1 max_pool_size = 5 } } output "linux_pool" { value = { name = fleets_pool.linux.name namespace = fleets_pool.linux.namespace target_replicas = fleets_pool.linux.replicas current_replicas = fleets_pool.linux.current_replicas ready_replicas = fleets_pool.linux.ready_replicas } } ``` ## Configure a Windows pool Use a distinct name for Windows. The Windows computer-server image requires UEFI, so this example sets `firmware = "efi"` and waits for the service on port `8000`. ```hcl title="main.tf" terraform { required_providers { fleets = { source = "trycua/fleets" version = "0.2.0" } } } provider "fleets" { endpoint = "https://run.cua.ai" } resource "fleets_pool" "windows" { name = "windows-pool" cpu_cores = 4 memory = "4Gi" container_disk_image = "public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3" runtime = "kubevirt" firmware = "efi" readiness_probe_json = jsonencode({ tcpSocket = { port = 8000 } initialDelaySeconds = 60 periodSeconds = 5 timeoutSeconds = 3 failureThreshold = 120 }) service { name = "computer-server" target_port = 8000 protocol = "TCP" } autoscaling { min_pool_size = 0 initial_pool_size = 1 max_pool_size = 5 } } output "windows_pool" { value = { name = fleets_pool.windows.name namespace = fleets_pool.windows.namespace target_replicas = fleets_pool.windows.replicas current_replicas = fleets_pool.windows.current_replicas ready_replicas = fleets_pool.windows.ready_replicas } } ``` ## Apply, verify, and destroy Initialize from the public Registry and run one example: ```bash terraform init terraform fmt -check terraform validate terraform plan terraform apply terraform output ``` Commit `.terraform.lock.hcl` with your configuration so future runs select the same provider build. Run `terraform plan` against the existing state before applying changes; refresh keeps out-of-band changes to the pool target and status visible in the plan and outputs. The provider returns the backing `namespace` and template name, the live `replicas` target, and the `current_replicas` and `ready_replicas` status counts. A ready pool has at least one ready sandbox after its warm capacity provisions. Destroy the pool when finished. This deletes both the pool and its same-named namespace: ```bash terraform destroy ``` ## Troubleshooting and security - Use a lowercase DNS label of at most 63 characters for `name`. Changing it replaces the pool because it also owns its namespace. - Check the image reference and image-pull-secret policy if creation returns `403`. `image_pull_secret` defaults to `ecr-credentials`. - Use `bios` for the Linux workspace image and `efi` for the Windows computer-server image. A firmware mismatch prevents the guest from booting. - Protect Terraform state and shell history as operational secrets. --- # Create a sandbox pool with Python Create a reusable warm sandbox pool, claim a sandbox, and run commands with the Cua Sandbox SDK. Use the Cua Sandbox SDK when you want to create and claim a reusable sandbox pool from Python. This guide applies a one-replica pool, connects to a named claim, takes a screenshot, and runs a shell command. ## Prerequisites - Python `>=3.11,<3.14`. - [`uv`](https://docs.astral.sh/uv/) to run the self-contained script. - A run.cua.ai access token or OAuth user key with permission to manage sandbox pools. The SDK connects to `https://run.cua.ai` by default. Authenticate with a Fleet access token: ```bash export FLEETS_TOKEN="" ``` Or authenticate with OAuth client credentials: ```bash export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" ``` Set `CUA_FLEET_BASE_URL` only when you need to use a different Fleet API endpoint. ## Create and claim the pool Set a globally unique pool name that you own before running the example: ```bash export CUA_POOL_NAME="my-team-sandbox-pool" export CUA_CLAIM_NAME="my-task" ``` The example uses `cua-sandbox==0.4.3`. Use `Pool.apply()` only when this script owns the pool configuration. For a pool managed by another process or Terraform, use [an existing pool](#use-an-existing-pool) instead. Save the following script as `create_pool.py`: ```python title="create_pool.py" # /// script # requires-python = ">=3.11,<3.14" # dependencies = [ # "cua-sandbox==0.4.3", # ] # /// import asyncio import os from pathlib import Path from cua_sandbox import Image, Pool IMAGE = ( "public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04" "@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57" ) POOL_NAME = os.environ["CUA_POOL_NAME"] CLAIM_NAME = os.environ.get("CUA_CLAIM_NAME", POOL_NAME) async def main() -> None: # Create or reconcile the warm pool and its template. pool = await Pool.apply( Image.from_registry(IMAGE), name=POOL_NAME, replicas=1, cpu=4, memory_mb=4096, services={"server": 8000}, ) # Create or reconnect to the named claim. # Exiting this block releases the claim but leaves the pool warm. async with pool.claim( name=CLAIM_NAME, service="server", time_to_start=900, ) as sandbox: print(f"Pool: {sandbox.pool_name}") print(f"Claim: {sandbox.claim_name}") print(f"Sandbox: {sandbox.name}") screenshot = Path("sandbox.png") screenshot.write_bytes(await sandbox.screenshot()) print(f"Screenshot: {screenshot.resolve()}") result = await sandbox.shell.run("uname -a") if not result.success: raise RuntimeError(result.stderr) print(result.stdout.strip()) asyncio.run(main()) ``` Run it with `uv`. The script metadata installs `cua-sandbox` in an isolated environment automatically: ```bash uv run create_pool.py ``` `Pool.apply()` creates the pool and its sandbox template when they do not exist. If the named pool already exists, it reconciles the pool to the image, replica, CPU, memory, and service configuration in the script. `pool.claim()` creates or reconnects to the named claim, waits for binding, then waits up to 900 seconds for the `server` service on port `8000`. The service timeout is separate from claim binding. Exiting the `async with` block releases the claim, but the pool and its one warm replica remain available for the next claim. The script writes `sandbox.png` in the current directory and raises an error if `uname -a` does not complete successfully. ## Use an existing pool Replace the `Pool.apply(...)` call with a lookup to keep the pool's image, capacity, and services unchanged: ```python pool = await Pool.get(POOL_NAME) ``` Keep the claim block from the example. Give concurrent tasks different claim names, or omit `name=` to generate one. A named claim that still exists is reused; once released, reusing its name creates a new claim. Reusing a pool or claim name does not guarantee the same VM or preserve files from a released claim. Copy results out before releasing it. The claim context releases even an existing claim that it successfully attaches to. Use it only when this task owns that claim's release. If acquisition fails, the SDK attempts to release a claim it just created, but leaves a pre-existing claim alone. ## Keep a claim between processes To keep a claim after the connection closes, await the claim directly and save its reference before disconnecting: ```python import json sandbox = await pool.claim(name=CLAIM_NAME, service="server", time_to_start=900) try: Path("claim.json").write_text(json.dumps(sandbox.to_dict())) result = await sandbox.shell.run("uname -a") print(result.stdout) finally: await sandbox.disconnect() ``` `disconnect()` closes the client transport and leaves the claim held. A later process can reconnect using the saved reference without creating a pool or claim: ```python import json from pathlib import Path from cua_sandbox import Sandbox reference = json.loads(Path("claim.json").read_text()) async with Sandbox.from_dict(reference) as sandbox: result = await sandbox.shell.run("uname -a") print(result.stdout) ``` This reconnect context only disconnects on exit. When the task is finished, call `await sandbox.close()` to release its claim. A reference does not extend an expiry deadline. Protect it as resource metadata and keep authenticating with credentials authorized for that pool. For Fleet claims, use these explicit pool and claim APIs. The legacy `Sandbox.list()`, `Sandbox.suspend()`, and `Sandbox.resume()` methods are not per-claim Fleet lifecycle controls. ## Scale the pool with claim demand Instead of a static `replicas` count, pass `autoscaling=` to let the pool grow and shrink with claim demand. The pool scales toward `max_pool_size` while claims are pending and back down to `min_pool_size` as claims are released; `initial_pool_size` seeds a one-time warm head start when the pool is created: ```python from cua_sandbox import Image, Pool, WarmPoolAutoscaling pool = await Pool.apply( Image.from_registry(IMAGE), name=POOL_NAME, cpu=4, memory_mb=4096, autoscaling=WarmPoolAutoscaling( min_pool_size=0, initial_pool_size=2, max_pool_size=10, ), ) ``` Each field is optional (pass `None` to accept the server default). With `min_pool_size=0` the pool scales to zero when no claims are held, so the first claim after an idle period cold-starts a sandbox. ## Expire pools and claims automatically Pools and claims accept an optional creation-age TTL that deletes them a fixed time after creation. See [Expire pools and claims automatically](). ## Choose pool and claim names `CUA_POOL_NAME` is required. If `CUA_CLAIM_NAME` is omitted, the script uses the pool name for its claim. Override either name without editing the script: ```bash export CUA_POOL_NAME="my-sandbox-pool" export CUA_CLAIM_NAME="my-sandbox-claim" uv run create_pool.py ``` Use a stable pool name for shared capacity and a distinct claim name for each independent task. Names must be lowercase DNS labels. ## Pool names are globally unique A pool's name is also its namespace, and that namespace is shared across all Cua accounts. `Pool.apply()` therefore requires an explicit `name=`, and `Sandbox.create(image, ...)` refuses Fleet registry images without a pool — apply a named pool first and pass it as `pool=`: ```python pool = await Pool.apply(image, name="my-team-windows-pool") sandbox = await Sandbox.create(pool=pool, name="my-claim") ``` If the name belongs to another account, the SDK can raise `PoolAccessDeniedError` with a name-collision hint: ```text PoolAccessDeniedError: Fleet denied create pool on pool namespace 'my-team-windows-pool' (HTTP 403: k8s request is not allowed). Pool names are globally unique across accounts, so this name may already be taken — try a new pool name. If that does not work, contact support on Discord: https://discord.gg/mVnXXpdE85 ``` For a confirmed name collision, choose another pool name; changing claim names does not fix a pool namespace conflict. Claim names are scoped to their pool's namespace. The SDK generates random `claim-` claim names when omitted. The exception's name-collision hint is not a diagnosis for every HTTP 403. Inspect the operation and response body: a payment requirement or template policy denial needs a different action. See [Troubleshoot Fleet pools and claims](). `Sandbox.ephemeral(image)` remains a one-liner: it creates a disposable pool under a random `cua-eph-` name and deletes it on exit. Passing `keep_pool=True` requires `name=` so later runs can find the kept pool. ## Delete the pool The example intentionally leaves the pool warm, consuming capacity after the script exits. When you own the pool and all of its users have finished, call `await pool.delete()` after releasing every claim. Pool deletion also deletes the pool namespace and its remaining resources. Do not put it in a worker's per-task cleanup for a shared pool. `Pool.apply()` tracks its reconciled template for cleanup, including when that template already existed. In `cua-sandbox==0.4.3`, a template reconciliation failure also triggers an attempt to delete the reconciled pool. It is not a transaction that preserves an existing pool on failure. Use `Pool.get()` when you only need to consume existing capacity. If a process exits before cleanup completes, follow [Recover interrupted cleanup](). Do not rerun provisioning merely to obtain a handle to delete. --- # Create a sandbox pool with TypeScript Create a reusable Fleet pool and capture a sandbox screenshot from browser or Node.js TypeScript. Use `@trycua/fleet` to reconcile a reusable sandbox pool from TypeScript. This guide applies a one-replica Linux pool configuration, claims a sandbox, captures a screenshot, and releases the claim while leaving the pool warm. The package provides separate runtime entry points: - `@trycua/fleet/browser` loads the Fleet WebAssembly module in a browser. - `@trycua/fleet/node` loads the same WebAssembly module from disk and provides a fetch-based HTTP transport. ## Prerequisites - Node.js 20 or newer. - OAuth client credentials with permission to manage sandbox pools. - A globally unique, lowercase DNS-label pool name. This guide uses the public Linux computer-server image: ```text public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57 ``` **Warning** Browser code can read every `VITE_*` value bundled into the application. Use the browser example only in a trusted local or internal application with temporary, narrowly scoped client credentials. Keep long-lived credentials in Node.js or on your backend. ## Choose a runtime **Node.js** Install the Fleet lifecycle SDK and a TypeScript runner: ```bash npm install @trycua/fleet@0.1.1 npm install --save-dev tsx typescript npm pkg set type=module ``` Set your OAuth client credentials and a unique pool name: ```bash export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" export CUA_POOL_NAME="my-team-js-pool" ``` Set `CUA_FLEET_BASE_URL` or `CUA_TOKEN_URL` only when you need to use different Fleet or OAuth endpoints. Save this script as `create-pool.ts`: ```ts title="create-pool.ts" import { writeFile } from 'node:fs/promises'; import { CreateClaimRequestBuilder, CreatePoolRequestBuilder, CreateTemplateRequestBuilder, CyclopsClient, CyclopsCredentials, FetchHttpClient, OsGymSandboxTemplateSpecBuilder, OsGymSandboxWarmPoolSpecBuilder, SandboxServiceBuilder, SandboxTemplateRefBuilder, VmTemplateBuilder, type Claim, type CyclopsConfiguration, uniffiInitAsync, } from '@trycua/fleet/node'; const IMAGE = 'public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04' + '@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57'; const BASE_URL = process.env.CUA_FLEET_BASE_URL ?? 'https://run.cua.ai'; const TOKEN_URL = process.env.CUA_TOKEN_URL ?? 'https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token'; const CLIENT_ID = requiredEnv('CUA_CLIENT_ID'); const CLIENT_SECRET = requiredEnv('CUA_CLIENT_SECRET'); const POOL_NAME = requiredEnv('CUA_POOL_NAME'); const TEMPLATE_NAME = `${POOL_NAME}-template`; function requiredEnv(name: string): string { const value = process.env[name]; if (!value) throw new Error(`Missing environment variable: ${name}`); return value; } function screenshotBase64(responseBody: ArrayBuffer): string { const text = new TextDecoder().decode(responseBody); const dataLine = text.split('\n').find((line) => line.startsWith('data: ')); if (!dataLine) throw new Error(`No screenshot data frame: ${text.slice(0, 200)}`); const payload = JSON.parse(dataLine.slice(6)); if (!payload.success) throw new Error(payload.error ?? 'Screenshot failed'); const encoded = payload.image_data ?? payload.result?.image_data; if (typeof encoded !== 'string') throw new Error('Screenshot response has no image data'); return encoded; } await uniffiInitAsync(); const configuration: CyclopsConfiguration = { baseUrl: BASE_URL, tokenUrl: TOKEN_URL, credentials: new CyclopsCredentials(CLIENT_ID, CLIENT_SECRET), poolPollIntervalMs: 5_000n, poolPollLimit: 120, claimPollIntervalMs: 5_000n, claimPollLimit: 120, }; const client = CyclopsClient.connect(configuration, new FetchHttpClient()) as CyclopsClient; let claim: Claim | undefined; try { const serverService = new SandboxServiceBuilder().name('server').targetPort(8000).build(); const mcpService = new SandboxServiceBuilder().name('mcp').targetPort(3000).build(); const vm = new VmTemplateBuilder() .containerDiskImage(IMAGE) .cpuCores(4) .memory('4Gi') .services([serverService, mcpService]) .build(); const templateSpec = new OsGymSandboxTemplateSpecBuilder().vmTemplate(vm).build(); const templateRef = new SandboxTemplateRefBuilder().name(TEMPLATE_NAME).build(); const poolSpec = new OsGymSandboxWarmPoolSpecBuilder() .replicas(1) .sandboxTemplateRef(templateRef) .build(); const pool = await client.reconcilePool( new CreatePoolRequestBuilder().namespace(POOL_NAME).spec(poolSpec).build() ); const template = await client.reconcileTemplate( new CreateTemplateRequestBuilder() .namespace(POOL_NAME) .name(TEMPLATE_NAME) .spec(templateSpec) .build() ); console.log(`Reconciled pool ${pool.metadata.name}`); console.log(`Reconciled template ${template.metadata.name}`); claim = await client.createClaim(new CreateClaimRequestBuilder().pool(pool).build()); console.log(`Claim: ${claim.metadata.namespace}/${claim.metadata.name}`); const sandbox = await client.waitClaim(claim); const requestBody = await new Blob([JSON.stringify({ command: 'screenshot' })]).arrayBuffer(); const response = await client.serviceRequest(sandbox, 'server', '/cmd', { method: 'POST', url: 'https://service.invalid/cmd', headers: [{ name: 'content-type', value: 'application/json' }], body: requestBody, timeoutSecs: 60n, }); if (response.status < 200 || response.status >= 300) { throw new Error(`Screenshot request failed with HTTP ${response.status}`); } await writeFile('sandbox.png', Buffer.from(screenshotBase64(response.body), 'base64')); console.log('Wrote sandbox.png'); } finally { try { // Release the claim but leave the reusable pool warm. if (claim) await client.deleteClaim(claim); } finally { client.uniffiDestroy(); } } ``` Run it: ```bash npx tsx create-pool.ts ``` The script writes `sandbox.png` in the current directory. The Fleet client creates and waits for the claim, routes `POST /cmd` through the authenticated `server` service proxy, and deletes the claim before closing. The template also declares the Cua Driver MCP endpoint as the `mcp` service on port `3000`, so the same pool works with the signed-service URL guide. **Browser** Create a Vite TypeScript project and install the browser SDK: ```bash npm create vite@latest fleet-browser -- --template vanilla-ts cd fleet-browser npm install npm install @trycua/fleet@0.1.1 ``` Create `.env.local` with temporary, narrowly scoped OAuth client credentials and a unique pool name: ```bash title=".env.local" VITE_CUA_CLIENT_ID= VITE_CUA_CLIENT_SECRET= VITE_CUA_POOL_NAME=my-team-browser-pool ``` Set `VITE_CUA_FLEET_BASE_URL` or `VITE_CUA_TOKEN_URL` only when you need to use different Fleet or OAuth endpoints. Replace `src/main.ts` with the following code: ```ts title="src/main.ts" import { CreateClaimRequestBuilder, CreatePoolRequestBuilder, CreateTemplateRequestBuilder, CyclopsClient, CyclopsCredentials, OsGymSandboxTemplateSpecBuilder, OsGymSandboxWarmPoolSpecBuilder, SandboxServiceBuilder, SandboxTemplateRefBuilder, VmTemplateBuilder, type Claim, type CyclopsConfiguration, type HttpClient, type HttpRequest, type HttpResponse, uniffiInitAsync, } from '@trycua/fleet/browser'; const IMAGE = 'public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04' + '@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57'; const BASE_URL = import.meta.env.VITE_CUA_FLEET_BASE_URL ?? 'https://run.cua.ai'; const TOKEN_URL = import.meta.env.VITE_CUA_TOKEN_URL ?? 'https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token'; const CLIENT_ID = requiredEnv('VITE_CUA_CLIENT_ID'); const CLIENT_SECRET = requiredEnv('VITE_CUA_CLIENT_SECRET'); const POOL_NAME = requiredEnv('VITE_CUA_POOL_NAME'); const TEMPLATE_NAME = `${POOL_NAME}-template`; function requiredEnv(name: string): string { const value = import.meta.env[name]; if (!value) throw new Error(`Missing environment variable: ${name}`); return value; } function screenshotBase64(responseBody: ArrayBuffer): string { const text = new TextDecoder().decode(responseBody); const dataLine = text.split('\n').find((line) => line.startsWith('data: ')); if (!dataLine) throw new Error(`No screenshot data frame: ${text.slice(0, 200)}`); const payload = JSON.parse(dataLine.slice(6)); if (!payload.success) throw new Error(payload.error ?? 'Screenshot failed'); const encoded = payload.image_data ?? payload.result?.image_data; if (typeof encoded !== 'string') throw new Error('Screenshot response has no image data'); return encoded; } function pngBlob(encoded: string): Blob { const bytes = Uint8Array.from(atob(encoded), (character) => character.charCodeAt(0)); return new Blob([bytes], { type: 'image/png' }); } class FetchHttpClient implements HttpClient { async execute(request: HttpRequest, asyncOpts_?: { signal: AbortSignal }): Promise { const timeoutSignal = request.timeoutSecs ? AbortSignal.timeout(Number(request.timeoutSecs) * 1_000) : undefined; const response = await fetch(request.url, { method: request.method, headers: request.headers.map(({ name, value }) => [name, value] as [string, string]), body: request.body, signal: asyncOpts_?.signal ?? timeoutSignal, redirect: 'manual', }); return { status: response.status, headers: [...response.headers].map(([name, value]) => ({ name, value })), body: await response.arrayBuffer(), }; } } await uniffiInitAsync(); const configuration: CyclopsConfiguration = { baseUrl: BASE_URL, tokenUrl: TOKEN_URL, credentials: new CyclopsCredentials(CLIENT_ID, CLIENT_SECRET), poolPollIntervalMs: 5_000n, poolPollLimit: 120, claimPollIntervalMs: 5_000n, claimPollLimit: 120, }; const client = CyclopsClient.connect(configuration, new FetchHttpClient()) as CyclopsClient; let claim: Claim | undefined; try { const serverService = new SandboxServiceBuilder().name('server').targetPort(8000).build(); const mcpService = new SandboxServiceBuilder().name('mcp').targetPort(3000).build(); const vm = new VmTemplateBuilder() .containerDiskImage(IMAGE) .cpuCores(4) .memory('4Gi') .services([serverService, mcpService]) .build(); const templateSpec = new OsGymSandboxTemplateSpecBuilder().vmTemplate(vm).build(); const templateRef = new SandboxTemplateRefBuilder().name(TEMPLATE_NAME).build(); const poolSpec = new OsGymSandboxWarmPoolSpecBuilder() .replicas(1) .sandboxTemplateRef(templateRef) .build(); const pool = await client.reconcilePool( new CreatePoolRequestBuilder().namespace(POOL_NAME).spec(poolSpec).build() ); const template = await client.reconcileTemplate( new CreateTemplateRequestBuilder() .namespace(POOL_NAME) .name(TEMPLATE_NAME) .spec(templateSpec) .build() ); console.log(`Reconciled pool ${pool.metadata.name} with template ${template.metadata.name}`); claim = await client.createClaim(new CreateClaimRequestBuilder().pool(pool).build()); console.log(`Claim: ${claim.metadata.namespace}/${claim.metadata.name}`); const sandbox = await client.waitClaim(claim); const requestBody = await new Blob([JSON.stringify({ command: 'screenshot' })]).arrayBuffer(); const response = await client.serviceRequest(sandbox, 'server', '/cmd', { method: 'POST', url: 'https://service.invalid/cmd', headers: [{ name: 'content-type', value: 'application/json' }], body: requestBody, timeoutSecs: 60n, }); if (response.status < 200 || response.status >= 300) { throw new Error(`Screenshot request failed with HTTP ${response.status}`); } const image = document.querySelector('#screenshot'); if (!image) throw new Error('Missing #screenshot image element'); image.src = URL.createObjectURL(pngBlob(screenshotBase64(response.body))); } finally { try { // Release the claim but leave the reusable pool warm. if (claim) await client.deleteClaim(claim); } finally { client.uniffiDestroy(); } } ``` Replace `index.html` with a minimal screenshot page: ```html title="index.html" Fleet screenshot

Fleet sandbox screenshot

Fleet sandbox desktop ``` Start the development server and open the printed URL: ```bash npm run dev ``` The browser SDK authenticates the Fleet API request, routes `POST /cmd` through the named `server` service, decodes the computer-server event-stream response, and displays the PNG. The pool also exposes the Cua Driver MCP endpoint as the `mcp` service on port `3000`, matching the service used by the signed-service URL guide. ## Keep or delete the pool Both examples release only the claim, so the one-replica pool remains warm for the next task and continues consuming capacity. Each run creates a new claim; the logged namespace and claim name identify it if cleanup is interrupted. Copy results out before claim release. Reusing the pool does not guarantee the same VM or preserve files from the previous claim. The `@trycua/fleet@0.1.1` SDK does not expose `pool.apply`. Instead, `reconcilePool()` and `reconcileTemplate()` are the lower-level UniFFI reconciliation primitives that Python `Pool.apply` delegates to internally. They do not provide `Pool.apply`'s request derivation, validation, ownership, rollback, or cleanup. Each method looks up its named resource, updates the existing resource's specification, or creates the resource when the lookup returns HTTP 403 or 404. Calling both methods lets you rerun the complete example without maintaining separate get-or-create branches. Use reconciliation only when this application owns the pool and template configuration. To use a pool managed by another process or Terraform, replace both reconciliation calls and their resource-specific log statements with a lookup: ```ts const pool = await client.getPool(POOL_NAME); ``` Keep the `createClaim`, `waitClaim`, and `deleteClaim` flow. This lookup does not create or reconfigure the pool. Avoid `Sandbox.suspend` or `Sandbox.resume` from other SDKs as a way to pause one Fleet claim: scaling shared capacity is not a per-claim operation. `client.uniffiDestroy()` only disposes of the local client. Remote claim release requires `deleteClaim()`. The `finally` blocks attempt that release even if claim readiness or the screenshot fails, provided `createClaim()` returned the claim. An interrupted process or a lost create response still needs inventory inspection; see [Recover interrupted cleanup](). To remove a pool you own permanently, retain its `pool` value and call the following while the client is still open, after every user has released their claims: ```ts await client.deletePool(pool); ``` Pool deletion also deletes its namespace, including remaining templates and claims. Do not use it as per-task cleanup for a shared pool. The examples leave the pool in place if template reconciliation or later work fails; inspect what exists before deciding whether to repair or remove a dedicated pool. Pool names are globally unique across Cua accounts. For a confirmed namespace collision, choose another pool name; changing the claim name does not resolve it. HTTP 403 can also indicate a payment requirement, authorization, or a template policy denial. Read the failing operation and response body before choosing a fix in [Troubleshoot Fleet pools and claims](). ## Use autoscaling instead of one replica Replace `.replicas(1)` with an autoscaling policy and omit `replicas`: ```ts import { WarmPoolAutoscalingBuilder } from '@trycua/fleet/node'; const autoscaling = new WarmPoolAutoscalingBuilder() .minPoolSize(0) .initialPoolSize(2) .maxPoolSize(10) .build(); const poolSpec = new OsGymSandboxWarmPoolSpecBuilder() .autoscaling(autoscaling) .sandboxTemplateRef(templateRef) .build(); ``` Use the matching `@trycua/fleet/browser` import in browser code. With a minimum size of zero, the first claim after an idle period cold-starts a sandbox. --- # Run a Cursor Cloud Agent worker on Cua Cloud Fleet Start a Cursor self-hosted Team Pool worker inside a claimed Cua Cloud Fleet Linux desktop. Use this guide to run one Cursor self-hosted Cloud Agent worker inside a Cua Cloud Fleet VM. A local Python controller creates a reusable Cua pool, claims a VM, starts the Cursor worker in that VM, and keeps the claim alive until you stop the controller. ```text Cursor Cloud Agent control plane | | outbound worker connection v Cursor worker in a claimed Cua Fleet VM | +-- repository checkout and terminal tools +-- XFCE/X11 desktop and browser +-- private services reachable from the VM ``` **Warning** This is a manual integration path, not a built-in Cursor provider. The controller must remain running because the Cua claim owns the VM used by the Cursor worker. Scaling Cua capacity from Cursor's pending-request queue requires an additional controller and is not covered here. ## Prerequisites You need: - Python `>=3.11,<3.14` and [`uv`](https://docs.astral.sh/uv/); - Cua Fleet credentials that can create pools and claims; - a Cursor Enterprise plan with Self-Hosted Agents enabled; - a Cursor service-account API key for Team Pool workers; and - a globally unique, lowercase DNS-label name for the Cua pool. This guide uses the public Cua Ubuntu 24.04 desktop image: ```text public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57 ``` The image provides a glibc-based Linux VM, an XFCE/X11 desktop, and the Cua computer server on port `8000`. The Cursor CLI and worker are third-party software installed at runtime from Cursor's official installer. ## Configure credentials and names Authenticate to Cua Fleet with an access token: ```bash export FLEETS_TOKEN="" ``` Or use Cua OAuth client credentials: ```bash export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" ``` Set the Cua pool name, Cursor Team Pool name, and Cursor service-account key: ```bash export CUA_POOL_NAME="" export CURSOR_WORKER_POOL_NAME="cua-linux" export CURSOR_API_KEY="" ``` The two pool names identify different resources: | Variable | Meaning | | --- | --- | | `CUA_POOL_NAME` | The Cua Fleet pool that supplies VMs | | `CURSOR_WORKER_POOL_NAME` | The Cursor Team Pool that receives Cloud Agent requests | Keep both providers' credentials in environment variables or a secret manager. Do not store them in an image definition or commit them to source control. ## Start the Fleet-backed worker Save this script as `run_cursor_worker.py`: ```python title="run_cursor_worker.py" # /// script # requires-python = ">=3.11,<3.14" # dependencies = [ # "cua-sandbox", # ] # /// import asyncio import os import shlex from cua_sandbox import Image, Pool IMAGE = ( "public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04" "@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57" ) CUA_POOL_NAME = os.environ["CUA_POOL_NAME"] CURSOR_POOL_NAME = os.environ["CURSOR_WORKER_POOL_NAME"] CURSOR_API_KEY = os.environ["CURSOR_API_KEY"] async def checked(sandbox, command: str, timeout: int = 300) -> str: result = await sandbox.shell.run(command, timeout=timeout) if not result.success: raise RuntimeError(f"{command}\n{result.stderr}") return result.stdout.strip() async def main() -> None: pool = await Pool.apply( Image.from_registry(IMAGE, os_type="linux", kind="vm"), name=CUA_POOL_NAME, replicas=1, cpu=4, memory_mb=8192, services={"server": 8000}, ) async with pool.claim(service="server", time_to_start=1800) as sandbox: print(f"Cua pool: {sandbox.pool_name}") print(f"Cua claim: {sandbox.claim_name}") print(f"Sandbox: {sandbox.name}") await checked(sandbox, "curl -fsSL https://cursor.com/install -o /tmp/cursor-install.sh") await checked(sandbox, "bash /tmp/cursor-install.sh") await checked(sandbox, "test -x /root/.local/bin/agent") await checked( sandbox, "mkdir -p /root/.config /root/cursor-workspaces/default", ) # Write the credential through the file API so it is not interpolated # into a shell command or printed by the controller. env_file = ( f"CURSOR_API_KEY={shlex.quote(CURSOR_API_KEY)}\n" f"CURSOR_WORKER_POOL_NAME={shlex.quote(CURSOR_POOL_NAME)}\n" ) await sandbox.files.write_bytes( "/root/.config/cursor-worker.env", env_file.encode(), ) await checked( sandbox, "chmod 600 /root/.config/cursor-worker.env", ) command = """bash -lc ' set -a . /root/.config/cursor-worker.env set +a export PATH=/root/.local/bin:$PATH exec agent worker \\ --pool "$CURSOR_WORKER_POOL_NAME" \\ --worker-dir /root/cursor-workspaces/default \\ --clone-git-repos \\ start > /root/cursor-worker.log 2>&1 '""" worker = await sandbox.shell.run(command, background=True) if not worker.success: raise RuntimeError(worker.stderr) try: print(f"Cursor worker process: {worker.stdout or 'started'}") print("The Cua claim is active. Press Ctrl-C to release it.") while True: await asyncio.sleep(15) status = await sandbox.shell.run( "pgrep -af '[a]gent worker'", timeout=15, ) if not status.success: raise RuntimeError( "Cursor worker exited; inspect /root/cursor-worker.log" ) finally: await sandbox.shell.run("pkill -f '[a]gent worker' || true") await sandbox.shell.run("rm -f /root/.config/cursor-worker.env") try: asyncio.run(main()) except KeyboardInterrupt: print("Releasing the Cua claim; the reusable pool remains available") ``` Run the controller: ```bash uv run run_cursor_worker.py ``` The first run can take several minutes while Fleet provisions the VM. The script then installs the Cursor CLI, starts an any-repository Team Pool worker, and checks every 15 seconds that the worker process remains present. The `--clone-git-repos` option lets Cursor prepare the assigned repository in the otherwise empty worker directory. Use this option only for a named any-repository Team Pool, as described in Cursor's Team Pools documentation. **Note** The script keeps the Cua pool after you stop it. Later runs can claim its warm capacity without rebuilding the pool, but the runtime Cursor CLI installation is not part of the immutable base image. For repeated production use, publish a private worker image with a pinned Cursor CLI and use that image digest in `Image.from_registry()`. ## Verify the worker In the Cursor dashboard, open **Cloud Agents → Self-Hosted Agents** and confirm that an idle worker appears in the Team Pool named by `CURSOR_WORKER_POOL_NAME`. Start a Cloud Agent run and select that Team Pool. Ask the agent to perform a small, reversible check, such as: ```text Report the operating system, current working directory, and git remote. Do not edit files. ``` Confirm that: 1. Cursor assigns the run to the registered worker. 2. The reported operating system is Linux. 3. The repository is cloned under the worker's workspace. 4. The local controller continues to report a live claim. If you intend to use graphical tools, add Cursor's `--computer-use` worker option and follow Cursor's Linux computer-use dependency and display checks. The Cua image already provides an X11 desktop, but Cursor's computer-use helper is separate from Cua Driver and must pass Cursor's own preflight. ## Stop and clean up Press `Ctrl-C` in the local controller. The script stops the worker and removes its credential file before exiting the claim context. Exiting the context then releases the claim. The Cua pool and its warm replica remain for the next run. Delete the pool only when you no longer need it. Add the following after the claim context, or use a separate cleanup script that resolves the same named pool: ```python await pool.delete() ``` Deleting the pool removes its warm capacity and same-named Fleet namespace. ## Production hardening Before using this integration for production workloads: - build and pin a private worker image instead of downloading the CLI at each start; - run the worker as a dedicated non-root user; - restrict outbound access to the endpoints required by Cursor and your source provider; - decide whether Cursor artifact uploads are allowed by your data-handling policy; - prevent secrets from appearing in terminal output, screenshots, diffs, and artifacts; - set claim and pool TTLs appropriate for abandoned-controller recovery; - collect worker logs without recording API keys or repository contents; and - test claim loss, worker failure, cancellation, and controller restart behavior. Cursor documents that the worker sends agent-required content to Cursor, including file contents, terminal output, diffs, screenshots, local MCP results, and routing metadata. Cursor also documents that Cloud Agent artifacts may be uploaded to Cursor-managed storage. Running the worker on Cua Fleet keeps execution in the Fleet VM; it does not make the Cursor agent loop or its data plane self-hosted. ## Understand the scaling boundary Cursor and Cua each use a pool, but at different layers: ```text Cursor Team Pool pending request | | needs an idle Cursor worker v integration controller | | creates and holds a Cua claim v Cua Fleet pool -> VM -> Cursor worker process ``` The script in this guide maintains one warm Cua VM and one Cursor worker. A production autoscaler must observe Cursor pending requests, create or release Cua claims, start exactly one Team Pool worker per claimed VM, and reconcile both systems after failures. Do not configure independent autoscalers on both sides without defining ownership; they can otherwise disagree about desired capacity or leave workers and claims orphaned. ## Troubleshooting | Symptom | Check | | --- | --- | | No worker appears in Cursor | Confirm Self-Hosted Agents is enabled and `CURSOR_API_KEY` belongs to a Cursor service account | | Worker exits immediately | Run Cursor's `agent worker ... debug --json` command in the claimed VM and inspect the result | | Repository is not cloned | Confirm the worker uses a named pool and `--clone-git-repos`, and that Cursor can access the source repository | | Fleet claim never becomes ready | Verify Cua credentials, pool-name uniqueness, image access, and the `server` service on port `8000` | | Agent cannot reach an internal service | Test DNS, routing, and outbound policy from inside the Fleet VM | | Cursor shows no screenshots or artifacts | Review Cursor's required artifact-storage endpoint and your outbound network policy | | Controller stops and the worker disappears | Expected: the controller owns the Cua claim; run it under a supervised service for persistent use | ## Related documentation - [Create a sandbox pool with Python]() - [Pass secrets into a sandbox]() - [Run sandboxes in parallel]() - [Cursor Self-Hosted Machines](https://cursor.com/docs/cloud-agent/self-hosted) - [Cursor Team Pools](https://cursor.com/docs/cloud-agent/self-hosted/pool) --- # Expire pools and claims automatically Set a creation-age TTL so sandbox pools and claims delete themselves. Pools and claims accept an optional creation-age TTL. Pass `ttl_seconds_after_created=` (requires `cua-sandbox>=0.4.2`) and the resource is deleted once it has existed that many seconds, whether or not it is in use. Without this argument, there is no creation-age TTL from this setting. Explicit claim lifecycle deadlines, normal claim release, pool deletion, and platform policies can still end resources. This guide assumes a pool created as in [Create a sandbox pool with Python](). ## Expire a pool Pass the TTL to `Pool.apply()` for a pool you own to request automatic deletion a fixed time after the pool was first created: ```python from cua_sandbox import Image, Pool # Delete the whole pool 24 hours after it was first created. pool = await Pool.apply( Image.from_registry(IMAGE), name=POOL_NAME, replicas=1, ttl_seconds_after_created=86400, ) ``` ## Expire a claim Pass the same argument to `pool.claim()` (or `pool.create_claim()`) to delete one claim and its sandbox a fixed time after the claim was created: ```python # Delete this claim and its sandbox one hour after the claim was created. async with pool.claim( name=CLAIM_NAME, ttl_seconds_after_created=3600, ) as sandbox: ... ``` A claim TTL fills in the claim's lifecycle shutdown time with a `Delete` policy; a claim that already carries an explicit lifecycle keeps it. Exiting the claim context still releases it immediately, even if its TTL has not elapsed. Disconnecting alone does not release it or cancel its deadline. ## The clock starts at creation The TTL counts from the resource's original creation, not its last use: re-running `Pool.apply()` reconciles the existing pool without resetting its age, and reconnecting to a named claim keeps the deadline set when the claim was first created. Passing a TTL when attaching to an existing named claim does not replace that claim's spec or renew its deadline. A pool may therefore expire shortly after a rerun if it is already near the end of its lifetime. Use a TTL as a cleanup backstop for interrupted jobs, and still release claims when work finishes. It is not an inactivity timeout or an automatic heartbeat. Keep the pool lifetime longer than the intended work, including startup time, and copy results out before the deadline. Controller-driven deletion is asynchronous; verify resource absence after expiry instead of treating the deadline as proof that cleanup has finished. ## Hand-built claim specs When you build a claim spec by hand, put the TTL inside it — passing both `spec=` and `ttl_seconds_after_created=` raises `ValueError`: ```python from cua_sandbox import ClaimSpec spec = ClaimSpec( sandbox_template_ref=pool.resource.spec.sandbox_template_ref, warmpool=None, bind_deadline=None, lifecycle=None, ttl_seconds_after_created=3600, ) async with pool.claim(spec=spec) as sandbox: ... ``` ## Pool expiry under live claims **Warning** Treat a pool TTL that can expire while claims are held as a capacity event, not just cleanup: deleting the pool drains its sandboxes, and a claim that outlives its pool destroys its sandbox on release instead of returning it to the pool. If cleanup was interrupted, use the resource identities you recorded and [inspect the remaining pool and claims](). Reapplying a pool is a configuration write, not a cleanup operation. --- # Troubleshoot Fleet pools and claims Diagnose Fleet pool errors and recover interrupted claim or pool cleanup without provisioning replacements. Use this guide when a Fleet pool or claim fails to start, an existing pool cannot be reused, or a process exits before releasing its resources. It applies to the [Python pool guide]() and [TypeScript pool guide](). ## Identify the failing operation Record the SDK versions, pool namespace, claim name if known, failing operation, HTTP status, and sanitized error body. Keep credentials, tokens, signed URLs, and guest content out of shared logs. Read the operation as well as the exception class. `PoolAccessDeniedError` includes a name-collision hint, but HTTP 403 alone does not establish that the name belongs to another account. ## Resolve authentication or access errors If token exchange fails or the API returns HTTP 401, verify that the credential has not expired or been revoked and that your process uses the intended Fleet and token endpoints. Follow the credential setup in the guide for your SDK, then retry an existing-resource lookup before attempting creation again. For HTTP 403, inspect the response body and operation: - If the body says a payment method is required, review the account's billing setup with its owner. A different pool name or repeated retries will not satisfy that requirement. - If an initial pool creation is denied and ownership checks show that the namespace is unavailable to your account, choose a different globally unique pool name. A different claim name does not resolve this conflict. - If access to a known pool is denied, verify that the credential belongs to an account authorized for that pool. Do not interpret 403 as proof that the resource is absent or that cleanup succeeded. - If `create template` or `update template` is denied, inspect the image reference and image-pull configuration against the deployment's policy. A successful pool creation does not prove the template is permitted. Preserve the failing operation and sanitized response when asking for help. For local validation errors about names, use lowercase DNS labels: lowercase letters, digits, and hyphens, with a letter or digit at each end. Keep names at most 63 characters. Use a distinct claim name for each concurrent task. ## Diagnose a claim that does not become ready Check the pool, template, and claim in [Fleet](https://run.cua.ai) using the recorded namespace and claim name. Separate these stages: 1. Verify that the pool and its referenced template exist. Inspect the template reference rather than assuming its name equals the pool name. 2. Inspect available capacity and the claim's binding status. A pool scaled to zero can require a cold start. A missing template, unavailable image, or exhausted capacity is not fixed by extending a guest-service timeout. 3. Once the claim is bound, verify that the selected service exists and the guest process listens on its configured port. The Python example expects `server` on port `8000` and a compatible computer-server image. 4. If the configuration is correct and startup is still progressing, adjust the relevant wait budget. Python's `time_to_start` controls the service readiness wait after binding; TypeScript's `claimPollIntervalMs` and `claimPollLimit` control claim polling. Neither setting extends a resource's TTL or lifecycle deadline. An API connection, a created pool, and a bound claim are different checkpoints. Verify a successful service request before treating the guest as usable. On a readiness timeout, inspect whether the claim still exists before retrying: Python attempts to release newly created claims on acquisition failure, while a pre-existing named claim remains held. The TypeScript examples attempt release in `finally` if creation returned a claim handle. ## Recover interrupted cleanup Use lookups and deletion with the identities recorded before the failure. Do not rerun `Pool.apply()`, `reconcilePool()`, or `createClaim()` to recover a handle: they can create or modify resources. Disconnecting a client or closing a browser tab does not prove that remote resources were released. First, inspect the named pool and its claims in Fleet. If the pool is shared, release only your task's claim. If a create response was lost before its claim name was recorded, inspect inventory and establish ownership before deleting anything. A timestamp or similar-looking name alone is not proof of ownership. For scripted recovery, initialize the TypeScript client using only the imports, configuration, and `CyclopsClient.connect(...)` setup from the [Node.js example](). Do not run its reconciliation or claim-creation block. The following snippets use that open `client` and the exact `POOL_NAME` you recorded. List claim identities without waiting for a guest to become ready: ```ts const claims = await client.listClaims(POOL_NAME); for (const claim of claims) { console.log(`${claim.metadata.namespace}/${claim.metadata.name}`); } ``` After identifying a claim that your task owns, set `CUA_CLAIM_NAME` to that exact name and release it. This does not create a replacement claim or wait for service readiness: ```ts const claimName = process.env.CUA_CLAIM_NAME; if (!claimName) throw new Error('Set CUA_CLAIM_NAME to the claim you own'); const claims = await client.listClaims(POOL_NAME); const claim = claims.find((item) => item.metadata.name === claimName); if (claim) await client.deleteClaim(claim); const remaining = await client.listClaims(POOL_NAME); if (remaining.some((item) => item.metadata.name === claimName)) { throw new Error('Claim deletion is still pending; inspect again'); } ``` If you own the entire pool and all users have finished, fetch and delete it: ```ts const pool = await client.getPool(POOL_NAME); await client.deletePool(pool); ``` This deletes the pool namespace and its remaining resources, including other claims and templates. Use this operation only for a pool you intend to remove completely. In Python, the equivalent is `pool = await Pool.get(POOL_NAME)` followed by `await pool.delete()`. If the pool lookup returns 404 but namespace inventory still shows the exact dedicated namespace, inspect its remaining claims and templates. Once you have confirmed that all remaining resources belong to the cleanup task, delete the namespace directly: ```ts await client.deleteNamespace(POOL_NAME); ``` In `@trycua/fleet@0.1.1`, namespace deletion accepts an already-absent namespace as success, so the same exact deletion can be retried after interruption. Close the local client with `client.uniffiDestroy()` in a `finally` block when the recovery script ends. ## Verify cleanup finished After claim-only cleanup, verify that the claim is absent and the shared pool still exists. After whole-pool cleanup, check namespace inventory again until the namespace and its resources are absent. An accepted DELETE can precede asynchronous removal. If a request returns 403, resolve authorization before claiming cleanup is complete. For future disposable work, configure a [creation-age TTL]() as a backstop and retain explicit cleanup. A TTL does not replace copying out results or checking that resources were removed. --- # Run OpenClaw on Cloud Fleet Run OpenClaw's bundled CUA computer-use provider in an isolated Linux desktop on Cua Cloud Fleet. This guide provisions one Linux desktop on [Cua Cloud Fleet](), installs OpenClaw, and connects a local Gateway and node runtime in the same VM. The node's bundled `cua-computer` plugin calls the pinned `@trycua/cua-driver` SDK directly, so the normal OpenClaw `computer` tool can control the VM's graphical desktop. ```text OpenClaw agent -> Gateway -> paired local node -> computer.act / screen.snapshot -> bundled cua-computer provider -> @trycua/cua-driver SDK -> X11 desktop in the same Fleet VM ``` **Warning** The supported Linux proof path is a glibc-based Linux x64 or ARM64 guest with an X11 desktop. Do not use native Wayland as evidence for this integration: OpenClaw documents it as unsupported for the bundled Linux provider. ## Before you start You need: - Python `>=3.11,<3.14` and [`uv`](https://docs.astral.sh/uv/); - Node.js and npm in the guest image (the public image used below includes both); - Fleet credentials that can create pools and claims; and - a vision-capable model credential for the OpenClaw agent. The example uses this immutable, public KubeVirt containerDisk: ```text public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57 ``` It boots Ubuntu 24.04 with XFCE on X11 and exposes the Cua computer server as the Fleet `server` service on port `8000`. Use an image with the same properties if you substitute your own image. ## Authenticate with Fleet The SDK connects to `https://run.cua.ai` by default. Export either a Fleet access token or OAuth client credentials: ```bash export FLEETS_TOKEN="" # Or: export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" ``` Keep credentials in the environment or a secret manager. Never put them in an image definition or commit them to source control. Choose a globally unique, lowercase pool name: ```bash export CUA_POOL_NAME="" ``` ## Provision and claim a desktop Save this script as `run_openclaw_fleet.py`: ```python title="run_openclaw_fleet.py" # /// script # requires-python = ">=3.11,<3.14" # dependencies = [ # "cua-sandbox", # ] # /// import asyncio import os from pathlib import Path from cua_sandbox import Image, Pool IMAGE = ( "public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04" "@sha256:c1e601dbb748fdc467c663136f7592e308a91a3c19c309b75261544432826a57" ) POOL_NAME = os.environ["CUA_POOL_NAME"] async def run(sandbox, command: str) -> None: result = await sandbox.shell.run(command) if not result.success: raise RuntimeError(f"{command}\n{result.stderr}") if result.stdout.strip(): print(result.stdout.strip()) async def main() -> None: pool = await Pool.apply( Image.from_registry(IMAGE, os_type="linux", kind="vm"), name=POOL_NAME, replicas=1, cpu=4, memory_mb=8192, services={"server": 8000}, ttl_seconds_after_created=21600, ) try: async with pool.claim(service="server", time_to_start=1800) as sandbox: print(f"Sandbox: {sandbox.name}") await run(sandbox, "uname -a") await run(sandbox, "printf 'DISPLAY=%s\\n' \"$DISPLAY\"") await run(sandbox, "test -S /tmp/.X11-unix/X1") # Install OpenClaw in the disposable VM. Pin the version you tested. await run(sandbox, "npm install --global openclaw@2026.8.1") await run(sandbox, "openclaw plugins enable cua-computer") await run( sandbox, "openclaw doctor --lint --only cua-computer/driver-artifacts", ) print("Desktop preflight and bundled CUA artifact check passed") screenshot = Path("openclaw-fleet-preflight.png") screenshot.write_bytes(await sandbox.screenshot()) print(f"Screenshot: {screenshot.resolve()}") finally: await pool.delete() asyncio.run(main()) ``` Run it with `uv`: ```bash uv run run_openclaw_fleet.py ``` `Pool.apply()` reconciles the named pool and its VM template. `pool.claim()` waits for the `server` service and releases the VM when its context exits. The `finally` block deletes the pool, including warm replicas, even when setup fails. Save any screenshots or files you need before cleanup. The script is a disposable preflight. To continue into an agent session, keep the claim open and run the Gateway, node, pairing, and agent commands in the same `async with pool.claim(...)` block (or remove the `finally` deletion and use a second controller that holds the claim). Do not let the claim exit before the Gateway and node have finished their work. ## Configure the Gateway and node For a disposable single-VM setup, keep the Gateway on loopback and run the node in the same user session. Set a temporary Gateway token and configure the tool policy: ```bash export OPENCLAW_GATEWAY_TOKEN="" openclaw config set gateway.mode local openclaw config set gateway.bind loopback openclaw config set gateway.auth.mode token openclaw config set gateway.auth.token "$OPENCLAW_GATEWAY_TOKEN" openclaw config set tools.alsoAllow '["computer"]' ``` Start both processes from the X11 session. Keep the logs so you can inspect pairing and provider advertisements: ```bash nohup openclaw gateway run --token "$OPENCLAW_GATEWAY_TOKEN" \ > ~/openclaw-gateway.log 2>&1 & DISPLAY="$DISPLAY" nohup openclaw node run \ --host 127.0.0.1 --port 18789 \ --token "$OPENCLAW_GATEWAY_TOKEN" \ --display-name "Fleet Linux node" \ > ~/openclaw-node.log 2>&1 & ``` The node must advertise both `computer.act` and `screen.snapshot`. Approve the new command surface on the Gateway: ```bash openclaw nodes pending openclaw nodes approve openclaw nodes status ``` Repeat `openclaw nodes status` after approval and confirm that the connected node reports provider id `cua-computer`. If you change the node's advertised capabilities, approve the new pending request again. ## Run an agent turn Configure a model using the normal OpenClaw onboarding or provider setup, then run a Gateway-backed agent turn: ```bash openclaw agent --message \ "Use the computer tool to open a visible desktop message that says OPENCLAW CUA FLEET PASSED." \ --json ``` The agent should make a fresh `computer` screenshot, perform input through `computer.act`, and observe the resulting screen. Verify the effect in the VM with the Cua Sandbox screenshot API or your image's desktop viewer. A passing run has all of these properties: - the model request includes the normal `computer` tool; - the node is paired and connected with `computer.act` and `screen.snapshot`; - the node provider is `cua-computer`; - the OpenClaw agent completes the turn without a `COMPUTER_*` error; and - the requested text is visible in an X11 window in the Fleet VM. The external `cua-driver mcp` server is not part of this path. The bundled provider uses the npm SDK in-process on Linux; do not install a standalone driver executable or configure an MCP server for this proof. ## Troubleshoot - **No `DISPLAY` or an empty screenshot:** run the node from the interactive X11 session and confirm `/tmp/.X11-unix/X1` exists. A headless shell without the desktop session cannot provide computer use. - **Wayland session:** switch the image to X11. Native Wayland is not a supported proof path for the bundled Linux provider. - **`computer` is missing:** add `computer` to `tools.alsoAllow`. Sandboxed agents also need `tools.sandbox.tools.alsoAllow: ["computer"]`. - **Node is connected but has no computer commands:** run `openclaw plugins enable cua-computer`, rerun the focused doctor check, and restart `openclaw node run`. Approve the resulting request with `openclaw nodes approve`. - **Driver artifact errors:** run `openclaw doctor --lint --only cua-computer/driver-artifacts`. Reinstall the same OpenClaw version; do not replace native package files manually. - **Fleet claim timeout or HTTP 403:** verify that the image is an admitted, immutable KubeVirt containerDisk and that your credentials can create pools. Pool names are globally unique, so choose another name if the name is taken. ## Validated configuration This procedure was exercised end to end with: | Component | Tested value | | -------------- | ------------------------------------------------------------- | | OpenClaw | `2026.8.1`, commit `6dc72d7ee21947f8bec897f76de5214e3830ffd4` | | Cua | commit `e7295472e196b22e1d02a69441b315a3776d50fb` | | Cua Driver SDK | `@trycua/cua-driver` `0.21.0` | | Guest | Ubuntu `24.04.4`, Linux x86_64 | | Desktop | XFCE on X11, `DISPLAY=:1` | | Fleet | KubeVirt, BIOS, 4 vCPU, 8192 MB, one replica, `server:8000` | | Image | Ubuntu 24.04 containerDisk pinned by the digest above | The exercised agent session completed five `computer` calls with zero failures and produced a visible desktop result. The pool was deleted after the run. ## Clean up an interrupted run If the process is interrupted after `Pool.apply()`, rerun it with the same `CUA_POOL_NAME`. `Pool.apply()` reconciles the existing pool, and the `finally` block reaches the deletion path when connectivity returns. You can also delete the pool explicitly after reconnecting: ```python await pool.delete() ``` For reusable warm capacity, omit the deletion and set a suitable pool and claim TTL. See [Create a reusable sandbox pool with Python](). --- # Run Omarchy on Fleet Provision an amd64 Omarchy desktop on Cua Fleet and control it with the Cua Sandbox SDK. Use the Cua Sandbox SDK to provision an amd64 Omarchy desktop on [Cua Fleet](). Fleet runs the Omarchy system as a KubeVirt `containerDisk`; the guest starts Hyprland and exposes both the `cua-computer-server` API and the Cua Driver MCP service. For the desktop stack and input boundaries, see [Linux desktops and computer use]() and [Hyprland support](). Provisioning a private VM does not establish isolated background input between applications inside that VM. **Note** This guide is for the amd64 Fleet image. The [Omarchy on Apple Silicon guide]() builds a separate ARM64 VM for local Lume use and does not run on Fleet. ## Before you start You need: - Python `>=3.11,<3.14`; - [`uv`](https://docs.astral.sh/uv/); - `cua-sandbox==0.4.3`; and - a Fleet access token or OAuth client credentials that can manage pools. The Fleet-verified public image reference is: ```text public.ecr.aws/k5j5w0x5/cua-omarchy-workspace@sha256:d9b7be06beac425084eaa99eb912589b38b5cc86ae3e3ec45c9c5d59d4b3a7ab ``` The image is an amd64 KubeVirt containerDisk. It is not an ordinary OCI application image: the image contains a bootable disk at `/disk/disk.img`. Fleet pulls the public image, so you do not need AWS credentials on the machine that runs this script. ## Authenticate with Fleet The SDK connects to `https://run.cua.ai` by default. Export one supported credential set before running the example: ```bash export FLEETS_TOKEN="" ``` Or use OAuth client credentials: ```bash export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" ``` Keep credentials in your shell environment or a secret manager. Do not put them in the image definition or commit them to source control. Choose a globally unique, lowercase DNS-label pool name: ```bash export CUA_POOL_NAME="" ``` ## Provision and claim an Omarchy desktop Save this script as `run_omarchy_fleet.py`: ```python title="run_omarchy_fleet.py" # /// script # requires-python = ">=3.11,<3.14" # dependencies = [ # "cua-sandbox==0.4.3", # ] # /// import asyncio import os from pathlib import Path from cua_sandbox import Image, Pool IMAGE = os.environ.get( "OMARCHY_IMAGE", "public.ecr.aws/k5j5w0x5/cua-omarchy-workspace" "@sha256:d9b7be06beac425084eaa99eb912589b38b5cc86ae3e3ec45c9c5d59d4b3a7ab", ) POOL_NAME = os.environ["CUA_POOL_NAME"] async def main() -> None: image = Image.from_registry(IMAGE, os_type="linux", kind="vm") pool = await Pool.apply( image, name=POOL_NAME, replicas=1, cpu=4, memory_mb=6144, services={"server": 8000, "mcp": 3000}, ttl_seconds_after_created=21600, ) try: async with pool.claim(service="server", time_to_start=1800) as sandbox: print(f"Sandbox: {sandbox.name}") print(f"Screen: {await sandbox.get_dimensions()}") result = await sandbox.shell.run("pgrep -a Hyprland") if not result.success: raise RuntimeError(result.stderr) print(result.stdout.strip()) screenshot = Path("omarchy-fleet.png") screenshot.write_bytes(await sandbox.screenshot()) print(f"Screenshot: {screenshot.resolve()}") await sandbox.clipboard.set("hello from Omarchy Fleet") if await sandbox.clipboard.get() != "hello from Omarchy Fleet": raise RuntimeError("clipboard round trip failed") width, height = await sandbox.get_dimensions() await sandbox.mouse.click(width // 2, height // 2) await sandbox.keyboard.keypress(["cmd", "2"]) print("Screenshot, shell, clipboard, click, and workspace hotkey passed") finally: await pool.delete() asyncio.run(main()) ``` Run the script: ```bash uv run run_omarchy_fleet.py ``` `Pool.apply()` creates or reconciles the named pool and its template. The `server` service on port `8000` carries screenshot, shell, keyboard, mouse, and clipboard operations. The `mcp` service on port `3000` carries the Cua Driver MCP endpoint at `/mcp` for an MCP client. The pool name is globally unique across Cua accounts. If the name is already in use, choose another name and run the script again: ```bash export CUA_POOL_NAME="my-team-omarchy-pool" uv run run_omarchy_fleet.py ``` ## View the desktop Use `sandbox.screenshot()` to inspect the desktop and the mouse, keyboard, shell, clipboard, and window interfaces to control it without a live display. For an interactive browser display, expose a Fleet service named `vnc` and configure the guest's existing WayVNC process with a noVNC bridge. Follow [Connect to an Omarchy Fleet desktop with noVNC](). Fleet carries the display through its authenticated HTTP and WebSocket service proxy; it does not expose a public raw VNC TCP socket. ## Connect an MCP client The claim exposes the Cua Driver MCP service as the named Fleet service `mcp`. Use the SDK service interface while the claim is active: ```python response = await sandbox.services.request( "mcp", method="POST", path="/mcp", headers={ "content-type": "application/json", "accept": "application/json, text/event-stream", }, json={ "jsonrpc": "2.0", "id": 1, "method": "initialize", "params": { "protocolVersion": "2025-03-26", "capabilities": {}, "clientInfo": {"name": "my-agent", "version": "0.1.0"}, }, }, ) response.raise_for_status() ``` For a complete MCP client, use the [Cua Driver MCP tool reference]() and keep the authenticated Fleet claim alive for the duration of the client session. ## Keep or delete the pool The example sets a six-hour creation-age TTL and deletes the pool in `finally`. That is a good default for jobs and CI. If you want a reusable warm pool, omit the `finally` deletion and release only the claim; delete the pool explicitly when you are finished: ```python await pool.delete() ``` Deleting the pool removes its template and sandboxes. Save screenshots or files that you need before the claim and pool are deleted. ## Troubleshoot startup - **HTTP 403 during `Pool.apply()`:** confirm the image repository is included in Fleet admission policy and that your credentials can create a pool in the selected namespace. - **Claim timeout:** verify that the image is an amd64 KubeVirt containerDisk, the image reference includes the exact digest, and the `server` service is configured on port `8000`. - **Black or empty screenshot:** check the image's unattended Hyprland boot and the `cua-computer-server` service before debugging the Fleet transport. - **MCP connection failure:** claim the sandbox with the `mcp` service exposed on port `3000` and send requests to `/mcp` through `sandbox.services`. For local Omarchy development and ARM64 compatibility notes, see [Run Omarchy on Apple Silicon](). --- # Connect to an Omarchy Fleet desktop with noVNC Add browser-based desktop access to an Omarchy VM running on Cua Fleet. Use noVNC to view and control an Omarchy Fleet desktop from an authenticated browser. This guide connects four components: ```text Browser -> Fleet /api/svc proxy -> websockify :5900 -> WayVNC :5902 -> Hyprland ``` Fleet exposes the HTTP and WebSocket endpoint through a service named `vnc`. It does not expose the guest's raw VNC port to the internet. ## Before you start You need: - an Omarchy pool definition that you can update; - an active Fleet claim with shell access to its guest; and - a browser that can sign in to Cua. Follow [Run Omarchy on Fleet]() to create the pool and claim. ## Add the Fleet service Add a service named `vnc` on guest port `5900` to the pool definition: ```python pool = await Pool.apply( image, name=POOL_NAME, replicas=1, cpu=4, memory_mb=6144, services={"server": 8000, "mcp": 3000, "vnc": 5900}, ttl_seconds_after_created=21600, ) ``` Apply the pool definition, then create a claim. If you add the service while a sandbox is already bound, create a new claim if the current sandbox does not show `vnc` in its service list. **Warning** Do not expose port `5902` as another Fleet service. WayVNC uses that raw RFB port inside the guest. Only the noVNC bridge on port `5900` needs a Fleet service. ## Install noVNC and websockify Open a shell in the claimed Omarchy guest. Install noVNC `1.7.0` and websockify `0.13.0` under your user account: ```bash NOVNC_VERSION="1.7.0" WEBSOCKIFY_VERSION="0.13.0" VNC_ROOT="$HOME/.local/share/omarchy-vnc" NOVNC_ARCHIVE="$VNC_ROOT/noVNC-v${NOVNC_VERSION}.tar.gz" WEBSOCKIFY_VENV="$VNC_ROOT/websockify-venv" install -d "$VNC_ROOT" curl --fail --location \ "https://github.com/novnc/noVNC/archive/refs/tags/v${NOVNC_VERSION}.tar.gz" \ --output "$NOVNC_ARCHIVE" tar --extract --gzip --file "$NOVNC_ARCHIVE" --directory "$VNC_ROOT" python3 -m venv "$WEBSOCKIFY_VENV" "$WEBSOCKIFY_VENV/bin/python" -m pip install \ "websockify==${WEBSOCKIFY_VERSION}" ``` The Omarchy Fleet image starts WayVNC with the Hyprland graphical session. The verified setup moves the raw RFB listener to loopback port `5902` and leaves port `5900` to the Fleet-facing bridge: ```bash install -d "$HOME/.config/systemd/user/wayvnc.service.d" tee "$HOME/.config/systemd/user/wayvnc.service.d/override.conf" >/dev/null <<'EOF' [Service] ExecStart= ExecStart=/usr/bin/wayvnc 127.0.0.1 5902 EOF systemctl --user daemon-reload systemctl --user restart wayvnc.service ``` websockify serves the noVNC files on port `5900` and bridges WebSocket traffic to the WayVNC listener on `5902`. ## Run websockify as a user service Create a systemd user service: ```bash install -d "$HOME/.config/systemd/user" tee "$HOME/.config/systemd/user/omarchy-websockify.service" >/dev/null <<'EOF' [Unit] Description=noVNC bridge for the Omarchy graphical session After=wayvnc.service Requires=wayvnc.service PartOf=graphical-session.target ConditionEnvironment=WAYLAND_DISPLAY [Service] Type=simple ExecStart=%h/.local/share/omarchy-vnc/websockify-venv/bin/websockify --web=%h/.local/share/omarchy-vnc/noVNC-1.7.0 0.0.0.0:5900 127.0.0.1:5902 Restart=always RestartSec=3 [Install] WantedBy=graphical-session.target EOF systemctl --user daemon-reload systemctl --user enable --now omarchy-websockify.service ``` The service restarts with the graphical session and waits for the existing `wayvnc.service` unit. ## Verify the guest services Check both services and request the noVNC page from inside the guest: ```bash systemctl --user is-active wayvnc.service omarchy-websockify.service curl --fail --silent --show-error --output /dev/null \ http://127.0.0.1:5900/vnc.html ``` Both systemd checks must print `active`, and the HTTP request must return a successful response. To inspect a failure, run: ```bash journalctl --user \ --unit wayvnc.service \ --unit omarchy-websockify.service \ --no-pager --lines 100 ``` ## Authenticate your browser Open [Cua authentication](https://cua.ai/auth) and sign in. Keep the same browser profile open when you connect to the Fleet service. **Note** A Fleet access token or OAuth client ID and secret authenticate the Sandbox SDK. They do not create a browser session. An unauthenticated browser request to `/api/svc` redirects to the interactive sign-in flow. ## Open the desktop from the claim Open the claim details page in Fleet. When the bound sandbox is ready and the pool exposes a service named `vnc`, the page displays a **Desktop** pane. Wait for its status to change to **Connected**, then click or type in the desktop to confirm that input reaches Omarchy. If the pane disconnected while the sandbox was starting, select **Reconnect**. ## Open the direct noVNC URL You can also open the noVNC page directly. Copy the namespace and bound sandbox name from the claim details page, then replace the placeholders in this URL: ```text https://run.cua.ai/api/svc//-vnc/vnc.html?autoconnect=1&resize=scale&reconnect=1 ``` The page connects to this WebSocket endpoint through the same authenticated service proxy: ```text wss://run.cua.ai/api/svc//-vnc/websockify ``` Use the bound sandbox name, not the claim name. The `-vnc` suffix comes from the Fleet service name. ## Troubleshoot the connection - **The URL redirects to sign-in:** sign in at [Cua authentication](https://cua.ai/auth), then reopen the noVNC URL in the same browser profile. SDK OAuth credentials do not replace this step. - **The service URL returns 404:** confirm that the namespace and bound sandbox name match the claim details page and that the pool exposes a service named `vnc` on port `5900`. - **The page loads but noVNC disconnects:** verify that both user services are active and that ports `5900` and `5902` are listening inside the guest with `ss --listening --tcp --numeric --processes`. - **The desktop is black or frozen:** confirm that Hyprland is running with `pgrep -a Hyprland`, then restart `wayvnc.service` and `omarchy-websockify.service`. - **A native VNC client cannot connect:** this is expected. Fleet's service route carries authenticated HTTPS and WebSocket traffic, not a public raw VNC TCP socket. Use the claim's Desktop pane or the noVNC page. ## Stop browser access To stop the bridge in the guest, disable its user service: ```bash systemctl --user disable --now omarchy-websockify.service ``` Remove the `vnc` entry from the pool's `services` mapping before you apply the next pool template. Delete the claim or pool when you no longer need the VM. --- # Pass secrets into a sandbox Read secrets from the host and inject them into a sandbox at runtime. Pass task-specific credentials from the host into a sandbox at runtime. Keep secrets out of image specifications, build commands, and copied application files. These examples use a connected Linux sandbox named `sb` with a POSIX shell and Python 3. Put your application at `/app/main.py` before running them. Use credentials with only the permissions and lifetime the task needs. ## Inject secrets at runtime Set `DATABASE_URL` and `GITHUB_TOKEN` in the host environment through your secret manager. Then call `await run_with_secrets(sb)` from your async code: ```python import os import shlex from cua_sandbox import Sandbox async def run_with_secrets(sb: Sandbox): db_url = os.environ['DATABASE_URL'] gh_token = os.environ['GITHUB_TOKEN'] command = ( f'DATABASE_URL={shlex.quote(db_url)} ' f'GITHUB_TOKEN={shlex.quote(gh_token)} ' 'python3 /app/main.py' ) result = await sb.shell.run(command) if not result.success: raise RuntimeError('Application failed; inspect sanitized diagnostics') ``` Keep the assignments and application in the same `shell.run()` call. An `export` in one call does not persist into another call. `shlex.quote()` preserves spaces, quotes, and newlines as literal values in a POSIX shell; environment values cannot contain a NUL byte. The assignments apply to the application and its child processes. They do not configure later shell calls. Wait for the application to finish, and avoid leaving background children running with inherited credentials. **Warning** These commands contain the secret values. Shell quoting prevents shell interpretation; it does not redact command logs, traces, process inspection, or application output. Use a trusted sandbox, restrict access to its diagnostics, and do not print the command or enable shell tracing (`set -x`). ## Keep image configuration non-sensitive Use `Image.env()` for non-sensitive config, **not for secrets**. ```python from cua_sandbox import Image img = ( Image.linux() .apt_install('python3') .env( LOG_LEVEL='info', APP_ENV='production', PORT='8080', ) ) ``` **Warning** .env() values are stored in the Image spec and visible to anyone who can inspect it. Do not use .env() for API keys, passwords, or tokens. ## Provide a temporary credential file For an application that reads an SSH key file, create a credential for the task and set `TASK_SSH_KEY_FILE` to its host file path. Restrict that host file to its owner (`chmod 600`). Configure `/app/main.py` to read the sandbox file path from `TASK_SSH_KEY_FILE`, then call `await run_with_key_file(sb)`: ```python import os from pathlib import Path import shlex from cua_sandbox import Sandbox async def run_with_key_file(sb: Sandbox): key_content = Path(os.environ['TASK_SSH_KEY_FILE']).read_bytes().decode('utf-8') script = f""" set -eu umask 077 secret_dir=$(mktemp -d /tmp/cua-task-secret.XXXXXXXXXX) trap 'rm -f "$secret_dir/task-key"; rmdir "$secret_dir"' EXIT trap 'exit 1' HUP INT TERM printf '%s' {shlex.quote(key_content)} > "$secret_dir/task-key" chmod 600 "$secret_dir/task-key" TASK_SSH_KEY_FILE="$secret_dir/task-key" python3 /app/main.py """ result = await sb.shell.run(script) if not result.success: raise RuntimeError('Credential task failed; inspect sanitized diagnostics') ``` This example accepts a UTF-8 text key without NUL bytes. `printf '%s'` writes its content without adding or removing a trailing newline. `mktemp -d` creates a private directory, and `umask 077` restricts the file to its owner from creation. Mode `600` permits only the owner to read and write the key; it does not protect against root or other processes running as that owner. The exit trap removes the temporary file and directory after the application exits, including an ordinary nonzero exit. Forced termination, a shell timeout, or loss of the sandbox connection can prevent cleanup. The host key remains in place. Do not snapshot the sandbox while it holds credentials, and revoke the task credential when finished. Removing the file does not securely erase copies, logs, or snapshots. ## Finish the task Check the application's exit status without printing credentials or unreviewed output. Confirm that credential-consuming processes have stopped and that temporary files were removed. If cleanup was interrupted, remove the remaining task files or destroy the disposable sandbox before reusing it. Revoke or expire credentials through the issuing service, and keep host secret files out of version control. --- # Run sandboxes in parallel Run many sandbox jobs at once, limit concurrency, and handle failures. Because `Sandbox.ephemeral` and `Sandbox.create` are async, you can **run many sandboxes at the same time** using `asyncio`. **Note** Every local sandbox consumes host CPU, memory, and disk. Start with low concurrency and increase it only after observing host resource use. ## Run a fixed set concurrently Use `asyncio.gather` when you already have the full list of sandbox jobs. ```python import asyncio from cua import Sandbox, Image async def run_task(task: str) -> str: async with Sandbox.ephemeral(Image.linux(), local=True) as sb: result = await sb.shell.run(f"echo '{task}'") return result.stdout.strip() async def main(): tasks = ['task-1', 'task-2', 'task-3', 'task-4'] results = await asyncio.gather(*[run_task(t) for t in tasks]) print(results) asyncio.run(main()) ``` ## Cap the number of concurrent sandboxes Use an `asyncio.Semaphore` when you need *rate limiting*. ```python import asyncio from cua import Sandbox, Image MAX_CONCURRENT = 5 async def process_item(sem: asyncio.Semaphore, item: str) -> dict: async with sem: async with Sandbox.ephemeral(Image.linux(), local=True) as sb: result = await sb.shell.run(f"python /app/process.py '{item}'") return {'item': item, 'output': result.stdout, 'ok': result.success} async def main(): items = [f'item-{i}' for i in range(20)] sem = asyncio.Semaphore(MAX_CONCURRENT) results = await asyncio.gather(*[process_item(sem, item) for item in items]) print(f"{sum(r['ok'] for r in results)}/{len(results)} succeeded") asyncio.run(main()) ``` ## Process a dynamic queue of tasks Use `asyncio.Queue` when tasks arrive at runtime. ```python import asyncio from cua import Sandbox, Image async def worker(queue: asyncio.Queue, worker_id: int): while True: task = await queue.get() if task is None: break async with Sandbox.ephemeral(Image.linux(), local=True) as sb: result = await sb.shell.run(task) print(f'[worker-{worker_id}] {result.stdout.strip()}') queue.task_done() async def main(): queue: asyncio.Queue = asyncio.Queue() num_workers = 4 workers = [asyncio.create_task(worker(queue, i)) for i in range(num_workers)] for i in range(10): await queue.put(f"echo 'job {i}'") for _ in range(num_workers): await queue.put(None) await asyncio.gather(*workers) asyncio.run(main()) ``` ## Handle failures Set `return_exceptions=True` when one failed sandbox job **should not cancel the rest**. ```python results = await asyncio.gather( *[run_task(t) for t in tasks], return_exceptions=True ) for task, result in zip(tasks, results): if isinstance(result, Exception): print(f'{task}: FAILED — {result}') else: print(f'{task}: {result}') ``` --- # Forward a port from a sandbox Reach a sandbox service through a local startup mapping, a Fleet service request, or a supported tunnel transport. Choose the connection method for your sandbox's transport. In `cua-sandbox` 0.4.3, local VM `HTTPTransport` and pool-backed `FleetTransport` do not implement `sb.tunnel.forward()`. Use the supported paths below; see [Sandbox runtime support]() for the versioned transport contract. ## Expose a local QEMU service at startup Declare the port on the image before starting a bare-metal QEMU VM, then read the host mapping from `sb.exposed_ports`. This example requires QEMU, a compatible host, `cua==0.1.6`, and `httpx`: ```python import asyncio import httpx from cua import Image, QEMURuntime, Sandbox async def main(): image = Image.linux().expose(8080) async with Sandbox.ephemeral( image, local=True, runtime=QEMURuntime(mode='bare-metal') ) as sb: await sb.shell.run( 'nohup python3 -m http.server 8080 --bind 0.0.0.0 ' '>/tmp/docs-http-server.log 2>&1 ) and use a pool whose guest starts the desired service. Within the claimed sandbox's context, send an authenticated request by service name: ```python response = await sb.services.request('port-3000', method='GET', path='/healthz') response.raise_for_status() print(response.text) ``` An image's `.expose(3000)` declaration creates the `port-3000` service in the default pool template. It does not start an application on that port or create a localhost listener. See [Prepare and reference a Fleet image]() for the complete image and claim workflow. External clients need the Fleet service URL and authentication; an endpoint URL is not anonymous access. ## Forward through SSH or ADB For an existing sandbox connected through `SSHTransport` or `ADBTransport`, the tunnel helper creates a localhost forwarding endpoint. Start the guest service first, then keep its tunnel open for as long as the client needs it: ```python # sb must use a transport that implements forwarding, such as SSH or ADB. async with sb.tunnel.forward(8080) as tunnel: print(tunnel.url) # Connect your local client before leaving this context. ``` For several ports, `sb.tunnel.forward(8080, 9222)` returns a dictionary keyed by guest port. When using `tunnel = await sb.tunnel.forward(8080)` without a context manager, call `await tunnel.close()` after the client finishes. ### Android abstract sockets ADB also accepts a guest abstract socket name, such as the DevTools socket of a running Android Chrome instance: ```python # sb must be connected to the Android guest through ADBTransport. async with sb.tunnel.forward('chrome_devtools_remote') as tunnel: print(tunnel.url) ``` A socket name or integer port does not enable forwarding on an unsupported transport. See [Sandbox SDK interfaces]() for the tunnel handle's fields and cleanup behavior. --- # Share a sandbox service with a signed URL Create, list, and revoke time-limited public URLs for sandbox services with the TypeScript Fleet SDK. Use a signed service URL when a person or external system needs temporary access to a service running inside a Fleet sandbox. The URL is public, time-limited, and revocable. Requests still route only to the named service on the bound sandbox. **Warning** A signed service URL is a bearer credential. Anyone who has it can access the service until the URL expires or is revoked. Do not put signed URLs in source control, logs, analytics events, or public chat channels. ## Prerequisites - A Fleet pool whose template exposes the service you want to share. - A bound sandbox claim. The signed URL does not create or keep a claim alive. - OAuth client credentials with permission to manage the claim. - The published `@trycua/fleet` 0.1.1 TypeScript package and a Node.js environment with `fetch` and `AbortSignal.timeout`. The published Python `cua-sandbox` 0.4.3 package exposes `sandbox.services.request()`, but does not expose `create_signed_url()`, `list_signed_urls()`, or `revoke_signed_url()`. Use the TypeScript API below for signed URLs. See the [service reference](). If you do not have a pool yet, create one with [Python]() or [TypeScript](). The service name passed to the signed URL API must match a service declared on that pool's template. Signed URLs accept expiration times from 60 seconds through 24 hours. The optional label is useful for recording why a link was created and is limited to 120 UTF-8 bytes. ## Authenticate Obtain credentials through [Fleet authentication](). This example uses OAuth client credentials and the `https://run.cua.ai` Fleet endpoint. Set the credentials and resource names before running it: ```bash export CUA_CLIENT_ID="" export CUA_CLIENT_SECRET="" export CUA_POOL_NAME="my-team-pool" export CUA_CLAIM_NAME="signed-url-demo" ``` Set `CUA_FLEET_BASE_URL` or `CUA_TOKEN_URL` only when you use non-default Fleet or OAuth endpoints. ## Create and revoke a signed URL The example claims a sandbox from an existing pool, creates a one-hour URL for the `mcp` service, waits while you use the URL, and then revokes it before releasing the claim. Install the Fleet SDK and a TypeScript runner: ```bash npm install @trycua/fleet@0.1.1 npm install --save-dev tsx typescript npm pkg set type=module ``` Save the following script as `share-service.ts`: ```ts title="share-service.ts" import { createInterface } from 'node:readline/promises'; import { CreateClaimRequestBuilder, CyclopsClient, CyclopsCredentials, FetchHttpClient, type Claim, type CyclopsConfiguration, type SignedServiceUrl, uniffiInitAsync, } from '@trycua/fleet/node'; const BASE_URL = process.env.CUA_FLEET_BASE_URL ?? 'https://run.cua.ai'; const TOKEN_URL = process.env.CUA_TOKEN_URL ?? 'https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token'; const CLIENT_ID = requiredEnv('CUA_CLIENT_ID'); const CLIENT_SECRET = requiredEnv('CUA_CLIENT_SECRET'); const POOL_NAME = requiredEnv('CUA_POOL_NAME'); const CLAIM_NAME = process.env.CUA_CLAIM_NAME ?? 'signed-url-demo'; const SERVICE_NAME = 'mcp'; function requiredEnv(name: string): string { const value = process.env[name]; if (!value) throw new Error(`Missing environment variable: ${name}`); return value; } await uniffiInitAsync(); const configuration: CyclopsConfiguration = { baseUrl: BASE_URL, tokenUrl: TOKEN_URL, credentials: new CyclopsCredentials(CLIENT_ID, CLIENT_SECRET), poolPollIntervalMs: 5_000n, poolPollLimit: 120, claimPollIntervalMs: 5_000n, claimPollLimit: 120, }; const client = CyclopsClient.connect(configuration, new FetchHttpClient()) as CyclopsClient; let claim: Claim | undefined; let signedUrl: SignedServiceUrl | undefined; try { const pool = await client.getPool(POOL_NAME); claim = await client.createClaim( new CreateClaimRequestBuilder().pool(pool).name(CLAIM_NAME).build() ); const sandbox = await client.waitClaim(claim); signedUrl = await client.createSignedServiceUrl({ sandbox, service: SERVICE_NAME, label: 'Customer demo', expiresInSeconds: 3600, }); console.log(`Share this URL: ${signedUrl.url}`); const urls = await client.listSignedServiceUrls(sandbox); for (const item of urls) { const state = item.revokedAt ? 'revoked' : 'available'; console.log(`${item.label ?? item.id}: ${state}, expires ${item.expiresAt}`); } const readline = createInterface({ input: process.stdin, output: process.stdout }); await readline.question('Press Enter to revoke the URL... '); readline.close(); } finally { try { if (signedUrl) await client.revokeSignedServiceUrl(signedUrl); } finally { try { if (claim) await client.deleteClaim(claim); } finally { client.uniffiDestroy(); } } } ``` Run it: ```bash npx tsx share-service.ts ``` The Fleet client manages signed URLs directly. Keep the returned `SignedServiceUrl` record because `revokeSignedServiceUrl()` uses it to identify the URL and namespace. The nested cleanup blocks revoke the URL, release the claim, and destroy the native client in that order. ## Use the URL safely - Keep the claim bound while clients use the URL. Releasing or deleting the claim can remove the backing service before the URL expires. - Choose the shortest practical expiration time. Create a new URL instead of extending access through a long-lived link. - Use labels that identify the recipient or purpose without including secrets or personal data. - Revoke the URL immediately when sharing is complete. Expiration is a fallback, not a substitute for revocation. - Treat a `503` response from URL management operations as signed URLs being unavailable in the current Fleet environment. ## Troubleshoot ### The SDK reports an unknown service The service name must be present in the bound sandbox's service list. Reconcile the pool template with the service and target port, then create a new claim. ### The URL stops working before its expiration time Confirm that the claim is still bound and that the backing service still exists. A signed URL grants access to a service; it does not extend the claim's lifecycle. ### The SDK says signed service URLs are unavailable Confirm that the Fleet environment has signed service URLs configured. The TypeScript `@trycua/fleet` 0.1.1 package includes the create, list, and revoke operations, but package availability does not enable the feature on the server. The Python `cua-sandbox` 0.4.3 service interface does not include these methods. --- # Choose and build a sandbox image Select a sandbox base image and add packages, files, setup commands, ports, and inspection. Use an **Image** to define the OS and software environment for a sandbox before you start it. Start with [Choose a sandbox image]() if you need to decide between using a published Fleet artifact, customizing a local image, or publishing a Fleet artifact. For Fleet, [choose a published image](#choose-a-published-fleet-image). For local execution, [choose a base image](#choose-a-base-image) and add the setup steps below. **Warning** **SDK builder customization is local-only in `cua-sandbox` 0.4.3.** `apt_install()`, `pip_install()`, `env()`, `run()`, `copy()` and the other builder methods are applied when you boot the image locally (`local=True`). Fleet runs a prebuilt registry artifact and rejects a customized one with `NotImplementedError: Fleet cloud supports registry images with optional exposed services only`. For Fleet customization, [prepare and publish the guest image first](). See [Sandbox runtime support]() for versioned acceptance and verification limits. ## Choose a published Fleet image 1. Open the [OS and image catalog]() and choose a Fleet VM artifact that matches your guest OS and workload. Check its architecture and evidence limits. Local Docker and Lume entries are separate runtime choices, not built-in Fleet mappings. 2. Check the artifact's guest services against your client. The Sandbox SDK's default connection expects computer-server on port `8000`; an MCP client needs an image that starts an MCP service. Declaring a port does not install either service. See the [Fleet guest-service contract](). 3. Use the built-in constructor for a mapped artifact, or copy the exact registry reference into `Image.from_registry()`. Set the guest OS and `kind='vm'` explicitly. Use a digest when you need an identity independent of tag changes. 4. Follow [Create a sandbox pool with Python]() to apply the image and claim a replica. Confirm registry access, guest readiness, and the operation your workload needs before reusing that pool. For an explicit registry reference, set `FLEET_IMAGE_REF` to the artifact you selected, then construct the image specification: ```python import os from cua_sandbox import Image image = Image.from_registry( os.environ['FLEET_IMAGE_REF'], os_type='linux', kind='vm', ) ``` For a Windows artifact, change `os_type` to `'windows'`; this selects EFI in the generated Fleet template. This snippet defines an image without provisioning it. For image-specific service configuration, use the catalogue's linked recipe. Windows images do not include a Windows license. Before provisioning one, confirm that your own licenses cover your intended use, including applicable hosting and virtualization rights. A successful boot is not a licensing check. If the artifact lacks required software, [prepare and publish a guest image]() before creating the pool. The local builder examples below do not publish a Fleet artifact or run during a Fleet claim. ## Choose a base image An Image describes the OS and software environment. Images are _immutable_ and _chainable_. Each builder method returns a new Image. Start with the built-in constructor that matches the operating system and isolation level you need. ```python from cua import Image Image.linux() # Ubuntu 24.04 VM (default, QEMU) Image.linux(kind='container') # Ubuntu 24.04 container (lighter, Docker/XFCE) Image.linux('ubuntu', '22.04') # Older Ubuntu Image.macos() # macOS Tahoe (version '26') Image.macos('15') # macOS Sequoia Image.windows() # Windows Server 2022 Image.windows('11') # Windows 11 (local ISO install only) Image.android() # Android 14 ``` Choose a constructor using the [OS and runtime table](). The built-in Ubuntu 24.04 and Windows Server 2022 VMs map to registry artifacts for Fleet; the other built-in inputs do not. A constructor returning an `Image` does not verify that its guest can boot on your host or Fleet deployment. For Windows, released 0.4.3 selects EFI in the Fleet template. See [Windows firmware]() before diagnosing an unready Windows guest as a BIOS mismatch. If a local Docker image times out on readiness, check the guest server's listening address. An earlier `trycua/cua-xfce:latest` recipe reported the server bound to `127.0.0.1:8000` inside the container, which Docker's published port could not reach. That observation does not establish the state of every later mutable image tag. ### How a Linux image runs locally `Image.linux()` is a VM. Locally it boots the same pinned containerDisk that Cua Fleet references, under QEMU on the host. Sharing an artifact does not make host networking, firmware, resources, or available SDK operations identical: ```python from cua import Image, Sandbox async with Sandbox.ephemeral(Image.linux(), local=True) as sb: print((await sb.shell.run('uname -a')).stdout) ``` That path needs `qemu-system-x86_64` on the host, and uses `/dev/kvm` when it is available. Without QEMU the call raises rather than quietly starting something else: ``` RuntimeError: Image.linux() is a VM and needs QEMU, which was not found on this host. ``` `Image.linux(kind='container')` is the Docker path instead, and needs no QEMU. **Warning** Passing `runtime=QEMURuntime()` explicitly selects `mode='docker'`, which wraps QEMU in a container and does **not** boot the pinned containerDisk. Use `QEMURuntime(mode='bare-metal')` if you are naming a runtime by hand, or leave `runtime=` off and let the default pick it. ## Install packages Use the package manager for the target operating system. ```python Image.linux().apt_install('curl', 'git', 'ffmpeg') Image.macos().brew_install('ffmpeg', 'jq') Image.windows().choco_install('nodejs', 'git') Image.windows().winget_install('Microsoft.VisualStudioCode') Image.android().apk_install('/path/to/app.apk') ``` Install Python packages with `pip_install()` or `uv_install()`. ```python Image.linux().pip_install('numpy', 'pandas', 'playwright') Image.linux().uv_install('numpy', 'pandas') # faster, installs into cua-server project ``` ## Set environment variables Set non-sensitive environment variables with `.env()`. ```python Image.linux().env(DATABASE_URL='postgres://localhost/mydb', LOG_LEVEL='debug') ``` Variables are baked in as `/etc/profile.d/cua-env.sh` on Linux and macOS, and as machine-scoped `setx` variables on Windows. On Linux that file is read by _login_ shells, so a bare `sb.shell.run()` does not see them — ask for a login shell: ```python await sb.shell.run("echo $MY_TOKEN") # empty await sb.shell.run("bash -lc 'echo $MY_TOKEN'") # abc123 ``` **Warning** Avoid putting real secrets in .env() because anyone who can read the Image spec can see the values. ## Run setup commands Run arbitrary shell commands during setup with `.run()`. ```python Image.linux().run('curl -fsSL https://deb.nodesource.com/setup_20.x | bash -') ``` ## Copy files Copy local files into the image with `.copy()`. ```python Image.linux().copy('./config.json', '/app/config.json') ``` ## Expose ports Expose every port the sandbox workload must listen on. ```python Image.linux().expose(8080).expose(5432) ``` **Warning** In 0.4.3, bare-metal QEMU forwards exposed ports at startup and reports their host ports in `sb.exposed_ports`. Other local backends can differ. Fleet creates named services instead. `expose()` does not make `tunnel.forward()` available on every transport; see [ports and transports](). ## Chain builder calls Chaining: builder calls are composable, each returns a new Image: ```python img = ( Image.linux() .apt_install('curl', 'git', 'ffmpeg') .pip_install('requests', 'Pillow') .env(MY_TOKEN='abc123', DEBUG='1') .run('mkdir -p /app/data') .expose(8080) ) ``` Fork a base image for different use cases: ```python base = Image.linux().apt_install('curl', 'git') dev = base.pip_install('ipython', 'rich').env(DEBUG='1') prod = base.pip_install('gunicorn').run('useradd -m appuser') ``` ## Use a custom OCI image Use `Image.from_registry()` when your base image already exists in a registry. The reference must be a **KubeVirt containerDisk** — an image carrying a bootable disk at `/disk/disk.img`. An ordinary OCI application image such as `ubuntu:22.04` is not one, and fails with `FileNotFoundError: OCI image 'ubuntu:22.04' does not contain /disk/disk.img`. ```python img = Image.from_registry( 'registry.example/workspace@sha256:...', os_type='linux', kind='vm', ) # Local setup only; Fleet rejects builder layers. img = img.run('echo hi') ``` A registry image defaults to `kind=None`, so the SDK cannot automatically select a runtime unless you set the kind. For local use, select a runtime compatible with the actual guest disk. The Linux VM example uses bare-metal QEMU: ```python from cua import QEMURuntime, Sandbox async with Sandbox.ephemeral(img, local=True, runtime=QEMURuntime(mode='bare-metal')) as sb: ... ``` **Warning** Replace the example reference with an artifact you published. `Image.from_registry()` defaults to `os_type='linux'`; set `os_type='windows', kind='vm'` for a Windows VM disk. The registry reference does not prove that the artifact is accessible or bootable. Local pull authentication and Fleet registry access are separate requirements. ## Use a local disk image Use `Image.from_file()` for qcow2, vhdx, raw, or iso disk images. URLs are also supported and cached automatically. ```python Image.from_file('/path/to/disk.qcow2', os_type='linux') Image.from_file('/path/to/windows.vhdx', os_type='windows') Image.from_file('https://example.com/disk.qcow2', os_type='linux') ``` Disk images downloaded from a URL are cached in `~/.cua/cua-sandbox/image-cache/`, keyed by a hash of the URL. This is a different cache from the one used for registry containerDisks, which land in `~/.cua/cua-sandbox/images/container-disks//disk.qcow2`. Delete either directory to force a re-download. ## Inspect an image Call `.to_dict()` to inspect the final image spec before using it. ```python img = Image.linux().apt_install('curl').pip_install('requests') print(img.to_dict()) # {'os_type': 'linux', 'distro': 'ubuntu', 'version': '24.04', 'kind': 'vm', # 'layers': [{'type': 'apt_install', 'packages': ['curl']}, # {'type': 'pip_install', 'packages': ['requests']}]} ``` `env`, `ports` and `files` keys appear only when you have set them: ```python img = Image.linux().env(DEBUG='1').expose(8080).copy('./a.json', '/app/a.json') print(img.to_dict()) # {..., 'kind': 'vm', 'layers': [], 'env': {'DEBUG': '1'}, # 'ports': [8080], 'files': [['./a.json', '/app/a.json']]} ``` --- # Prepare and reference a Fleet image Publish a Fleet-compatible OCI image and reference it from the Sandbox SDK. Fleet does not build a sandbox environment from the full SDK image-builder surface. Instead, Fleet expects a **prebuilt OCI artifact** that already carries the bootable guest image for the pool. **Note** Configure [Fleet pool credentials]() before running the SDK example. The image-based `Sandbox.ephemeral()` path in `cua-sandbox` 0.4.3 accepts only the default region, `us-east-1`. ## Before you start This guide uses `cua-sandbox` 0.4.3 through `cua` 0.1.6. Choose a bootable guest artifact compatible with your Fleet deployment and a registry that deployment can pull from. SDK acceptance does not establish registry access or successful boot. You can use the built-in Ubuntu 24.04 or Windows Server 2022 VM image without publishing your own artifact. Follow this guide when you need software or startup configuration baked into a custom image. See [Sandbox runtime support]() for the accepted image inputs and customization limits. **Warning** Treat Fleet as a published-image workflow. Build first, publish first, then reference the resulting registry artifact from the SDK. ## 1. Build the guest image outside the claim path Prepare the operating system, installed software, startup behavior, and service layout before you ask Fleet to boot it. The published OCI artifact must already carry the bootable guest image Fleet expects. The Sandbox SDK's Fleet VM template references a containerDisk artifact, with a guest disk at `/disk/disk.img`; an ordinary application container is not a substitute. The SDK does not build that artifact, convert a disk file, or apply SDK layers on your behalf. ## 2. Publish the OCI artifact immutably Push the finished artifact to a registry and prefer an immutable digest over a mutable tag. ```text registry.example/fleet/workspace@sha256:... ``` Pinned digests make it possible to say exactly which boot image a pool launched. Mutable tags weaken that guarantee because the same pool spec can resolve to different guest images over time. ## 3. Reference the published image from the SDK Use `Image.from_registry(...)` with any additional Fleet services declared through `.expose(...)`. With the SDK's default service configuration used here, your published image must already start the computer-server service on port `8000`. The generated template declares the `server` service and probes that port for readiness; the SDK also waits for its `/status` endpoint. Extra `.expose(...)` ports declare additional services your image starts. Explicit pool configurations can use a different service layout; see the [Pool reference](). Replace the example reference with your published Linux VM artifact. For a Windows VM artifact, use `os_type='windows', kind='vm'`; the OS hint controls the generated firmware configuration and is not inferred from the disk. ```python import asyncio from cua import Image, Sandbox async def main(): image = ( Image.from_registry( "registry.example/fleet/workspace@sha256:...", os_type="linux", kind="vm", ) .expose(3000) ) async with Sandbox.ephemeral(image) as sb: result = await sb.shell.run("uname -a") print(result.stdout) asyncio.run(main()) ``` ## 4. Expose only the services your image actually starts `expose()` is how you declare additional named Fleet services that your guest brings up after boot. If your image only needs the default computer-server endpoint on `8000`, you do not need extra exposed ports. If it serves additional application traffic on another port, expose that port on the registry image you reference. ## 5. Request the Fleet service through the authenticated connection Within the sandbox context, use `sb.services.request()` to send an authenticated HTTP request to the named service. Replace `/healthz` with a route your service implements: ```python response = await sb.services.request("port-3000", method="GET", path="/healthz") response.raise_for_status() print(response.text) ``` The image-based `Sandbox.ephemeral()` example creates a temporary pool and claim and uses the same `FleetTransport` as `Pool.claim()` and `Sandbox.create(pool=...)`. It does not create a localhost listener, and that transport does not implement `tunnel.forward()` in 0.4.3. See [ports and transports](). ## Related pages - [How Fleet images work]() - [Sandbox runtime support]() - [Image reference]() --- # Open an interactive shell in a sandbox Open an interactive PTY session inside the code half of a sandbox from the CLI or Python SDK. A **Cua Sandbox** is a full isolated computer where an agent can run code and drive the GUI. Use `cua do shell` or the Sandbox SDK terminal API to open an interactive PTY session in the code half of that sandbox, similar to SSH but without network setup. The same sandbox is also driven through the GUI via computer-use with screenshots, clicks, and typing. Because both share one machine, you can run code and automate the UI against the same filesystem and state. ## From the CLI Switch to the sandbox target first. ```bash # Switch to a target first cua do switch docker my-container # or cua do switch host # Open an interactive shell (bash / PowerShell on host) cua do shell # Run a specific program interactively cua do shell python3 cua do shell vim /etc/hosts # Override terminal dimensions cua do shell --cols 220 --rows 50 ``` The shell is fully interactive: arrow keys, tab completion, and Ctrl+C work. When stdin is piped (non-TTY), the command runs non-interactively. ## From the Python SDK Create a sandbox, open a PTY session with `sb.terminal.create`, send input as text, read session info, then close the session. ```python from cua import Sandbox, Image async with Sandbox.ephemeral(Image.linux(), local=True) as sb: session = await sb.terminal.create(cols=80, rows=24) pid = session['pid'] await sb.terminal.send_input(pid, 'echo hello\n') info = await sb.terminal.info(pid) print(info) closed = await sb.terminal.close(pid) print(f'closed: {closed}') ``` Pass a `command` to run a specific program instead of the default login shell: ```python session = await sb.terminal.create(command='python3', cols=80, rows=24) ``` ## Reference See [Sandbox SDK interfaces]() for the full terminal method signatures and return shapes. --- # Run Minecraft in a Windows sandbox Boot a Windows sandbox, install Minecraft Java Edition, and drive it with an agent through the cua-driver MCP server running inside the sandbox. Minecraft exercises almost everything a Windows sandbox can do: it needs internet access, a Java runtime, working OpenGL, and a GUI that only clicks can drive. This guide boots a Windows sandbox, installs Minecraft Java Edition, and hands it to an agent that talks to **cua-driver's MCP server inside the sandbox** — the same loop against a local sandbox and against Fleet. The timings, guest hardware, graphics behavior, and deployment errors below are observations from this recipe's environment. They are not a certification of every Fleet deployment or local host. The [runtime support reference]() separates those observations from the released 0.4.3 image and transport contracts. ## Before you start - **cua-sandbox 0.4.3** for the examples below, including the explicit Fleet pool and claim workflow. The local recipe was originally developed against 0.3.3; its recorded timings and guest behavior are observations from that environment, not a new end-to-end test of 0.4.3. - **A host with hardware virtualisation** for the local path — a Linux x86_64 machine with `/dev/kvm`, or an Intel Mac. This guide passes `-cpu host`, which QEMU only accepts with KVM or HVF. An x86_64 guest on Apple Silicon runs under TCG emulation, where `-cpu host` is rejected outright. The Fleet path runs there instead, including the game, with the one extra environment variable described in the Fleet section below. - **A Microsoft account that owns Minecraft Java Edition.** Signing in uses Microsoft device authorization, so one step in the middle is manual: a code appears inside the sandbox and you approve it in your own browser. - **A vision-capable LLM endpoint** for the agent loop. ## Boot a Windows sandbox `Image.windows()` resolves to a pinned Windows Server 2022 containerDisk. Three things get added on top of the defaults: - **`.expose(3000)`** publishes cua-driver's MCP server, which already runs inside the guest, so the agent can reach it. - **A second network interface.** The bare-metal runtime attaches its NIC with `restrict=on`, which isolates the guest. `sb.shell.run()` still works over the forwarded port, but nothing inside Windows can reach the internet — and Minecraft needs to. - **`-cpu host`.** The default `qemu64` model is too thin for a software OpenGL driver: Minecraft creates its window and then dies while loading resources, with no Java exception and no crash log. The last `-cpu` on the command line wins, so appending it is enough. ```python import asyncio from cua import Image, QEMURuntime, Sandbox EXTRA_ARGS = [ # a second, unrestricted user-mode NIC — the default one is restrict=on '-netdev', 'user,id=net1,net=10.0.3.0/24,host=10.0.3.2,dns=10.0.3.3', '-device', 'virtio-net-pci,netdev=net1,mac=52:55:00:d1:55:02', # a CPU the software OpenGL driver can actually use '-cpu', 'host', ] async def main(): sb = await Sandbox.create( Image.windows().expose(3000), name='mc-win', local=True, runtime=QEMURuntime( mode='bare-metal', cpu_count=12, memory_mb=16384, extra_args=EXTRA_ARGS, ), ) mcp_port = sb.exposed_ports[3000] print(f'cua-driver MCP on http://127.0.0.1:{mcp_port}/mcp') await sb.disconnect() # the sandbox keeps running asyncio.run(main()) ``` A warm boot takes about 30 seconds. `exposed_ports` maps each exposed guest port to the host port it landed on, and `GET /healthz` on that port answers `ok` once cua-driver is up. **Warning** **Read the port from `sb.exposed_ports` on this bare-metal QEMU path.** `sb.tunnel.forward(3000)` raises `NotImplementedError: HTTPTransport does not support port forwarding` on the local HTTP transport in 0.4.3. This runtime picks a host port at boot and saves the mapping with the sandbox state so a later `Sandbox.connect()` can read it back. The property is backend-specific. Fleet publishes services instead; use an authenticated named-service request or the service URL as shown below. **Warning** Give the second NIC its own subnet. Both user-mode networks default to `10.0.2.0/24` and both offer the guest `10.0.2.15`, so Windows drops one interface to a `169.254.x.x` link-local address with no gateway and no working DNS. Confirm the guest really has internet before installing anything. ```python async with Sandbox.connect('mc-win', local=True) as sb: check = await sb.shell.run( 'powershell -Command "(Invoke-WebRequest -UseBasicParsing ' 'https://piston-meta.mojang.com/mc/game/version_manifest.json).StatusCode"' ) print(check.stdout) # 200 ``` ## Install a launcher and a software OpenGL driver The sandbox GPU is the *Microsoft Basic Display Adapter*, which offers OpenGL 1.1. Minecraft 1.17 and later need OpenGL 3.2, so the game needs Mesa3D's `opengl32.dll` (llvmpipe), which implements OpenGL in software. Both downloads below are **MinGW** builds on purpose. The MSVC builds of Prism Launcher and Mesa both depend on the Visual C++ redistributable, which Windows Server 2022 does not ship: Prism then exits silently, and Mesa's DLL fails to load so Windows quietly falls back to the system `opengl32.dll`. ```powershell $ErrorActionPreference = 'Stop' $ProgressPreference = 'SilentlyContinue' New-Item -ItemType Directory -Force -Path C:\mc | Out-Null # Prism Launcher — signs in with Microsoft device authorization, which needs no browser Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\prism.zip ` 'https://github.com/PrismLauncher/PrismLauncher/releases/download/11.0.3/PrismLauncher-Windows-MinGW-w64-Portable-11.0.3.zip' Expand-Archive C:\mc\prism.zip -DestinationPath C:\mc\prismw -Force # 7-Zip, because Mesa ships as .7z Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\7z.msi 'https://www.7-zip.org/a/7z2408-x64.msi' Start-Process msiexec.exe -ArgumentList '/i','C:\mc\7z.msi','/qn' -Wait # Mesa3D software OpenGL Invoke-WebRequest -UseBasicParsing -OutFile C:\mc\mesa.7z ` 'https://github.com/pal1000/mesa-dist-win/releases/download/26.1.6/mesa3d-26.1.6-release-mingw.7z' & 'C:\Program Files\7-Zip\7z.exe' x C:\mc\mesa.7z -oC:\mc\mesamw -y | Out-Null Start-Process -FilePath C:\mc\prismw\prismlauncher.exe -WorkingDirectory C:\mc\prismw ``` Save that as `setup.ps1`, push it into the sandbox, and run it. It downloads roughly 100 MB, so allow a generous timeout. ```python from pathlib import Path async with Sandbox.connect('mc-win', local=True) as sb: await sb.shell.run('if not exist C:\\mc mkdir C:\\mc') await sb.files.write_text('C:\\mc\\setup.ps1', Path('setup.ps1').read_text()) result = await sb.shell.run( 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\mc\\setup.ps1', timeout=1800, ) print(result.stdout) ``` ## Sign in and create an instance Prism opens a **Quick Setup** wizard on first run. Screenshot the sandbox, click through it, and stop at the account page. ```python async with Sandbox.connect('mc-win', local=True) as sb: Path('sandbox.png').write_bytes(await sb.screenshot()) # look at it await sb.mouse.click(888, 678) # Next ``` 1. Work through the wizard to **Accounts → Add Microsoft**. Prism shows a QR code and an eight-character device code. 2. Read the code off a screenshot, open `https://www.microsoft.com/link` in your own browser, enter it, and approve the sign-in. The account then appears with status *Ready*. 3. Click **Add Instance**, search for a version such as `1.20.1`, and click **OK**. Prism downloads the client jar and assets. **Note** Device codes expire after about fifteen minutes, but Prism issues a fresh one automatically and keeps polling, so the dialog can be left open. Take a new screenshot to read the current code rather than reusing an old one. ## Point the software driver at the launcher's Java Click **Launch** once. Prism downloads its own Java runtime and the game fails with `GLFW error 65542: WGL: The driver does not appear to support OpenGL` — expected, because Mesa is not in place yet. Prism may keep using the runtime it downloaded even if you set `JavaPath` in its config, so copy the Mesa DLLs next to *every* `javaw.exe` under the install root. Windows loads `opengl32.dll` from the running executable's directory before the system directory, which is what makes this work. ```powershell $dirs = Get-ChildItem C:\mc -Recurse -Filter javaw.exe -ErrorAction SilentlyContinue | Select-Object -ExpandProperty DirectoryName -Unique foreach ($d in $dirs) { Copy-Item C:\mc\mesamw\x64\opengl32.dll, C:\mc\mesamw\x64\libgallium_wgl.dll $d -Force Write-Output "mesa -> $d" } ``` Deliver it the same way as the first script. ```python async with Sandbox.connect('mc-win', local=True) as sb: await sb.files.write_text('C:\\mc\\mesa.ps1', Path('mesa.ps1').read_text()) result = await sb.shell.run( 'powershell -NoProfile -ExecutionPolicy Bypass -File C:\\mc\\mesa.ps1', timeout=600, ) print(result.stdout) # mesa -> C:\mc\prismw\java\java-runtime-gamma\bin ``` Click **Launch** again. The Minecraft title screen appears after a minute or two. ## Drive it with an agent over MCP The sandbox already runs **cua-driver**, which serves an MCP endpoint on guest port 3000 — that is what `.expose(3000)` published. The agent is a small loop: list the MCP tools, hand them to a model as ordinary function tools, call whichever one it picks, feed the result back. Three things about cua-driver's tools shape the loop: - **A tool listing is not permission to execute a tool.** In the guest used for this recipe, `list_tools()` advertised 55 tools over both the local and Fleet transports, including tools the guest's YAML policy refused. A denied call returned `Permission denied: user policy: tool 'X' is not allowed by the YAML policy`. Check the policy and handle permission errors on the image you use; this observed tool list is not a contract for other Driver versions. See [Restrict tool access](). - **Clicks are addressed to an application, not the screen.** `click(pid=..., x=..., y=...)` targets a window belonging to that pid, which you find with `list_windows`. Add `delivery_mode='foreground'` when a background-delivered click does not land. - **There is no wait tool.** The loop waits by calling `get_desktop_state` again, so say that in the system prompt or the model will invent something worse. ```python import asyncio, json, os import litellm from fastmcp import Client from fastmcp.client.transports import StreamableHttpTransport SYSTEM = """You operate a computer through the provided tools. The desktop is Windows at 1280x800. Work in a loop: look at the screen with get_desktop_state, decide one action, call one tool, then look again. * click / type_text / press_key act on a specific application, addressed by `pid`. Use list_windows to find the pid, then pass pid with x/y. * There is no wait tool. If something is still loading, call get_desktop_state again — repeated looks are how you wait. Call exactly one tool per turn. When the task is complete, reply DONE.""" async def complete(**kwargs): """Two workarounds for the endpoint used here — yours may need neither. It is streaming-only (a plain request comes back with empty output), and it rejects role=system, so the system prompt travels as the first user turn. """ messages, system = [], [] for m in kwargs['messages']: (system if m.get('role') == 'system' else messages).append(m) if system: text = '\n\n'.join(m['content'] for m in system) messages = [{'role': 'user', 'content': text}] + messages kwargs['messages'] = messages stream = await litellm.acompletion(**kwargs, stream=True) chunks = [c async for c in stream] return litellm.stream_chunk_builder(chunks, messages=messages) def prune_images(messages, keep=3): """Each get_desktop_state returns a full screenshot; keep only the newest.""" seen = 0 for msg in reversed(messages): if not isinstance(msg.get('content'), list): continue for part in msg['content']: if part.get('type') == 'image_url': seen += 1 if seen > keep: part.clear() part.update({'type': 'text', 'text': '[older screenshot dropped]'}) return messages async def run(mcp_url, task, model, max_steps=60, headers=None): client = Client(StreamableHttpTransport(mcp_url, headers=headers)) async with client: # list_tools() returned 55 tools on the image used here, most of them # browser and recording plumbing this task never needs. Hand the model # only what the job requires: dozens of schemas is a lot of context to # spend, and a shorter menu is a shorter list of ways to go wrong. Worth # doing whether or not your driver already filters denied tools out. wanted = { 'get_desktop_state', 'list_windows', 'list_apps', 'click', 'double_click', 'type_text', 'press_key', 'hotkey', 'launch_app', 'bring_to_front', 'scroll', } mcp_tools = [t for t in await client.list_tools() if t.name in wanted] tools = [{ 'type': 'function', 'function': { 'name': t.name, 'description': (t.description or '')[:800], 'parameters': t.inputSchema or {'type': 'object', 'properties': {}}, }, } for t in mcp_tools] messages = [{'role': 'system', 'content': SYSTEM}, {'role': 'user', 'content': task}] for _ in range(max_steps): resp = await complete( model=model, messages=prune_images(messages), tools=tools, tool_choice='auto', temperature=0.0, ) msg = resp.choices[0].message messages.append(msg.model_dump()) if not msg.tool_calls: break # model said DONE for call in msg.tool_calls: args = json.loads(call.function.arguments or '{}') result = await client.call_tool(call.function.name, args, raise_on_error=False) text = ''.join(getattr(b, 'text', '') for b in (result.content or [])) messages.append({'role': 'tool', 'tool_call_id': call.id, 'name': call.function.name, 'content': text[:1500]}) shot = next((b.data for b in (result.content or []) if getattr(b, 'data', None)), None) if shot: messages.append({'role': 'user', 'content': [ {'type': 'text', 'text': 'screenshot after that action:'}, {'type': 'image_url', 'image_url': {'url': f'data:image/png;base64,{shot}'}}, ]}) ``` Point it at the exposed port and give it the task. ```python TASK = """ Prism Launcher is open, with a Minecraft instance and a signed-in account. Select the instance and click Launch. Minecraft uses a software renderer, so the window takes minutes to appear and repaints slowly — keep calling get_desktop_state to watch it, and do not restart anything. On the title screen click Singleplayer, then Create New World, then Create New World again. Stop as soon as you are inside the world (terrain in first person, hotbar and hearts visible) and reply DONE. Never press Escape while Minecraft is in the foreground. """ asyncio.run(run(f'http://127.0.0.1:{mcp_port}/mcp', TASK, 'your-model')) ``` Because the MCP tools are presented as **ordinary function tools**, this works against endpoints that reject the provider-native computer-use tool types. That is not hypothetical: on the gateway used here, the same model with the same image in the same second returned 200 for an ordinary function tool and 403 for Anthropic's `computer_20250124`, and `computer_use_preview` was refused outright. The `complete()` wrapper above exists only for that gateway — it is streaming-only, and it rejects `role: system`. Against an endpoint without those quirks, call `litellm.acompletion` directly. A full run — launcher to standing in a new world — took 52 steps locally and 51 on Fleet, roughly twenty minutes, most of it waiting on the software renderer. Expect the model to spend long stretches doing nothing but re-screenshotting. **Note** Give the model help with coordinates. A vision model without grounding guesses pixel positions and misses: in one run an ungrounded model clicked at (1226, 210) four times, nowhere near the button it wanted, then declared it had no desktop tool. cua-driver's `list_windows` and pid-scoped clicks avoid most of this, and a grounding pass over the screenshot removes the rest. ## Publish the installed sandbox as a containerDisk Everything above is a one-time cost, and none of it has to be repeated — least of all on Fleet, where a manual GUI install is the least pleasant part of this guide. A cua sandbox boots from a **containerDisk**: an OCI image whose entire content is one file at `/disk/disk.img`. Push the disk you just built as one, and every later sandbox, local or Fleet, starts with Prism, Java, Mesa and the game files already in place. Despite the name, `/disk/disk.img` is a **qcow2**, not a raw image. The puller looks for exactly `disk/disk.img` or `./disk/disk.img` inside the layer tarball and caches whatever it finds under `~/.cua/cua-sandbox/images/container-disks/`. Nothing reads the extension — it is a KubeVirt convention. `Sandbox.snapshot()` is not a substitute for publishing a containerDisk. In `cua-sandbox` 0.4.3, it is not implemented for local sandboxes or the current Fleet creation paths. The package's legacy cloud snapshot code does not enable this workflow. See [image customization support](). **Caution** **Build the image before you sign in, never after.** A disk that has ever held a signed-in Minecraft account cannot be reliably cleaned, and a containerDisk you publish is a disk anyone can pull. Deleting Prism's `accounts.json` is not enough, and neither is deleting it and then zero-filling the volume's free space. Both were done to a disk where the game had been played, and the Microsoft profile name, the profile UUID and a full Mojang access-token JWT were still recoverable from the exported image. Mapping the byte offsets back to files with `ntfscluster` put them in three places: - **`pagefile.sys`** — most of them. The JVM heap, swapped out, holding the `--accessToken` command line and raw HTTPS response bodies from `api.minecraftservices.com`. Free-space zeroing cannot reach it, because the pagefile is an allocated file. - **File slack inside a live log.** Clusters allocated to `instances/1.20.1/minecraft/logs/latest.log` past its valid-data length still held `Setting user: ` from a longer earlier run. This is also why searching from inside the guest proves nothing: `findstr` stops at end-of-file, the disk image does not. - **Unallocated clusters the zero-fill missed**, because NTFS does not reuse every freed cluster when you write one large file. No scrub turns "my search found nothing" into "no credential is present". Build the image without ever signing in and the question does not arise — and signing in is the reader's step anyway, since every reader needs their own Microsoft account. ### Build the image without an account Follow the walkthrough above but **skip the sign-in section entirely**. Prism's Quick Setup ends on an *Add Microsoft account* page that also has a **Finish** button; click Finish. Two steps that normally happen as a side effect of signing in and launching then have to be done explicitly: - **Create the instance.** *Add Instance → Custom*, search `1.20.1`, **OK**. Prism downloads the client jar, libraries and assets with no account attached. - **Fetch Java without launching.** The walkthrough gets Prism's Java runtime by clicking Launch, which needs an account. Use *Settings → Java → Installations → **Download*** instead and pick a Mojang **Java 17** runtime — `java-runtime-gamma` `17.0.15` for 1.20.1. It lands in `C:\mc\prismw\java\java-runtime-gamma\bin\`, which is where the Mesa script then copies `opengl32.dll`. Run that script *after* this, not before. Then close the launcher and make two edits. Prism rewrites its config on exit, so doing this while it is running achieves nothing. ```powershell # Prism auto-sizes -Xmx from the *build* host's RAM. A Fleet sandbox has 4 GB. (Get-Content C:\mc\prismw\prismlauncher.cfg) -replace '^MaxMemAlloc=.*','MaxMemAlloc=2048' | Set-Content C:\mc\prismw\prismlauncher.cfg # The installers are dead weight once unpacked — 243 MB of them. Remove-Item C:\mc\prism.zip, C:\mc\mesa.7z, C:\mc\7z.msi -Force ``` **Note** Prism refuses to add an **offline** account until a Microsoft account that owns Minecraft has been added at least once — *"You must add a Microsoft account that owns Minecraft before you can add an offline account."* So there is no way to smoke-test the game on the finished image without signing into it, which is exactly what you are avoiding. Test the game on the sandbox you built it from, before the export. ### Shut the guest down from inside **Warning** **Do not stop the sandbox with `sb.stop()`.** The QEMU runtime treats the session disk as ephemeral — `runtime.start()` reads `opts.pop("ephemeral", True)` and `Sandbox.create()` never passes the flag — so `stop()` unlinks `~/.cua/cua-sandbox/images/sessions/.qcow2`, which is the disk you just spent an hour building. Starting the same sandbox name again is no safer: `create_session_disk()` unlinks and recreates the overlay every time. Shut Windows down from inside instead, and wait for the QEMU process to exit before touching the file. ```python async with Sandbox.connect('mc-win', local=True) as sb: await sb.shell.run('shutdown /s /t 0') ``` ### Export the disk The session disk is a qcow2 overlay on the base containerDisk. `qemu-img convert` flattens the chain and `-c` compresses the result, so one command produces a standalone image. ```bash qemu-img convert -O qcow2 -c ~/.cua/cua-sandbox/images/sessions/mc-win.qcow2 disk.img ``` Expect it to be slow and CPU-bound rather than I/O-bound — `-c` is single-threaded zlib. For the image built here it took **8 min 44 s** and produced **7,697,072,128 bytes (7.14 GiB)** from a 3.40 GB overlay on the 5.62 GiB base disk, 64 GiB virtual. `qemu-img` itself needs almost nothing resident — under 20 MB — so the size of the host does not matter. **Note** Zero-filling the volume's free space from inside Windows before shutting down makes the export smaller, but only if QEMU is told to discard the zeroes instead of storing them. Attach the disk with `discard=unmap,detect-zeroes=unmap` and the writes are dropped, so the source qcow2 *shrinks* rather than growing toward its 64 GiB virtual size. ```bash -drive file=.qcow2,format=qcow2,if=virtio,discard=unmap,detect-zeroes=unmap ``` Inside the guest, write zeroes to a file until the volume is nearly full and then delete it. Leave about a gigabyte of headroom; filling `C:` completely destabilises Windows. Zeroing roughly 46 GB took under two minutes on the disks here, because QEMU drops the writes rather than committing them. ### Build the OCI image and push it The Dockerfile is two lines, and `FROM scratch` is not an optimisation — a containerDisk must contain nothing else. ```dockerfile FROM scratch ADD disk.img /disk/disk.img ``` ```bash docker buildx build --provenance=false --sbom=false \ -t ghcr.io//minecraft-workspace:1.20.1 --push . ``` That took **6 min 28 s** here — 3 min 48 s exporting the layer and 1 min 57 s pushing it. The layer came out at 7,631,366,006 bytes as `application/vnd.oci.image.layer.v1.tar+gzip`: gzip buys essentially nothing on a qcow2 that is already zlib-compressed, so budget for pushing the full size rather than expecting the progress bar to outrun it. `--provenance=false --sbom=false` suppresses buildx's attestation manifests. With them off, buildx publishes a plain `application/vnd.oci.image.manifest.v1+json` and no image index at all, which is the simplest thing for a single-platform disk to be. cua's puller does follow an index and skips attestation children — it filters on `os == "linux"` and on the `vnd.docker.reference.type` annotation — so an index is not fatal, but there is no reason to publish one here. **Warning** **A GHCR package is private when first pushed, and Fleet pulls anonymously.** Fleet's nodes have no credentials for your registry, so a private package fails there no matter how well `docker login` works on your own machine. Make the package public before testing on Fleet — *Package settings → Change visibility → Public* — and only publish an image you are willing to hand to strangers, which is what the sign-in warning above is about. A `gh auth login` token does not carry `write:packages`. `docker login ghcr.io` still succeeds with it, and the push then fails at the very end with `denied: permission_denied: The token provided does not match expected scopes`. Run `gh auth refresh -h github.com -s write:packages` first, or use a PAT that has the scope. ### Boot the published image `os_type` is what selects firmware on both paths: the local runtime only looks for OVMF when it is `"windows"`, and the Fleet transport only sets `Firmware.EFI` for it. A Windows containerDisk that claims to be Linux boots SeaBIOS against a GPT/ESP disk and never reaches the readiness probe, so say which one it is: ```python from cua import Image, QEMURuntime, Sandbox REF = 'ghcr.io//minecraft-workspace:1.20.1' IMAGE = Image.from_registry(REF, os_type='windows', kind='vm').expose(3000) ``` `IMAGE` is then a drop-in replacement for `Image.windows().expose(3000)` in both snippets earlier in this guide — the local `Sandbox.create(..., local=True, runtime=QEMURuntime(...))` call and the Fleet one. Nothing else changes: the same `EXTRA_ARGS` locally, the same agent loop, and on Fleet the same `GALLIUM_DRIVER=softpipe`. **Warning** A Fleet pool claim carries `FleetTransport`, which does not implement `sb.tunnel.forward(3000)` in 0.4.3. Image-based `Sandbox.ephemeral()` uses a pool claim too, and `Sandbox.create(image)` requires an explicitly named pool. For HTTP requests, use `sb.services.request('port-3000', method='GET', path='/healthz')`. A streaming MCP client instead needs the service URL and authentication shown below. Booted locally, that image printed `Image(windows/registry:latest, kind=vm, ...)`, came up on the first try, and had everything in it: `instances\1.20.1`, `java\java-runtime-gamma`, Mesa's `opengl32.dll` beside `javaw.exe`, `minecraft-1.20.1-client.jar`, `MaxMemAlloc=2048` — and `Test-Path C:\mc\prismw\accounts.json` returning `False`. Opening Prism shows the Quick Setup account page and *No accounts added!*, which is what a correctly-built image looks like. **Note** Use a registry that speaks **HTTPS**. The puller goes through `oras`, which never tries plain HTTP, so a scratch `docker run registry:2` on `localhost:5000` fails with `SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number')` before it ever fetches a manifest. Give the registry a certificate and point `REQUESTS_CA_BUNDLE` at the CA if you want to rehearse this locally. Expect the first Fleet boot on a given node to be slow: it has to pull the whole image before the sandbox can start, and Fleet enforces a **300-second bind deadline** that `time_to_start=` does not extend, so a cold pull can surface as `BindDeadlineExceeded: no adoptable Sandbox within 300s`. A second failure to expect is `403 k8s request is not allowed` on `update template`. A template can be **created but never updated**: both branches of the gateway's image policy are guarded by `input.method != "PATCH"`, so any request that takes the update path is refused, with the same opaque message you get for a disallowed image. Reusing the sandbox name guarantees it. A fresh name usually avoids it but **not reliably** — of four boots of the same image, one reused name and one fresh name both returned 403, while two other fresh names reached `READY` in 157 s and 187 s. Retry; it is intermittent, and nothing about your image changes the outcome. The SDK surfaces these 403s as `PoolAccessDeniedError`. The same error also appears when the pool name you chose is already owned by **another account** — pool names are globally unique across accounts — and in that case the fix is a different pool name, not a retry. The reader's remaining work is the part that has to be theirs: open Prism, *Accounts → Add Microsoft*, approve the device code, and click Launch. ## Run the same thing on Fleet The image and the agent loop are identical on Fleet. Two things change: there is no `local=True` and no `runtime=`, and the MCP endpoint is reached through Fleet's service proxy rather than a forwarded localhost port. `.expose(3000)` becomes a Fleet **service** named `port-3000`. Configure [Fleet pool credentials]() before running this `cua-sandbox` 0.4.3 example. It creates a dedicated pool with a unique name and claims a sandbox. The proxy URL uses the returned namespace and bound sandbox name, not the claim name. The MCP client sends a bearer token; the URL alone does not grant access. ```python import httpx import os import uuid from urllib.parse import quote from cua import Image, Sandbox from cua_sandbox import Pool pool = await Pool.apply( Image.windows().expose(3000), name=f'minecraft-{uuid.uuid4().hex[:12]}' ) try: async with Sandbox.ephemeral(pool=pool, name='mc-fleet', time_to_start=900) as sb: health = await sb.services.request('port-3000', method='GET', path='/healthz') health.raise_for_status() reference = sb.to_dict() namespace = quote(reference['namespace'], safe='') sandbox_name = quote(sb.name, safe='') service_url = f'https://run.cua.ai/api/svc/{namespace}/{sandbox_name}-port-3000/' async with httpx.AsyncClient(timeout=30.0) as client: response = await client.post( 'https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token', data={'grant_type': 'client_credentials', 'client_id': os.environ['CUA_CLIENT_ID'], 'client_secret': os.environ['CUA_CLIENT_SECRET']}, ) response.raise_for_status() token = response.json()['access_token'] await run(service_url + 'mcp', TASK, 'your-model', headers={'Authorization': f'Bearer {token}'}) finally: await pool.delete() ``` That is the same `run()` as above, with only the URL and an auth header changed. The context releases the claim, and `finally` deletes this example's dedicated pool. For interrupted runs, follow the [Fleet cleanup guidance](). In this recipe's environment, a Fleet Windows sandbox took about three minutes to become ready, against about thirty seconds for a warm local one. `GET healthz` on that service URL answers `ok` when cua-driver is up. Notice there is no `-cpu host` here. That flag exists on the local path because the bare-metal runtime defaults to the thin `qemu64` model; a Fleet sandbox is provisioned for you and already reports a full host CPU — `Intel Xeon Processor (SapphireRapids)`, with AVX, AVX2 and AVX-512 all present — so there is nothing to override, and no QEMU arguments to pass. `Pool.apply` also accepts `cpu` and `memory_mb`, but sizing a Fleet sandbox is account-dependent: passing them routes the request through a gated custom-resource path, which returned `403 create pool: k8s request is not allowed` at every size until a card-on-file requirement was lifted for the account. The default sandbox — 4 vCPU and 4 GB — is what this guide was written against, and it is enough. **Warning** **On Fleet the game needs one extra environment variable: `GALLIUM_DRIVER=softpipe`.** With Mesa's default llvmpipe renderer, Minecraft dies during resource loading every time — `Process crashed with exitcode -2147024809`, no Java exception, no `hs_err` file, nothing in the Windows event log, the log simply stopping after `Reloading ResourceManager`. Switching Mesa to its `softpipe` rasteriser fixes it, and the game runs. Set the variable in the process that launches the launcher, so the game inherits it — a machine-level variable does not reach an already-running process, and a test that silently did not apply looks exactly like a test that failed. `setup.ps1` above already started Prism without it, so close the running launcher first and start it again like this. ```powershell $env:GALLIUM_DRIVER = 'softpipe' Start-Process -FilePath C:\mc\prismw\prismlauncher.exe -WorkingDirectory C:\mc\prismw ``` Softpipe is a reference rasteriser with no JIT, so it is **considerably slower than llvmpipe** — allow several minutes for the title screen and longer again for world generation. **Warning** Use client credentials, not a `cua auth login` session token. The session token is short-lived and is held without refresh, so a long provisioning wait dies partway with `401 auth token is invalid`. Client-credential tokens expire too — the ones issued here came back with `expires_in` of 900 seconds — so a run longer than that has to re-mint the token and rebuild the MCP client. What that crash is *not*, since each obvious explanation was tested and eliminated: not the Minecraft version or the Java/LWJGL generation (1.20.1 on Java 17 with LWJGL 3 and 1.12.2 on Java 8 with LWJGL 2 fail identically), not the heap (forcing Prism's auto-sized `-Xmx2717m` down to `-Xmx1024m` changed nothing), and not the size of the machine (the same local session disk rebooted with `-m 4096 -smp 4`, matching Fleet exactly, runs the game fine). Narrowing llvmpipe's vectors with `LP_NATIVE_VECTOR_WIDTH=128` did **not** help either, which argues the fault is not simply wide-vector code generation. ## Troubleshooting | Symptom | Cause | Fix | |---------|-------|-----| | QEMU refuses to start with `-cpu host` | no KVM or HVF — for example an x86_64 guest on Apple Silicon, which runs under TCG | use a host with hardware virtualisation, or the Fleet path | | Launcher never appears, no error | MSVC build without the VC++ redistributable | use the MinGW portable build | | Guest has an IP address but cannot resolve names | both user-mode NICs offered the same address | give the second NIC its own subnet | | `GLFW error 65542: WGL: The driver does not appear to support OpenGL` | Mesa DLLs missing beside the `javaw.exe` actually in use, or the MSVC Mesa build failed to load | copy the MinGW Mesa DLLs into every `javaw.exe` directory | | Game exits during resource loading with no Java exception, **local** | default `qemu64` CPU model | append `-cpu host` to `extra_args` | | Game exits during resource loading with `exitcode -2147024809`, **Fleet** | Mesa's default llvmpipe renderer | set `GALLIUM_DRIVER=softpipe` in the process that starts the launcher, and restart the launcher if it is already running | | `Permission denied: user policy: tool 'X' is not allowed` | The guest's YAML policy refuses that tool; the recipe's guest advertised some denied tools in `list_tools()` | Check the policy and use an allowed tool; do not infer permission from the listing alone | | Model replies with empty output on the first call | endpoint is streaming-only | issue `stream=True` and rebuild with `litellm.stream_chunk_builder` | | Sandbox from `Image.from_registry()` never becomes ready | `os_type` defaults to `"linux"`, so a Windows disk gets BIOS instead of UEFI | pass `os_type="windows", kind="vm"` (needs cua-sandbox 0.3.3; before that, `dataclasses.replace()` on the result) | | `SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number')` while pulling | the registry speaks plain HTTP; `oras` only speaks HTTPS | give the registry a certificate, and set `REQUESTS_CA_BUNDLE` for a self-signed one | | `denied: permission_denied: The token provided does not match expected scopes` at the end of a push | the `gh` OAuth token carries no `write:packages` | `gh auth refresh -h github.com -s write:packages`, or use a PAT that has it | | Fleet cannot pull the image you just published | GHCR packages are private on first push, and Fleet pulls anonymously | make the package public | | The session disk vanished after a run | `stop()` unlinks the ephemeral session overlay, and starting the same name recreates it | shut the guest down from inside, and copy the qcow2 before anything else touches it | --- # Install and run Lume Install Lume, Cua's local Apple Silicon VM manager for macOS and Linux guests. Lume is Cua's local VM manager for Apple Silicon Macs. Use it when you want to create, run, or serve macOS and Linux VMs on your own machine. It can provide local VM substrate for Cua workflows, but the primary product choice is still [Cua Driver]() for an existing machine or [Cua Sandbox]() for a fresh isolated cloud desktop. ## Requirements - Apple Silicon Mac (M1, M2, M3, etc.) - macOS 13.0 or later - At least 8GB of RAM (16GB recommended) - At least 50GB of free disk space ## Install Lume Install with a single command: ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/lume/install.sh)" ``` ## Verify the install ```bash lume --version ``` ## Update Lume Check whether a newer release is available: ```bash lume check-update ``` Apply an available update: ```bash lume update --apply ``` The installer no longer creates scheduled auto-updaters. Older `lume-update` cron jobs and LaunchAgents are removed the next time the installer runs. ## Choose a release channel To select nightlies for future checks and explicit updates, install the newest nightly and persist the channel in one command: ```bash curl -fsSL https://cua.ai/lume/install.sh | bash -s -- --channel nightly ``` Later checks and explicit updates stay on nightly. Switch back with: ```bash lume channel set stable lume update --apply ``` `channel set` never replaces the running binary by itself. Stable remains the default on machines without saved channel state. ### Pin one exact nightly Nightlies are immutable builds of exact `main` commits. Copy the full `nightly-lume-v…` tag from the component's GitHub release and pass it as an explicit, one-shot pin: ```bash curl -fsSL https://cua.ai/lume/install.sh | \ LUME_VERSION=nightly-lume-v0.5.4-nightly.20260812.123456789 bash ``` The pin does not change the saved channel and cannot be combined with `--channel`. Stable discovery remains blind to nightly tags, and nightly discovery remains blind to stable tags. ## Manual installation You can also download the `lume.pkg.tar.gz` archive from the [latest release](https://github.com/trycua/cua/releases?q=lume&expanded=true), extract it, and install the package manually. ## Next steps - [Create your first local VM](): follow the Tahoe happy path from IPSW to SSH. - [Create a vanilla Tahoe VM](): create, run, inspect, and remove a local VM. - [Manage local VMs](): share files, use storage locations, and clone disks. - [Change SIP on a macOS VM](): change the signed policy from paired Recovery. - [Serve the Lume API](): run the local HTTP API for tools and scripts. - [Use Lume with MCP](): connect an AI client to local VM tools. - [Lume CLI reference](): inspect every command and option. - [Lume HTTP API reference](): use the local API from tools and scripts. --- # Create a vanilla Tahoe VM Create and manage a vanilla macOS Tahoe VM from a local Apple restore image. Use this guide when you already have Lume installed and need a repeatable local Tahoe VM. The offline unattended setup does not drive Setup Assistant through the display. ## Download an IPSW Use the latest supported Apple restore image, or provide an IPSW you already downloaded: ```bash IPSW_URL="$(lume ipsw | tail -n 1)" curl -L "$IPSW_URL" -o ~/Downloads/macos-tahoe.ipsw ``` ## Create the VM ```bash lume create macos-tahoe \ --ipsw ~/Downloads/macos-tahoe.ipsw \ --unattended tahoe ``` The `tahoe` preset creates the `lume` user, enables SSH, configures autologin, and disables sleep and screen locking. The default credentials are `lume` / `lume`. ## Run and inspect it ```bash lume run macos-tahoe lume ls lume ssh macos-tahoe 'sw_vers; id -un' ``` `lume run` opens the native viewer by default while keeping VNC available for automation and remote access. Use `--display vnc` to open Screen Sharing instead, or `--display none` (or `--no-display`) to start without a viewer. ## Stop or remove it ```bash lume shutdown macos-tahoe # graceful guest shutdown lume stop macos-tahoe # immediate process-level stop lume delete macos-tahoe ``` Deletion removes the VM disk and its configuration. Clone the VM first if you need a reusable copy: ```bash lume clone macos-tahoe macos-tahoe-backup ``` ## Troubleshoot common failures Run Lume as the same user that owns its VM directory. `sudo lume` uses a different home directory and cannot see VMs created under your account. If Lume reports that auxiliary storage is locked, another `lume run` process is still using the VM. Stop that process or run `lume stop macos-tahoe` before starting it again. Tahoe is the verified unattended preset. Sequoia may still show the Accessibility step of Setup Assistant on its first display boot; track that behavior in [issue #2155](https://github.com/trycua/cua/issues/2155). Continue with [Manage local VMs](), [Serve the Lume API](), or [Use Lume with MCP](). --- # Run Omarchy on Apple Silicon Build an experimental ARM64 Omarchy VM from official source with Lume, Arch Linux ARM, and optional Rosetta translation. You can run the official Omarchy 4.0.1 source in an ARM64 Linux VM backed by Apple's Virtualization.framework. The tested VM boots into Hyprland, renders the Omarchy shell, accepts keyboard and pointer input, runs native ARM64 applications, and translates self-contained x86-64 Linux programs with Rosetta. For how Omarchy, Hyprland, and Wayland fit together, read [Linux desktops and computer use](). The VM results in this guide do not certify Cua Driver application support; see the separate [Hyprland support entry](). **Warning: Experimental compatibility install** This is official Omarchy source installed on an Arch Linux ARM base. It is not an official Omarchy ARM64 release, and it does not use Omarchy's stock ISO installer. The published Omarchy ISO and package repository are x86-64-only. ![Omarchy running in an ARM64 Lume VM]() ## Before you start You need: - an Apple Silicon Mac running macOS 13 or later; - at least 50 GiB of free disk space; - [Lume 0.5.3 or later](); and - GnuPG and a BLAKE2b-512 checksum tool for installer verification. The tested VM used eight virtual CPUs, 8 GiB of memory, an 80 GiB disk, Arch Linux ARM kernel `7.2.0-2-aarch64-ARCH`, and Omarchy tag `v4.0.1`. The tag resolves to source commit `13f18b2cb7286fb54f87daf571a031aa6af3d8f0`. ## Understand the limitations The tested installation has these known gaps: - Graphics use Mesa `llvmpipe`. OpenGL reports `Accelerated: no`, and Vulkan cannot enumerate a physical device. - Lume exposes a virtio-sound device, but the tested Arch Linux ARM kernel has `CONFIG_SND_VIRTIO` disabled. PipeWire therefore exposes only `Dummy Output`. - 27 package names from Omarchy's base set are absent from the tested ARM repositories. A follow-up audit found that 16 are packaging or substitution gaps, seven are plausible native source builds, one needs an alternative, and three are not useful in this VM. These paths are not yet E2E-validated. - Rosetta translates x86-64 user-space programs. It cannot boot the x86-64 Omarchy ISO, translate its kernel or bootloader, or supply missing x86-64 shared libraries. ## Download and verify Archboot The official Omarchy ISO cannot boot an ARM64 VM. Download the tested ARM64 Archboot environment instead: ```bash curl -fLO https://release.archboot.com/aarch64/2026.08/iso/archboot-2026.08.25-02.28-7.2.0-2-aarch64-ARCH-aarch64.iso curl -fLO https://release.archboot.com/aarch64/2026.08/b2sum.txt curl -fLO https://release.archboot.com/aarch64/2026.08/b2sum.txt.sig ``` Import the signing key, inspect its fingerprint, and verify the signed checksum file: ```bash gpg --keyserver hkps://keyserver.ubuntu.com \ --recv-keys 5B7E3FB71B7F10329A1C03AB771DF6627EDF681F gpg --fingerprint 5B7E3FB71B7F10329A1C03AB771DF6627EDF681F gpg --verify b2sum.txt.sig b2sum.txt ``` The tested signature was valid for Tobias Powalowski's key with fingerprint `5B7E 3FB7 1B7F 1032 9A1C 03AB 771D F662 7EDF 681F`. Importing a key from a keyserver does not establish its identity. Verify the fingerprint through an independent Archboot or Arch Linux channel before trusting it. Verify the ISO against the signed file: ```bash b2sum archboot-2026.08.25-02.28-7.2.0-2-aarch64-ARCH-aarch64.iso ``` The tested BLAKE2b-512 digest is: ```text 562bd4c54d879d7d2b80f565186b3804456f99a6e3913f7eda7bf9b0e9f132d6e9fcbd083998cb3dcbbf065528a2a404c4ce6151f90b3228dade788237ae1bc7 ``` ## Create and boot the VM Create a blank ARM64 Linux VM: ```bash lume create omarchy-arm64 \ --os linux \ --cpu 8 \ --memory 8GB \ --disk-size 80GB \ --display 1920x1200 ``` Boot the Archboot ISO and open its display: ```bash lume run omarchy-arm64 \ --mount "$PWD/archboot-2026.08.25-02.28-7.2.0-2-aarch64-ARCH-aarch64.iso" \ --display vnc ``` Wait for networking, then open a root shell in Archboot. ## Install Arch Linux ARM Download the companion installation script inside the Archboot shell: ```bash curl -fLO https://raw.githubusercontent.com/trycua/cua/main/docs/public/scripts/omarchy-arm64-lume/install-arch.sh chmod +x install-arch.sh TIMEZONE=UTC ./install-arch.sh /dev/vda ``` **Warning: The script erases the VM disk** `install-arch.sh` repartitions and formats all of `/dev/vda`. Run it only in the new Lume VM. The script rejects other disk names and requires you to type `ERASE` before it proceeds. Replace `UTC` with a valid name from `/usr/share/zoneinfo` if needed. The script installs `archlinuxarm-keyring` with the base system because the ARM repositories use their own signing keys. Set a password for the desktop user, unmount the installed system, and turn off the guest: ```bash arch-chroot /mnt passwd omarchy umount -R /mnt poweroff ``` Boot the VM without the ISO, then sign in as `omarchy` on the text console: ```bash lume run omarchy-arm64 --display vnc ``` ## Install the pinned Omarchy source Download and run the compatibility installer in the ARM64 guest: ```bash curl -fLO https://raw.githubusercontent.com/trycua/cua/main/docs/public/scripts/omarchy-arm64-lume/install-omarchy.sh chmod +x install-omarchy.sh sudo ./install-omarchy.sh sudo reboot ``` The script installs the tested native ARM64 package subset, checks out official Omarchy source commit `13f18b2cb7286fb54f87daf571a031aa6af3d8f0`, installs the desktop configuration, and forces Hyprland onto its software-rendering path. The guest should reboot directly into the Omarchy desktop. ## Enable Rosetta translation **Optional:** If Rosetta is installed on the macOS host, Lume 0.5.3 attaches its translator as a `rosetta` virtiofs share. Mount the share and register it with Linux `binfmt_misc`: ```bash curl -fLO https://raw.githubusercontent.com/trycua/cua/main/docs/public/scripts/omarchy-arm64-lume/enable-rosetta.sh chmod +x enable-rosetta.sh sudo ./enable-rosetta.sh sudo reboot ``` Verify transparent translation with the tested, self-contained x86-64 BusyBox binary: ```bash curl -fsSL \ https://busybox.net/downloads/binaries/1.35.0-x86_64-linux-musl/busybox \ -o /tmp/busybox-x86_64 echo '6e123e7f3202a8c1e9b1f94d8941580a25135382b99e8d3e34fb858bba311348 /tmp/busybox-x86_64' \ | sha256sum -c - chmod +x /tmp/busybox-x86_64 file /tmp/busybox-x86_64 /tmp/busybox-x86_64 uname -m ``` The final command should print `x86_64`. Dynamically linked x86-64 programs also need compatible x86-64 libraries. Rosetta provides translation, not a complete x86-64 Linux filesystem. ## Verify the installation Check the architecture, kernel, desktop processes, pinned source, and network: ```bash uname -m uname -r pgrep -a Hyprland pgrep -a quickshell sudo git -C /usr/share/omarchy rev-parse HEAD curl -fsSI https://example.com | sed -n '1p' ``` The tested VM reports `aarch64`, runs Hyprland and Quickshell, returns the pinned Omarchy commit, and receives an HTTP success response. Keyboard and pointer input also work through Lume's VNC display. ![Text entered into Foot through the Lume VNC display]() Native ARM64 Chromium can load HTTPS pages: ![Chromium loading example.com in the Omarchy VM]() ## Inspect graphics and audio Inspect the graphics path from a terminal in the desktop session: ```bash glxinfo -B vulkaninfo --summary ``` The tested VM reports software rendering through `llvmpipe`, no acceleration, and `ERROR_INITIALIZATION_FAILED` when Vulkan tries to enumerate a device. Inspect audio support: ```bash wpctl status aplay -l zgrep CONFIG_SND_VIRTIO /proc/config.gz ``` The tested kernel reports `# CONFIG_SND_VIRTIO is not set`, ALSA finds no sound cards, and PipeWire exposes only `Dummy Output`. A guest kernel built with `CONFIG_SND_VIRTIO` is the likely path to audio support because Lume attaches a virtio-sound PCI device. ## Review package compatibility The compatibility installer records the 27 package names that were absent from the tested repositories in `/var/log/omarchy-v4.0.1-unavailable-arm64-packages.txt`: ```bash cat /var/log/omarchy-v4.0.1-unavailable-arm64-packages.txt ``` The raw count does not represent 27 hard blockers. An August 26, 2026 source and package-metadata audit produced these categories: | Category | Count | Packages | Assessment | | ----------------------------- | ----: | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Packaging or substitution | 16 | `aether`, `cliamp`, `dotnet-runtime`, `herdr`, `hyprland-preview-share-picker`, `localsend`, `mise-bin`, `nvim`, `omarchy-nvim`, `tobi-try`, `ttf-ia-writer`, `ttf-jetbrains-mono-nerd-basic`, `ufw-docker`, `xdg-terminal-exec`, `yaru-icon-theme`, `yay` | Upstream recipes or releases declare ARM64 support, the package is architecture-independent, or a native substitute already exists. `nvim` is covered by the installed `neovim` package. | | Native source-build candidate | 7 | `omacalc`, `omacut`, `omawrite`, `pinta`, `tensaku`, `ttfx`, `tzupdate` | Their C++, .NET, or Rust sources are plausible ARM64 builds, but their current Omarchy or AUR package is x86-64-only or has an ARM64 dependency gap. Each still needs a reproducible build and runtime test. | | Alternative needed | 1 | `obsidian` | Obsidian does not publish a native Linux ARM64 desktop artifact. Use its web app, another Markdown editor, or test the x86-64 app with Rosetta and a compatible x86-64 userspace. | | Defer for this VM | 3 | `asdcontrol`, `obs-studio`, `qemu-user-static-binfmt` | `asdcontrol` targets physical Apple displays; Rosetta covers the tested x86-64 translation use case instead of QEMU; and OBS is not useful here until accelerated graphics and audio work. | The low-risk group is based on package metadata, not a completed install matrix. For example, current AUR recipes for `aether`, `cliamp`, `herdr`, `hyprland-preview-share-picker`, `localsend`, `mise-bin`, and `yay` explicitly include `aarch64`; several font, configuration, and shell packages declare `any`; and Microsoft publishes a Linux ARM64 .NET runtime. Omarchy's package repository itself still has no `aarch64` database, so these packages need an ARM64 build channel or installation path before this guide can include them by default. Do not install arbitrary x86-64 packages into the ARM root filesystem. Add one package at a time, preserve its source and license, and verify its desktop integration after installation. ## What you built The result is a reboot-persistent Omarchy desktop with native ARM64 applications and optional x86-64 user-space translation. Its provenance is official Omarchy 4.0.1 source pinned to a specific commit and adapted onto an Arch Linux ARM base. It remains an experimental compatibility installation with software-only graphics, no audio on the tested kernel, and incomplete package integration. ## Related guides - [Install Lume]() - [Create a vanilla Linux VM]() - [Manage local Lume VMs]() - [Lume CLI reference]() --- # Run OpenClaw in a local macOS VM Install OpenClaw in an isolated Lume VM and keep its gateway running headlessly. This guide shows you how to run an OpenClaw gateway in a local macOS VM on an Apple Silicon Mac. The VM keeps OpenClaw separate from your daily macOS environment and can run headlessly after setup. ## Before you start - Use an Apple Silicon Mac with at least 60GB of free disk space. - [Install Lume](). - Have credentials for the model provider you will select during OpenClaw onboarding. ## Create the VM Create a macOS Tahoe VM with Lume's unattended setup: ```bash lume create openclaw --ipsw latest --unattended tahoe ``` The unattended preset creates a `lume` account, enables SSH, and configures automatic login. Its initial username and password are both `lume`. Start the VM without opening the VNC client: ```bash lume run --no-display openclaw ``` ## Install OpenClaw in the VM Open an interactive shell in the guest: ```bash lume ssh openclaw ``` Run the official OpenClaw installer without its automatic onboarding step, then start onboarding explicitly so it installs the gateway service: ```bash curl -fsSL --proto '=https' --tlsv1.2 https://openclaw.ai/install.sh \ | bash -s -- --no-onboard openclaw onboard --install-daemon ``` Choose your model provider and channels when prompted. OpenClaw stores its configuration inside the VM. Verify the installation before leaving the guest shell: ```bash openclaw --version openclaw doctor openclaw gateway status ``` Exit the guest shell when the gateway reports that it is running: ```bash exit ``` ## Access the gateway from the host OpenClaw binds its gateway to the guest loopback interface by default. Get the VM IP address: ```bash lume get openclaw ``` In a separate host terminal, forward the default gateway port through SSH: ```bash ssh -N -L 18789:127.0.0.1:18789 lume@ ``` Keep that command running and open `http://127.0.0.1:18789` on the host. Enter the gateway credentials created during onboarding when prompted. ## Add macOS-only channels Run the VM with its display when you need to sign in to Messages or grant macOS permissions: ```bash lume stop openclaw lume run openclaw ``` For iMessage, follow OpenClaw's [iMessage setup guide](https://docs.openclaw.ai/providers/imessage). Messages must be signed in inside the VM, and the process running OpenClaw needs Full Disk Access and Automation permission. Return the VM to headless operation after you finish the interactive setup: ```bash lume stop openclaw lume run --no-display openclaw ``` ## Save a reusable VM Stop and clone the configured VM before making further changes: ```bash lume stop openclaw lume clone openclaw openclaw-golden lume run --no-display openclaw ``` The clone contains the guest disk and its OpenClaw configuration. Protect it as you would the original VM because it may contain provider and channel credentials. ## Troubleshoot the setup ### `lume ssh` cannot connect Run `lume ls` and confirm that `openclaw` is running and has an IP address. If the VM is still booting, wait and run `lume ssh openclaw` again. ### The gateway does not start after boot Run `lume ssh openclaw 'openclaw gateway status'`. If the service is missing, open an interactive guest shell and run `openclaw gateway install`. ### A macOS permission prompt never appears Restart the VM with `lume run openclaw`, open its display with `lume attach openclaw`, then run the command that needs permission from a guest terminal in that logged-in session. ## Related guides - [Manage local Lume VMs]() - [How Lume unattended setup works]() - [Lume CLI reference]() - [OpenClaw macOS VM guide](https://docs.openclaw.ai/install/macos-vm) --- # Manage local Lume VMs Run, inspect, clone, share files with, and remove local Lume VMs. Use this guide after you have created a VM and need the everyday lifecycle and storage commands. ## Inspect VMs List all VMs: ```bash lume ls ``` Get details for one VM: ```bash lume get macos-tahoe ``` The details include the VM state, IP address, display session, resources, and SSH availability when the VM is running. ## Run, attach to, and stop a VM Run Tahoe with the low-latency native viewer. VNC remains available for automation and remote access, and the owning process stays attached to the terminal: ```bash lume run macos-tahoe ``` Use `--detach` to run it in the background without opening a viewer. Lume prints its PID and writes logs to `~/Library/Logs/lume/macos-tahoe.log` by default: ```bash lume run macos-tahoe --detach ``` VNC remains active in the background. From another terminal, attach the preferred viewer; Lume uses its native viewer when the owning process supports live attachment and otherwise falls back to VNC: ```bash lume attach macos-tahoe ``` Select a viewer explicitly when needed: ```bash lume attach macos-tahoe --display native lume attach macos-tahoe --display vnc ``` Select Screen Sharing or suppress the startup viewer explicitly when needed: ```bash lume run macos-tahoe --display vnc lume run macos-tahoe --display none ``` Closing the native window only hides it; it does not stop the guest. Reopen it from Lume's Dock icon, **Window → Show VM Window**, or another `lume attach` command. The older `--no-display` flag remains an alias for `--display none`. ### Run without a VNC server `--display` chooses a local viewer, but Lume still starts a VNC server in every display mode so automation and `lume attach --display vnc` keep working. Opt out of the VNC server entirely with `--vnc disabled`: ```bash lume run macos-tahoe --display none --vnc disabled ``` That run binds no VNC port, writes no VNC password to the guest, and reports `"vncUrl": null` in `lume get` and `lume list`. `lume stop`, `lume delete`, and running status still work across processes, including for `--detach` runs. `--vnc disabled` is rejected together with `--display vnc`, `--vnc-port`, or `--vnc-password`, since each of those needs a VNC server. Guest clipboard shortcuts in the native viewer travel over VNC, so they are unavailable in a VNC-disabled run. ### Copy files into a native guest Drop one or more files or folders from Finder onto the native VM screen. Lume shows the transfer status while recursively copying the items over SSH to `/Users/lume/Desktop`. File drop is available only in the native viewer and transfers from host to guest. SSH for the default `lume` guest account must be available. Lume rejects the entire drop rather than overwriting an existing Desktop item; items in the same drop must also have distinct names. Gracefully restart or shut down a guest over SSH: ```bash lume restart macos-tahoe lume shutdown macos-tahoe ``` Keep `stop` for an immediate process-level stop: ```bash lume stop macos-tahoe ``` Only one `lume run` process can hold a VM's auxiliary storage at a time. If a second process reports that auxiliary storage is locked, stop the existing process before starting the VM again. ## Share a host directory Pass a host directory when you run the VM: ```bash lume run macos-tahoe --shared-dir ~/Projects ``` The directory appears in the guest at `/Volumes/My Shared Files`. In the native viewer, **Share Folder** can also add a directory while the guest is running, including the first folder added to a VM that started without shared folders; no guest restart or remount is needed. Folders added from the toolbar last for that run only. Use a read-only mount when the guest only needs to read the files: ```bash lume run macos-tahoe --shared-dir ~/Projects:ro ``` ## Store VMs on another volume Add a named storage location and make it the default: ```bash lume config storage add external /Volumes/External/lume lume config storage list lume config storage default external ``` Commands also accept a storage name directly: ```bash lume create macos-tahoe \ --ipsw ~/Downloads/macos-tahoe.ipsw \ --unattended tahoe \ --storage external lume run macos-tahoe --storage external ``` The default VM directory is `~/.lume`. Run Lume as the same user that owns this directory; `sudo lume` uses a different home directory and cannot see those VMs. ## Expand a VM disk Follow [Expand a VM disk]() to preview the change, grow the disk, and verify the guest capacity. ## Clone and remove VMs Clone a VM before making changes or using it as a reusable starting point: ```bash lume clone macos-tahoe macos-tahoe-backup ``` Remove a VM and its disk when you no longer need it: ```bash lume delete macos-tahoe-backup ``` macOS VM disks can be sparse. The configured capacity and the host space used by the disk can therefore differ. To disable or enable System Integrity Protection, follow [Change SIP on a macOS VM](). --- # Expand a VM disk Increase a stopped Lume VM's disk and verify the guest capacity. Use this guide to increase an existing VM's total disk capacity. Lume does not support shrinking VM disks. ## Preview the resize Stop the VM and validate the proposed size without changing the disk: ```bash lume stop macos-tahoe lume set macos-tahoe --disk-size 120GB --dry-run ``` Resolve any reported layout, encryption, host-space, or VM-state error before continuing. ## Expand the disk Set the new total capacity: ```bash lume set macos-tahoe --disk-size 120GB ``` Lume creates a rollback copy for macOS VMs. The operation can take several minutes while Lume preserves RecoveryOS and expands the APFS container. Use `--keep-backup` to retain the rollback files after success: ```bash lume set macos-tahoe --disk-size 120GB --keep-backup ``` Use `--no-backup` only when you have another verified copy of the VM. Automatic rollback is unavailable when a resize without a backup fails. ## Verify the guest capacity Boot the VM and inspect its root container: ```bash lume run --no-display macos-tahoe lume ssh macos-tahoe 'diskutil info / | grep "Container Total Space"' lume stop macos-tahoe ``` For Linux VMs, grow the guest partition and filesystem after Lume increases the image. The exact commands depend on the partition table and filesystem. See [Disk resizing limits]() for supported macOS layouts. Read [How Lume expands macOS disks]() for the RecoveryOS and rollback model. --- # Change SIP on a macOS VM Disable or enable System Integrity Protection from paired macOS Recovery and verify the result. Use `lume sip` to change System Integrity Protection (SIP) on a stopped macOS VM. Lume runs `csrutil` in the VM's paired Recovery environment and verifies the result after a normal boot. For the policy model behind this workflow, read [How SIP works in Lume VMs](). ## Before you start The VM must: - run macOS; - be stopped; - have a working administrator account; and - have Remote Login enabled. VMs created with Lume's unattended setup meet these requirements. Their initial username and password are both `lume`. Install the optional Recovery VNC driver: ```bash python3 -m pip install --user vncdotool ``` Recovery input currently supports administrator passwords made from lowercase ASCII letters, digits, and hyphens. The Recovery password prompt must name the account passed with `--admin-user`. ## Disable SIP With the default `lume` account, run: ```bash lume sip off macos-tahoe ``` Confirm the change when prompted. Lume validates the account, changes the policy in Recovery, boots normally to run `csrutil status`, and leaves the VM stopped. A successful run ends with: ```text System Integrity Protection status: disabled. SIP is disabled. ``` For another administrator account, enter its password without exposing it in the process arguments: ```bash lume sip off macos-tahoe \ --admin-user admin \ --admin-password-stdin ``` For non-interactive use, provide confirmation with `--yes` so the password is the only line read from standard input: ```bash printf '%s\n' "$LUME_ADMIN_PASSWORD" | \ lume sip off macos-tahoe \ --admin-user admin \ --admin-password-stdin \ --yes ``` ## Enable SIP Run the same command with `on`: ```bash lume sip on macos-tahoe \ --admin-user admin \ --admin-password-stdin ``` A successful run ends with the canonical enabled status and leaves the VM stopped. ## Prepare a reusable SIP-disabled VM Change SIP on a stopped seed, then clone it: ```bash lume sip off macos-tahoe-seed --yes lume clone macos-tahoe-seed macos-tahoe-worker-001 ``` Lume copies `disk.img` and `nvram.bin` together. Do not replace either file independently because the signed policy, personalized boot files, and anti-replay state belong to one paired VM state. Boot a worker and inspect its policy when you need an extra check: ```bash lume run --no-display macos-tahoe-worker-001 lume ssh macos-tahoe-worker-001 'csrutil status' lume stop macos-tahoe-worker-001 ``` ## Related pages - [How SIP works in Lume VMs]() - [Manage local Lume VMs]() - [Lume limits]() - [Lume CLI reference]() --- # Metal capability unlock Enable a configured, tested Metal capability profile for one workload inside a Lume macOS VM. Lume VMs use Apple's paravirtualized GPU bridge, but a stock macOS guest can report conservative Metal capabilities. Applications that query those values may skip newer GPU paths even when the paravirtualized device can execute the work. This guide builds a small MIT-licensed shim and activates it for one process. The shim changes selected capability answers; it does not pass a physical GPU to the guest, patch the host, or change the guest kernel. If you're looking for computer-use automation instead, start with [Cua Driver]() or [Cua Sandbox](). Read the [technical writeup](/blog/gpu-passthrough-macos-vms) for the mechanism, benchmark evidence, and architectural context. Apple's [Metal capability tables](https://developer.apple.com/metal/capabilities/) and [feature-detection guide](https://developer.apple.com/documentation/metal/detecting-gpu-features-and-metal-software-versions) explain how applications select behavior from reported device support. ## What the shim changes The tested profile can raise two values returned to the injected process: - the result of `supportsFamily:` through a configured Apple-family ceiling; and - the reported maximum threadgroup memory. The release candidate contains no timing or clock interposition, mesh substitution, ray-tracing override, private feature-profile hook, argument-layout guard, or pipeline-compilation fallback. It leaves the Common, Mac, and Metal family ranges unchanged. A missing, out-of-range, or malformed `LUME_METAL_APPLE_FAMILY_MAX` leaves the process on its stock capability path. Reported support is not proof that every Metal API associated with a family works through virtualization. Test each workload and host/guest version independently. ## Requirements - Apple Silicon host - A [Lume macOS VM]() - Xcode Command Line Tools on the machine where you build the shim - A workload you control; hardened or platform-protected executables may reject injected libraries Our release-candidate validation used Lume 0.5.1, an Apple M1 Ultra host running macOS 26.6.1, and the public Tahoe Cua guest image at macOS 26.5.2. Other combinations are experimental until tested. ## Build and verify the shim Clone the Cua repository on an Apple Silicon Mac, then build the architecture-specific dylibs and capability probe: ```bash git clone https://github.com/trycua/cua.git cd cua/libs/lume/metal-capability-shim ./Scripts/build.sh ./Scripts/verify.sh ``` `verify.sh` checks the architectures, ad-hoc code signatures, absence of research-only timing and compatibility symbols, and the generated checksums. The output is written to `dist/`: ```text dist/ ├── LumeMetalCapabilities-arm64.dylib ├── LumeMetalCapabilities-arm64e.dylib ├── SHA256SUMS └── metal-capabilities ``` Most guest workloads are `arm64`. Check the target executable before selecting a dylib: ```bash lipo -archs /absolute/path/to/your-workload shasum -a 256 -c dist/SHA256SUMS ``` ## Enable the host capability path The preference below applies to VMs launched by your macOS user. Stop the VM before changing it so the graphics device is recreated with the requested feature level: ```bash lume stop my-vm defaults write com.apple.gpusw.ParavirtualizedGraphics \ ForceUnrestrictedDeviceFeatureLevel -bool true defaults read com.apple.gpusw.ParavirtualizedGraphics \ ForceUnrestrictedDeviceFeatureLevel lume run my-vm ``` The `defaults read` command should print `1`. To restore the stock host setting later, stop the VM, delete the preference, and start the VM again: ```bash lume stop my-vm defaults delete com.apple.gpusw.ParavirtualizedGraphics \ ForceUnrestrictedDeviceFeatureLevel lume run my-vm ``` ## Copy and verify the artifact in the guest Copy the `arm64` dylib and probe into a stable, workload-specific directory. Replace the local path below with the path to your Cua checkout: ```bash lume ssh my-vm "mkdir -p '/Users/lume/.local/share/lume/metal-capabilities'" VM_IP=$(lume get my-vm --format json | jq -r '.[0].ipAddress') scp ./dist/LumeMetalCapabilities-arm64.dylib ./dist/metal-capabilities \ "lume@${VM_IP}:/Users/lume/.local/share/lume/metal-capabilities/" ``` For a standard unattended Lume image, `scp` prompts for the default guest password, `lume`. Use the credentials configured for your image if they differ. Record the local checksums, then compare them with the guest copies: ```bash shasum -a 256 dist/LumeMetalCapabilities-arm64.dylib dist/metal-capabilities lume ssh my-vm \ "shasum -a 256 \ '/Users/lume/.local/share/lume/metal-capabilities/LumeMetalCapabilities-arm64.dylib' \ '/Users/lume/.local/share/lume/metal-capabilities/metal-capabilities'" ``` ## Verify stock and unlocked capability reporting Run the probe without injection first: ```bash lume ssh my-vm \ "'/Users/lume/.local/share/lume/metal-capabilities/metal-capabilities' 1009" ``` Then run the same probe with the tested profile: ```bash lume ssh my-vm \ "DYLD_INSERT_LIBRARIES='/Users/lume/.local/share/lume/metal-capabilities/LumeMetalCapabilities-arm64.dylib' \ LUME_METAL_APPLE_FAMILY_MAX=1009 \ '/Users/lume/.local/share/lume/metal-capabilities/metal-capabilities' 1009" ``` In our Tahoe guest, `supportsFamily:1009` changed from `false` to `true`, and maximum threadgroup memory changed from 32,768 to 65,536 bytes. Your result may differ on another host or guest. Stop if the injected process crashes, reports Metal errors, or behaves incorrectly. The tested command uses these values: | Variable | Default when active | Behavior | | ----------------------------------------- | ------------------: | ------------------------------------------------------------------------------------------------- | | `LUME_METAL_APPLE_FAMILY_MAX` | required | Answer `supportsFamily:` through this Apple-family ceiling. The tested value is Apple 9 (`1009`). | | `LUME_METAL_MAX_THREADGROUP_MEMORY` | `65536` | Raise maximum threadgroup memory to at least this many bytes. | | `LUME_METAL_RECOMMENDED_WORKING_SET_SIZE` | unchanged | Raise the reported working-set size only when explicitly set. | Keep the defaults unless you have isolated evidence for another profile. Do not substitute a Common, Mac, or Metal-family enum: the shim intentionally accepts only Apple-family values from `1001` to `1999`. ## Run one workload Injection is per-process, so only the selected workload and its child processes see the changed answers: ```bash lume ssh my-vm \ "DYLD_INSERT_LIBRARIES='/Users/lume/.local/share/lume/metal-capabilities/LumeMetalCapabilities-arm64.dylib' \ LUME_METAL_APPLE_FAMILY_MAX=1009 \ /absolute/path/to/your-workload first-argument" ``` Do not use `launchctl setenv DYLD_INSERT_LIBRARIES ...`; that applies injection across the login session and can affect unrelated applications. ## Run a workload with launchd For a long-running inference server, renderer, or worker, create a per-user LaunchAgent inside the guest. The environment applies only to that program and its child processes. Save this as `/Users/lume/Library/LaunchAgents/com.trycua.lume.metal-workload.plist`, replacing the executable and arguments with absolute paths that exist in the guest: ```xml Label com.trycua.lume.metal-workload ProgramArguments /absolute/path/to/your-workload first-argument EnvironmentVariables DYLD_INSERT_LIBRARIES /Users/lume/.local/share/lume/metal-capabilities/LumeMetalCapabilities-arm64.dylib LUME_METAL_APPLE_FAMILY_MAX 1009 StandardOutPath /Users/lume/Library/Logs/Lume/metal-workload.stdout.log StandardErrorPath /Users/lume/Library/Logs/Lume/metal-workload.stderr.log ``` Each `ProgramArguments` entry is one argument. Remove `first-argument` if the program takes no arguments, or add one `` per argument. Do not combine the executable and arguments into one string. Create the required directories, validate the plist, and start the LaunchAgent from a logged-in guest session: ```bash mkdir -p "$HOME/Library/LaunchAgents" "$HOME/Library/Logs/Lume" PLIST="$HOME/Library/LaunchAgents/com.trycua.lume.metal-workload.plist" SERVICE="gui/$(id -u)/com.trycua.lume.metal-workload" plutil -lint "$PLIST" launchctl bootstrap "gui/$(id -u)" "$PLIST" launchctl kickstart -k "$SERVICE" launchctl print "$SERVICE" ``` Inspect the service output: ```bash tail -f "$HOME/Library/Logs/Lume/metal-workload.stdout.log" tail -f "$HOME/Library/Logs/Lume/metal-workload.stderr.log" ``` If the program fails after injection, unload the service and run the executable directly without the shim before debugging further: ```bash launchctl bootout "gui/$(id -u)/com.trycua.lume.metal-workload" /absolute/path/to/your-workload first-argument ``` ## Remove the configuration For a one-shot command, omit `DYLD_INSERT_LIBRARIES` and all `LUME_METAL_*` variables the next time you start the workload. For the LaunchAgent: ```bash SERVICE="gui/$(id -u)/com.trycua.lume.metal-workload" PLIST="$HOME/Library/LaunchAgents/com.trycua.lume.metal-workload.plist" launchctl bootout "$SERVICE" mv "$PLIST" "$HOME/.Trash/com.trycua.lume.metal-workload.plist" ``` Restore the host preference with the earlier `defaults delete` sequence if you no longer need the unrestricted feature level. ## Limitations - **Experimental and version-sensitive.** The shim relies on private guest Metal implementation details that may change in any macOS release. - **Per-process, not global.** Only an injected process and its children receive changed answers. - **Configured profile, not hardware discovery.** The shim raises selected Apple-family values; it does not mirror every property of the physical GPU. - **Narrow test scope.** The current release-candidate evidence covers a capability probe, two llama.cpp workloads, and one MLX-LM compatibility run on the documented M1 Ultra/Tahoe configuration. MLX-LM was already fast in the stock VM and showed no material uplift. - **Reported support is not complete support.** A positive family response does not establish that every shader, renderer, framework, or API in that family works correctly in the VM. - **Still a VM.** Existing `Virtualization.framework` limits remain. ## See also - [Lume source tree](https://github.com/trycua/cua/tree/main/libs/lume) - [Technical writeup and benchmark evidence](/blog/gpu-passthrough-macos-vms) - [Install and run Lume]() - [How SIP works in Lume VMs]() --- # Serve the Lume API Run Lume's local HTTP API for tools and scripts. Use the Lume API when another local process needs to list, create, run, or stop VMs through HTTP. ## Start the server Start the API on its default port: ```bash lume serve ``` To use another port: ```bash lume serve --port 7778 ``` The API listens on `localhost`. Keep the server bound to the local host unless you have a specific network boundary and authentication plan for exposing it. ## Check the server List the local VMs: ```bash curl http://localhost:7777/lume/vms ``` Use the [HTTP API reference]() for request bodies, responses, and the complete endpoint list. ## Run it at login The standard installer starts Lume's background service at login. Install with `--no-background-service` when you want to manage the API process yourself: ```bash /bin/bash -c "$(curl -fsSL https://cua.ai/lume/install.sh) --no-background-service" ``` After that installation, start the server explicitly with `lume serve`. --- # Use Lume with MCP Connect an MCP client to Lume for local VM management. Lume can expose its VM management tools through the Model Context Protocol (MCP). The MCP server uses stdio, so the client starts one Lume process and communicates with it directly on the host. ## Configure an MCP client Add a server entry using the path returned by `which lume`: ```json { "mcpServers": { "lume": { "command": "/Users/your-name/.local/bin/lume", "args": ["serve", "--mcp"] } } } ``` Replace the command path with the result of: ```bash which lume ``` Restart the MCP client after saving its configuration. The client can then list VMs, create Tahoe guests, run and stop VMs, clone or resize disks, and execute SSH commands inside unattended guests. See [Lume MCP tools]() for the tool schemas. ## Test the server Use the MCP Inspector to test the connection: ```bash npx @modelcontextprotocol/inspector lume serve --mcp ``` The server uses stdio for the MCP protocol. Keep its standard output reserved for MCP messages when embedding it in another client. ## Choose MCP or HTTP Use MCP when an AI client should manage VMs through tool calls. Use the [Lume HTTP API]() when a script or service should make HTTP requests. Both interfaces control the same local VM store. ## Security The MCP server can create, delete, start, stop, clone, and access VMs through SSH. Configure it only for clients you trust, and use a dedicated vanilla VM for agent tasks when the guest may be modified. --- # Run and validate a Cua-Bench task Inspect a task, verify its oracle, and exercise a variant manually. Use this guide to validate an existing task before running it with an agent or including it in a dataset. ## Before you start - Install Cua-Bench and any dependencies required by the task's provider. - Have a local task directory containing `main.py`. ## Inspect the task Load the task before starting an environment: ```bash cb task info ``` Confirm: - the provider and operating-system types are the ones you expect; - the selected variant exists; - setup and evaluation show check marks; - solve shows a check mark when the task is supposed to include an oracle. ## Verify the oracle and evaluator Run the selected variant with its oracle and close it after evaluation: ```bash cb interact \ --variant-id 0 \ --oracle \ --no-wait ``` Check the reported evaluation result against the reward the task considers successful. A failed oracle is a task or environment validation failure; do not use that result as an agent score. ## Exercise the task manually ```bash cb interact --variant-id 0 ``` Complete the task in its visible environment. Return to the terminal and press Enter to evaluate and close it. ## Check additional variants Repeat the oracle check for each variant: ```bash cb interact --variant-id --oracle --no-wait ``` Changing only `--variant-id` exercises the same lifecycle functions with the selected variant's prompt, metadata, and computer configuration. ## Troubleshooting ### Playwright reports that the browser executable is missing For a simulated task, install Chromium in the same environment as Cua-Bench: ```bash uv tool run --from 'cua-bench[browser]' playwright install chromium ``` ### A lifecycle function is missing Confirm that the task's `main.py` decorates each function with the same split. Refer to the [task definition reference]() for the supported decorators and signatures. If you have not created a task before, follow [Build your first Cua-Bench task](). --- # Automate a legacy Windows app behind a VPN Use Cua Driver on a VPN-connected Windows machine to drive a legacy desktop app that has no API. When a desktop app has **no API** and only runs **behind the corporate VPN**, drive it where it already runs, on the Windows box inside the network. **Cua Driver** runs on that machine and exposes the app through MCP stdio tools, so the desktop session, app traffic, and staged files stay inside the VPN boundary. **No data leaves the VPN boundary.** [Drive a legacy postal app with Cua Driver]() Claude Code fills shipment details and prints receipts in a Windows desktop app with no API. The [original launch thread](https://x.com/trycua/status/2059688960838828391) explains the use case. **Note** Before you start: [install Cua Driver](), [connect your agent](), and configure [Keep Cua Driver running](). If you connect over SSH, also read [Drive a Windows app over SSH](). ### Start from the Windows machine inside the VPN Use a workstation, jump box, or RDP VM that is already connected to the VPN and can open the legacy app. Connect with RDP if needed: ```powershell mstsc /v:payroll-jumpbox.corp.example ``` Confirm the app is installed or reachable from that desktop session before you install anything. For this recipe, assume the payroll client opens normally when you launch it by hand. ### Check that Cua Driver sees the real GUI Run tool calls from the same Windows machine: ```powershell cua-driver call list_apps cua-driver call list_windows ``` The output should include the legacy app, such as `Payroll Client` or `eGecko`, and its active windows. It should not be an empty array: ```json [] ``` Empty output usually means the command is running outside the interactive desktop. Recheck `cua-driver status`, `query session`, and the autostart task. ### Drive the legacy app Stage the input files on the Windows box first, for example `C:\creds.txt` and `C:\Users\you\Desktop\new_hire.xlsx`. Cua Driver does not read or write files directly. It drives the apps that open those files. Start the agent from the Windows machine, then give it a concrete task: ```text Open the payroll client, log in with the credentials in C:\creds.txt, create a new employee from new_hire.xlsx, generate the onboarding report, and export it as PDF to the Desktop. ``` The agent should use `launch_app` to open the client, `list_apps` and `list_windows` to find the running process and target window, `get_window_state` to inspect the UI (it returns the accessibility tree and a screenshot), then `click` and `type_text` to complete the workflow. ## Scale this out When the same workflow must run concurrently, use local Cua Sandbox Windows VMs and cap concurrency to the resources available on the host. Start with [Run sandboxes in parallel]() and [Sandbox lifecycle](). The driving logic stays the same. Each local Windows VM needs the VPN configuration and application included in its starting image. See [Choose and build a sandbox image](). --- # Export contacts overnight Extract LinkedIn or X contacts overnight from a logged-in browser session into a clean CSV. After a networking event, use Cua Driver on the machine where you already have a logged-in browser. The agent inherits that **real authenticated session** from the browser profile, so there are no credentials in the script and no anti-bot fight. Let it run overnight, then wake up to a CSV. **Note** Before you start: [install Cua Driver](), grant host permissions, [connect your agent](), and configure [Keep Cua Driver running](). ### Log in once On your own machine, open the browser profile that Cua Driver will drive and log into LinkedIn, X, or both. Do the normal login yourself, including any two-factor prompts. The _session persistence_ is the whole trick: the overnight job starts from a browser that is already authenticated. **Warning** Keep the machine awake for the whole run. Disable sleep, leave the browser profile available, and keep the network connection up so the overnight run does not pause. ### Give the overnight task Ask the agent for the exact extraction you want. For LinkedIn, start from the connections page and cap the run with a clear `N` while you test: ```text Using cua-driver, open the browser to https://www.linkedin.com/mynetwork/invite-connect/connections/ . For each of the first 50 connections: 1. Open the profile. 2. Read first name, last name, role, company, and profile URL. 3. Append one row to ~/contacts.csv. 4. Set met_at to "Google Devfest Toronto" for every row. Use this CSV header: first,last,role,company,met_at,linkedin If a value is missing, leave that field empty and keep going. ``` The agent will use `page` for browser navigation, then `click`, `type_text`, and `get_window_state` to inspect and drive the browser. It may call `list_windows` first if more than one browser window is open. Cua Driver **does not expose a filesystem write tool**. The agent writes `~/contacts.csv` by driving an app that can write files, for example a terminal running shell commands or a text editor with the CSV open. ### Let it run Kick off the task before you leave. In the morning, open `~/contacts.csv` and spot-check the rows before importing them into a CRM or email tool. ## Scale this out One logged-in browser profile handles one account at a time. To process several accounts, events, or platforms in parallel overnight, use one local Cua Sandbox per authenticated session and cap concurrency to the resources available on the host. Use [Run sandboxes in parallel]() to run several local desktops, [Images]() to prepare their starting state, and [Sandbox lifecycle]() for the basic local SDK lifecycle. --- # Build a report in a native app Generate a financial report in Numbers on macOS by driving the native app with Cua Driver. Numbers is a **macOS-only** app with **no clean automation API** for building a spreadsheet, charting it, and exporting a polished PDF. With Cua Driver, the agent drives the *real Numbers app on a real Mac*, the same way a person would, through the GUI. **Warning** This recipe is macOS-only. Numbers, Keynote, and Pages do not exist on Linux or Windows, so use a real Mac or a local macOS VM. **Note** Before you start: [install Cua Driver](), grant [macOS permissions](), and [connect your agent](). For long-running jobs, configure [Keep Cua Driver running](). ### Check Numbers.app On your Mac, make sure Numbers.app is installed: ```bash open -a Numbers ``` Numbers ships with macOS. If you removed it, install it again from the App Store before continuing. ### Ask the agent to build the report Give the agent the task in plain language: ```text Using cua-driver, fetch the last 30 days of AAPL daily closing prices. Open Numbers, create a new spreadsheet, enter Date and Close columns, add a 2D line chart, then export the document as a PDF to ~/reports/aapl-analysis.pdf. ``` The agent drives Numbers with `launch_app`, `list_windows`, `click`, `type_text`, `set_value`, and `get_window_state`. Cua Driver **does not include HTTP or filesystem tools**. To fetch prices, the agent can drive Terminal and run `curl`, or drive a browser and copy the data from the page. For example: ```bash curl "https://query1.finance.yahoo.com/v8/finance/chart/AAPL?interval=1d&range=1mo" ``` This Yahoo Finance endpoint is unofficial and may require a `User-Agent` header. If your data source needs authentication or a different format, have the agent use Terminal or the browser to retrieve it, then paste or enter the rows into Numbers. AppleScript can help with small app commands if the agent runs it through Terminal, but the main workflow is still the agent driving the Numbers GUI through Cua Driver. ### Confirm the PDF Confirm the exported report exists: ```bash ls -lh ~/reports/aapl-analysis.pdf ``` Open it to verify the chart and table landed in the PDF: ```bash open ~/reports/aapl-analysis.pdf ``` ## Scale this out One Mac builds one Numbers report at a time. To generate several reports concurrently, run the workflow in local macOS sandboxes backed by Lume and limit concurrency to the CPU and memory available on the host. Use [Run sandboxes in parallel]() to run the jobs concurrently, [Choose and build a sandbox image]() to prepare Numbers and any helper tools, and [Local macOS sandboxes with Lume]() for the VM foundation. Numbers cannot run in Linux containers, so each concurrent worker needs its own local macOS VM. --- # Fill a form from a local file Drive a browser form from a PDF or CSV file on your local machine. Use this recipe when the data already lives on your machine, for example in a PDF resume or a CSV file, and the form lives in a browser on the same desktop. **Cua Driver** drives both, so the **data never has to be uploaded anywhere** because the agent reads it locally and types it into the form. **Note** Before you start: [install Cua Driver](), grant host permissions, and [connect your agent](). ### Put the source file on the machine Save the source file somewhere the desktop user can open it, for example: ```text ~/Downloads/resume.pdf ``` Cua Driver has **no filesystem read tool**. The agent reads local files by opening them in an app it can see, for example Preview, another PDF viewer, or a text editor, then reading the on-screen content with the accessibility tree and screenshots from `get_window_state`. For a CSV, open the file in a text editor, spreadsheet app, or browser tab. If the file contains many rows, start with one row and name the row or record the agent should use. ### Give the agent the task In your MCP client, describe the local file, the fields to read, and the form URL: ```text Open ~/Downloads/resume.pdf in the PDF viewer. Read the name, email, phone, and work history. Then open https://form.jotform.com/ in the browser, fill each matching field from the PDF, and submit the form. ``` The agent can use `launch_app` to open the PDF viewer, `page` to drive the browser, `get_window_state` or `get_accessibility_tree` to read visible fields, and `click`, `type_text`, or `set_value` to fill the form. `get_window_state` returns a screenshot alongside the tree by default, so the agent can ground on both. ### Watch the field mapping Keep the browser visible while the agent maps source fields to form fields. If a form label is ambiguous, correct the mapping in plain language, for example "use the resume email for Work Email" or "leave Current Employer blank". **Note** Forms vary, so the agent works best when you name the exact fields to fill, such as name, email, and phone, instead of asking it to fill the whole form. ## Scale this out One desktop fills one form at a time. To submit several forms in parallel, for example one per CSV row or one per applicant, run one local Cua Sandbox per job and cap concurrency to the resources available on the host. Use [Run sandboxes in parallel]() to run the jobs concurrently. Use [Choose and build a sandbox image]() to stage the source files in an image or mount them into each sandbox. Use [Sandbox lifecycle]() for the basic local SDK lifecycle. --- # Connect your agent to Cua docs Point your coding agent at the hosted Docs & Code MCP Server so it can search Cua's docs and source. The Docs & Code MCP Server lets coding agents **ground answers and generated code** in Cua's real docs and source *instead of guessing*. Point your agent at the endpoint and it can query the actual APIs across released versions. The Docs & Code MCP Server speaks streamable-HTTP at `https://vk-mcp.cua.ai/mcp`. ## Cursor Open settings with `Shift+Command+J` on Mac, or `Shift+Ctrl+J` on Windows and Linux. Find `MCP Tools` in the sidebar. Click `Add Custom MCP` to open `mcp.json`. Add this server entry: ```json { "mcpServers": { "Cua Docs & Code": { "url": "https://vk-mcp.cua.ai/mcp" } } } ``` ## Claude Code Add the Docs & Code MCP Server over the CLI: ```bash claude mcp add --transport http cua-docs https://vk-mcp.cua.ai/mcp ``` Verify the connection: ```bash claude mcp list ``` Expect a `Connected` line for `cua-docs`. Inspect the server configuration when needed: ```bash claude mcp get cua-docs ``` Remove the local server entry when needed: ```bash claude mcp remove cua-docs -s local ``` ## Windsurf Open Windsurf Settings. Go to `Cascade > Model Context Protocol`. Click `Add Server > Add custom server`. Select `Streamable HTTP` transport. Enter this URL: ```text https://vk-mcp.cua.ai/mcp ``` Edit the config file directly when needed. On macOS and Linux, use `~/.codeium/windsurf/mcp_config.json`. On Windows, use `%USERPROFILE%\.codeium\windsurf\mcp_config.json`. ## Cline (VS Code) Open the Cline panel. Click the menu in the top right. Select `MCP Servers`. Open the `Remote Servers` tab. Enter `Cua Docs & Code` as the server name. Enter this URL: ```text https://vk-mcp.cua.ai/mcp ``` Click `Add Server`. ## GitHub Copilot Enable MCP in VS Code. Open Settings, search for `MCP`, and enable `chat.mcp.enabled`. Create `.vscode/mcp.json` in the project with this server entry: ```json { "servers": { "Cua Docs & Code": { "url": "https://vk-mcp.cua.ai/mcp" } } } ``` Open Copilot Chat. Switch to Agent mode. Use the Docs & Code MCP Server tools from the agent. ## Verify it works Once connected, the agent exposes four read-only tools: `query_docs_db`, `query_docs_vectors`, `query_code_db`, and `query_code_vectors`. Ask the agent something that forces a lookup, such as a Cua API signature in a specific release. Confirm that it calls one of the Docs & Code MCP Server tools before it answers. See the [Docs & Code MCP reference]() for tool details. --- # Reference Index of reference material for the Cua CLI, Cua Driver, Lume, the Sandbox SDK, and Cua-Bench. Reference documents the full technical specification for Cua's subsystems. It covers the `cua` CLI, `cua-driver`, Lume, the Sandbox SDK, Cua-Bench, and the Docs & Code MCP server. The pages define commands, APIs, exposed tools, types, and stated limits. | Section | What it documents | |---------|-------------------| | [Cua CLI]() | Commands for the `cua` CLI, how it authenticates and stores credentials, and the MCP server it serves to AI assistants. | | [`cua-driver`]() | CLI commands, the generated MCP tool reference for the stdio MCP server, and known behavioral limits for best-effort background automation. | | [Lume]() | CLI commands, local HTTP API, and host limits for creating, running, and managing macOS and Linux VMs on Apple Silicon Macs. | | [Sandbox SDK]() | Python API reference for building images, creating sandboxes, and driving their interfaces. | | [Cua-Bench]() | CLI command groups and the Python task-definition contract. | | [Docs & Code MCP]() | Hosted read-only search over Cua docs and versioned source code. | --- # CLI reference Command reference for the cua CLI: authentication, sandboxes, images, platforms, one-shot automation, skills, and trajectories. `cua` is the unified command-line interface for Cua. It authenticates against Cua cloud, manages sandboxes and images, drives a running machine one command at a time, and serves an MCP endpoint for AI assistants. Documented against `cua-cli` **0.1.14**. Run `cua --version` for your installed version, and `cua --help` for the options your build accepts. ## Install The CLI is published to Cua's own wheel index, so the index must be passed on the install command: ```bash pip install --extra-index-url https://wheels.cua.ai/simple cua-cli ``` Optional extras add the MCP server (`mcp`), skill recording with VLM captioning (`skills`), or both (`all`): ```bash pip install --extra-index-url https://wheels.cua.ai/simple "cua-cli[all]" ``` To keep the CLI isolated from your project environments, install it as a tool: ```bash uv tool install cua-cli --index https://wheels.cua.ai/simple ``` **Warning** On a headless Linux host, add a keyring backend at install time or the first authenticated command will fail. See [Authentication](). ## Conventions Several command groups have short aliases, and some subcommands have their own: | Canonical | Alias | |-----------|-------| | `cua sandbox` | `cua sb` | | `cua image` | `cua img` | | `cua trajectory` | `cua traj` | | `cua sb ls` | `cua sb list` | | `cua sb info` | `cua sb get` | | `cua image list` | `cua image ls` | | `cua skills list` | `cua skills ls` | Most commands that produce structured data accept `--json`. Two take `--format json` instead: `cua platform list`, and `cua image list` when it is listing local images. Commands that act on a machine take `--local` to target a local sandbox rather than a cloud one. Exit codes: | Code | Meaning | |------|---------| | `0` | Success. Also returned by bare `cua`, which prints help. | | `1` | The command ran and failed. The reason is printed as `Error: ...`. | | `2` | Argument parsing failed. `argparse` prints the usage line and the offending value. | ## Global options | Option | Description | |--------|-------------| | `-h, --help` | Show help for the CLI or for any subcommand. | | `-v, --version` | Print the installed `cua-cli` version and exit. | ## `cua auth` Manage the Cua cloud session. Tokens are stored in the operating system credential vault, never in a file the CLI writes itself. | Command | Description | |---------|-------------| | `cua auth login` | Log in through OAuth device authorization. | | `cua auth logout` | Revoke the refresh token and remove local credentials. | | `cua auth status` | Report whether a session exists and when its access token expires. | `cua auth login` takes one flag: | Option | Description | |--------|-------------| | `--no-browser` | Print the verification URL instead of trying to open a browser. | ```console $ cua auth status Logged in to run.cua.ai. Access token expires 2026-08-12T22:36:31.604247+00:00. ``` See [Authentication]() for the device flow, the headless-host workaround, and the environment variables the SDK reads. ## `cua sandbox` Create and control sandboxes, cloud or local. Aliased as `cua sb`. **Warning** **There are two different cloud paths behind these commands, and only one of them is current.** Passing `--pool` claims a sandbox from a Fleet pool, which is the supported cloud path. Passing an image without `--pool` goes through the older VM API, whose host `api.cua.ai` has been retired; those routes now live elsewhere and the CLI has not been repointed at them, so image-based cloud launches do not work today. Use `--pool` for cloud work, or `--local` for a sandbox on your own machine. ### `cua sb launch` Launch a new sandbox. Exactly one of the positional image or `--pool` is required; passing both, or neither, is an error. **Arguments:** | Name | Required | Description | |------|----------|-------------| | `` | No | Image to launch, for example `macos`, `ubuntu:24.04`, `windows:11`, or a registry reference such as `ghcr.io/trycua/mini-swe:latest`. | **Options:** | Name | Default | Description | |------|---------|-------------| | `--pool` | — | Claim the named pre-created Fleet pool instead of launching an image. Requires `--name`. | | `--local` | false | Launch a local sandbox. Requires a working local runtime. | | `--name` | generated | Sandbox name. | | `--vm` | false | Force VM kind for Linux images. Linux defaults to a container. | | `--cpu` | — | Number of vCPUs. | | `--memory` | — | Memory, as `8GB` or `4096MB`. A bare number is read as GB. | | `--disk` | — | Disk size, as `50GB`. A bare number is read as GB. | | `--region` | — | Cloud region. | | `--json` | false | Print `{"name": ..., "status": "ready"}` instead of a status line. | Bare image names are expanded before the request is made: | You write | You get | |-----------|---------| | `linux`, `ubuntu`, `debian`, `fedora` | Linux container, defaulting to Ubuntu 24.04 | | `windows`, `win` | Windows, defaulting to `2022` | | `macos`, `mac`, `osx` | macOS, defaulting to `26` | | `android` | Android, defaulting to `14` | | Anything with a registry host, such as `ghcr.io/org/image` | Pulled from that registry | A tag after a colon overrides the default version, so `ubuntu:22.04`, `windows:11`, and `macos:sequoia` all work. **Warning** **`--pool` needs Fleet credentials in the environment; logging in is not enough.** Every other cloud command reuses your `cua auth login` session, but the Fleet path does not pass that session through. Without either `FLEETS_TOKEN` or a `CUA_CLIENT_ID` and `CUA_CLIENT_SECRET` pair exported, `cua sb launch --pool` fails with: ``` Error: Fleet cloud sandboxes require CUA_CLIENT_ID and CUA_CLIENT_SECRET, or cua.configure(client_id=..., client_secret=...). ``` This is easy to misread as "you are not logged in" when `cua auth status` cheerfully reports a valid session. Export the credentials, or use `cua wif-token github` in CI, before reaching for `--pool`. ### `cua sb ls` List sandboxes. With no flags, lists cloud sandboxes. | Name | Description | |------|-------------| | `--local` | List local sandboxes. | | `--all` | List both local and cloud sandboxes. | | `--json` | Output as JSON. | ```console $ cua sb ls --local ┏━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━━━━━┓ ┃ NAME ┃ STATUS ┃ SOURCE ┃ ┡━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━━━━━┩ │ mc-win │ running │ qemu-baremetal │ │ linux-demo │ running │ docker │ └─────────────┴─────────┴────────────────┘ ``` The `SOURCE` column names the runtime backing each sandbox — `docker`, `qemu-baremetal`, `lume`, or `cloud`. **Warning** **There is no way to list cloud sandboxes, and no configuration that enables it.** Fleet has no list operation at all — with `FLEETS_TOKEN` set, `cua sb ls` refuses outright with `Listing Fleet sandboxes is not supported; use 'cua sb info NAME'.` Without it, the listing falls to the retired VM API described above and cannot succeed either. Address cloud sandboxes by name with `cua sb info`. `--local` is unaffected and lists normally. Older versions report this as `No sandboxes found.` rather than as an error, because the cloud listing swallowed its exception — so on those versions an empty cloud list is indistinguishable from a failure. Either way, an empty cloud list is not evidence that the account has no sandboxes. ### `cua sb info` Show a sandbox's name, status, and — when the provider reports them — OS, host, region, and creation time. Aliased as `cua sb get`. | Name | Description | |------|-------------| | `` | Sandbox name. Required. | | `--local` | Target a local sandbox. | | `--json` | Output as JSON. | ### Lifecycle commands Each takes a sandbox name and an optional `--local`: | Command | Description | |---------|-------------| | `cua sb suspend ` | Suspend the sandbox, preserving memory state. | | `cua sb resume ` | Resume a suspended sandbox. | | `cua sb restart ` | Restart the sandbox. | | `cua sb vnc ` | Open the sandbox's display in a browser. | ### `cua sb delete` Delete a sandbox. | Name | Description | |------|-------------| | `` | Sandbox name. Required. | | `--local` | Target a local sandbox. | | `--force` | Skip the confirmation prompt. | The prompt is also skipped automatically when stdin is not a terminal, so a piped or CI invocation deletes without `--force`. Pass it anyway to make the intent explicit. ### `cua sb shell` Open an interactive shell in the sandbox, or run a single command with a TTY attached. Everything after the sandbox name is passed through. | Name | Default | Description | |------|---------|-------------| | `` | — | Sandbox name. Required. | | `` | — | Command to run. Omit for an interactive shell. | | `--local` | false | Target a local sandbox. | | `--cols` | auto-detect | Terminal width. | | `--rows` | auto-detect | Terminal height. | ### `cua sb exec` Run a command non-interactively and exit. Use `--` before the command so its own flags are not parsed by `cua`. ```bash cua sb exec my-sandbox -- pwd ``` | Name | Description | |------|-------------| | `` | Sandbox name. Required. | | `` | Command to execute. | | `--local` | Target a local sandbox. | | `--json` | Output as JSON. | ## `cua image` Manage images. Cloud images live in your Cua workspace; local images live under `~/.local/share/cua/images/`. Aliased as `cua img`. ### `cua image list` List images. Cloud is the default. Aliased as `cua image ls`. | Name | Default | Description | |------|---------|-------------| | `--cloud` | true | List cloud images. | | `--local` | false | List local images. | | `--platform` | — | Filter local images by platform. | | `--format` | `table` | Output format for local images: `table` or `json`. | | `--json` | false | Output cloud images as JSON. | ### `cua image push` Upload a local image file to cloud storage. | Name | Default | Description | |------|---------|-------------| | `` | — | Image name. Required. | | `--file`, `-f` | `~/.local/share/cua/images//data.img` | Path to the image file. | | `--tag` | `latest` | Image tag. | | `--type` | `qcow2` | Image type: `qcow2`, `raw`, or `vmdk`. | ### `cua image pull` Download an image from cloud storage. | Name | Default | Description | |------|---------|-------------| | `` | — | Image name. Required. | | `--tag` | `latest` | Image tag. | | `--output`, `-o` | — | Output file path. | ### `cua image delete` Delete an image. Cloud by default. | Name | Default | Description | |------|---------|-------------| | `` | — | Image name. Required. | | `--tag` | `latest` | Image tag, for cloud images. | | `--local` | false | Delete a local image instead. | | `--force` | false | Skip confirmation. | ### `cua image create` Build a local image from a platform definition. See [`cua platform`](#cua-platform) for the platform list and each one's requirements. | Name | Default | Description | |------|---------|-------------| | `` | — | Platform name, for example `linux-docker` or `windows-qemu`. Required. | | `--name` | same as platform | Image name. | | `--iso` | — | Path to an ISO, for QEMU platforms. | | `--download-iso` | false | Download the Windows 11 ISO (~6 GB). | | `--docker-image` | platform default | Override the Docker image. | | `--distro` | `ubuntu` | Linux distribution: `ubuntu` or `fedora`. | | `--version` | `14` | OS version, for example `14` for Android or `sonoma` for macOS. | | `--disk` | `64G` | Disk size. | | `--memory` | `8G` | Memory. | | `--cpus` | `8` | CPU cores. | | `--winarena-apps` | false | Install the WinArena benchmark apps. | | `--detach`, `-d` | false | Run in the background. | | `--force` | false | Recreate an existing image. | | `--skip-pull` | false | Do not pull the Docker image. | | `--no-kvm` | false | Disable KVM acceleration. | | `--vnc-port` | auto from 8006 | VNC port. | | `--api-port` | auto from 5000 | API port. | ### Local image commands | Command | Description | |---------|-------------| | `cua image info ` | Show a local image's details. | | `cua image clone ` | Clone a local image. Add `--force` to overwrite the target. | | `cua image shell ` | Boot the image and open a shell in it. | `cua image shell` writes to a disposable overlay by default: | Name | Default | Description | |------|---------|-------------| | `--writable` | false | Modify the golden image directly. Destructive. | | `--detach`, `-d` | false | Run in the background. | | `--memory` | `8G` | Memory. | | `--cpus` | `8` | CPU cores. | | `--no-kvm` | false | Disable KVM acceleration. | | `--vnc-port` | auto from 8006 | VNC port. | | `--api-port` | auto from 5000 | API port. | ## `cua platform` Inspect the platform definitions `cua image create` can build from, and whether this host meets their requirements. | Command | Description | |---------|-------------| | `cua platform list` | List platforms with their status on this host. Accepts `--format table\|json`. Also the default when `cua platform` is run bare. | | `cua platform info ` | Show a platform's Docker image, ports, KVM requirement, and boot timeout. | ```console $ cua platform list Platforms ================================================================================ System: Docker: ✗ Not running KVM: ✓ Available -------------------------------------------------------------------------------- PLATFORM DESCRIPTION STATUS -------------------------------------------------------------------------------- linux-docker Linux GUI container (no KVM required) no Docker linux-qemu Linux VM with QEMU/KVM (OSWorld) no Docker windows-qemu Windows VM with QEMU/KVM (Windows Arena) no Docker android-qemu Android VM with QEMU/KVM no Docker macos-lume macOS VM with Apple Virtualization (Lume, Ap macOS only ``` Every platform except `macos-lume` runs through Docker, so a host without Docker reports `no Docker` for all of them regardless of KVM. `macos-lume` requires an Apple Silicon Mac running [Lume](). ## `cua do` Send one automation command to a target machine and exit. `cua do` keeps a selected target in `~/.cua/do_target.json`, so the target survives between invocations. Output is a single line beginning with `✅` or `❌`, followed by a context line showing the current machine and zoom state. Coordinates are in screenshot-image space; when a zoom is active the CLI translates them for you. | Option | Description | |--------|-------------| | `--no-record` | Disable trajectory recording for this command. Goes before the action: `cua do --no-record status`. | ### Target selection | Command | Description | |---------|-------------| | `cua do switch [name]` | Select the target. Providers: `cloud`, `cloudv2`, `local`, `lume`, `lumier`, `docker`, `winsandbox`, `host`. | | `cua do ls [provider]` | List machines for one provider. With no provider, lists the host plus every local and cloud sandbox. | | `cua do status` | Show the current target and zoom state. | | `cua do-host-consent` | Grant consent for `cua do switch host` and switch to it. | `host` means your own desktop. Selecting it is refused until consent is granted: ```console $ cua do switch host ❌ Warning: you are about to allow an AI to control your host PC directly. This grants full keyboard, mouse, and screen access to your local desktop. To continue, please run: cua do-host-consent ``` **Warning** `cua do-host-consent` is a top-level command, not a subcommand of `cua do`, and it writes a persistent marker at `~/.cua/host_consented`. Consent stays granted until that file is removed. ### Screen and framing | Command | Description | |---------|-------------| | `cua do screenshot [--save PATH]` | Capture the screen. Saved to a temp directory unless `--save`/`-s` is given. | | `cua do snapshot [instructions…]` | Screenshot plus an AI summary of the screen and its interactive elements. Requires `ANTHROPIC_API_KEY`. | | `cua do zoom ` | Crop every subsequent screenshot to that window and translate coordinates into it. | | `cua do unzoom` | Return to full-screen screenshots. | ### Input | Command | Description | |---------|-------------| | `cua do click [left\|right\|middle]` | Click. Defaults to `left`. | | `cua do dclick ` | Double-click. | | `cua do move ` | Move the cursor. | | `cua do drag ` | Drag between two points. | | `cua do type ` | Type text. | | `cua do key ` | Press one key, such as `enter`, `escape`, or `tab`. | | `cua do hotkey ` | Press a shortcut, such as `cmd+c` or `ctrl+shift+s`. | | `cua do scroll [amount]` | Scroll. Amount defaults to `3`. | ### Shell and files | Command | Description | |---------|-------------| | `cua do shell [command…]` | Run a shell command in the target, or open an interactive terminal when no command is given. Accepts `--cols` and `--rows`. | | `cua do open ` | Open a file or URL in the target. | ### Windows | Command | Description | |---------|-------------| | `cua do window ls [app]` | List windows, optionally filtered by application. | | `cua do window focus ` | Focus a window. `activate` is accepted as well. | | `cua do window unfocus` | Remove focus from the current window. | | `cua do window minimize\|maximize\|close ` | Change a window's state. | | `cua do window resize ` | Resize a window. | | `cua do window move ` | Move a window. | | `cua do window info ` | Show a window's details. | ## `cua skills` Record demonstrations on a sandbox and keep them as skills for agents to follow. Skills are stored in `~/.cua/skills/`. | Command | Description | |---------|-------------| | `cua skills list` | List saved skills. Accepts `--json`. Aliased as `cua skills ls`. | | `cua skills read ` | Print a skill. `--format`/`-f` selects `md` (default) or `json`. | | `cua skills replay ` | Open the skill's video recording. | | `cua skills delete ` | Delete one skill. | | `cua skills clean` | Delete every skill, after confirmation. | ### `cua skills record` Connect to a machine, record what you do, and caption the result with a vision model. Requires the `skills` extra. | Name | Default | Description | |------|---------|-------------| | `--sandbox`, `-s` | — | Sandbox name to connect to. | | `--vnc-url`, `-u` | — | Connect to a VNC URL directly instead. | | `--provider`, `-p` | `anthropic` | Captioning provider: `anthropic` or `openai`. | | `--model`, `-m` | provider default | Captioning model. | | `--api-key`, `-k` | from environment | API key for the captioning provider. | | `--name`, `-n` | prompted | Skill name. Supplying it skips the prompt. | | `--description`, `-d` | prompted | Skill description. Supplying it skips the prompt. | ## `cua trajectory` Every `cua do` command is recorded into a trajectory session under `~/.cua/trajectories///` unless `--no-record` was passed. Aliased as `cua traj`. | Command | Description | |---------|-------------| | `cua trajectory ls [machine]` | List sessions, optionally for one machine. Accepts `--json`. | | `cua trajectory view [target]` | Zip the session, serve it locally, and open it in the hosted trajectory viewer. Defaults to the newest session. | | `cua trajectory stop` | Stop the local file server started by `view`. | | `cua trajectory clean` | Delete sessions. | ```console $ cua trajectory ls Machine Session Turns Created ---------------------------------------------------------------------- my-container 20260812-224500 1 2026-08-12T22:45:00 ``` `view` accepts a machine name, a session timestamp, or a path, and takes `--port`/`-p` to move the local file server off its default port `8089`. It starts a background HTTP server on `127.0.0.1` and prints a `https://cua.ai/trajectory-viewer?zip=...` URL that points back at it, so the viewer only works while that server is running. Stop it with `cua trajectory stop`. `clean` takes `--older-than DAYS`, `--machine NAME`, and `-y`/`--yes` to skip the confirmation prompt. ## `cua serve-mcp` Start a Model Context Protocol server over stdio so an AI assistant can drive Cua directly. | Name | Default | Description | |------|---------|-------------| | `--permissions` | `CUA_MCP_PERMISSIONS`, else all | Comma-separated permission list. | | `--sandbox` | `CUA_SANDBOX` | Default sandbox for computer tools. | See [MCP server]() for the permission grammar and the full tool catalogue. ## `cua wif-token` Request a workload identity federation token for Fleets from a CI provider. `cua wif-token github` requests a GitHub Actions OIDC token with the `fleets` audience and prints only the raw token; it does not use the interactive session. It only runs inside GitHub Actions: ```console $ cua wif-token github Error: ACTIONS_ID_TOKEN_REQUEST_URL is missing; run in GitHub Actions with permissions: id-token: write. ``` See [Authentication]() for the workflow snippet. ## Files the CLI writes | Path | Written by | |------|------------| | OS credential vault, service `run.cua.ai`, account `cua-cli` | `cua auth login` | | `~/.cua/do_target.json` | `cua do switch`, `cua do zoom` | | `~/.cua/host_consented` | `cua do-host-consent` | | `~/.cua/skills/` | `cua skills record` | | `~/.cua/trajectories/` | `cua do` | | `~/.cua/sandboxes/` | local sandbox state, written when a local launch completes | | `~/.cua/cua-sandbox/` | disk images and overlays for local sandboxes | | `~/.local/share/cua/images/` | `cua image create`, `cua image pull` | | `~/.local/state/cua/images.json` | the local image registry | Local sandboxes and local images are stored in two separate trees, which is worth knowing when you are reclaiming disk space. `cua sb launch --local` works under `~/.cua/cua-sandbox/`: | Path | Contents | |------|----------| | `images/container-disks//disk.qcow2` | pulled container disks, content-addressed, reused across launches | | `images/-/disk.qcow2` | built base images | | `images/sessions/.qcow2` | per-sandbox overlay for a running sandbox, plus `.efivars.fd` for EFI guests | | `image-cache/` | downloaded image files, keyed by hash and original filename | `cua image create` and `cua image pull` write to `~/.local/share/cua/images/` instead. That path and `~/.local/state/cua/` follow the XDG base directory spec and move with `XDG_DATA_HOME` and `XDG_STATE_HOME`; `~/.cua/` does not. --- # CLI authentication Authenticate the cua CLI, choose credential storage, and configure workload identity. `cua` authenticates with OAuth device authorization. Endpoints are discovered from the issuer `https://auth.cua.ai/realms/cyclops-cs`; authenticated cloud requests then go to `https://run.cua.ai`. For the Sandbox SDK, Terraform, or `cua sb launch --pool`, use the separate [Fleet credential setup](#fleet-pools-need-their-own-credentials). A successful CLI login does not export credentials for those clients. ## Logging in ```bash cua auth login ``` The CLI prints a verification URL and a user code, tries to open a browser, and then polls until you approve. Because the code is printed, the flow works over SSH and in terminals with no browser at all — pass `--no-browser` to skip the browser attempt entirely: ```console $ cua auth login --no-browser Open this URL in any browser: https://auth.cua.ai/realms/cyclops-cs/device?user_code=QYNE-RIGM Enter this code if prompted: QYNE-RIGM ``` Approve in any browser, on any machine, then return to the terminal. Check the result at any time: ```console $ cua auth status Logged in to run.cua.ai. Access token expires 2026-08-12T22:36:31.604247+00:00. ``` Access tokens are refreshed automatically before authenticated requests, so the expiry shown by `cua auth status` moves forward on its own. `cua auth logout` asks the issuer to revoke the refresh token and removes the local credentials either way, even if the network is unavailable. The CLI never prints access or refresh tokens. ## Where credentials are stored Tokens go into the operating system credential vault through the `keyring` library, under service `run.cua.ai` and account `cua-cli`. The CLI does not read an API key from the environment and does not write one to a `.env` file. That vault is macOS Keychain, Windows Credential Manager, or — on Linux — a Secret Service provider such as `gnome-keyring` or KWallet. A server typically has none of these. ## Headless hosts On a Linux host with no keyring backend, `keyring` falls back to `keyring.backends.fail.Keyring` and every credential operation raises: ```console $ cua auth login --no-browser Open this URL in any browser: https://auth.cua.ai/realms/cyclops-cs/device?user_code=BCOP-JOSP Enter this code if prompted: BCOP-JOSP Error: Login failed: No secure credential store is available. Configure an OS keyring before logging in. ``` Two things are worth knowing about this failure. It happens **after** you have already approved in the browser, because the store is only written once the device flow completes — so the approval is wasted and has to be repeated. And it is not limited to `login`: `cua auth status` and every authenticated command raise the same error, because reading the vault fails the same way writing does. **Warning** **What is at stake is a refresh token, not just an access token.** The stored credential re-mints access tokens on demand, so it stays useful long after any access token it produced has expired, and automatic refresh keeps rewriting it so it never goes stale on its own. Whoever reads it holds your Cua session until it is revoked. Choose the storage backend accordingly, and prefer an encrypted one wherever the host outlives the job. ### Option 1: an encrypted file keyring The best fit for most headless hosts. `keyrings.cryptfile` writes to a file like the other file-backed backends but encrypts the store with a passphrase (Argon2 key derivation, AES-GCM): ```bash pip install keyrings.cryptfile export PYTHON_KEYRING_BACKEND=keyrings.cryptfile.cryptfile.CryptFileKeyring cua auth login --no-browser ``` The store lives at `~/.local/share/python_keyring/cryptfile_pass.cfg`, mode `600`. Every command that touches credentials prompts once for the passphrase: ```console $ cua auth status Please enter password for encrypted keyring: Logged in to run.cua.ai. Access token expires 2026-08-12T22:36:31.604247+00:00. ``` That prompt is the tradeoff: fine for a host you log into and work on, unworkable for an unattended job. `keyrings.alt` also ships an `EncryptedKeyring`, which stores to `crypted_pass.cfg` at mode `600` and prompts the same way. It needs a crypto library that `keyrings.alt` does not itself install, so add it explicitly or the backend fails to load with `ModuleNotFoundError: No module named 'Crypto'`: ```bash pip install keyrings.alt pycryptodome export PYTHON_KEYRING_BACKEND=keyrings.alt.file.EncryptedKeyring ``` ### Option 2: a plaintext file keyring, for hosts that are thrown away Use this only where the machine is destroyed after the job — a CI runner, an ephemeral VM — and never on anything that persists. ```bash pip install keyrings.alt export PYTHON_KEYRING_BACKEND=keyrings.alt.file.PlaintextKeyring cua auth login --no-browser ``` Or install it alongside the CLI in one step: ```bash uv tool install cua-cli --with keyrings.alt --index https://wheels.cua.ai/simple ``` Credentials then land in **`~/.local/share/python_keyring/keyring_pass.cfg`**, created with mode `600`. That is the file to protect, and the file to delete when you are finished with a host that outlived its purpose — removing it, or running `cua auth logout`, is what actually gets the credential off the disk. Setting `PYTHON_KEYRING_BACKEND` is not strictly required — `PlaintextKeyring` registers at a higher priority than the failing backend, so `keyring` selects it on its own once installed. Set it anyway: it pins the choice, so installing another backend later cannot silently move where your tokens are kept. **Warning** **`PlaintextKeyring` does not encrypt anything.** Values are base64-encoded, which is an encoding and not a protection — anyone who can read the file can recover the token with one command. File permissions are the only real barrier, and they do not survive a backup, a snapshot, a stray `tar`, or another user with root. Prefer Option 1 whenever the host persists. ### Option 3: a real keyring On a persistent Linux server, install and unlock a Secret Service provider such as `gnome-keyring` and let the default backend find it. This keeps the CLI on the same storage path it uses on a desktop, at the cost of having to unlock the keyring for each session — with `dbus-run-session` or a PAM module, depending on how the host is administered. ### Option 4: do not log in at all For CI, skip the interactive session and hand the CLI a workload token instead. See [GitHub Actions](#github-actions) below. ## Environment variables The CLI itself reads only these: | Variable | Effect | | --------------------------------- | ------------------------------------------------------------------------------------------------------ | | `FLEETS_TOKEN` | Fleet workload token. When set, it takes precedence over the interactive session for Fleet operations. | | `CUA_MCP_PERMISSIONS` | Default permission list for `cua serve-mcp`, overridden by `--permissions`. | | `CUA_SANDBOX` | Default sandbox for MCP computer tools, overridden by `--sandbox`. | | `ANTHROPIC_API_KEY` | Used by `cua do snapshot` and by `cua skills record --provider anthropic`. | | `OPENAI_API_KEY` | Used by `cua skills record --provider openai`. | | `PYTHON_KEYRING_BACKEND` | Read by the `keyring` library to select where credentials are stored. | | `XDG_DATA_HOME`, `XDG_STATE_HOME` | Move `~/.local/share/cua` and `~/.local/state/cua`. | The Sandbox SDK reads a further set. These apply when you import `cua_sandbox` in your own code: | Variable | Effect | | ------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------ | | `CUA_API_KEY` | API key for cloud sandboxes. | | `CUA_BASE_URL` | Base URL for the older VM API. Its default, `https://api.cua.ai`, is a retired host — set this explicitly if you use that API. | | `CUA_FLEET_BASE_URL` | Base URL for the Fleet API. Defaults to `https://run.cua.ai`. | | `CUA_TOKEN_URL` | OAuth token endpoint for client-credentials auth. | | `CUA_CLIENT_ID`, `CUA_CLIENT_SECRET` | OAuth client credentials for Fleet. | | `FLEETS_TOKEN` | Static Fleet workload token, checked before client credentials. | **Note** `CUA_BASE_URL` has no effect on `cua` commands. The CLI overwrites it at startup so that cloud requests always go to `https://run.cua.ai`, whatever the environment says. Set it only for direct SDK use. Fleet requests — the `--pool` path and everything `cua wif-token` feeds — go to `CUA_FLEET_BASE_URL`, which defaults to `https://run.cua.ai` and is the current cloud API. `CUA_BASE_URL` addresses the older VM API instead, and its default host is retired. **Warning** **`CUA_API_KEY` is not a way to skip logging in, and setting it makes things worse rather than better.** The SDK picks its transport with `_uses_fleet(api_key)`, which is true only when **no** API key is supplied. So providing one routes the call away from Fleet and onto the older VM API — whose host is retired — and it will fail there no matter how valid the key is. `FLEETS_TOKEN` is the environment variable that actually reaches a working API. `CUA_API_KEY` is not an authentication path for the CLI either: `cua` passes its own session token explicitly on every cloud call, and an explicitly passed key takes precedence over the environment. ## Fleet pools need their own credentials CLI login does not configure the Sandbox SDK or `cua sb launch --pool`. Follow [Set up Fleet credentials]() for the complete procedure. ### Create a Fleet user API key See [Create a Fleet user API key](). ### Configure the Sandbox SDK See [Configure the Sandbox SDK](). ### Check access before provisioning See [Check access before provisioning](). ### Credential lifetime and revocation See [Credential lifetime and revocation](). ### Existing access tokens and GitHub Actions See [Existing access tokens and GitHub Actions](). ## GitHub Actions `cua wif-token github` exchanges the job's GitHub OIDC identity for a Fleets token. It runs only inside GitHub Actions, requests the `fleets` audience, and prints nothing but the raw token: ```console $ cua wif-token github Error: ACTIONS_ID_TOKEN_REQUEST_URL is missing; run in GitHub Actions with permissions: id-token: write. ``` The job needs `id-token: write`. `FLEETS_TOKEN` is process-scoped and ephemeral; while it is set, it takes precedence over any interactive session: ```yaml permissions: id-token: write contents: read steps: - name: Run a non-interactive Fleets sandbox run: | export FLEETS_TOKEN="$(cua wif-token github)" cua sb launch ghcr.io/trycua/mini-swe:latest --name sandbox cua sb exec sandbox -- pwd cua sb delete sandbox --force ``` Use the GitHub-authorized sandbox name `sandbox`. `cua sb delete sandbox --force` releases the claim while preserving the reconciled one-replica pool, template, and namespace for the next claim. **Note** Under `FLEETS_TOKEN`, `cua sb ls` exits with `Listing Fleet sandboxes is not supported; use 'cua sb info NAME'.` Address Fleet sandboxes by name. --- # MCP server The cua serve-mcp stdio server, its permission grammar, and the tools each permission exposes. `cua serve-mcp` starts a Model Context Protocol server on stdio that exposes sandbox management, computer control, and skills to an MCP client. It requires the `mcp` extra: ```bash pip install --extra-index-url https://wheels.cua.ai/simple "cua-cli[mcp]" ``` Without it the command exits 1 with `MCP support not installed. Run: pip install cua-cli[mcp]`. ## Running it The server speaks MCP over stdin/stdout and logs to stderr, so it is started by the client rather than by you. Register it with Claude Code: ```bash claude mcp add cua -- cua serve-mcp ``` | Option | Default | Description | |--------|---------|-------------| | `--permissions` | `CUA_MCP_PERMISSIONS`, else every permission | Comma-separated permissions. | | `--sandbox` | `CUA_SANDBOX` | Default sandbox name for the computer tools. | ```bash claude mcp add cua -- cua serve-mcp --permissions sandbox:readonly,computer:readonly --sandbox my-sandbox ``` The server reports its enabled permissions on startup, which is the quickest way to confirm a grant landed as intended: ``` 2026-08-12 22:20:50,897 - cua-mcp - INFO - Enabled permissions: ['sandbox:list', 'sandbox:get'] 2026-08-12 22:20:50,909 - cua-mcp - INFO - Starting CUA MCP server... ``` ## Permissions A permission is either a single `group:action` string or one of the shorthand groups below. Only tools covered by the granted permissions are registered, so an ungranted tool is not merely refused at call time — the client never sees it. | Group | Expands to | |-------|-----------| | `all` | Every permission. | | `sandbox:all` | `sandbox:list`, `create`, `delete`, `start`, `stop`, `restart`, `suspend`, `get`, `vnc` | | `sandbox:readonly` | `sandbox:list`, `sandbox:get` | | `computer:all` | `computer:screenshot`, `click`, `type`, `key`, `scroll`, `drag`, `hotkey`, `clipboard`, `file`, `shell`, `window` | | `computer:readonly` | `computer:screenshot` | | `skills:all` | `skills:list`, `read`, `record`, `delete` | | `skills:readonly` | `skills:list`, `skills:read` | **Warning** **An empty or unrecognized permission list grants everything.** With no `--permissions` and no `CUA_MCP_PERMISSIONS`, the server logs `No permissions specified, granting all permissions` and registers all 47 tools. A misspelled permission is skipped with a `WARNING: Unknown permission` line — and if it was the only one you passed, the resulting empty set is treated as "unspecified" and again grants everything. Read the `Enabled permissions:` line on startup rather than assuming the flag was understood. ## Tools The permission that registers each tool: ### Sandbox | Permission | Tools | |------------|-------| | `sandbox:list` | `sandbox_list` | | `sandbox:get` | `sandbox_get` | | `sandbox:create` | `sandbox_create` | | `sandbox:delete` | `sandbox_delete` | | `sandbox:start` | `sandbox_start` | | `sandbox:stop` | `sandbox_stop` | | `sandbox:restart` | `sandbox_restart` | | `sandbox:suspend` | `sandbox_suspend` | | `sandbox:vnc` | `sandbox_vnc` | ### Computer | Permission | Tools | |------------|-------| | `computer:screenshot` | `computer_screenshot`, `computer_get_screen_size`, `computer_get_cursor_position`, `computer_get_accessibility_tree`, `computer_get_current_window` | | `computer:click` | `computer_click`, `computer_double_click`, `computer_move_cursor`, `computer_mouse_down`, `computer_mouse_up` | | `computer:type` | `computer_type` | | `computer:key` | `computer_key`, `computer_key_down`, `computer_key_up` | | `computer:hotkey` | `computer_hotkey` | | `computer:scroll` | `computer_scroll` | | `computer:drag` | `computer_drag` | | `computer:clipboard` | `computer_clipboard_get`, `computer_clipboard_set` | | `computer:file` | `computer_file_read`, `computer_file_write`, `computer_file_list` | | `computer:shell` | `computer_shell` | | `computer:window` | `computer_window_list`, `computer_window_open`, `computer_window_focus`, `computer_window_unfocus`, `computer_window_minimize`, `computer_window_maximize`, `computer_window_close`, `computer_window_resize`, `computer_window_move`, `computer_window_get_info`, `computer_launch` | ### Skills | Permission | Tools | |------------|-------| | `skills:list` | `skills_list` | | `skills:read` | `skills_read` | | `skills:record` | `skills_record` | | `skills:delete` | `skills_delete` | **Note** `computer:readonly` is not screenshot-only in practice: `computer:screenshot` also registers screen size, cursor position, the accessibility tree, and the current window. It reads the screen and never acts on it, but it reads more than a picture. ## Choosing a grant `computer:shell` and `computer:file` give the client arbitrary command execution and filesystem access inside the target machine, and `sandbox:delete` destroys machines. Grant the narrowest set that lets the assistant do its job: | Intent | Grant | |--------|-------| | Let an assistant look, not touch | `sandbox:readonly,computer:readonly` | | Drive a UI without a shell | `computer:screenshot,computer:click,computer:type,computer:key,computer:scroll` | | Full automation of one sandbox | `computer:all` plus `--sandbox ` | | Everything | `all` | Authentication comes from the same session as the rest of the CLI, so `cua auth status` must report a session before the sandbox tools can reach the cloud. See [Authentication](). --- # CLI Reference Command-line interface specification for Cua Driver Cross-platform computer-use automation driver. Install via the official script: ```sh curl -fsSL https://cua.ai/driver/install.sh | bash ``` Documented against Cua Driver **0.23.2**. Run `cua-driver --version` for your installed version. The macOS-only `cua-driver permissions` command is documented separately in [macOS permissions](). --- ## Tool dispatch ### `cua-driver list-tools` List every registered MCP tool with a one-line description. ### `cua-driver describe` Print a tool's full description and JSON input schema. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the MCP tool to describe. | ### `cua-driver call` Invoke an MCP tool through the running daemon. Requires a Cua Driver daemon. JSON arguments may be passed as a positional JSON object or through stdin. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the MCP tool to invoke. | | `` | String | No | JSON object for the tool input schema. If omitted, stdin is read when piped. | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--screenshot-out-file` | String | — | Write the first image content block from the response to this path. | | `--socket` | String | — | Override the daemon socket or named-pipe path. | ## Daemon management ### `cua-driver mcp` Run the stdio MCP server. On Windows and Linux, bare cua-driver mcp owns its runtime directly and shuts it down on stdin EOF. On macOS it proxies to CuaDriver.app so desktop permissions retain the app identity. Pass --direct to make the macOS MCP process own the runtime and TCC attribution, or --socket to select an explicit daemon endpoint. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--socket` | String | — | Select an explicit daemon socket or named-pipe endpoint. | | `--host-bundle-id` | String | — | Advisory host bundle id label echoed in check_permissions output (embedded mode). | | `--cursor-theme` | String | cua.default | Select an installed cursor theme id. | | `--cursor-reduced-motion` | String | auto | Follow the OS setting, force still frames, or allow animation: auto, on, or off. | | `--grant` | String | — | Pre-authorize a residual standard-mode boundary for a newly launched runtime. Repeatable; supported value: existing-profile. | **Flags:** | Name | Description | | ---- | ----------- | | `--direct` | Own the runtime in this MCP process; mutually exclusive with --socket. | | `--claude-code-computer-use-compat` | Accepted for older Claude Code setup snippets; no standalone screenshot tool — use get_window_state for window screenshots. | | `--embedded` | Declare embedding-host mode. Without --direct, require the host's private service through --socket instead of auto-launching the standalone app. | ### `cua-driver serve` Run Cua Driver as a long-running daemon. The daemon owns per-process state such as element-index caches, recording state, and cursor overlay state. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--socket` | String | — | Override the daemon socket or named-pipe path. | | `--pid-file` | String | — | Override the pid-file path on Unix targets. | | `--permission-mode` | String | standard | Immutable agent authorization mode: standard, bounded, or unrestricted. | | `--grant` | String | — | Pre-authorize a residual standard-mode boundary. Repeatable; supported value: existing-profile. | | `--capability-manifest` | String | — | Optional narrow-only tool/resource ceiling; required in bounded mode. | | `--session-policy` | String | — | Deprecated alias for capability-manifest. | | `--host-bundle-id` | String | — | Advisory host bundle id label echoed in check_permissions output (embedded mode). | **Flags:** | Name | Description | | ---- | ----------- | | `--dangerously-bypass-approvals` | Select unrestricted mode and acknowledge its risk. | | `--approve-capability-manifest` | Trusted-launcher confirmation that the exact capability manifest was reviewed. | | `--approve-session-policy` | Deprecated alias for approve-capability-manifest. | | `--no-permissions-gate` | Skip the macOS first-launch permissions gate. | | `--embedded` | Run embedded inside a host app: inherit the host's TCC grants, never prompt or relaunch. Also CUA_DRIVER_EMBEDDED=1. | | `--no-overlay` | Disable the agent cursor overlay for this daemon. | ### `cua-driver stop` Ask the running daemon to exit gracefully. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--socket` | String | — | Override the daemon socket or named-pipe path. | ### `cua-driver status` Report whether a Cua Driver daemon is running. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--socket` | String | — | Override the daemon socket or named-pipe path. | | `--pid-file` | String | — | Override the pid-file path on Unix targets. | ### `cua-driver mcp-config` Print client-specific connection guidance (MCP config where supported). Supported clients include claude, codex, cursor, antigravity, openclaw, opencode, hermes, pi, prime-agent, qwen, droid, and zcode. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--client` | String | — | Client name to print configuration for. | ## Trajectory recording ### `cua-driver recording` Control trajectory recording on a running daemon. Recording state lives in the required daemon and survives client reconnects. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--socket` | String | — | Override the daemon socket or named-pipe path. | #### `cua-driver recording start` Start trajectory recording to a directory. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Directory to write turn folders into. | #### `cua-driver recording stop` Stop trajectory recording. #### `cua-driver recording status` Print the current recording state. #### `cua-driver recording render` Render a recorded trajectory directory to an MP4. This pure file-to-file path does not require a running daemon. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Trajectory directory containing recorded turn folders. | | `` | String | Yes | Output MP4 path. | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--scale` | Number | — | Scale factor for rendered frames. | **Flags:** | Name | Description | | ---- | ----------- | | `--no-zoom` | Disable cursor/action zoom effects in the rendered video. | ## Configuration ### `cua-driver config` Read or mutate persistent driver configuration. Without a subcommand, prints the full config. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--socket` | String | — | Override the daemon socket or named-pipe path. | #### `cua-driver config show` Print the full config. #### `cua-driver config get` Print one config key. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Config key to read. | #### `cua-driver config set` Set one config key. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Config key to write. | | `` | String | Yes | Value to store. | #### `cua-driver config reset` Reset config to defaults. ## Diagnostics ### `cua-driver check-update` Check whether a newer cua-driver release is available. Read-only. Uses the same update-state payload as the check_for_update MCP tool. **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit a machine-readable JSON payload. | | `--no-cache` | Skip the 20-hour on-disk cache and force a GitHub request. | ### `cua-driver update` Check for an update and optionally apply it. The apply path delegates to the canonical platform installer scripts. **Flags:** | Name | Description | | ---- | ----------- | | `--apply` | Download and install the latest release when one is available. | | `--json` | Emit the structured update-state payload. | ### `cua-driver doctor` Run platform-aware diagnostic probes. Exit code is non-zero when any probe is an error. **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit the probe report as JSON. | ### `cua-driver diagnose` Print a pasteable install-layout and permission-attribution report. ## Other commands ### `cua-driver revoke` Revoke one or all live authorization/session scopes. Revocation is deny-only and never accepts an approval token. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--session` | String | — | Exact session id to stop and revoke. | | `--socket` | String | — | Override the daemon socket or named-pipe path. | **Flags:** | Name | Description | | ---- | ----------- | | `--all` | Stop and revoke every live session. | ### `cua-driver telemetry` Inspect or change content-free product telemetry. Telemetry is default-on. Disable retains the pseudonymous installation ID; reset-id erases the ID and event markers while preserving the preference. #### `cua-driver telemetry enable` Persistently enable telemetry. #### `cua-driver telemetry disable` Persistently disable every telemetry request. Retains the local installation ID. #### `cua-driver telemetry status` Show the effective setting and redacted identity state. **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit JSON. | #### `cua-driver telemetry reset-id` Erase the installation ID and event markers. The persisted enabled/disabled preference is retained. #### `cua-driver telemetry inspect` Build a fixed event payload without sending it. The distinct ID is replaced with a redacted placeholder. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Fixed telemetry event name. | **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit JSON. | ### `cua-driver channel` Inspect or change the stable/nightly update channel. Selection is persistent but never installs by itself; use cua-driver update --apply after changing it. #### `cua-driver channel status` Show selected and current release channels. **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit machine-readable channel state. | #### `cua-driver channel set` Save stable or nightly as the update channel. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | stable or nightly | **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit machine-readable channel state. | ### `cua-driver autostart` Manage platform-native daemon autostart. Windows registers a logon Scheduled Task. macOS and Linux currently print manual-recipe guidance. #### `cua-driver autostart enable` Register the autostart entry. #### `cua-driver autostart disable` Remove the autostart entry. #### `cua-driver autostart status` Print whether autostart is registered and running. `not-registered` is emitted only when Task Scheduler explicitly reports that the named task does not exist. If the task cannot be inspected, the command exits non-zero and reports `permission-denied` or `unknown` together with the original diagnostic. #### `cua-driver autostart kick` Start the autostart entry now without re-logging. ### `cua-driver skills` Install, update, inspect, or remove the optional agent skill pack. The install script never touches agent skill directories automatically. #### `cua-driver skills install` Fetch the versioned skill pack and link detected agents. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--agent` | String | — | Restrict linking to one agent. | | `--from` | String | — | Fetch from a source such as main instead of the tagged release. | **Flags:** | Name | Description | | ---- | ----------- | | `--all-platforms` | Keep platform-specific skill files for every platform. | #### `cua-driver skills update` Refresh the local skill pack and links. #### `cua-driver skills uninstall` Remove agent skill links. **Flags:** | Name | Description | | ---- | ----------- | | `--all` | Also delete the local skill-pack copy. | #### `cua-driver skills status` Report local skill-pack and per-agent link state. #### `cua-driver skills path` Print the local skill-pack path. ### `cua-driver manifest` Emit a stable JSON description of the CLI surface. Consumers can use this instead of hardcoding launch arguments such as the MCP invocation. **Flags:** | Name | Description | | ---- | ----------- | | `-p`, `--pretty` | Pretty-print JSON. | ### `cua-driver cursor-theme` Validate, compile, inspect, preview, install, or remove a local cursor theme. This is a trusted local authoring workflow. Agent-facing tools may select an installed theme id, but cannot install source or compiled theme data. #### `cua-driver cursor-theme validate` Validate a bounded dotLottie source archive. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Path to the source .lottie archive. | **Flags:** | Name | Description | | ---- | ----------- | | `--development` | Allow the reserved com.example development namespace. | #### `cua-driver cursor-theme build` Compile a validated dotLottie archive into a bounded .cua-theme artifact. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Path to the source .lottie archive. | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--output` | String | — | Output .cua-theme path. | **Flags:** | Name | Description | | ---- | ----------- | | `--development` | Allow the reserved com.example development namespace. | #### `cua-driver cursor-theme inspect` Inspect metadata in a compiled .cua-theme artifact. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Path to the compiled .cua-theme artifact. | **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit machine-readable JSON. | #### `cua-driver cursor-theme preview` Render a compiled theme's representative still frames to a directory. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Path to the compiled .cua-theme artifact. | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--output` | String | — | Preview output directory. | #### `cua-driver cursor-theme install` Install a compiled theme into the current user's theme store. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Path to the compiled .cua-theme artifact. | #### `cua-driver cursor-theme list` List the built-in and installed cursor themes. **Flags:** | Name | Description | | ---- | ----------- | | `--json` | Emit machine-readable JSON. | #### `cua-driver cursor-theme uninstall` Remove a custom theme from the current user's theme store. The built-in cua.default theme cannot be removed. **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Installed custom theme id. | ### `cua-driver dump-docs` Output machine-readable CLI and MCP documentation JSON. Used by the docs generator to keep reference pages in sync with the live binary. **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--type` | String | all | Which docs to emit: all, cli, or mcp. | **Flags:** | Name | Description | | ---- | ----------- | | `-p`, `--pretty` | Pretty-print JSON. | ## Global options Available on all commands: - `--help` — Show help information. - `--version` — Show version number. --- # SDK reference Python and TypeScript packages, constructors, methods, results, and errors for the typed Cua Driver SDK. ## Packages | Runtime | Package | Import | | -------------------- | -------------------- | ------------------------------------------------ | | Python | `cua-driver` | `from cua_driver import CuaDriver` | | Node.js / TypeScript | `@trycua/cua-driver` | `import { CuaDriver } from "@trycua/cua-driver"` | Both packages are generated from the same Rust and UniFFI contract. Methods are asynchronous. Python uses `snake_case`; TypeScript uses `camelCase`. ## Constructors | Python | TypeScript | Behavior | | ------------------------------------- | -------------------------------- | ------------------------------------------------------------------------------------------ | | `CuaDriver.create(options=None)` | `CuaDriver.create(options?)` | Creates the primary same-process runtime. No daemon, executable, or IPC is required. | | `CuaDriver.create_configured(options)` | `CuaDriver.createConfigured(options)` | Trusted-host constructor with an immutable runtime ceiling and compatibility mode. | | `CuaDriver.create_configured_with_authorization_host(options, host)` | `CuaDriver.createConfiguredWithAuthorizationHost(options, host)` | Adds a trusted host callback for residual authorization boundaries. | | `CuaDriver.create_configured_with_activity_observer(options, observer)` | `CuaDriver.createConfiguredWithActivityObserver(options, observer)` | Adds content-free activity events without changing authorization. | | `CuaDriver.create_configured_with_host_integrations(options, host, observer)` | `CuaDriver.createConfiguredWithHostIntegrations(options, host, observer)` | Installs both optional host integrations. | | `CuaDriver.create_private_worker(options)` | `CuaDriver.createPrivateWorker(options)` | Spawns one supervised process-isolated runtime over inherited pipes; no reusable endpoint is created. | | `CuaDriver.connect(socket_path=None)` | `CuaDriver.connect(socketPath?)` | Connects the same typed surface to a daemon. This is a compatibility and app-hosting path. | `DriverOptions` currently contains `claude_code_compatibility` / `claudeCodeCompatibility`, which defaults to `false`. `create()` validates the immutable permission-mode, managed-policy, user-policy, and bounded-manifest startup configuration before constructing the native runtime. Every subsequent same-process call passes the same native authorization coordinator used by daemon-backed calls. An invalid configuration returns `DriverError.Configuration`; a denied generic call returns a tool error with stable code `permission_denied`. `create_configured()` accepts `ConfiguredDriverOptions`, whose `authorization` field is `RuntimeAuthorizationOptions`: an allowed-mode set, the compatibility mode inherited by ordinary `CuaDriver` calls, bounded manifest path when that compatibility mode is bounded, maximum absolute and idle session TTLs, and the separate unrestricted-risk acknowledgement. Applications may create more than one same-process runtime. Each runtime owns an independent authorization ceiling, session registry, browser bindings, recording state, and shutdown lifecycle. Trusted host code may call the top-level `create_trusted_session(driver, …)` / `createTrustedSession(driver, …)` factory and receive a `CuaDriverSession`. Keeping the factory separate preserves the released structural `CuaDriverProtocol` and `CuaDriverLike` interfaces. The returned object is already bound to one immutable authorization context. Its public session string remains only a lifecycle label: a tool argument cannot select, replay, or widen the context. Call `close()` at the host lifecycle boundary; dropping the object, ending the session, or shutting down the runtime also revokes it. `PrivateWorkerOptions` adds the absolute Cua Driver binary path, host identity, startup and shutdown timeouts, configured runtime options, a safe environment allowlist, and optional stderr inheritance. Worker configuration is sent after spawn on the inherited channel rather than in argv. Caller-provided environment entries cannot set authorization controls; process-admin managed policy and disablement variables are inherited by both private workers and embedded services so a child topology cannot relax its parent's ceiling. Closing the host channel terminates the worker; a worker cannot be discovered or reattached. Daemon `connect()` clients retain the daemon compatibility mode. An embedded host may create a trusted session only over its original authenticated service connection; an ordinary or standalone daemon client cannot. ## Authorization host Routine standard-mode automation does not call the authorization host. The callback currently handles the existing logged-in Chromium profile boundary. It receives an attested, expiring request and must return the same `request_digest` with `Allow`, `Deny`, or `Cancel`. Python: ```python from cua_driver import ( CuaDriver, DriverAuthorizationAction, DriverAuthorizationDecision, DriverAuthorizationHost, ) class AppAuthorization(DriverAuthorizationHost): async def authorize(self, request): allowed = await app_ui.confirm(request.human_summary) return DriverAuthorizationDecision( action=( DriverAuthorizationAction.ALLOW if allowed else DriverAuthorizationAction.DENY ), request_digest=request.request_digest, ) driver = CuaDriver.create_configured_with_authorization_host( options, AppAuthorization(), ) ``` TypeScript: ```ts import { CuaDriver, DriverAuthorizationAction, type DriverAuthorizationHost, } from "@trycua/cua-driver"; const authorizationHost: DriverAuthorizationHost = { async authorize(request) { const allowed = await appUi.confirm(request.humanSummary); return { action: allowed ? DriverAuthorizationAction.Allow : DriverAuthorizationAction.Deny, requestDigest: request.requestDigest, }; }, }; const driver = CuaDriver.createConfiguredWithAuthorizationHost( options, authorizationHost, ); ``` The callback is trusted application code. Do not expose it as a model tool, forward `resource_json` / `resourceJson` to an agent, or auto-accept every request. Cua Driver intentionally does not prescribe or render the host UI. ## Activity observer `DriverActivityObserver` receives content-free events for authorized actions, authorization refusals, ordinary action failures, grant issue and revocation, and session start and end. Events include the tool name, risk class, adapter IDs, public session label, and stable refusal code. They never include tool arguments, text, images, URLs, file paths, or attested resource JSON. The observer cannot authorize, deny, or change a result. Return quickly and hand expensive work to an application queue. `execution_mode()` / `executionMode()` reports the generated `DriverExecutionMode` value. The enum variants are `Embedded`, `PrivateWorker`, `Daemon`, and the Rust-only carrier-backed `Remote`. ## Lifecycle and metadata | Python | TypeScript | Result | | ------------------ | ----------------- | ------------------------------------------------------- | | `metadata()` | `metadata()` | Driver version, protocol version, and platform metadata | | `execution_mode()` | `executionMode()` | `DriverExecutionMode` | | `is_available()` | `isAvailable()` | Whether the runtime can serve calls | | `shutdown()` | `shutdown()` | Stops admission and awaits admitted work | In TypeScript, `shutdown()` and `uniffiDestroy()` have separate jobs: `shutdown()` closes runtime admission and awaits admitted operations; `uniffiDestroy()` immediately frees the generated UniFFI object handle. JavaScript's `FinalizationRegistry` is only a nondeterministic fallback, so orderly applications call both, in that order. Python finalization releases its handle automatically, but applications must still await `shutdown()`. ## Session methods | Python | TypeScript | Input | | ------------------- | ----------------- | ---------------------- | | `start_session` | `startSession` | `StartSessionInput` | | `get_session` | `getSession` | `GetSessionInput` | | `list_sessions` | `listSessions` | `ListSessionsInput` | | `end_session` | `endSession` | `EndSessionInput` | | `get_session_state` | `getSessionState` | `GetSessionStateInput` (deprecated) | | `escalate_session` | `escalateSession` | `EscalateSessionInput` (deprecated) | `start_session` is optional. The first ordinary call creates an implicit session owned by the SDK transport, and later calls on that transport reuse it. The default idle TTL is five minutes. Shutdown, explicit end, and transport close run the same cleanup hooks. A call that is still in flight cannot expire, and only completed calls refresh the idle timer. For multi-call work, prefer a short public `session` label and pass the same value on every call that accepts it. Passing it once is not sticky: a later call that omits the field uses the transport's implicit session instead. Use `start_session` to name or configure a run before acting, or to revive a public name after it has ended. For one-off or deliberately unlabeled work, omitting `session` is valid. `get_session` and `list_sessions` return content-free state only for sessions visible to that transport. The public label is a map key, never an authorization credential. A trusted host can inspect its wider runtime namespace through `list_host_sessions_json()` / `listHostSessionsJson()`; agent tools cannot use that operator view. `StartSessionInput.capture_scope` / `captureScope`, `get_session_state`, and `escalate_session` remain available for legacy capture-scope sessions during the compatibility window. New code selects a target on each action. There is no `deescalate_session` method. ## Typed desktop methods | Python | TypeScript | Input | | --------------------- | ------------------- | ------------------------ | | `get_desktop_state` | `getDesktopState` | `GetDesktopStateInput` | | `get_screen_size` | `getScreenSize` | `GetScreenSizeInput` | | `get_cursor_position` | `getCursorPosition` | `GetCursorPositionInput` | | `move_cursor` | `moveCursor` | `MoveCursorInput` | | `click` | `click` | `ClickInput` | | `drag` | `drag` | `DragInput` | | `scroll` | `scroll` | `ScrollInput` | | `type_text` | `typeText` | `TypeTextInput` | | `press_key` | `pressKey` | `PressKeyInput` | | `hotkey` | `hotkey` | `HotkeyInput` | `move_cursor`, `click`, `drag`, `scroll`, `type_text`, `press_key`, and `hotkey` accept an `ActionTarget` / `actionTarget` on each call: ```text window: { kind: "window", pid, window_id } desktop: { kind: "desktop", display_id: "primary" } ``` The desktop target uses screen coordinates and foreground delivery. The window target uses coordinates from that window's screenshot and keeps the existing background or foreground delivery ladder. Legacy flat `scope`, `pid`, and `window_id` fields remain accepted but cannot be combined with `target`. ## Typed cursor methods | Python | TypeScript | Input | | -------------------------- | ----------------------- | ---------------------------- | | `set_agent_cursor_enabled` | `setAgentCursorEnabled` | `SetAgentCursorEnabledInput` | | `set_agent_cursor_motion` | `setAgentCursorMotion` | `SetAgentCursorMotionInput` | | `set_agent_cursor_theme` | `setAgentCursorTheme` | `SetAgentCursorThemeInput` | | `get_agent_cursor_state` | `getAgentCursorState` | `GetAgentCursorStateInput` | Cursor-bearing actions create or wake the transport's implicit session even when the caller does not supply a public session. Cursor configuration methods still take a public session label. `cua.default` is the only built-in theme. A trusted local operator may validate, compile, and install a custom theme with `cua-driver cursor-theme`; agent-facing tools may select an installed theme ID but cannot install source or compiled theme data. See [Agent cursor themes]() for the semantic profile and authoring workflow. Use `list_tools_json()` / `listToolsJson()` and `call_tool()` / `callTool()` for the generic, platform-extensible tool surface. Its top-level `enforcement_adapters` array distinguishes active, metadata-only, and not-exposed permission adapters. Prefer typed methods when one is available. ## `ToolResult` Desktop methods return `ToolResult`. | Python field | TypeScript field | Meaning | | ----------------- | ---------------- | ------------------------------------------ | | `text` | `text` | Human-readable result text | | `images` | `images` | Returned images and MIME types | | `structured_json` | `structuredJson` | Structured platform result | | `is_error` | `isError` | Whether the tool reported failure | | `error_code` | `errorCode` | Stable tool error code, when present | | `verified` | `verified` | Whether post-action verification succeeded | | `degraded` | `degraded` | Whether a reduced-capability path was used | | `raw_json` | `rawJson` | Complete extensible result envelope | ## Errors The generated `DriverError` variants are: - `Configuration` - `InvalidArguments` - `Transport` - `Protocol` - `Tool` - `Shutdown` - `RuntimeAlreadyExists` - `Worker` - `Remote` - `ActionInterrupted` Tool-level failures may also arrive as `ToolResult.is_error` / `isError`. Inspect the result before consuming images or structured fields. `ActionInterrupted` includes an `ActionCompletion` value: `NotStarted`, `Completed`, or `Unknown`. Do not automatically retry a non-idempotent action when completion is unknown. `RuntimeAlreadyExists` remains in the stable error vocabulary for hosts or platform facilities that cannot safely establish another owner. Ordinary second-runtime creation no longer returns it. See [MCP tools]() for the agent-facing tool catalog and [SDK, MCP, and process hosting]() for the architecture boundary. --- # Telemetry and privacy Exact Cua Driver telemetry controls, event schema, and privacy boundaries. Cua Driver sends content-free product telemetry to the PostHog EU ingest endpoint. Telemetry is enabled by default. The installer prints a notice before it sends the first event, and the preference persists across driver processes and upgrades. If an installation or release event fails, the driver leaves its success marker unset and waits 15 minutes before retrying. Commands during that interval do not wait for another telemetry request. ## Control telemetry ```bash cua-driver telemetry status --json cua-driver telemetry disable cua-driver telemetry enable ``` An environment override takes precedence over the saved preference: ```bash CUA_DRIVER_RS_TELEMETRY_ENABLED=false cua-driver mcp ``` Disabling telemetry stops telemetry requests. It does not delete the local installation ID, so enabling telemetry later resumes the same pseudonymous installation identity. A normal uninstall also preserves the ID and saved preference. To create a new identity, run: ```bash cua-driver telemetry reset-id ``` Use `uninstall --purge` when uninstalling if you also want to remove retained Cua Driver state. ## Separate update check The telemetry setting does not control Cua Driver's update check. On `mcp`, `serve`, and `doctor` startup, Cua Driver may make a content-free request to the public GitHub Releases API and cache the result locally for 20 hours. This request does not include the telemetry installation ID or command usage. Disable the update check separately when you need to prevent that GitHub request and cache write: ```bash CUA_DRIVER_RS_UPDATE_CHECK=false cua-driver mcp ``` Set the same environment variable in your MCP server configuration to keep the update check disabled across sessions. ## Inspect events locally `inspect` builds the event without sending a network request. It replaces the installation ID with a redacted placeholder. ```bash cua-driver telemetry inspect cua_driver_mcp_session_started --json cua-driver telemetry inspect cua_driver_mcp_tool_completed --json cua-driver telemetry inspect cua_driver_agent_session_started --json cua-driver telemetry inspect cua_driver_agent_session_ended --json ``` ## Common client event properties Every schema-v3 client event has this fixed envelope: | Property | Meaning | | -------------------------- | --------------------------------------------------------- | | `telemetry_schema_version` | Fixed integer `3` | | `product_version` | Installed Cua Driver version | | `os_family` | `macos`, `windows`, `linux`, or `other` | | `os_major` | Major operating-system version only | | `arch` | Bounded CPU architecture | | `is_ci` | Whether a recognized CI environment is present | | `is_synthetic` | Explicit Cua-owned test marker; `false` by default | | `transport` | `cli`, `daemon`, `mcp_stdio`, `mcp_http`, or `unknown` | | `process_session_id` | Random ID created once per process and never persisted | | `id_persisted` | Whether the installation ID came from durable local state | Client events set `$process_person_profile` to `false`. The installation UUID is used only as the PostHog `distinct_id`; it is not duplicated into event properties. This is pseudonymous installation telemetry, not anonymous telemetry. The client does not send an IP address as an event property. PostHog derives a country code and country name from the network request during ingestion, then discards the client IP before storing the event. A country-only transformation removes continent, city, subdivision, postal-code, coordinate, accuracy-radius, and time-zone properties. ## Fixed events | Event | Additional bounded properties | | ---------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | | `cua_driver_installation_registered` | `install_channel` | | `cua_driver_release_installed` | `install_channel`, `product_version` | | `cua_driver_serve` | None. Transitional process-start event for `cua-driver serve`; `transport` is `daemon` | | `cua_driver_cli_completed` | Fixed `command`, allowlisted tool for `call`, bounded `operation`, `computer_action`, bounded MCP `client_kind`, `success`, `exit_class`, and `duration_bucket` | | `cua_driver_mcp_startup_completed` | execution path, bounded daemon state, success, duration bucket, and `execution_mode` | | `cua_driver_mcp_session_started` | normalized MCP client and protocol, capability booleans, optional reported agent context, and `execution_mode` | | `cua_driver_mcp_tool_completed` | allowlisted tool, bounded compound-tool `operation`, `computer_action`, protocol success, error class, structured-refusal code, duration bucket, output shape and size buckets, and `execution_mode` | | `cua_driver_agent_session_started` | declaration kind, revival flag, concurrent-session bucket, entry transport, client kind, capture scope, and `execution_mode` | | `cua_driver_agent_session_ended` | end reason, duration and count buckets, browser-refusal bucket, success flags, feature-family flags, multi-transport flag, client kind, capture scope, bounded auto-escalation outcome, and `execution_mode` | | `cua_driver_permissions_gate_started` | missing-permission booleans | | `cua_driver_permissions_gate_dismissed` | missing-permission booleans and duration bucket | | `cua_driver_permissions_gate_completed` | missing-permission booleans, panel and dismissal flags, fixed resolution, and duration bucket | | `cua_driver_update_checked` | fixed source and outcome, strict public target release version or `unknown`, and cache-hit flag | | `cua_driver_update_apply_started` | strict public target release version and whether a daemon was running | | `cua_driver_update_apply_completed` | strict public target release version or `unknown`, fixed outcome and failure class, prior daemon state, and duration bucket | The transitional `cua_driver_serve` event carries the common event properties and no additional properties. It does not contain command arguments or process output. The legacy `cua_driver_install` event remains part of lifetime registration dashboards so existing installations are not registered a second time. ## Reported model and agent context An MCP host can explicitly report agent context in the initialize request: ```json { "params": { "_meta": { "ai.cua/agent-context": { "provider": "anthropic", "model": "claude-sonnet-4-5", "agent_name": "claude-code", "agent_version": "1.x" } } } } ``` These values are self-reported and normalized through allowlists. Unknown values become `custom` or `unknown`. Cua Driver does not infer a model from the MCP client name, does not use MCP Sampling to identify the parent model, and does not collect the complete initialize payload. Stdio emits once per process. A long-lived HTTP process emits once per normalized client category, so the event measures HTTP client-category adoption rather than every connection. ## Cua agent sessions A Cua agent session is one lifecycle episode owned by a trusted transport. It may have an optional public `session` label, but telemetry never sends that label or the runtime's private transport identity. The start event is emitted for a successful `start_session` or the first admitted session-requiring call on a transport. Its `declaration` is `start_session` or `implicit_first_action`. Repeated calls in the same live episode do not emit another start. Explicitly reviving an ended public label creates a new episode with `revived=true`. The end event is emitted for explicit end, idle expiry, transport cleanup, and revocation. It contains aggregate counters and booleans. When the platform has a cursor entry, the event includes bounded cursor categories: enabled state, built-in/default/custom icon class, automatic/custom color source, label presence, motion customization, and an active-cursor count bucket. `cursor_outcome_observed=false` distinguishes a missing cursor entry from a disabled or default cursor. Both session events carry the closed `client_kind` (`cli`, `direct`, `mcp`, `python_sdk`, or `typescript_sdk`). The Python and TypeScript package roots select their category automatically; applications cannot attach a free-form client label. `execution_mode` independently distinguishes `embedded` from `standalone` hosts. The end event adds `used_window_modality` and `used_desktop_modality`, which are content-free booleans derived from admitted calls. `capture_scope`, `auto_escalated_to_desktop`, and `escalation_reason` remain bounded compatibility fields for legacy capture-scope sessions. New per-call targets do not mutate session capture state. Free-form escalation detail is never observed or sent. Computer-action success covers fixed pointer and keyboard capabilities, app launch and kill, window activation, state-changing `page` operations, and successful browser navigation, click, type, pointer, file-assignment, download, or dialog-resolution calls. A structured browser refusal does not count as a completed computer action. `get_browser_state` and `browser_dialog` inspection are reads for this classifier, and `browser_prepare` is tracked as browser use rather than a computer action because preparation may either reuse an endpoint or produce an approved visible side effect. The end event includes `used_browser` and a bounded `browser_refusal_count_bucket`. It does not include browser targets, pages, profiles, or per-site information. Per-call `computer_action` is derived from the same fixed classifier used by the session aggregate. It describes the tool category; combine it with `success=true` when measuring value. Cua-owned automated tests can set `CUA_DRIVER_TELEMETRY_SYNTHETIC=true`. This marks every event from that process with `is_synthetic=true` so product metrics can exclude test traffic without relying on installation IDs. Third-party CI remains independently identified by `is_ci`. For finite CLI commands, `operation` distinguishes only reviewed verbs for recording, permissions, config, autostart, skills, and update. `client_kind` is populated only for `mcp-config`. Unknown values become `other`; commands without a meaningful sub-operation use `not_applicable`. ## Update funnel Explicit CLI and MCP update checks emit `cua_driver_update_checked`. Its `source` is `cli` or `mcp`; its `outcome` is `up_to_date`, `available`, or `unavailable`. Background checks emit only `available` on the freshness-bounded network path, rather than on every startup. Cached startup banners do not emit another event. `cua-driver update --apply` emits `cua_driver_update_apply_started` immediately before launching the canonical installer and `cua_driver_update_apply_completed` after the attempt. Completion outcomes are `installed`, `already_current`, or `failed`. Failure classes are limited to `none`, `check_failed`, `installer_exit`, or `installer_launch`; raw errors and exit codes are not collected. The common `product_version` is the version that initiated the check or update. `target_version` is accepted only when it is a strict public SemVer value. The existing `cua_driver_release_installed` event with `install_channel=update_apply` independently confirms that the new binary recorded the installed release. Together these events support the funnel from update availability through apply and first use of the installed release. For the compound `page` tool, `operation` is one of `execute_javascript`, `get_text`, `query_dom`, `click_element`, `insert_text`, `type_keystrokes`, `enable_javascript_apple_events`, or `other`. Typed browser tools use these reviewed operation values: | Tool | `operation` values | | --- | --- | | `get_browser_state` | `browser_bind`, `browser_snapshot_dom_refs_v1`, `browser_snapshot_semantic_v2`, or `other` | | `browser_prepare` | `browser_prepare_isolated`, `browser_prepare_existing_profile`, or `other` | | `browser_click` | `browser_click_trusted`, `browser_click_dom_event`, or `other` | | `browser_type` | `browser_type_insert_text`, `browser_type_keystrokes`, or `other` | | `browser_dialog` | `browser_dialog_inspect`, `browser_dialog_accept`, `browser_dialog_dismiss`, or `other` | | `browser_set_input_files` | `browser_set_input_files` | | `browser_download` | `browser_download` | | `browser_pointer` | Action-and-route values such as `browser_pointer_scroll_trusted` or `browser_pointer_drag_dom_event`; invalid combinations become `other` | | `browser_navigate` | `not_applicable`; the tool name already identifies the operation | ### Structured browser refusals A browser refusal is a successful MCP exchange with a behavioral result of `refused`. The tool-completion event therefore keeps `success=true` and `error_class=none`, while `refusal_code` records one closed, content-free code. The allowed values are: - `none` - `browser_route_unavailable` - `browser_requires_setup` - `browser_binding_ambiguous` - `browser_binding_stale` - `browser_wrong_target_refused` - `browser_tab_required` - `browser_tab_not_found` - `browser_ref_stale` - `browser_input_trust_unavailable` - `browser_endpoint_owner_mismatch` - `browser_consent_required` - `browser_consent_revoked` - `browser_reconnect_exhausted` - `browser_input_incomplete` - `browser_action_unavailable` - `other` for an unrecognized future refusal until it is reviewed for telemetry `browser_consent_revoked` indicates only that the browser consent flow was dismissed or denied. It does not include who acted, the browser profile, the prompt contents, or any associated page state. ## Data that is never collected Routine telemetry excludes task text, prompts, tool arguments, tool response bodies, typed text, screenshots, accessibility trees, window titles, application names, free-form client labels, escalation detail, file paths, URLs, arbitrary MCP metadata, raw cursor IDs, labels, colors, icon paths, numeric motion values, and raw error messages. Browser telemetry also excludes target, tab, frame, ref, process, window, and session identifiers; profile names and paths; queries and coordinates; approval tokens; endpoint URLs and ports; refusal messages and details; and requested or delivered text lengths. Tool-completion events retain only coarse result shape: text, image, mixed, empty, or unknown; a size bucket; a duration bucket; a fixed error class; and, for reviewed structured browser refusals, the fixed code above. The content itself never crosses the telemetry observer boundary. A proxy and daemon negotiate one completion-event owner. Mixed-version pairs retain proxy ownership, which prevents duplicate events during upgrades. To bound client and ingestion load, each Cua Driver process emits at most 1,000 routine tool-completion events in an hour. If that ceiling is reached before a successful computer action, the process may emit that first value event as well. Lifecycle, permissions, and aggregate session events are not subject to this ceiling. A CLI `call` is always delegated to the daemon. The daemon owns the completion event and reports `transport=daemon`, so the event is emitted only once. ## Region and deletion controls Client events are sent to PostHog's EU ingest endpoint. PostHog processes the request IP long enough to derive country-level distribution, then discards the IP before event storage. Stored telemetry keeps only the derived country code and country name; it excludes continent, city, subdivision, postal code, coordinates, accuracy radius, and time zone. PostHog does not create person profiles for these events. Retention and server-side deletion are governed by the [Cua privacy policy](https://cua.ai/privacy-policy); the client does not enforce a retention window or submit deletion requests. `telemetry reset-id` and uninstall purge controls remove local identity state only. Do not post the installation UUID in a public issue. --- # Computer History Agent Integration Tool discovery, permissions, query semantics, privacy boundaries, and consultation behavior for agent hosts. Computer History is an opt-in encrypted local record of actions performed through Cua Driver. Agent runtimes can read bounded metadata through two permission-gated tools. They cannot control capture or access the encryption key and encrypted chunks. **Note** This contract covers the early preview in nightly macOS, Windows, and Linux builds. Discover tools at runtime and tolerate their absence. ## Integration flow ```mermaid sequenceDiagram participant A as Agent runtime participant T as Cua tool registry participant P as Cua permission system participant H as Encrypted local history A->>T: Discover tools alt History tools absent T-->>A: Continue without history else History tools present A->>T: history_status({}) T->>P: Authorize history.status P-->>T: Allow or deny T-->>A: Structured status or denial opt A bounded read is useful A->>T: history_query(filters) T->>P: Authorize history.query P-->>T: Allow or deny T->>H: Decrypt, validate, filter, and bound H-->>T: Metadata-only events T->>H: Append encrypted access record T-->>A: Events and context disclosure end end ``` The access record is appended only when a successful query returns at least one event. The response that caused it does not include the new access record. ## Availability and feature detection The runtime registers the tools only when the desktop daemon admits the preview. Tool presence does not prove that the user granted the calling agent access. | Observed state | Meaning | Host behavior | | --------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------- | | Neither tool is advertised | The runtime does not admit the preview or the platform does not support it. | Continue without history. | | `history_status` is advertised | The runtime admits the preview. | Request `history.status` permission before reading status. | | `enabled: false` | Capture is off. Earlier encrypted history may remain. | Query only when prior history is useful and authorized. | | `paused: true` | New capture is paused. | Treat history after the pause point as incomplete. | | Drops or unhealthy storage are reported | The record may contain gaps. | Preserve the warning and avoid completeness claims. | ## Consultation behavior Tool discovery makes history available to a model. It does not cause the model to call either tool. A history-aware host should consult history when the user asks it to continue, resume, recall recent Cua activity, explain a prior Cua run, or locate where a Cua-mediated workflow stopped. For a matching request, the host should: 1. discover the history tools before broader desktop inspection; 2. call `history_status` and retain disabled, paused, unhealthy, and dropped-event state; 3. when useful and authorized, call `history_query` with a bounded recent slice; 4. treat events as metadata evidence rather than a transcript; 5. keep omitted content, geometry, arguments, results, and user intent unknown; 6. use the returned application or capability as a lead and verify current state through the least intrusive source; and 7. continue without history after absence, denial, empty results, or a recoverable history failure. A trusted system instruction or bundled skill can guide this behavior. A host may instead perform the status and query preflight before model execution when deterministic consultation is required. | Integration level | Required behavior | Accurate claim | | -------------------------- | ---------------------------------------------------------------------------------- | ---------------------------------------------------------------------------- | | Tool-capable | The runtime advertises the tools and schemas. | Agents can query Computer History. | | History-aware | A trusted policy directs consultation for matching requests and handles fallbacks. | The agent checks Computer History for recent-work and continuation requests. | | Deterministic consultation | The host enforces status and bounded-query preflight before broader discovery. | The agent automatically checks Computer History for matching requests. | Prompt wording can guide tool selection but does not establish deterministic consultation. ## `history_status` Returns operational metadata without returning history events. **Required capability:** `history.status` **Properties:** read-only, non-destructive, idempotent, closed-world **Input:** an empty object. Unknown fields are rejected. Important response fields are: | Field | Type | Meaning | | ---------------- | ------- | ------------------------------------------------------------------ | | `supported` | boolean | The platform adapter supports the preview. | | `admitted` | boolean | The daemon admitted the experimental feature. | | `enabled` | boolean | The user enabled capture. | | `paused` | boolean | New action capture is paused. | | `encrypted` | boolean | The encrypted storage profile is active. Preview 0 returns `true`. | | `profile` | string | Storage profile identifier. | | `retention_days` | integer | Query-visible retention. Default: `7`. | | `quota_bytes` | integer | Encrypted-store quota. Default: `104857600`. | | `bytes_used` | integer | Current encrypted bytes under the history root. | | `dropped_events` | integer | Events dropped by the nonblocking capture path. | | `health` | string | Fixed health category. | Health categories are `ready`, `disabled`, `paused`, `not_admitted`, `key_unavailable`, `key_locked`, `key_corrupt`, `key_destroy_failed`, `storage_unavailable`, `storage_corrupt`, `quota_reached`, `events_dropped`, and `writer_stopped`. ## `history_query` Returns a bounded event slice that may enter the current model context. **Required capability:** `history.query` **Properties:** read-only, non-destructive, closed-world. A successful non-empty query appends an encrypted access record, so the call is not idempotent. | Field | Type | Required | Bounds | Meaning | | ---------------- | ------- | -------- | ------------------- | ------------------------------------------------------------------------------------------------- | | `limit` | integer | No | `1..200` | Maximum events. Default: `50`. | | `session_id` | string | No | `1..128` characters | Opaque ID returned by history, or a caller-known session label resolved in the history namespace. | | `since_sequence` | integer | No | `>=1` | Inclusive lower sequence bound. | | `until_sequence` | integer | No | `>=1` | Inclusive upper sequence bound. | Unknown fields are rejected. When both sequence bounds are present, `since_sequence` must not exceed `until_sequence`. Events are ordered by `data.sequence`. The query applies every filter, retains the newest `limit` matching events, and returns that slice in ascending order. The preview has no opaque pagination token. Missing sequence numbers are valid gap evidence because capture uses a bounded nonblocking queue. Every response includes: ```json { "events": [], "metadata_only": true, "model_context_disclosure": true } ``` ## Event contract Events use CloudEvents 1.0 JSON and `urn:cua-driver:schema:history-event:v0`. | Event type | Payload kind | Meaning | | ---------------------------------------- | ------------------ | --------------------------------------------------------- | | `cua-driver.history.control.v0` | `control` | User lifecycle operation such as enable, pause, or flush. | | `cua-driver.history.action_started.v0` | `action_started` | A Cua-mediated action began. | | `cua-driver.history.action_completed.v0` | `action_completed` | The validated action outcome. | | `cua-driver.history.session_started.v0` | `session` | A Cua Driver lifecycle session began. | | `cua-driver.history.session_ended.v0` | `session` | A Cua Driver lifecycle session ended. | | `cua-driver.history.access.v0` | `access` | A local CLI or agent query returned events. | | `cua-driver.history.health.v0` | `health` | A fixed writer-health or dropped-event marker. | Clients must branch on both `dataschema` and `type`. Stop interpreting an event whose schema is unsupported. ## Permission contract Status and query are separate private-observation operations. Permission for `history.status` does not grant `history.query`. In bounded mode, the approved manifest must name the tools and the matching computer-history operations: ```yaml version: 3 expires_after: 1h idle_timeout: 10m resources: computer_history: operations: - status - query allow: tools: - history_status - history_query ``` An agent may propose this manifest. The trusted launcher selects and approves it. Clients must surface a denial and continue without history. They must not read the store directly, change permission modes, or reconstruct denied history with another observation tool. ## Privacy and storage boundary Returned events may contain timestamps, opaque identifiers, capability names, fixed application identity fields, and fixed outcome, route, delivery, evidence, lifecycle, access, and health categories. The contract prohibits screenshots, video, audio, accessibility trees, typed text, raw keystrokes, clipboard contents, raw tool arguments or results, file paths, window titles, URLs, and free-form diagnostics. Each CloudEvent is encrypted and authenticated inside a COSE_Encrypt0 record. Records use RFC 8742 CBOR Sequence framing. A native credential-store key and per-chunk HKDF keys protect data at rest. History payloads remain local and history tools are excluded from per-tool and agent-session telemetry. ## Errors Tool failures return a structured `code`. Authorization may deny the call before tool execution through the normal authorization envelope. | Code | Host behavior | | ------------------------------ | ---------------------------------------------------------------------- | | `invalid_history_query` | Correct the request once. Do not retry it unchanged. | | `invalid_history_query_range` | Correct the sequence bounds. | | `history_preview_not_admitted` | Refresh tool discovery and continue without history. | | `history_key_unavailable` | Report history as unavailable. | | `history_key_locked` | Let the user unlock the native credential store. | | `history_key_corrupt` | Stop querying and direct the user to local recovery controls. | | `history_storage_unavailable` | Continue the primary task without history. | | `history_storage_corrupt` | Stop consuming results and direct the user to local recovery controls. | | `history_quota_reached` | Treat history after that point as incomplete. | | `history_events_dropped` | Treat the affected interval as incomplete. | | `history_writer_stopped` | Do not assume new events are recorded. | ## Version identifiers | Contract | Identifier | | ----------------- | --------------------------------------------------------------------- | | Status tool | `history_status` | | Query tool | `history_query` | | Status capability | `history.status` | | Query capability | `history.query` | | Event schema | `urn:cua-driver:schema:history-event:v0` | | Storage profile | `cua-history-profile-v1/cbor-sequence+cose-encrypt0+cloudevents-json` | The tool and capability names are intended to remain stable. The event schema is experimental. Clients must use runtime tool discovery and advertised input schemas. ## Source contracts - [Full agent integration RFC](https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-agent-integration-rfc.md) - [Event schema](https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-event-v0.schema.json) - [Encrypted storage profile](https://github.com/trycua/cua/blob/main/libs/cua-driver/docs/computer-history-profile-v1.cddl) - [Use Computer History]() --- # macOS Permissions The macOS-only `cua-driver permissions` command for inspecting and requesting Accessibility and Screen Recording TCC grants. `cua-driver permissions` is a macOS-only command. It is not part of the auto-generated [CLI reference]() because it has no Windows or Linux counterpart, so it is documented here. ## `cua-driver permissions` (macOS) Inspect or request the macOS TCC grants the driver needs (Accessibility and Screen Recording). Embedded-mode hosts do not use `cua-driver permissions grant`; the host app requests these grants itself, as described in [Embedding](). The same rule applies to an in-process `CuaDriver.create()` runtime and to `cua-driver mcp --direct`: `check_permissions` is read-only even if a caller passes `{"prompt": true}`. It reports `source.attribution: "host"`, `source.direct_runtime: true`, and leaves direct ScreenCaptureKit readiness as `not_checked`. The responsible host owns permission prompts, Settings navigation, and the restart flow. ```bash cua-driver permissions status # report grant status; read-only, no prompt cua-driver permissions grant # launch CuaDriver via LaunchServices so the prompt attributes to the app ``` **Flags:** | Flag | Description | | -------- | ------------------------------- | | `--json` | Machine-readable status output. | The JSON status distinguishes the ordinary Screen Recording preflight from direct ScreenCaptureKit readiness. Because macOS Tahoe can show a separate private-window-picker bypass dialog when ScreenCaptureKit is queried, `permissions status` never runs that probe: it returns `screen_recording_capturable: null` and `direct_capture_status: "not_checked"`. This preserves the command's read-only, no-prompt contract. After `permissions grant` completes a successful live probe, status also returns `direct_capture_verification` with the `permissions_grant` source, the UTC verification time, and the macOS bundle ID whose probe succeeded. ScreenCaptureKit confirms that the explicit probe succeeded but does not report whether consent was newly granted or already present. The permission service validates the stored schema, source, timestamp, and bundle ID, and only loads the record when the live status belongs to the driver daemon identity. The CLI only renders the complete live and historical observation. Historical evidence does not turn the read-only status call into a claim about current ScreenCaptureKit readiness. `permissions grant` never sends a prompt-capable request over the daemon socket. It launches a short-lived instance of the installed CuaDriver app through LaunchServices, explains the additional dialog before deliberately triggering it, and requires a successful live capture probe. The child returns only content-free grant status through a private temporary file; that file cannot grant permission or widen an agent session. macOS describes the combined privacy category as screen and system-audio recording even though Cua Driver's current ScreenCaptureKit recorder captures screen video only and does not enable audio capture. The consent is limited to macOS capture; it does not authorize browser-profile access, browser data, or CDP attachment. The LaunchServices-hosted process records successful verification before it returns because that process owns the probed macOS identity. Records are stored separately for the release and local identities and contain the matching bundle ID. A failed explicit probe clears the matching prior record. If the record cannot be written, `permissions grant` reports the live success but exits with an error instead of claiming that later status calls can corroborate it. If the installed app is not listed under **Screen & System Audio Recording** after the prompt, click **+**, add `/Applications/CuaDriver.app` (or `/Applications/CuaDriverLocal.app` for a local build), enable it, and run `permissions grant` again. This recovers a stale or missing macOS registration without granting the terminal or another build identity. Automation prompts can also appear when an explicitly requested browser or app operation uses Apple Events. Those grants are target-specific and remain outside the core `permissions status` payload. After Accessibility or Screen Recording grants change, fully quit and relaunch the responsible application before recreating its runtime. For standalone service mode that application is `CuaDriver.app`; for direct SDK or direct MCP mode it is the importing or spawning host. For the complete prompt sequence in a Lume guest, see [Run Cua Driver in a macOS Lume VM](). Source-built test seeds also need a stable certificate identity; follow [Run Cua Driver macOS tests in a Lume VM](). --- # App-hosted daemon reference Run a private Cua Driver daemon when a desktop app must expose MCP using its permission identity. This reference describes the daemon-backed host for an application that must expose MCP to an external agent. It is not required for ordinary SDK use: [`CuaDriver.create()`]() runs the driver inside the importing process with no daemon or socket. The app-hosted form runs a dedicated `cua-driver serve` daemon as a direct child of your host app instead of launching the standalone `CuaDriver.app`. On macOS, the daemon inherits the host app's Accessibility and Screen Recording grants, so users only approve your app. A second `cua-driver mcp` child proxies stdio MCP traffic to that daemon; it never executes tools itself. A complete macOS reference host and demo live in the repo at `libs/cua-driver/rust/examples/embedded-host-macos`. ## Target forms Cua Driver integrations have three target forms. Each form has a distinct owner and lifecycle: | Target | Connection contract | Owner | | --- | --- | --- | | Existing MCP configuration | Launch its configured `command`, `args`, and environment unchanged | The MCP client or user configuration | | Existing daemon | `CuaDriver.connect(socketPath)` or `cua-driver mcp --socket ` | The process that started the daemon | | App-hosted daemon | Start `EmbeddedCuaDriverHost` with an absolute binary path and use the returned `connection` | The permission-owning host app | The SDK does not scan `PATH`, search installation directories, or choose among these targets. The embedding application must select an explicit target. An app that starts an embedded host must use the `connection.mcp.command`, `connection.mcp.args`, and `connection.mcp.environment` values returned by that host instead of reconstructing the proxy invocation. `EmbeddedDriverHostOptions.binaryPath` / `binary_path` may point to any absolute executable path. The path does not determine macOS permission ownership. The process that calls `start()` determines the responsibility chain. Packaged applications normally keep the executable in their signed resources so the nested binary is covered by packaging, signing, and notarization. An embedded connection belongs to one host generation. After `restart()`, the host must replace every SDK client, MCP proxy, and copied MCP configuration with values from the new connection. ## Launch embedded The supported SDK host generates a private endpoint, clears unsafe ambient environment variables, starts the daemon, and waits for a versioned metadata handshake: ```ts import { CuaDriver, EmbeddedCuaDriverHost } from "@trycua/cua-driver" const host = new EmbeddedCuaDriverHost( "/path/to/cua-driver", "com.yourco.yourapp", ) const connection = await host.start() const driver = CuaDriver.connect(connection.socketPath) // Existing application SDK calls use `driver` unchanged. // Existing agent runtimes launch connection.mcp.command with // connection.mcp.args and connection.mcp.environment. ``` Python exposes the same generated Rust objects: ```python from cua_driver import CuaDriver, EmbeddedCuaDriverHost, get_binary_path host = EmbeddedCuaDriverHost(str(get_binary_path()), "com.yourco.yourapp") connection = await host.start() driver = CuaDriver.connect(connection.socket_path) ``` For non-SDK hosts, the equivalent low-level launch is: ```sh CUA_DRIVER_EMBEDDED=1 \ CUA_DRIVER_HOST_BUNDLE_ID=com.yourco.yourapp \ cua-driver serve --socket /tmp/yourapp-cua.sock ``` Then start the MCP proxy against that socket: ```sh cua-driver mcp --embedded --socket /tmp/yourapp-cua.sock \ --host-bundle-id com.yourco.yourapp ``` You can pass `--embedded --host-bundle-id com.yourco.yourapp` to `serve` instead of the environment variables. Only the exact value `CUA_DRIVER_EMBEDDED=1` enables environment-based embedded mode. The host bundle id declares the expected host: `health_report` compares it with the daemon's parent application, while trust still comes from macOS's responsibility chain. ## Node and Electron hosts Use the embedded host in `@trycua/cua-driver` instead of duplicating socket naming, readiness checks, restart handling, and process cleanup: ```ts import { CuaDriver, EmbeddedCuaDriverHost } from '@trycua/cua-driver'; const embedded = new EmbeddedCuaDriverHost( '/path/inside/YourApp.app/Contents/Resources/cua-driver', 'com.example.your-app', ); const connection = await embedded.start(); const driver = CuaDriver.connect(connection.socketPath); // Application calls use driver; an agent runtime uses connection.mcp. driver.uniffiDestroy(); await embedded.stop(); embedded.uniffiDestroy(); ``` The package does not install or bundle cua-driver. Ship a compatible executable outside Electron's ASAR archive, preserve its executable bit, and sign the nested executable before signing and notarizing the enclosing macOS app. The `/electron` entry point exposes compatibility-named Accessibility and Screen Recording helpers backed by the same generated Rust SDK; there is no second FFI library. The host remains responsible for permission UI, status, and restart policy, and must not start the daemon until both grants are active. Destroy the SDK client and call `await embedded.stop()` from every orderly shutdown path. Electron hosts should defer their first `before-quit` event until `stop()` completes because asynchronous cleanup cannot run after the host process has exited. If grants change while the daemon is running, destroy the SDK client, call `embedded.restart()`, and reconnect so macOS re-evaluates them in a fresh process. ## Lifecycle contract - `start()` is concurrency-safe and coalesces callers into one generation. - A connection is valid only for its returned generation. `restart()` changes the generation, PID, and usually the endpoint; discard every old SDK client and MCP proxy before reconnecting. - Stop accepting new work, end active sessions, close MCP proxies and SDK clients, then await `stop()`. `stop()` cancels startup and is idempotent. - Use `waitForExit(generation)` / `wait_for_exit(generation)` to observe an unexpected child exit. Do not automatically replay an action whose completion is unknown. - The host holds a parent-liveness pipe. EOF shuts down the daemon if the host exits; Rust destruction also requests a kill as a fallback. This is crash containment, not a substitute for orderly shutdown. - Capture modality is selected by each observation or action target, not by the host or lifecycle session. Concurrent sessions sharing one daemon may issue exact window and desktop calls independently; always close host leases and explicitly end named sessions when deterministic cleanup matters. - macOS grant changes require a daemon restart. Never start the daemon before the permission-owning host process is ready. - Custom endpoints are accepted only when they are absent or a stale socket. The host refuses regular files, symlinks, and live listeners and only removes the endpoint identity owned by the matching generation. ## Host requirements - Spawn `cua-driver serve --embedded` directly from your app, for example with `Process` / `NSTask`, `posix_spawn`, or `fork` / `exec`. - Wait for its private socket to become ready, then spawn `cua-driver mcp --embedded --socket ` and speak MCP over the proxy's stdin/stdout. - On macOS, do not launch the driver with `open(1)` or `NSWorkspace.open`; LaunchServices makes the launched app its own responsible process and breaks permission inheritance. - On macOS, request Accessibility and Screen Recording from the host app with `AXIsProcessTrustedWithOptions` and `CGRequestScreenCaptureAccess`. If macOS grants are added after the daemon has started, restart the daemon so TCC is re-queried with a fresh per-process cache. ## App + gateway architectures `--embedded` does not transfer a GUI app's grants to the driver; it only keeps the daemon inside its spawner's macOS responsibility chain. If a separate gateway or Node process spawns the daemon, the daemon inherits the gateway's identity, not the app's. Spawn `cua-driver serve --embedded` from the app process. Call `health_report` with `{"include":["bundle_identity"]}` after connecting. The check passes only when macOS can resolve the direct parent as an application and, when configured, its observed bundle identifier matches `CUA_DRIVER_HOST_BUNDLE_ID`. Normal OpenClaw gateway and Hermes YAML MCP configurations remain standalone integrations; do not set embedded mode merely because one of those agents is the client. A signed Node or Electron desktop host may use `@trycua/cua-driver/embedded`, but only its permission-owning app process may start the daemon. The MCP client can then launch the proxy described by `connection.mcp`. ```text Wrong (inherits the gateway's identity): Right: gateway / node daemon YourApp.app └─ cua-driver serve --embedded ├─ cua-driver serve --embedded └─ cua-driver mcp --socket ``` An Electron app may pass the returned MCP configuration to a separate backend over its existing bootstrap or IPC channel. That backend may launch the MCP proxy, but it must not construct `EmbeddedCuaDriverHost` or start the embedded daemon on the app's behalf. ## What changes | Behavior | Standalone | Embedded | | --- | --- | --- | | Process model | Standalone daemon + proxy | Host-spawned daemon + proxy | | Daemon launch | May auto-launch CuaDriver.app | Host starts private daemon | | macOS TCC identity | `com.trycua.driver` or caller | Host app | | macOS permission prompts | Driver may prompt | Driver never prompts | | macOS Settings entries | CuaDriver | Host app only | | `check_permissions` attribution | `driver-daemon` or `caller` | `host` on macOS embedded runs | Driver tools, screenshots, AX tree reads, background input, and the agent cursor overlay otherwise behave the same. ## macOS permission check Call the `check_permissions` MCP tool after the proxy connects to the embedded daemon. On macOS, embedded mode ignores prompt requests and should return `source.attribution: "host"`: ```json { "accessibility": true, "screen_recording": true, "screen_recording_capturable": null, "direct_capture_status": "not_checked", "source": { "attribution": "host", "host_bundle_id": "com.yourco.yourapp", "embedded": true } } ``` Embedded mode never lets the driver raise consent dialogs, so this read-only call deliberately does not run the prompt-capable ScreenCaptureKit probe. The host should present its own consent context and verify pixels with an explicit screenshot/capture operation. If `source.attribution` is not `host` on macOS, embedded mode is not active in the daemon handling your MCP calls. Check that `CUA_DRIVER_EMBEDDED=1` is passed to the `serve` child, that the daemon was spawned directly, and that the proxy uses the intended private socket. `source.attribution: "host"` means the daemon is running in embedded mode; it does not prove that your GUI app is the responsible process. If a gateway or Node process spawned the daemon, the reported grant state still belongs to that spawner. --- # MCP Tools Reference for every MCP tool Cua Driver exposes `cua-driver` exposes 56 MCP tools through a single stdio server (`cua-driver mcp`). Every tool is also callable from the shell as `cua-driver ''`. Tool names are `snake_case`. Responses are MCP `CallTool.Result` envelopes: a text content block prefixed with a `✅` summary (or the error reason on failure), plus optional image or structured-content blocks on tools that produce them. See the [CLI reference]() for CLI-specific options like `--socket` and `--screenshot-out-file`. For the cross-cutting parameter contract (shared parameters, required-parameter rules, platform-specific parameters) and the action response shape, see [MCP tool notes](). **Note** Tool names here match the CLI form exactly. `cua-driver list_apps` and the MCP `list_apps` tool run the same code path. **Note** **Runtime ownership.** On Windows and Linux, bare `cua-driver mcp` owns its SDK runtime directly and shuts it down on stdin EOF. On macOS it proxies to the installed `CuaDriver.app` daemon so AX and Screen Recording grants retain the app-bundle identity. Passing `--socket` selects an explicit daemon/service endpoint on every platform. See the [process model]() for the full lifecycle and wrapper-author guidance. ## Inspection tools ### `list_apps` List macOS apps — both currently running and installed-but-not-running — with per-app state flags: - running: is a process for this app live? (pid is 0 when false) - active: is it the system-frontmost app? (implies running) - launch_path: filesystem path to the `.app` bundle, when known. Pass this to `launch_app` to start the app cold. - kind: `"desktop"` for `.app` bundles on macOS. - last_used: RFC3339 timestamp from the bundle's filesystem mtime, when readable; otherwise null. Only apps with NSApplicationActivationPolicyRegular are included — background helpers and system UI agents are filtered out. Installed apps come from scanning /Applications, /Applications/Utilities, ~/Applications, /System/Applications, and /System/Applications/Utilities. Use this for "is X installed?" as well as "is X running?". For per-window state — on-screen, on-current-Space, minimized, window titles — call list_windows instead. For just opening an app — running or not — call launch_app({bundle_id: ...}) directly; list_apps is not a prerequisite. **Arguments:** none. ### `list_windows` List all layer-0 top-level windows currently known to WindowServer. Includes off-screen windows (minimized, on another Space, hidden-launched). Use this to find a window_id before calling get_window_state. Per-record fields: window_id, pid, app_name, title, bounds (x/y/width/height, top-left origin), z_index (integer or null; higher values are closer to the front; null means stacking order is unavailable and callers must not infer one), is_on_screen, space_ids, current_space_id (the active Space on that window's display), and on_current_space. The top-level current_space_id is WindowServer's main/global active Space and can differ from a record's current_space_id when displays use independent Spaces. To select a frontmost candidate, take the maximum integer z_index; if every value is null, use an explicit fallback instead of relying on array order. **Arguments:** - `on_screen_only` (boolean, optional): When true, drop windows not on the current Space. Default false. - `pid` (integer, optional): Optional pid filter. When set, only this pid's windows are returned. ### `get_window_state` Walk a running app's AX tree and return BOTH a structured `elements` array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as `element_index` in the structured array — pass those indices to click, type_text, press_key, etc. INVARIANT: call get_window_state once per turn per (pid, window_id) before any element-indexed action. The index map is replaced by the next snapshot. PREFERRED CONSUMERS read `structuredContent.elements` (one entry per indexed row with `element_index`, `role`, `label`, `value` (the element's text/AXValue when present — use it to verify what a field holds), `frame: {x,y,w,h}`, `parent_index`, `depth`). The markdown `tree_markdown` stays available and unchanged in shape for existing text-parsing callers — but new fields will only be added to the structured side. Always returns BOTH the element tree AND a screenshot — ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with `h:1` frames). You choose the modality at ACTION time, not here: an element ax action (pass `element_index`/`element_token` → the accessibility rung) or an element px action (pass `x`,`y` → the pixel rung, read straight off this screenshot). `capture_mode` is deprecated and ignored. Pass `include_screenshot:false` to skip the grab and get the tree only — the cheap path when you're just re-indexing before an element ax action. The mirror image: pass `include_accessibility_tree:false` to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata — `window_bounds`, `screenshot_scale`, `screenshot_width`/`screenshot_height`, `app_name`, and `window_title` — the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH `include_accessibility_tree:false` and `include_screenshot:false` is an error (nothing to return). Optional `max_dimension` caps the returned screenshot's long edge in pixels (aspect preserved) for a cheap thumbnail. The snapshot is SCOPED to `window_id`: a window_id that no longer exists is refused with `window_id_not_found`, and one owned by another process is refused with `window_owner_pid_mismatch` naming the real `owner_pid` to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with `degraded_reason: ax_window_unresolved` and the screenshot of the requested window — act by pixel there. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. `px_frame_mismatch` or `px_capture_unavailable` omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available. Optional `query` projects both tree_markdown and structured `elements` to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and `element_count` continues to report its total size; `filtered_element_count` reports the projected response size. Optional `max_elements` / `max_depth` bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25). **Arguments:** - `capture_mode` (string, optional): DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot — ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_index/element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. - `include_accessibility_tree` (boolean, optional): Default true — walk the AX tree and return `elements` + `tree_markdown` alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, up to 20 s) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title) — the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). - `include_screenshot` (boolean, optional): Default true — returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk. - `max_depth` (integer, optional): Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. range: 1–unbounded - `max_dimension` (integer, optional): Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved) — the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. range: 1–unbounded - `max_elements` (integer, optional): Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. range: 1–unbounded - `pid` (integer, required): Target process ID. - `query` (string, optional): Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values. - `screenshot_out_file` (string, optional): When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `window_id` (integer, required): Target window ID from list_windows. ```json {"pid":844,"window_id":10725} ``` ### `get_accessibility_tree` Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. For the full AX subtree of a single window (with interactive element indices you can click by), use `get_window_state` instead — that's the heavy per-window tool. This one is a fast discovery read that needs no TCC grants. **Arguments:** none. ### `get_desktop_state` Capture the full display in true screen pixels with no downscale. Use its native-size PNG as the coordinate source for actions whose target is {kind:"desktop",display_id:"primary"}. Returns the true screen size and backing scale factor. Vision-only: no AX tree walk. **Arguments:** - `screenshot_out_file` (string, optional): Write PNG here instead of base64. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. ### `get_screen_size` Return the logical size of the main display in points plus its backing scale factor. Agents click in points; Retina displays have scale_factor 2.0. Requires no TCC permissions. **Arguments:** - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. ### `get_cursor_position` Return the current mouse cursor position in screen points (origin top-left). **Arguments:** - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. ### `get_config` Return the current cua-driver-rs configuration. **Arguments:** none. ### `get_recording_state` Report the current trajectory recorder state: whether recording is enabled, the output directory (when enabled), and the 1-based counter for the next turn folder that will be written. Counter increments on every recorded action tool call and resets to 1 each time recording is (re-)enabled. Pure read-only. **Arguments:** none. ### `get_agent_cursor_state` Return the session cursor's theme, semantic playback, position, visibility, and motion. **Arguments:** - `session` (string, required) ```json {"session":"example"} ``` ## Action tools ### `launch_app` Launch a macOS app in the background — the target does NOT come to the foreground. Provide either `bundle_id` (preferred — unambiguous, e.g. `com.apple.calculator`) or `name` (e.g. "Calculator"). If both are given, bundle_id wins. Optional `urls` are handed to the app as open targets — for Finder, pass a folder path to open a backgrounded Finder window there. Browser DevTools setup belongs to `browser_prepare`, which can prove that a separate isolated profile is driver-owned before enabling CDP. Optional `webkit_inspector_port`: opens a WebKit inspector server on the specified port (sets WEBKIT_INSPECTOR_SERVER=127.0.0.1:N + TAURI_WEBVIEW_AUTOMATION=1). Use this for Tauri/WebKit-based apps. Optional `creates_new_application_instance`: when true, forces a new app instance even if one is already running (passes -n to open). Reach for this when another agent or session may drive the SAME app concurrently — it returns a fresh pid + window so each session acts on its own isolated window instead of clobbering one shared instance. Without it, single-instance apps (Calculator, many utilities) hand every caller the same window, so two sessions fight over it. Optional `additional_arguments`: extra argv strings appended after --args. Returns the launched app's pid, bundle_id, name, and a `windows` array (same shape as `list_windows`) so callers can skip an extra round-trip before `get_window_state(pid, window_id)`. `launch_state` distinguishes whether the request was sent, the process is running, and a window is ready. When the focus-steal belt-and-braces demotion check ran (target pid ≠ prior frontmost), the response also includes `self_activation_suppressed: bool` — true if focus stayed with the prior frontmost, false if the launched app held focus despite the re-demote attempt. **Arguments:** - `additional_arguments` (array of string, optional): Extra arguments appended after --args when launching. - `bundle_id` (string, optional): App bundle identifier, e.g. com.apple.calculator. Preferred over name. - `creates_new_application_instance` (boolean, optional): When true, force a new app instance even if already running (open -n). Use for concurrent multi-agent/multi-session work so each session gets an isolated instance + window instead of sharing one — on single-instance apps (e.g. Calculator) every caller otherwise gets the same window and the sessions clobber each other. - `name` (string, optional): App display name. Used only when bundle_id is absent. - `urls` (array of string, optional): Optional file paths or URLs to open with the app (e.g. a folder path for Finder). - `webkit_inspector_port` (integer, optional): Open a WebKit inspector server on this port (sets WEBKIT_INSPECTOR_SERVER env var). ### `kill_app` Force-terminate a process by pid (kill -9 equivalent on macOS / Linux; taskkill /F equivalent on Windows). Use as escalation when the cooperative close path (hotkey cmd+q on macOS, click-the-X on Windows) failed to make the process exit. Unsaved state is lost — prefer the cooperative path first. **Arguments:** - `pid` (integer, required): PID of the process to terminate. ```json {"pid":844} ``` ### `bring_to_front` Persistently activate an app and leave it in the foreground. Most input does not need this; use it only for a focus-proxy surface that must remain foreground across interactions. With window_id, success means the exact ordinary macOS window was independently verified as the focused window and first in WindowServer layer-0 order. Request acceptance alone is reported as a partial result, never as activation. This DOES steal foreground. **Arguments:** - `pid` (integer, required) - `window_id` (integer, optional) ```json {"pid":844} ``` ### `set_window_frame` Set one exact top-level window's frame in the desktop-coordinate space reported by list_windows and verify the resulting geometry through an independent readback. **Arguments:** - `height` (number, required): range: 1–unbounded - `pid` (integer, required): range: 1–unbounded - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `width` (number, required): range: 1–unbounded - `window_id` (integer, required): range: 1–unbounded - `x` (number, required) - `y` (number, required) ```json {"height":1,"pid":844,"width":1,"window_id":10725,"x":100,"y":200} ``` ### `click` Click against a target pid. **Prefer `element_token` over pixel coordinates** — the token works on backgrounded / minimized / hidden / off-Space windows, identifies one exact snapshot element, and tells you what you're clicking via the cached element's role + label. Reach for `x, y` only when the target is a canvas / video / WebGL / custom-drawn surface that doesn't appear in the AX tree. Two addressing modes: - element_token, or element_index + snapshot_id (from get_window_state): AX action path. Works on backgrounded/hidden windows. No cursor move, no focus steal. The snapshot cache is scoped per (pid, window_id) and is replaced by the next snapshot of the same window — re-snapshot every turn before clicking. - x, y (window-local screenshot pixels, top-left origin of the PNG returned by get_window_state): CGEvent path. Synthesizes mouse events and posts to pid. Use modifier for cmd/shift/option/ctrl. Needs a visible on-screen window to anchor the conversion. button: "left" (default), "right", or "middle". Defaults to left so the field is fully back-compat — omit it and you get the legacy left-click behaviour. Pixel path: routes through the CGEvent left/right/middle mouse-button primitives. AX path: "right" maps to AXShowMenu (same surface as the dedicated `right_click` tool); "middle" has no AX equivalent and falls back to a pixel middle-click at the element's center. action: press (default), show_menu, pick, confirm, cancel, open. from_zoom: set true after a zoom call to auto-translate zoom-image pixel coordinates to full-window space. **Arguments:** - `action` (string, optional): AX action: press, show_menu, pick, confirm, cancel, open. - `button` (string, optional): Mouse button. Default: "left" — omit for legacy left-click behaviour. Pixel path uses the matching CGEvent primitive; AX path maps "right" to AXShowMenu and falls back to a pixel middle-click at the element's center for "middle". - `count` (integer, optional): Click count (pixel path only). Default 1. - `debug_image_out` (string, optional): Optional file path. When set on a pixel-addressed click, captures a fresh screenshot, draws a red crosshair at (x, y), and writes the PNG. Use to verify coordinate spaces. Requires window_id; incompatible with from_zoom. - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": perform the AX action or post the CGEvent without fronting. "foreground": briefly front the window, act, let transient UI settle, then restore the prior frontmost app. Requires window_id. Modified clicks require "foreground" so macOS observes physical modifier-key state. A generic click has no independent postcondition read-back, except selection of list-like AX rows whose AXSelected state can be confirmed; otherwise confirm the effect from a fresh state snapshot. Use the agent loop: background AX (element_index) → snapshot → background pixel (x/y) → snapshot → delivery_mode:"foreground". - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `from_zoom` (boolean, optional): When true, x and y are in the last zoom image for this pid; driver translates back to full-window coordinates. - `modifier` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl. - `pid` (integer, optional): Target process ID. - `scope` (string, optional): Coordinate frame for a windowless screen-absolute click (default "window"). Pass "desktop" when sending x,y with NO pid/window_id — the coordinates are then true screen pixels (read from get_desktop_state with scope="desktop"). Per-call; not a setting. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. - `window_id` (integer, optional): Target window ID. Required for element_index. Optional when element_token is supplied (the token carries it). - `x` (number, optional): X in screenshot pixels. A window target uses the get_window_state PNG; a desktop target uses the native get_desktop_state PNG. The driver reverses Retina backing scale and any window-image downscale. - `y` (number, optional): Y in screenshot pixels from the image selected by target. ### `double_click` Double-click at (x, y) or on an AX element identified by element_index + window_id. AX path (element_index provided): performs `AXOpen` when the element advertises it (Finder items, openable list rows/cells); otherwise resolves the element's on-screen center and falls back to a pixel double-click there. Pixel path (x, y provided): two down/up pairs ~80 ms apart at the given coordinates. **Arguments:** - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `pid` (integer, required) - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it). - `x` (number, optional): Screen X coordinate (pixel path). - `y` (number, optional): Screen Y coordinate (pixel path). ```json {"pid":844} ``` ### `right_click` Right-click against a target pid. Two addressing modes: - `element_index` + `window_id` (from the last `get_window_state` snapshot) — performs `AXShowMenu` on the cached element. Pure AX RPC, works on backgrounded / hidden windows, no cursor move or focus steal. Requires a prior `get_window_state(pid, window_id)` in this turn. - `x`, `y` — synthesizes `rightMouseDown` / `rightMouseUp` CGEvent pair posted to the pid. Driver converts image-pixel → screen-point internally. `modifier` forces the CGEvent path (AX actions don't propagate modifier keys). Exactly one of `element_index` or (`x` AND `y`) must be provided. `pid` always required. `window_id` required when `element_index` is used. **Arguments:** - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `modifier` (array of string, optional): Modifier keys held during the right-click: cmd/shift/option/ctrl. Pixel path only. - `pid` (integer, required): Target process ID. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it). - `x` (number, optional): X in window-local screenshot pixels. Must be provided together with y. - `y` (number, optional): Y in window-local screenshot pixels. Must be provided together with x. ```json {"pid":844} ``` ### `drag` Press-drag-release gesture from (from_x, from_y) to (to_x, to_y) in window-local screenshot pixels — the same space get_window_state returns. Top-left origin of the target's window. Use for: marquee/lasso selection, drag-and-drop, resizing via a handle, scrubbing a slider, repositioning a panel. `duration_ms` (default 500) is the wall-clock budget for the path between mouse-down and mouse-up; `steps` (default 20) is the number of intermediate mouseDragged events linearly interpolated along the path. Increase both for slower, more human drags; decrease for snap gestures. `modifier` keys (cmd/shift/option/ctrl) are held across the entire gesture. When `from_zoom` is true, coordinates are in the last zoom image for this pid; the driver maps them back to window coordinates before dispatching. **Arguments:** - `button` (string, optional): Mouse button used for the drag. Default: left. - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. - `duration_ms` (integer, optional): Wall-clock duration of the drag path between mouseDown and mouseUp. Default: 500. range: 0–10000 - `from_x` (number, required): Drag-start X in window-local screenshot pixels. Top-left origin. - `from_y` (number, required): Drag-start Y in window-local screenshot pixels. Top-left origin. - `from_zoom` (boolean, optional): When true, coordinates are in the last zoom image for this pid; driver maps back to window coordinates. - `modifier` (array of string, optional): Modifier keys held across the entire gesture: cmd/shift/option/ctrl. - `pid` (integer, optional): Target process ID. - `scope` (string, optional): Use desktop with no pid/window_id for native get_desktop_state screenshot coordinates. default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `steps` (integer, optional): Number of intermediate mouseDragged events linearly interpolated along the path. Default: 20. range: 1–200 - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. - `to_x` (number, required): Drag-end X in window-local screenshot pixels. - `to_y` (number, required): Drag-end Y in window-local screenshot pixels. - `window_id` (integer, optional): CGWindowID for the window the pixel coordinates were measured against. Optional only when pid owns exactly one eligible top-level window; otherwise the action refuses with ambiguous_window_target. ```json {"from_x":100,"from_y":200,"to_x":100,"to_y":200} ``` ### `type_text` Insert text into the target pid via `AXSetAttribute(kAXSelectedText)`. Works for standard Cocoa text fields and text views. No keystrokes are synthesized — special keys (Return / Escape / arrows) go through `press_key` / `hotkey`. For Chromium / Electron inputs that don't implement `kAXSelectedText`, the tool falls back to CGEvent character synthesis automatically when the estimated route stays within the daemon transport budget. Longer synthesized routes are refused before character events and return a safe chunk size; one-call AX insertion remains uncapped. Optional `element_index` + `window_id` (from the last `get_window_state` snapshot) directs the write to a specific field. Without `element_index`, the write goes to the pid's currently focused element. WEB CONTENT (Chromium/WebKit/Electron — browser tabs, Slack, VS Code, X's compose box): AXValue is not independent proof that the renderer/DOM observed an AX write or synthesized keystrokes. The driver detects this at the element level (an AXWebArea ancestor) and refuses to trust AXValue-only read-back there — type_text returns effect:"unverifiable" + escalation, never a false "confirmed" (a browser's own native address bar/toolbar stays trusted). For a browser TAB the reliable path is the `page` tool (drives the DOM via CDP); for an embedded web view use this tool's px form: pass x,y (no element_index) to pixel-click the field then type, in one call. NOTE: a px focus-click won't reliably open+focus a CLOSED control; AX-press to open/activate it first (works in the background), then px-type. Always confirm via the screenshot; if px-background still drops, escalate to delivery_mode:"foreground". **Arguments:** - `delay_ms` (integer, optional): Milliseconds between characters in the CGEvent fallback path. Default 30. Ignored when the AX path succeeds. range: 0–200 - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": AX insert, then CGEvent keystrokes if needed — no focus steal; native controls can be confirmed via AXValue read-back, while web-content writes remain effect:"unverifiable". "foreground": briefly front the window, type, restore the prior frontmost — the explicit last resort for focus-sensitive surfaces (e.g. WhatsApp/Catalyst) where background keystrokes don't land. Re-call with "foreground" when a background attempt remains unverifiable and a fresh snapshot shows the text did not appear. - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `pid` (integer, optional): Target process ID. - `scope` (string, optional): Use desktop with no pid/window_id to type into the frontmost application. default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. - `text` (string, required): Text to insert at the target's cursor. - `window_id` (integer, optional): CGWindowID. Required when element_index is used. Optional when element_token is supplied (the token carries it). - `x` (number, optional): Screenshot-pixel X of the field to type into — the element px action form. Pass x,y (no element_index) and the tool pixel-clicks there to establish real renderer focus, then types. Use for Chromium/Electron inputs the AX path can't reach. Read straight off the get_window_state PNG, same convention as click. - `y` (number, optional): Screenshot-pixel Y of the field (see x). ```json {"text":"hello"} ``` ### `press_key` Press and release a single key. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default: • `background` (default): post to the pid WITHOUT fronting/raising — the auth-message path (Chromium-safe). With element_index it focuses that AX element first. `window_id` only targets; it does not raise. • `foreground`: guard and briefly front the exact window, focus an addressed AX element when supplied, send a genuine HID key transition so Chromium content, inline editors, and native menu equivalents receive it, then restore prior frontmost. Requires window_id. A key press is confirmed only when a bounded native AX value/selection read-back changes on the same control. Otherwise a successfully attempted post remains effect:"unverifiable" without implying delivery failure or recommending foreground. Key names: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, plus any letter or digit. Modifiers array: cmd, shift, option/alt, ctrl, fn. **Arguments:** - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `key` (string, required): Key name: return, tab, escape, up, down, etc. - `modifiers` (array of string, optional): Modifier keys: cmd, shift, option/alt, ctrl, fn. - `pid` (integer, optional) - `scope` (string, optional): Use desktop with no pid/window_id to send the key to the frontmost application. default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. - `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground". Does NOT itself raise the window — raising is gated on delivery_mode. - `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the key. Use when the key must go to a Chromium/Electron surface the AX path can't focus. Pass with y, no element_index. - `y` (number, optional): Screenshot-pixel Y (see x). ```json {"key":"return"} ``` ### `hotkey` Press a key combination — e.g. `["cmd", "c"]` for Copy, `["cmd", "shift", "4"]` for screenshot selection. Follows the same `delivery_mode` ladder as click/type_text — it does NOT raise the window by default: • `background` (default): post the combo to the target pid WITHOUT fronting or raising it — uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. With an AX target, focus that exact element first. No top-level focus steal. `window_id` here only targets the combo; it does not raise. • `foreground`: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost — the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With an AX target or x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id. A combo is never driver-verifiable (no read-back) → effect:"unverifiable"; confirm via screenshot. NOTE: a keyboard combo does NOT focus a text field — to type into a backgrounded Electron input, establish real renderer focus with a PIXEL click first, then `type_text`. If an app only accepts paste, call `clipboard_write`, then `clipboard_read` and verify its types (and text when applicable) before selecting or replacing editor content; only then send Cmd+V. Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modifier keys use the same vocabulary as `press_key`. Order: modifiers first, one non-modifier last. **Arguments:** - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `keys` (array of string, required): Modifier(s) and one non-modifier key, e.g. ["cmd", "c"]. items: 2–unbounded - `pid` (integer, optional): Target process ID. - `scope` (string, optional): Use desktop with no pid/window_id to send the chord to the frontmost application. default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. - `window_id` (integer, optional): Target window. Required for delivery_mode:"foreground" (the NSMenu activation needs a window). Does NOT itself raise the window — raising is gated on delivery_mode. - `x` (number, optional): Screenshot-pixel X — the element px action form: pixel-click there to focus, then send the combo (so e.g. Cmd+V pastes into that field). Pass with y. Use for Chromium/Electron surfaces the background combo can't reach. - `y` (number, optional): Screenshot-pixel Y (see x). ```json {"keys":["cmd","c"]} ``` ### `set_value` Set a value on a UI element. Two modes depending on element role: - **AXPopUpButton / select dropdown**: finds the child option whose title or value matches `value` (case-insensitive) and AXPresses it directly — the native macOS popup menu is never opened, so focus is never stolen. Use this for HTML <select> elements in Safari or any native NSPopUpButton. - **All other elements**: writes AXValue directly (sliders, steppers, date pickers, native text fields that expose settable AXValue). For free-form text entry into web inputs, prefer `type_text_chars` which synthesises key events — AXValue writes are ignored by WebKit. **Arguments:** - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `pid` (integer, required) - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `value` (string, required): New value. AX will coerce to the element's native type. - `window_id` (integer, optional): CGWindowID for the window whose get_window_state produced the element_index. Required when element_index is used; optional when element_token is supplied (the token carries it). ```json {"pid":844,"value":"42"} ``` ### `scroll` Scroll the target pid. Two paths, picked by how you address the scroll: • **Targeted wheel path** — when you pass a target, either `element_index`/`element_token` (preferred) or window-local `x, y` pixels: the driver synthesizes a real mouse-wheel event (CGEventCreateScrollWheelEvent, at that screen point. The renderer hit-tests the wheel at the cursor, so the scroll lands on whatever element is under the point — exactly like physically rolling the wheel over it. This is the ONLY way to scroll a nested `overflow:auto` region (e.g. a scrollable <div> with no tabindex): such regions never take keyboard focus, so the keystroke path below no-ops on them. Use this for inner/nested scrollers in web views. • **Keystroke path (focused region)** — when you pass NO target (just pid + direction): synthesizes PageDown/PageUp (by='page') or Down/Up arrows (by='line'); horizontal uses Left/Right arrows. Drives the focused / page scroller only. Mapping: by='page' → larger step; by='line' → smaller step; amount = number of wheel notches (targeted path) or keystroke repetitions (keystroke path). **Arguments:** - `amount` (integer, optional): Pixel-wheel path: number of wheel notches. Keystroke path: number of keystroke repetitions. Default: 3. range: 1–50 - `by` (string, optional): Scroll granularity. Default: line. - `delivery_mode` (string, optional): Best-effort-background ladder rung (default "background"). "background": inject without fronting or raising the target — no focus steal. "foreground": briefly front the target, act, then restore the prior frontmost — the explicit last resort when a background attempt didn't land. Re-call with "foreground" only for the action that needs it. - `direction` (string, required): Scroll direction. - `element_index` (integer, optional): Element index from get_window_state. Requires the matching `snapshot_id` alongside it. Prefer `element_token`, which carries both values. - `element_token` (string, optional): Opaque per-snapshot element handle from `structuredContent.elements[].element_token`. If element_index, snapshot_id, or window_id are also supplied they must agree. Returns an explicit stale error once a newer snapshot supersedes it. - `pid` (integer, optional) - `scope` (string, optional): Use desktop with x,y and no pid/window_id for native get_desktop_state screenshot coordinates. default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `snapshot_id` (string, optional): Snapshot handle from get_window_state. Required when targeting by element_index; stale snapshots fail closed. - `target` (window target or desktop target, optional): Exact capture/input target selected independently for each action. `display_id="primary"` is the portable desktop target in this release. Platforms that cannot address another display reject it explicitly rather than silently changing coordinate spaces. - `window_id` (integer, optional) - `x` (number, optional): Window-local screenshot X (top-left origin of the PNG from get_window_state). With `y`, routes through the pixel-wheel path at this point — use for a scrollable surface that isn't in the AX tree. Requires window_id to anchor the window→screen conversion. - `y` (number, optional): Window-local screenshot Y. See `x`. ```json {"direction":"up"} ``` ### `move_cursor` Move a cursor to (x, y). In window scope (default), moves only the agent cursor overlay. With scope=desktop, moves the real OS pointer in native get_desktop_state screenshot coordinates. **Arguments:** - `cursor_id` (string, optional): Cursor instance to move. Default: 'default'. - `scope` (string, optional): default: `"window"` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `target` (window target or desktop target, optional): Preferred per-call target. New callers should set this field. - `x` (number, required) - `y` (number, required) ```json {"x":100,"y":200} ``` ### `zoom` Capture a cropped JPEG of a window region (x1,y1)–(x2,y2) in screenshot pixel coordinates, with 20% padding added on each side. The output image is at most 500 px wide. After a zoom, pass `from_zoom=true` to click/type_text to auto-translate coordinates back to full-window space. **Arguments:** - `pid` (integer, optional): Target pid — required for from_zoom click/type translation. - `window_id` (integer, required): CGWindowID from list_windows. - `x1` (number, required): Left edge of region in screenshot pixels. - `x2` (number, required): Right edge of region in screenshot pixels. - `y1` (number, required): Top edge of region in screenshot pixels. - `y2` (number, required): Bottom edge of region in screenshot pixels. ```json {"window_id":10725,"x1":100,"x2":100,"y1":200,"y2":200} ``` ## Browser tools ### `page` Legacy browser compatibility tool. Prefer get_browser_state and the typed browser_* tools for exact targeting, endpoint ownership, and consent. Read-only get_text and query_dom remain available by default. Mutating actions require the daemon operator to set CUA_DRIVER_ENABLE_LEGACY_PAGE_MUTATIONS=1 before daemon startup (restart the daemon after changing it); this escape hatch does not provide the typed browser surface's exact binding or existing-profile grant guarantees. Supports Chrome, Brave, Edge, Safari (via AppleScript on macOS), Electron apps (via CDP), Chromium/Firefox on Windows (via UIA for read; CDP for execute_javascript when --remote-debugging-port is set), and WKWebView/Tauri/AT-SPI fallbacks. Actions: - execute_javascript: Run JS and return the result. - get_text: Extract visible text from the page. - query_dom: Find elements matching a CSS selector. - click_element: Click a CSS-selected element AND animate the agent cursor to its on-screen center first (so the user sees what the agent is doing). Prefer over `execute_javascript('el.click()')` whenever you want visible cursor feedback. - insert_text: Insert `text` at whatever currently holds DOM focus in one native operation (CDP Input.insertText) — no synthesized key events, but more durable than a one-shot execute_javascript write since rich-text editors already have to treat it like an IME commit. Try this before type_keystrokes on a contenteditable that discarded an execute_javascript write. Click/focus the target field first. - type_keystrokes: Type `text` via real per-character keystroke events into whatever currently holds DOM focus. Slower than insert_text but the most durable rung — use it when insert_text also gets discarded, or the editor's own keydown/keyup handlers need to see real keys. Click/focus the target field first. - enable_javascript_apple_events: macOS-only — patch the browser's Preferences to allow JS from Apple Events (Chrome/Brave/Edge, requires user confirmation and a browser restart). **Arguments:** - `action` (string, required): Action to perform. - `attributes` (array of string, optional): Element attributes to include in query_dom results. - `bundle_id` (string, optional): Bundle ID of the browser. Required for enable_javascript_apple_events (macOS only). - `cdp_port` (integer, optional): Optional, for execute_javascript/insert_text/type_keystrokes: use this exact CDP port instead of auto-discovering one from pid. Needed when the port was opened via the browser's own remote-debugging toggle rather than a launch-time flag, since that path may not answer the auto-discovery probe. range: 1–65535 - `css_selector` (string, optional): CSS selector for query_dom (e.g. 'a', 'button', 'input', 'h1'-'h6', 'p', 'img', 'select', '*'). - `javascript` (string, optional): JavaScript to execute. Required for execute_javascript. - `pid` (integer, optional): Target process ID. - `selector` (string, optional): CSS selector for click_element (e.g. 'button.submit', '#login a'). - `target_url_contains` (string, optional): Optional, for execute_javascript/insert_text/type_keystrokes: require exactly one browser tab whose URL contains this substring. Use this on a multi-tab browser — there's no built-in link between window_id and which tab a CDP call reaches. - `text` (string, optional): Text to insert or type. Required for insert_text and type_keystrokes. The target field must already have DOM focus (click/focus it first). - `user_has_confirmed_enabling` (boolean, optional): Must be true to proceed with enable_javascript_apple_events. This will quit and relaunch the browser. - `window_id` (integer, optional): Target window ID from list_windows. ```json {"action":"execute_javascript"} ``` ## Clipboard tools ### `clipboard_read` List available system clipboard types and optionally return privacy-sensitive plain text. Clipboard content is never retained in telemetry. **Arguments:** - `include_text` (boolean, optional): Return plain-text clipboard content in addition to the available types. Clipboard content is privacy-sensitive and is never retained in telemetry. default: `false` - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. ### `clipboard_write` Replace the system clipboard with exactly one value: plain text, an image from an absolute local path, or a file URL from an absolute local path. Returns the available types for read-back before paste. **Arguments:** - `file_path` (string, optional): Absolute path to a local file to place on the clipboard as a file URL. - `image_path` (string, optional): Absolute path to a local image to place on the clipboard. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `text` (string, optional): Plain text to place on the clipboard. ## Recording tools ### `start_recording` Start trajectory recording. Every subsequent action-tool invocation (click, right_click, scroll, type_text, press_key, hotkey, set_value) writes a turn folder under `output_dir`: - `before_state.json` / `after_state.json` — application AX/UIA/AT-SPI state immediately before and after the action. - `before.png` / `after.png` — target-window screenshots immediately before and after the action. - `evidence.json` — capture status and a stable classification when an expected artifact could not be captured. - `app_state.json` — post-action AX/UIA snapshot for the target pid. - `screenshot.png` — compatibility alias of `after.png`. - `action.json` — tool name, full input arguments, result summary, result-error flag, pid, click point (when applicable), ISO-8601 timestamp. - `click.png` — for dispatched click-family actions only, `before.png` with a red marker at the click point. A call refused before target resolution is explicitly not applicable instead. Turn folders are named `turn-00001/`, `turn-00002/`, etc. Turn numbering restarts at 1 each time recording is (re-)started. **Video is off by default.** Pass `record_video: true` to also capture the main display to `/recording.mp4` (H.264 / 30 fps) for the lifetime of the session. The recording is torn down automatically when the MCP client disconnects. **macOS uses native ScreenCaptureKit** (daemon-owned SCStream + SCRecordingOutput) so video inherits the daemon's Screen Recording grant — no extra TCC prompt, no ffmpeg subprocess. Requires macOS 15.0+. **Windows + Linux use an ffmpeg subprocess** (`gdigrab` / `x11grab` + libx264). Requires ffmpeg on PATH (winget install Gyan.FFmpeg / apt install ffmpeg); when ffmpeg is missing or fails on startup the per-turn capture (screenshots + action.json) still runs and the session's `last_error` field carries the diagnostic. State persists for the life of the daemon; a restart resets to disabled with no on-disk state. Call `stop_recording` to disable + finalize the mp4. **Arguments:** - `output_dir` (string, required): Absolute or ~-rooted directory where turn folders and (when enabled) the video file are written. - `record_video` (boolean, optional): Capture the main display to <output_dir>/recording.mp4. Default: false. Set to true to also capture the main display to recording.mp4 (otherwise only the per-turn screenshots + JSON are recorded). On macOS this uses native ScreenCaptureKit (no extra TCC prompt, macOS 15.0+); on Windows + Linux it requires ffmpeg on PATH. ```json {"output_dir":"~/cua-trajectories/demo1"} ``` ### `stop_recording` Stop trajectory recording. Disables further per-turn capture and, when video was enabled, gracefully terminates the ffmpeg subprocess so the mp4's moov atom is finalized (the file is playable). Calling stop on an already-stopped session is a no-op. The response carries `last_video_path` pointing at the finalized mp4 (when video was on). A manual `stop_recording` is **unconditional** — it stops whatever recording is active regardless of which session started it. Ownership-scoped teardown (so one client disconnecting can't stop a recording a later client started) is handled by the registry's `session_end` lifecycle hook, not by this tool. **Arguments:** none. ### `replay_trajectory` Replay a recorded trajectory by re-invoking every turn's tool call in lexical order. `dir` must point at a directory previously written by `start_recording`. Each `turn-NNNNN/` is parsed for `action.json`, and the recorded tool is called with its recorded `arguments` via the same dispatch path an MCP / CLI call uses. Caveats: - Element-indexed actions (`click({pid, element_index})` etc.) will fail because element indices are per-snapshot and don't survive across sessions. Pixel clicks (`click({pid, x, y})`) and all keyboard tools replay cleanly. Failures are reported but don't stop replay unless `stop_on_error` is true. - `get_window_state` and other read-only tools are NOT currently recorded, so replays do not re-populate the per-(pid, window_id) element cache. - If recording is ENABLED while replay runs, the replay itself is recorded into the currently configured output directory. That's deliberate: recording a replay against a new build and diffing the two trajectories is the regression-test workflow. **Arguments:** - `delay_ms` (integer, optional): Milliseconds to sleep between turns, for human-observable pacing. Default 500. range: 0–10000 - `dir` (string, required): Trajectory directory previously written by `start_recording`. Absolute or ~-rooted. - `stop_on_error` (boolean, optional): Stop replay on the first tool-call error. Default true — set false to best-effort through the full trajectory. ```json {"dir":"~/cua-trajectories/demo1"} ``` ## Configuration tools ### `set_config` Update cua-driver-rs configuration. Changes to max_image_dimension take effect immediately. The experimental_pip keys are persisted to ~/.cua-driver/config.json and take effect on the next daemon restart (the PiP backend is initialised once at startup). Note: capture_mode is a per-call param (on get_window_state / click), not a stored setting. Capture modality is selected by each action's target; the old capture_scope config key is retired. **Arguments:** - `experimental_pip` (boolean, optional): Enable the experimental picture-in-picture preview window. Applies on next daemon restart. - `experimental_pip_geometry` (string, optional): PiP window size + optional position in `WxH` or `WxH+X+Y` form (e.g. `320x200+24+24`). Applies on next daemon restart. - `key` (string, optional): Name of a single config field to write ({key, value} shape, matching the CLI `config set` and the Windows/Linux tools). Pair with `value`. Equivalent to passing the field directly. - `max_image_dimension` (integer, optional): Max dimension for screenshot resizing (0 = no limit). - `value` (unknown, optional): New value for `key`. JSON type depends on the key. ### `start_session` Optionally create or return a lifecycle session before acting. For multi-call work, prefer a short public `session` label and repeat it on every call that accepts it; an omitted value uses the authenticated transport lease's implicit session instead. This tool is optional because an ordinary action can create or reuse a named run directly. Use it to set the initial cursor theme before acting or to revive a public name after it has ended; ordinary actions never revive ended names. `capture_scope` is deprecated compatibility input; new callers select window or desktop modality per action. Idempotent. **Arguments:** - `capture_scope` (string, optional): Deprecated compatibility policy. New callers select window or desktop modality on each action instead of storing it on the session. - `cursor_theme` (object or null, optional): Optional initial cursor theme. The host applies it before the cursor is first made visible, avoiding a flash of the default theme. - `session` (string, optional): Optional stable public label for this run (e.g. "research-run-1"). When omitted, the authenticated transport lease's implicit session is created or returned. ### `end_session` End one visible lifecycle session and run its cursor, recording, configuration, and other cleanup hooks exactly once. Omit `session` to end the authenticated transport's implicit session. Idempotent. **Arguments:** - `session` (string, optional): Optional public label to end. When omitted, end the caller's attached implicit session. ### `set_agent_cursor_enabled` Show or hide the agent cursor owned by a session. **Arguments:** - `enabled` (boolean, required) - `session` (string, required) ```json {"enabled":false,"session":"example"} ``` ### `set_agent_cursor_motion` Configure only movement physics and visibility timing for a session cursor. **Arguments:** - `arc_flow` (number or null, optional) - `arc_size` (number or null, optional) - `dwell_after_click_ms` (number or null, optional) - `end_handle` (number or null, optional) - `glide_duration_ms` (number or null, optional) - `idle_hide_ms` (number or null, optional) - `session` (string, required) - `spring` (number or null, optional) - `start_handle` (number or null, optional) - `turn_radius` (number or null, optional) ```json {"session":"example"} ``` ## Maintenance tools ### `check_permissions` Report TCC permission status for Accessibility and Screen Recording. By default also raises the system permission dialogs for any missing grants — Apple's request APIs are no-ops when the grant is already active, so this is safe to call repeatedly. Pass {"prompt": false} for a purely read-only status check. Returns: `accessibility` + `screen_recording` (booleans from the TCC preflight APIs), `screen_recording_capturable` (a live ScreenCaptureKit probe when `prompt` is true; null on read-only calls), `direct_capture_status` (`ready`, `unavailable`, `timed_out`, `probe_failed`, `blocked_by_screen_recording`, or `not_checked`), `direct_capture_error` (a structured timeout/probe failure when applicable), `direct_capture_verification` (validated source, UTC time, and bundle identity from an explicit grant probe), and `source` (which TCC identity the booleans reflect: the CuaDriver daemon vs the launching terminal/IDE). macOS attributes grants to the responsible process, so a standalone call from a terminal reports the terminal's grants, not the driver's. The prompt-capable ScreenCaptureKit probe never runs when `prompt` is false. Pass `probe_direct_capture:false` with `prompt:true` to register/request only the two required TCC grants before separately explaining Tahoe's direct-capture consent. **Arguments:** - `probe_direct_capture` (boolean, optional): When prompting and Screen Recording is granted, also run the live ScreenCaptureKit probe that may raise Tahoe's direct-capture consent. Default true. Set false for a staged Accessibility/Screen Recording request. - `prompt` (boolean, optional): Raise the system permission prompts for missing grants. Default false; only a trusted host setup route may set true. default: `false` ### `health_report` Single-call end-to-end driver diagnostics. Designed to let downstream consumers ship one stable call instead of stitching together check_permissions, doctor, version, bundle attribution, and platform capability status. On macOS, prompt-capable direct capture is deliberately skipped; use `cua-driver permissions grant` to verify it explicitly. cua-driver owns the health model; consumers stay thin. Input — all optional: { "include": ["<check_name>", ...], // run only these "skip": ["<check_name>", ...] // skip these } If both are given, `include` wins. Canonical check names: macOS : binary_version, platform_supported, session_active, bundle_identity, tcc_accessibility, tcc_screen_recording, ax_capability, screen_capture_capability Windows: binary_version, platform_supported, session_active, ax_capability (via UIA), screen_capture_capability (via DXGI) Linux : binary_version, platform_supported, session_active, ax_capability (via AT-SPI), screen_capture_capability (via X11) Output — stable contract, schema_version="1": { "schema_version": "1", "platform": "darwin" | "win32" | "linux", "driver_version": "<semver>", "overall": "ok" | "degraded" | "failed", "checks": [ { "name": "<one of the canonical names above>", "status": "pass" | "fail" | "skip", "message": "<one-line summary, always present>", "hint": "<remediation step, present when status=fail>", "data": { /* check-specific structured fields */ } }, ... ] } `overall` rules: - `ok` — every non-skipped check passes - `degraded` — at least one non-core check fails (binary is still usable) - `failed` — any core check fails (binary_version, platform_supported, session_active) Stability: schema_version="1" is the contract. Future breaking changes will be `"2"`. Adding new check names under the same schema_version is non-breaking; consumers must tolerate unknown check names. **Arguments:** - `include` (array of string, optional): Only run these checks (canonical names). Wins over `skip`. - `skip` (array of string, optional): Skip these checks (canonical names). Ignored when `include` is set. ### `check_for_update` Check the saved stable/nightly Cua Driver channel for a release on GitHub. Returns current and selected channels, current and latest versions, an `update_available` boolean, the install one-liner, and the release notes URL. Read-only — never installs. Mirror of `cua-driver check-update --json`. **Arguments:** none. ### `install_ffmpeg` Install the ffmpeg binary used by start_recording's video capture (Linux/Windows; macOS records natively and needs no ffmpeg). Two-step and confirmed: called without `confirm` it only REPORTS the exact install command for this platform's package manager; pass `confirm: true` to actually run it. No-op if ffmpeg is already on PATH. ffmpeg is run as a separate process, never linked into the driver. **Arguments:** - `confirm` (boolean, optional): Run the install command. Without it, only the planned command is reported. ## Other tools ### `verify_state` Deterministically verify bounded predicates against one exact window. The driver evaluates structured window/accessibility state and may return the final screenshot as uninterpreted visual evidence for a multimodal caller. Predicate results are satisfied, unsatisfied, or unknown; unknown never implies success. Accessibility projections are conservative: absence remains unknown unless the observed search domain is proven exhaustive. **Arguments:** - `expect` (array of object, required): One to eight predicates, combined with logical AND. items: 1–8 - `include_screenshot` (boolean or null, optional): Return the final window screenshot as image content for a multimodal caller. The driver does not interpret that image. - `pid` (integer, required): Exact process whose window may be observed. range: 1–unbounded - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. This field never selects capture modality or authorization. - `stable_samples` (integer, optional): Consecutive satisfied samples required before returning success. default: `2`; range: 1–5 - `timeout_ms` (integer, optional): Bounded wait. Zero performs one sample. default: `5000`; range: 0–10000 - `window_id` (integer, required): Exact native window identifier. ```json {"expect":[{"window":{"exists":true}}],"pid":844,"window_id":10725} ``` ### `invoke_menu` Resolve an exact application-menu path one live native level at a time and invoke its final item through accessibility APIs. Missing, ambiguous, disabled, or structurally mismatched segments fail closed; this tool never falls back to pixels. **Arguments:** - `path` (array of string, required): items: 1–16 - `pid` (integer, required): range: 1–unbounded - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. - `window_id` (integer, required): range: 1–unbounded ```json {"path":["example"],"pid":844,"window_id":10725} ``` ### `set_agent_cursor_theme` Select an already-installed cursor theme for a session. **Arguments:** - `reduced_motion` (string, optional): default: `"auto"` - `session` (string, required) - `theme_id` (string, required) ```json {"session":"example","theme_id":"example"} ``` ### `get_browser_state` Read-only browser inspection. Mode 1 (bind): pass pid + window_id of a native browser window to classify it, correlate it to a CDP target (exact-or-refuse), and mint a session-scoped target id plus tab ids. Mode 2 (snapshot): pass target_id + tab_id. The dom_refs_v1 compatibility format returns composed DOM refs. semantic_v2 joins accessibility, DOM, layout, and viewport state; ranks visible content before retained/offscreen state; and returns a semantic outline, typed action refs, content refs, scoped reads, and opaque continuation. Never performs setup — a missing endpoint is a structured browser_requires_setup refusal pointing at browser_prepare. **Arguments:** - `continuation` (string, optional): Opaque continuation minted by an earlier semantic_v2 response. - `include_screenshot` (boolean, optional): Capture the exact tab viewport as PNG through CDP without selecting the tab or foregrounding its native window. The request refuses if capture cannot be completed. default: `false` - `pid` (integer, optional): Native browser process id (bind mode). - `query` (string, optional): Read-only semantic match over role, accessible name, and visible text. - `scope_ref` (string, optional): Current semantic/content ref whose subtree should be observed. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session. - `snapshot_format` (string, optional): Versioned snapshot contract. dom_refs_v1 remains the compatibility default. - `tab_id` (string, optional): Opaque tab id from get_browser_state (session-scoped). - `target_id` (string, optional): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id). - `window_id` (integer, optional): Native window id owned by pid (bind mode). ### `browser_prepare` Explicitly prepare an owned DevTools endpoint for a browser. pid is required for an existing process or existing-profile attachment, and optional only for allow_launch=true with an isolated profile. Existing endpoints are detected without side effects. Acting setup for an isolated profile follows the runtime permission mode and optional capability manifest. It requires allow_launch=true, launches a separate browser, and never copies, modifies, or terminates the requested user profile. Without pid, only a platform-attested system Chrome/Edge installation (or a root-owned package payload on Linux) is eligible; redirects and user-controlled locations fail closed. Existing-profile attachment is explicit and follows the runtime's immutable permission mode: standard requires an explicit --grant existing-profile launch grant or an embedding authorization host, bounded requires a launch-approved exact resource manifest, and unrestricted requires explicit trusted startup risk acceptance. Ordinary MCP transport approval never proves profile authorization. On proven platforms, an authorized request also permits one bounded exact-window setup: open the recognized browser product's fixed remote-debugging page, toggle its uniquely matched per-instance checkbox, prove the PID-owned loopback endpoint, and close the temporary tab. Every visible effect is reported; ambiguity is refused. **Arguments:** - `allow_launch` (boolean, optional): Allow a separate driver-owned isolated Chromium process to be launched (default false). - `pid` (integer, optional): Browser process id to prepare. Required except for a driver-owned isolated_new/isolated_named launch with allow_launch=true. - `profile` (object, optional) - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session. - `strategy` (object, optional) - `window_id` (integer, optional): Exact native window approval anchor; required for strategy.kind=existing_profile. ### `browser_navigate` Navigate one tab of an exactly-bound browser target to a new URL (http/https/about only). Refused for heuristic bindings. Navigation invalidates all p<snapshot>:<index> refs for the tab. **Arguments:** - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session. - `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped). - `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id). - `url` (string, required): Destination URL (http:, https:, or about:). ```json {"tab_id":"example","target_id":"example","url":"example"} ``` ### `browser_click` Click a page element (by ref) or viewport coordinates in an exactly-bound tab. Default route is trusted hardware-like input (Input.dispatchMouseEvent), and refuses where that route cannot preserve standalone-browser background posture. input_route="dom_event" (synthetic el.click(), ref required) is used only when explicitly requested; it proves dispatch, not control activation, because trust-gated controls may ignore synthetic events. Refused for heuristic bindings. **Arguments:** - `input_route` (string, optional): "trusted" (default): Input.dispatchMouseEvent. It refuses rather than foregrounding a standalone browser. "dom_event": synthetic full-background DOM click, only when explicitly requested. Dispatch does not prove the control activated; refresh page state and verify the expected postcondition. - `ref` (string, optional): Page element ref in the p<snapshot>:<index> namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session. - `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped). - `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id). - `x` (number, optional): Viewport x (CSS px) — alternative to ref. - `y` (number, optional): Viewport y (CSS px) — alternative to ref. ```json {"tab_id":"example","target_id":"example"} ``` ### `browser_type` Type text into an exactly-bound tab via the Input domain. mode="insert_text" (default) uses Input.insertText; mode="keystrokes" dispatches per-character key events. Both insert at the caret, so typing into a field that already holds text appends to it; pass replace=true to set the field instead, or to clear it by typing an empty string. Pass a ref to an editable element from the latest snapshot. A ref is required; heuristic bindings are refused. **Arguments:** - `mode` (string, optional): insert_text (default): bulk Input.insertText. keystrokes: per-character Input.dispatchKeyEvent. - `ref` (string, required): Page element ref in the p<snapshot>:<index> namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab. - `replace` (boolean, optional): false (default): insert at the caret, appending to whatever the field already holds. true: select the element's whole content first so the text replaces it — with an empty text this clears the field. Replacement goes through the selection, so beforeinput/input still fire and framework state stays consistent. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session. - `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped). - `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id). - `text` (string, required): Text to type. ```json {"ref":"example","tab_id":"example","target_id":"example","text":"hello"} ``` ### `browser_dialog` Inspect or resolve a page-owned JavaScript alert, confirm, prompt, or beforeunload dialog on one exactly-bound tab. This never handles browser permission UI, extension UI, native dialogs, or file pickers. Inspect returns an opaque dialog_id; accept/dismiss require that exact current id. Resolution defaults to background delivery; Linux callers must explicitly request foreground delivery because Chromium's native modal cannot be resolved there without changing foreground posture. **Arguments:** - `action` (string, required) - `delivery_mode` (string, optional): Requested foreground posture for accept/dismiss. Linux Chromium requires foreground; inspect is read-only. default: `"background"` - `dialog_id` (string, optional): Opaque current dialog generation returned by action=inspect. - `prompt_text` (string, optional): Sensitive response text, valid only when accepting a prompt dialog. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session. - `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped). - `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id). ```json {"action":"inspect","tab_id":"example","target_id":"example"} ``` ### `browser_set_input_files` Assign one or more explicit absolute local files to an exact live <input type=file> ref through CDP. This bypasses native file pickers, rejects symlinks and non-regular files, and never returns local paths. **Arguments:** - `files` (array of string, required): items: 1–32 - `ref` (string, required): Page element ref in the p<snapshot>:<index> namespace from get_browser_state. Refs are invalidated by navigation and by newer snapshots of the same tab. - `session` (string, optional): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. Browser targets, tabs, and refs belong to the resolved lifecycle session. - `tab_id` (string, required): Opaque tab id from get_browser_state (session-scoped). - `target_id` (string, required): Opaque browser target id minted by get_browser_state (session-scoped; never a CDP id). ```json {"files":["example"],"ref":"example","tab_id":"example","target_id":"example"} ``` ### `browser_download` Trigger one download through an exact live browser ref and save it inside an explicitly approved directory. Requires MCP-host destructive-tool approval, refuses ambiguous or stale capabilities, and never returns the source URL, filename, or destination path. **Arguments:** - `destination_root` (string, required): Absolute, existing, canonical directory approved to receive the download. - `ref` (string, required): Live page ref whose activation initiates the download. - `session` (string, required): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. This tool requires the label that owns its browser target, tab, and refs. - `tab_id` (string, required): Opaque exact tab id from get_browser_state. - `target_id` (string, required): Opaque exact browser target id from get_browser_state. ```json {"destination_root":"example","ref":"example","session":"example","tab_id":"example","target_id":"example"} ``` ### `browser_pointer` Perform hover, right-click, double-click, scroll, or drag in an exactly-bound browser tab. Semantic refs must declare pointer for hover, right-click, double-click, and drag; scroll accepts a scroll or pointer capability. The trusted route uses CDP Input events and refuses if standalone background posture cannot be preserved. The explicit dom_event route requires a page ref and synthesizes full-background DOM events. Never activates or brings a tab to the foreground. **Arguments:** - `action` (string, required) - `delta_x` (number, optional): Horizontal scroll delta in CSS pixels. - `delta_y` (number, optional): Vertical scroll delta in CSS pixels. - `destination_ref` (string, optional): Drag destination page ref in the exact same frame. - `input_route` (string, optional): default: `"trusted"` - `ref` (string, optional): Origin page ref. Alternative to x/y. - `session` (string, required): For multi-call work, prefer a short public session label and repeat it on every call that accepts it. This tool requires the label that owns its browser target, tab, and refs. - `tab_id` (string, required): Opaque tab id minted by get_browser_state. - `target_id` (string, required): Opaque target id minted by get_browser_state. - `to_x` (number, optional): Drag destination viewport x in CSS pixels. - `to_y` (number, optional): Drag destination viewport y in CSS pixels. - `x` (number, optional): Origin viewport x in CSS pixels. - `y` (number, optional): Origin viewport y in CSS pixels. ```json {"action":"hover","session":"example","tab_id":"example","target_id":"example"} ``` ### `escalate_session` Deprecated compatibility tool for legacy capture-scope sessions. New callers select window or desktop modality on each action. No deescalate_session tool exists. **Arguments:** - `detail` (string, optional): Optional bounded diagnostic detail. Never use secrets or page content. - `reason` (string, required) - `session` (string, required) ```json {"reason":"ax_tree_pixel_mismatch","session":"example"} ``` ### `get_session` Read content-free lifecycle, cursor, recording, and idle status for one session visible to this authenticated transport. Omit `session` to inspect its implicit session. **Arguments:** - `session` (string, optional): Optional public label. When omitted, inspect the caller's attached implicit session. ### `list_sessions` List content-free lifecycle summaries attached to this authenticated transport lease. It does not enumerate other callers' sessions. **Arguments:** - `cursor` (string, optional): Opaque continuation cursor returned by a previous call. - `limit` (integer or null, optional): Maximum number of content-free summaries to return (default 50, max 100). Ordinary agent transports are scoped to their own lease. range: 0–unbounded ### `get_session_state` Deprecated compatibility alias that reads a live legacy session's capture policy. Use get_session for lifecycle state. **Arguments:** - `session` (string, optional): Optional public label. When omitted, inspect the caller's attached implicit session. --- # MCP Tool Notes Cross-cutting MCP tool contracts: shared parameters, required-parameter rules, platform-specific parameters, and the action response shape. These notes are hand-maintained companions to the auto-generated [MCP Tools]() reference. They document the cross-cutting parameter contract and response shape that span multiple tools and are not derivable from any single tool's schema. ## Common parameters Several parameters are a **shared cross-platform contract**: the same JSON shape on Windows, macOS, and Linux, composed from canonical schema fragments and enforced by a CI consistency gate so the three platforms cannot drift. Tools accept them uniformly. | Parameter | Where | Notes | | --------- | ----- | ----- | | `session` | every action and cursor tool | Optional public run label. For multi-call work, prefer a short label and repeat it on every call that accepts it; it is not sticky. When this field is absent, the call uses the authenticated transport's private implicit session. A public label is never caller identity or authorization evidence. Accepted on all three platforms. | | `target` | `move_cursor`, `click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey` | Preferred tagged per-call target: `{kind:"window", pid, window_id}` or `{kind:"desktop", display_id:"primary"}`. It cannot be combined with legacy `scope`, `pid`, or `window_id` fields. | | `delivery_mode` | the input family (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`) | `"background"` (default) tries to inject without fronting or raising the target. See [Best-effort background](). `"foreground"` briefly fronts the target, acts, then restores the prior frontmost. Use it when a background attempt did not land. Legacy `"auto"` is removed; omitted or unknown values fall back to `"background"` for safety. | | `capture_mode` | `get_window_state` | **Deprecated and ignored.** Still accepted for back-compat so old callers do not error, but it has no effect. `get_window_state` always returns both the accessibility tree and a screenshot by default. There is no `ax`/`vision`/`som` capture choice; the modality (`ax` vs `px`) is chosen at action time by how you address the target. | | `include_screenshot` | `get_window_state` | Boolean, default `true` (returns the tree **and** a screenshot). Set `false` to skip the screenshot grab and return the tree only when re-indexing before an element-`ax` action. | | `modifier`, `button`, `element_index`, `element_token` | pointer and element tools | Held modifier keys, mouse button, and the two element-addressing handles. | ### Required parameters The `required` set is uniform across platforms: `click` requires nothing, `scroll` requires `direction`, and `zoom` requires `window_id` plus `x1`/`y1`/`x2`/`y2`. The preferred `target` makes the modality explicit. Legacy flat calls still validate `pid` conditionally: a window action needs it, while a desktop action omits it. The first admitted stateful call creates the transport's implicit lifecycle session. Repeated unnamed clicks on the same MCP connection belong to that one session; they do not create one session per click. The default idle TTL is five minutes, separate from the cursor's visibility timeout. Transport close, explicit end, or idle expiry runs the same cleanup path. ### PID-only window targets A window-scoped action may omit `window_id` only when its `pid` owns exactly one eligible top-level window. The driver resolves that unique window before dispatch. When the PID owns multiple windows, it sends no input and returns `code: "ambiguous_window_target"`, `effect: "refused"`, and candidate metadata. Select a candidate from that result or `list_windows({pid})`, then retry with the exact `window_id`. A PID with no eligible windows returns `window_target_not_found`. Explicit `(pid, window_id)` and `element_token` targets keep their exact resolution behavior. ### Platform-specific parameters A few parameters are platform-specific by design and are intentionally NOT part of the shared contract: | Parameter or tool | Platform | Why | | ----------------- | -------- | --- | | `launch_app` identifiers | macOS: `bundle_id`, `urls`. Windows: `aumid`, `launch_path`, `path`, `start_minimized`. | App launch is OS-native. `name` is the portable fallback that works on both. | | `debug_window_info` | Windows only | A window-handle / class / rect / z-order debug tool with no macOS or Linux counterpart. | | `check_permissions.prompt` | macOS only | Public tool calls are read-only by default. `prompt:true` is refused before platform dispatch; use the human-run `cua-driver permissions grant` setup command, which launches the installed app through LaunchServices. There is no Windows or Linux equivalent of TCC. | | Per-call window or desktop target | all three platforms | `target` has the same tagged shape on each platform. `display_id:"primary"` is the portable desktop target in this release; unsupported display IDs fail explicitly. Legacy scope/config fields remain only for compatibility. | | `modifier` on a *background* click | honored on macOS and Linux; dropped on a Windows background click | A Windows background click goes through UIA Invoke / PostMessage, which carry no live keyboard state, so `modifier` takes effect only on the `delivery_mode:"foreground"` (SendInput) rung there. | ## Action response shape Action tools (`click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`) return these structured fields: | Field | Type | Meaning | Value set / presence | | ----- | ---- | ------- | -------------------- | | `path` | string | Delivery rung that ran. | `"ax"`, `"cgevent"`, `"cgevent_fg"`, `"key_events"`, `"key_events_fg"`, `"pixel"`, `"x11_atspi"`, `"x11_pixel"`, `"x11_pixel_fg"`, `"msaa"`. | | `verified` | boolean or absent | AX read-back verification result. `true` means the driver read the effect back through AX; `false` means the action ran but is unconfirmed; absent means the tool does not carry this field. | `true`, `false`, or absent. | | `effect` | string | Action confidence signal. | `"confirmed"`, `"unverifiable"`, `"suspected_noop"`. | | `escalation` | object or absent | Machine-readable next-rung recommendation. Present only when the driver recommends climbing the ladder. | `{ recommended: "px" \| "foreground" \| "page", reason: string }`, or absent. | ## Per-tool notes ### `get_window_state` degraded results On Linux (and macOS/Windows), the structured result may include `degraded: true` alongside a `degraded_reason` string when the accessibility walk completed but found no actionable elements. This distinguishes "a11y bridge not up, daemon not on the session D-Bus, or non-AX surface" from a window that genuinely has no controls. Treat `elements: []` as incomplete when `degraded: true` is set, and act by `px` off the screenshot returned in the same response. ### `page` platform support `get_text`, `query_dom`, `click_element`, and `execute_javascript` work cross-platform (macOS, Windows, Linux). `insert_text` and `type_keystrokes` are implemented on macOS only for now; Windows and Linux return a clear "not implemented" error rather than a silent no-op. Tracked in [trycua/cua#2084](https://github.com/trycua/cua/issues/2084). --- # Browser Profile Attachment Contract, lifetimes, refusals, and platform scope for attaching Cua Driver to an existing Chromium profile. Existing-profile attachment is an additive `browser_prepare` strategy. It does not change the lifecycle of driver-owned `isolated_new` and `isolated_named` profiles. ## Request ```jsonc { "pid": 844, "window_id": 10725, "session": "research-1", "strategy": { "kind": "existing_profile" }, } ``` `pid`, `window_id`, and `session` are required. `strategy` cannot be combined with legacy `allow_launch` or `profile` fields. Authorization depends on the runtime's immutable [permission mode](): - `standard` requires either the trusted launch option `--grant existing-profile` or an authorization callback supplied by the embedding application. - `bounded` requires the approved manifest to match the browser profile, application identity, and requested operation. - `unrestricted` attaches without a second authorization after the trusted launcher supplied `--dangerously-bypass-approvals`. Cua Driver does not display its own confirmation modal or persistent banner. The CLI launch grant is useful when a person or trusted supervisor starts the runtime. Embedded applications can collect authorization in their own UI and return the decision through `DriverAuthorizationHost`. The ordinary MCP destructive-tool marker does not authorize this strategy. It proves transport provenance, not approval to use an authenticated browser profile. A standalone standard-mode runtime without a launch grant or host callback fails closed. ## Success result A successful call returns `action: "attached_existing_profile"` and an `attachment` object with: | Field | Value | | -------------------------- | ------------------- | | `kind` | `existing_profile` | | `browser` | `chromium` | | `capabilities_invalidated` | `true` | | `next_action` | `get_browser_state` | `side_effects` reports the bounded setup and browser-owned connection effects: | Field | Meaning | | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `opened_setup_page` | A temporary tab was opened in the approved native window. | | `closed_setup_page` | That temporary setup tab was closed successfully. | | `focused_setup_address_field` | The temporary tab's address field received in-app focus for exact navigation. | | `enabled_remote_debugging` | The exact per-instance Chrome checkbox was toggled from off to on and the same control's resulting state was verified. | | `used_bounded_pixel_fallback` | macOS used its setup-page-only pixel route to read or change the checkbox after the web AX subtree was unavailable. | | `foregrounded_window` | Setup temporarily foregrounded the exact approved browser window for a bounded local action. | | `injected_global_input` | Setup delivered global mouse or keyboard input only after revalidating the approved browser process and window. | | `changed_preferences` | Mirrors `enabled_remote_debugging` for the generic prepare contract. | | `displayed_consent_prompt` | Chrome displayed its browser-owned connection prompt. | All fields are `false` when the endpoint was already available. The driver does not return the profile identity, endpoint address, port, connection generation, grant identifier, or internal authorization details. A refusal after setup begins may include `detail.setup_side_effects`. This uses the same setup fields and adds `restored_remote_debugging` when the driver had to reverse its own checkbox change. It never claims cleanup succeeded unless the exact checkbox returned to the off state. On macOS, Cua Driver prefers the checkbox's semantic AX action. Current Chrome versions can withhold the internal page's web AX subtree even while exposing the native address field and selected setup tab. The bounded fallback runs only in the temporary tab Cua Driver created and navigated, after the fixed internal URL is committed, the expected title is selected, and no omnibox edit is in progress. It requires one unique checkbox-shaped control in the setup-page region, revalidates the unchanged window, sends the click only to the approved browser PID, and verifies the visual state change. That bounded global-input route may briefly foreground the exact approved window and then restores the previous frontmost app; `foregrounded_window` and `injected_global_input` report what occurred. Unsupported appearance, scale, zoom, window-size, or browser-toolbar geometry—including layouts shifted by a bookmarks bar—is refused without a click. It is not available for ordinary web content. On refusal, `restored_remote_debugging: true` means the exact semantic checkbox, or the setup page's sole bounded pixel checkbox when web AX became unavailable, was proven off during cleanup. A false value means cleanup could not prove the state; callers must not infer that remote debugging is disabled. ## Chrome 144+ auto-connect Chrome 144 and later can publish an agent auto-connect endpoint for its running profile after remote debugging is enabled. Cua Driver reads the endpoint from that browser process's `DevToolsActivePort` file, proves that the live loopback listener belongs to the approved process, and connects without restarting the browser. The profile keeps its current tabs, extensions, cookies, and sign-in state. See Chrome's [agent auto-connect guide](https://developer.chrome.com/docs/devtools/agents/use-cases/auto-connect). Endpoint discovery is provenance rather than authorization. A `DevToolsActivePort` file, custom user-data path, or process-owned socket never grants access to a standalone profile. The caller must complete `browser_prepare` for the exact PID, window, and session first. After binding, `get_browser_state` reports the non-sensitive connection class: | Field | Existing-profile value | Meaning | | ----------------------- | ---------------------------- | -------------------------------------------------------------- | | `endpoint_transport` | `dev_tools_active_port` | Chrome published the endpoint through its auto-connect bridge. | | `endpoint_access_class` | `existing_profile_approved` | A live existing-profile grant authorizes this connection. | The result never exposes the port, WebSocket path, or profile path. Cua Driver also rejects CDP commands that add persistent scripts, intercept requests, or override browser identity on this connection. Ending the Cua session revokes its in-memory grant and closes its socket. It does not close Chrome or disable Chrome's remote-debugging preference. Disable that setting in Chrome when the profile should stop accepting local DevTools connections. This attachment route does not promise fewer CAPTCHAs or bypass site checks. Sites can still use network reputation, account history, browser state, and interaction patterns. ## Grant lifetime The grant is scoped to the runtime instance, public session, transport session, browser process fingerprint, approved native window, and proven loopback endpoint. It expires after 30 minutes of inactivity or eight hours absolutely, whichever comes first. Session end, browser prompt dismissal, identity mismatch, reconnect exhaustion, and runtime shutdown revoke it. One browser-level connection is owned per generation. At most three reconnect attempts are made after a socket loss. A successful reconnect increments the generation and invalidates every target, tab, snapshot, frame, and ref minted by the previous generation. ## Security properties | Property | Guarantee | | -------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Network exposure | The accepted DevTools endpoint must be loopback-only. | | Process binding | The endpoint, native window, and live process fingerprint must agree before every mutation. | | Driver authorization | Standard requires a trusted launch grant or host callback. Bounded requires a matching approved manifest. A model-supplied Boolean and ordinary MCP approval are insufficient. | | Revocation | `cua-driver revoke`, session end, browser identity changes, expiry, and runtime shutdown revoke the attachment grant. | | Capability scope | Targets, tabs, dialogs, frames, and refs are opaque, session-scoped, and invalidated after navigation or connection-generation changes. | | Profile handling | Cua Driver does not copy, edit, restart, or terminate the selected existing profile. | | Residual local risk | CDP itself is not authenticated. Another process running as the same OS user may connect to an exposed endpoint outside Cua Driver. Cua Driver is a same-user tool boundary, not a defense against a hostile process with the same OS account. | An isolated profile is the recommended default. Existing-profile attachment is appropriate only when the task requires its authenticated state and the host is trusted. See [Browser targeting and background delivery]() for why Cua Driver uses CDP and which risks remain outside the driver. ## Refusals | Code | Meaning | | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `browser_consent_required` | Standard has no launch grant or host decision, or bounded has no matching manifest grant. | | `browser_consent_revoked` | The host denied the request or the browser-owned connection prompt was dismissed. | | `browser_requires_setup` | No unique approved browser endpoint is available. | | `browser_endpoint_owner_mismatch` | Endpoint ownership or identity changed. | | `browser_wrong_target_refused` | The native window, prompt, or browser target is not exact. | | `browser_binding_stale` | A connection generation changed; bind again. | | `browser_reconnect_exhausted` | The bounded reconnect policy could not establish a proven socket. | | `browser_input_incomplete` | Keystroke typing delivered only a reported prefix. | | `browser_route_unavailable` | The classified engine or platform has no accepted typed route. For Firefox and Safari, bounded detail identifies the required protocol and current lifecycle limitation. | | `browser_input_trust_unavailable` | Trusted input cannot preserve background posture. Bounded detail identifies the explicit `dom_event` alternative when a synthetic ref-targeted action is acceptable. | | `browser_origin_outside_scope` | A bounded-mode tab left the manifest's allowed origin set; further mutation pauses. | ## Recording and telemetry Host authorization events contain only bounded metadata and the structured outcome. They do not include browser screenshots or accessibility snapshots. Public results, recordings, and telemetry must not contain profile paths or hashes, endpoint URLs or ports, page content, tab URLs, cookies, or storage. The content-free browser operation and refusal fields are documented in [Telemetry and privacy](). ## Platform scope The browser-setup route is implemented for Google Chrome and Microsoft Edge on macOS and Windows. Chrome and Edge have product-specific acceptance evidence on Linux X11. Chrome is also accepted on GNOME Wayland, and Chromium is accepted alongside Chrome and Edge on the validated native-Wayland Sway lane. The Chromium-family products use the same descriptor-backed Linux route, but still need product-specific acceptance evidence on each desktop before they are listed as validated. Every route requires an exact authorized PID and native window, one uniquely matched setup control, a loopback endpoint owned by that process, one exact browser-owned consent action, and a fresh native/CDP rebind. Linux additionally requires the browser's complete AT-SPI renderer tree. Start the Chromium-family process with `--force-renderer-accessibility`, or run a screen reader that enables full renderer accessibility. On Sway and validated GNOME Shell sessions, Cua Driver briefly focuses the exact compositor-attested window for fixed internal-page navigation and restores the previous window before continuing. GNOME requires the maintained WinRects helper API v4 and an immutable D-Bus owner that resolves to the current user's system-installed GNOME Shell process. Cua Driver refuses generic Wayland sessions that cannot prove exact process, window, and geometry identity. The setup adapters currently recognize the products' English accessibility labels. Other UI locales fail closed without toggling an unrecognized control. Safari, Firefox, Brave, Vivaldi, Opera, Arc, and Electron do not have an existing-profile setup descriptor and return a structured refusal. An already available endpoint does not bypass existing-profile authorization for a standalone consumer browser. Driver-owned profiles and supported embedded applications can still use their ordinary exact binding routes when the platform independently proves the endpoint. See [Drive a web page]() for the task workflow and [Browser targeting and background delivery]() for the security and generation model. --- # Browser semantic snapshots Reference for semantic_v2 browser state, refs, scopes, and continuation capabilities. `get_browser_state` snapshot mode supports two versioned contracts: | Format | Status | Output | | --- | --- | --- | | `semantic_v2` | Recommended for new browser workflows | Semantic outline, typed action refs, content refs, visibility, omission counts, and continuation | | `dom_refs_v1` | Compatibility default when `snapshot_format` is omitted | Interactive DOM refs and a `truncated` flag | Bind a native browser window first. Snapshot mode requires the returned opaque `target_id`, one returned `tab_id`, and the same explicit driver `session`. ## Request ```json { "session": "browser-run-1", "target_id": "bt-...", "tab_id": "tab-...", "snapshot_format": "semantic_v2" } ``` Semantic snapshot requests may also contain one of these read scopes: | Field | Meaning | | --- | --- | | `query` | Match role, accessible name, or visible text and retain semantic ancestor context | | `scope_ref` | Read the subtree rooted at a current action ref or content ref | | `continuation` | Read the next ranked segment from the same stored snapshot | `continuation` cannot be combined with a new `query` or `scope_ref`. ## Response ```json { "status": "ok", "mode": "snapshot", "target_id": "bt-...", "tab_id": "tab-...", "snapshot": { "id": "p42", "format": "semantic_v2", "complete": false, "scope": "viewport", "selected_nodes": 300, "total_nodes": 318, "node_budget": 300, "omitted": { "css_hidden": 410, "offscreen": 82, "page_occluded": 1, "no_layout": 0, "unknown": 0, "budget": 18, "unprovable_frame": 0 }, "continuation": "bc-..." }, "page": { "url": "https://example.test/inbox", "title": "Inbox" }, "outline": "- heading \"Message\"\n- textbox \"Reply body\"", "refs": [], "content_refs": [], "oopif": { "status": "attached", "frames": 1 } } ``` `complete` is true only when collection is complete and no ranked state remains behind the output budget. A non-null continuation identifies that remaining state. Known oversized-DOM fallback produces a partial snapshot with `complete: false`; unrelated transport failures remain errors. ## Action refs Entries in `refs` declare their supported action kinds: ```json { "ref": "p42:8", "role": "button", "name": "Reply", "value": null, "states": { "disabled": false }, "actions": ["click"], "frame": "main", "visibility": "in_viewport" } ``` The closed action vocabulary is `click`, `type`, and `upload`. `browser_click`, `browser_type`, `browser_set_input_files`, and `browser_download` reject a semantic ref when the required action is absent, returning `browser_action_unavailable` before delivery. A file input declares `upload`, not `type`. Legacy `dom_refs_v1` refs retain their existing behavior. ## Content refs `content_refs` identify readable semantic nodes that have a live backend node but no declared mutation. They exist so a caller can pass `scope_ref` without making static text clickable or editable. ## Visibility `visibility` has this closed vocabulary: | Value | Meaning | | --- | --- | | `in_viewport` | Layout bounds intersect the current page viewport | | `near_viewport` | Layout bounds are within the bounded near-viewport margin | | `offscreen` | Layout exists outside that margin | | `css_hidden` | DOM or computed style proves the node hidden | | `no_layout` | No non-empty layout box is available | | `page_occluded` | A higher-painted fixed or absolute page overlay conservatively covers the node | | `unknown` | Available evidence cannot classify layout visibility | Page visibility is independent of native window foreground or desktop occlusion. It does not indicate whether the browser window is covered by another application. ## Capability lifetime Targets, tabs, snapshots, refs, and continuations are opaque capabilities. They do not expose CDP target IDs, backend node IDs, object IDs, or selectors. - A newer snapshot of the tab invalidates older refs and continuations. - Navigation invalidates every snapshot of the tab. - Browser reconnect or process replacement invalidates the target generation. - Ending the driver session removes the complete capability namespace. - Continuations are single-use and resolve only in their owning session, target, tab, snapshot, and generation. Every mutation re-proves the native binding, endpoint ownership, tab target, frame loader identity, and backend-node liveness. ## Composition Semantic snapshots join the browser accessibility tree, pierced author DOM, layout snapshot, viewport metrics, and frame tree. They include the main frame, author shadow roots, proven same-process frames, and capability-tested out-of-process frames. User-agent shadow roots and unprovable frame content are omitted. CSS-hidden retained state is removed before ranking. Active dialogs and focused context rank first, followed by visible actions, visible content, near-viewport state, and offscreen state. Document order is stable within each tier. --- # Agent action policy Behavior an agent should follow when choosing element, pixel, page, and foreground actions in Cua Driver. This policy describes how an agent should choose Cua Driver action parameters. It is meant for agent prompts, wrappers, and evaluators. It is not a user walkthrough. Use it when the agent must decide what to pass to `click`, `type_text`, `get_window_state`, and related tools: `element_index` versus `x, y`, `delivery_mode: background` versus `foreground`, and a window versus desktop target. ## Preconditions The agent should already be connected to the driver and able to launch an app and read a window. Perception is no longer a mode to pick: `get_window_state` returns both the accessibility tree and a screenshot in one call by default, and the agent chooses the rung at action time. For the concepts behind the axes, see [Capture and delivery modalities](). ## Start on the accessibility path (the default) The agent should default to the **element ax action with background delivery**. Act by `element_index`. It is the only rung the driver can verify, it avoids foregrounding when the target surface supports it, and it works on Windows, macOS, and Linux (X11 and Wayland). Read the window once, then act on an element from that snapshot. The snapshot already carries the screenshot too, so the agent does not re-capture to switch how it addresses the target: ```jsonc // 1. snapshot: returns the accessibility tree AND a screenshot by default get_window_state({ pid, window_id }) // → elements[], each with an element_index and a frame, plus a grounding screenshot // 2. act by element_index: the element ax action, background by default click({ pid, window_id, element_index: 12 }) type_text({ pid, text: "hello" }) ``` When the agent is only re-indexing before an element ax action and does not need fresh pixels, pass `include_screenshot: false` to skip the screen grab and get the tree alone. The `ax` versus `px` decision still happens at action time, by how the agent addresses the target. To pin the rendered frame to disk instead of inlining it, set `screenshot_out_file`. ## Follow the escalation ladder Every action response carries the signals that tell the agent the next rung. Walk the ladder in order, and only step down when the response says to: 1. **Element ax action, background (the default).** Act by `element_index`. If the response shows `effect: "confirmed"`, the driver read the result back. If `get_window_state` came back `degraded` (empty AX tree), an action returns `effect: "suspected_noop"` (the AX action ran but likely no-op'd), an action returns `effect: "unverifiable"` on an echo-prone surface, or the tree disagrees with the screenshot (an `h:1` or off-viewport row), follow `escalation.recommended`. 2. **Element px action, background.** When `escalation.recommended` is `"px"`, pick the target pixel from the screenshot already in the `get_window_state` response and click it. Coordinates are window-relative for a windowed target. ```jsonc // the screenshot is already in the snapshot above; read a pixel off it click({ pid, window_id, x: 320, y: 210 }) // → { path: "cgevent", effect: "unverifiable" } ``` Use the same px form for keyboard fallback. If AX `type_text`, `press_key`, or `hotkey` returns `effect: "unverifiable"` on Electron/Chromium, retry with `x, y`: the tool pixel-clicks to focus, then sends the keys. ```jsonc type_text({ pid, window_id, element_index: 18, text: "hello" }) // → { effect: "unverifiable", escalation: { recommended: "px", reason: "..." } } type_text({ pid, window_id, x: 320, y: 210, text: "hello" }) press_key({ pid, window_id, x: 320, y: 210, key: "return" }) hotkey({ pid, window_id, x: 320, y: 210, keys: ["cmd", "a"] }) ``` On Linux this still avoids synthetic input where it can: the driver resolves the pixel to the element under it and fires that element's action via AT-SPI `doAction` at that point. See [Known limits]() for the remaining Wayland keyboard gap. 3. **Browser-tab DOM.** When `escalation.recommended` is `"page"`, switch to the `page` tool for browser-tab DOM work instead of retrying the AX write. ```jsonc type_text({ pid, window_id, element_index: 18, text: "hello" }) // → { effect: "unverifiable", escalation: { recommended: "page", reason: "..." } } page({ pid, window_id, action: "execute_javascript", javascript: "document.querySelector('#search').value = 'hello'" }) ``` 4. **Foreground.** If the response recommends `"foreground"` or the pixel click still does not land, retry with **`delivery_mode: "foreground"`**. This activates the window first. Common cases include DirectInput games, raw-input canvases (Blender, Unity), and focus-polling apps. ```jsonc click({ pid, window_id, x: 320, y: 210, delivery_mode: "foreground" }) // → { path: "cgevent_fg", effect: "unverifiable" } ``` Use foreground only for the action that needs it, and only when the user is not actively working on the machine. It raises the window. See [Known limits]() for the specific apps. ### The escalation signal on the response Two additive fields make the ladder explicit, so the agent escalates from data rather than a hunch: - **`effect`**: `"confirmed"` (the driver verified the result through AX read-back), `"unverifiable"` (the rung fired but only the caller can confirm), or `"suspected_noop"` (an AX action ran but almost certainly did nothing). - **`escalation`**: present when there is a next rung: `{ recommended: "px" | "foreground" | "page", reason }`. A `degraded` `get_window_state` carries the same hint (recommending `px`). ```jsonc click({ pid, window_id, element_index: 12 }) // → { effect: "suspected_noop", escalation: { recommended: "px", reason: "..." } } ``` **Wayland exception.** On a native Wayland session, raw keyboard input has no universal background path. When an AX keyboard action no-ops there, prefer an accessible field action or run the app under XWayland; otherwise escalate only that action to foreground. See [Known limits](). ## Use a desktop target only for screen-absolute work Use a **desktop target** only when the action has no single window, such as dragging between windows or clicking absolute screen coordinates. It uses foreground pixel input, so it cannot provide best-effort background behavior. ```jsonc get_desktop_state() // full-screen screenshot click({ x: 1280, y: 40, target: { kind: "desktop", display_id: "primary" }, }) ``` Choose the target on each action. This does not change the session or disable window-targeted tools for later calls. ## Confirm the action landed Only AX read-back can produce `verified: true` (the driver read the result back). Echo-prone AX surfaces, pixel actions, and foreground actions return `verified: false` or omit it; use `effect` and `escalation` to decide the next call. After an unverifiable action, re-read and check (the re-read returns both the tree and the screenshot): ```jsonc click({ pid, window_id, x: 320, y: 210 }) // → { verified: false, path: "cgevent", effect: "unverifiable" } get_window_state({ pid, window_id }) // confirm the change against tree + screenshot ``` ## Troubleshooting **Problem: the call returned success but nothing changed (false success).** Do not trust the status code on a `verified: false` action. Re-read the window. The snapshot carries both the tree and the screenshot. Confirm the effect; if it did not land, switch rung (element ax action -> element px action off the same screenshot) or escalate delivery (background -> foreground). **Problem: `desktop_scope_disabled` on a window-less click.** This is a legacy flat call without an admitted desktop scope. Retry with an exact window target, or use `target: {kind: "desktop", display_id: "primary"}` for screen-absolute work. **Problem: keystrokes don't land on a Linux app.** On native Wayland, raw keys have no background path. Type into accessible fields with `type_text`, drive controls by `element_index`, or run the app under XWayland. See [Known limits](). ## Related - [Capture and delivery modalities](): the axes and the validity matrix - [MCP tools](): full parameters for every tool - [Known limits](): targets that need foreground - [Interface contracts](): valid combinations and platform support --- # Permission modes Choose standard, bounded, or unrestricted authorization for Cua Driver. Cua Driver enforces authorization inside the native runtime, after transport arguments are sanitized and before a platform action runs. CLI, MCP, direct SDK, private-worker, and service-backed calls all reach the same enforcement boundary. The three public names are authorization profiles over that shared engine. They select built-in capability and approval behavior; a capability manifest is a separate narrow-only ceiling. ## Choose a mode | Mode | Best for | Runtime behavior | | --- | --- | --- | | `standard` | Normal local CLI and MCP use | Routine operations follow the built-in profile. Residual boundaries still require a launch grant or trusted host decision. An optional capability manifest can narrow tools and resources. | | `bounded` | Unattended agents, gateways, and embedded applications | A capability manifest is required. In-scope work is silent and undeclared scope is denied. | | `unrestricted` | Disposable or fully trusted environments | Cua approval checks are bypassed after an explicit dangerous acknowledgement. An optional capability manifest limits that bypass to declared scope. | | Profile | Manifest absent | Manifest present | | --- | --- | --- | | `standard` | Preserve standard behavior | Standard behavior intersected with manifest scope; residual grants remain required | | `bounded` | Startup fails | Unattended, deny-by-default bounded behavior with required lifetimes | | `unrestricted` | Preserve the acknowledged approval bypass within existing ceilings | Bypass approval only inside manifest scope | `standard` is the default: ```bash cua-driver mcp ``` Use `bounded` with a reviewed manifest: ```bash cua-driver serve \ --permission-mode bounded \ --capability-manifest ./cua-capabilities.yaml \ --approve-capability-manifest ``` Use `unrestricted` only when full autonomous access is intentional: ```bash cua-driver serve --dangerously-bypass-approvals ``` Add the same two capability-manifest flags to `standard` or `unrestricted` when that runtime should have a smaller tool and resource surface. A manifest can narrow a profile but never widen it. Passing `--permission-mode unrestricted` without `--dangerously-bypass-approvals` fails closed. **Warning** Unrestricted mode does not defend against prompt injection or unintended model actions. Use it only where you accept the full effect of every capability allowed by the built-in, managed, and user policy ceilings. `autonomous` remains a manifest compatibility alias for `bounded`. `yolo` remains a configuration alias for `unrestricted`. New integrations should use the canonical names. ## Configure a mode without CLI flags The mode is read once, when the runtime that owns the driver starts. Only `cua-driver serve` takes the flags above. A launcher that cannot pass them — `cua-driver mcp` owning its own runtime on Windows and Linux, a Windows Scheduled Task registered by `cua-driver autostart enable`, or an embedding host — uses the equivalent environment variables: | Flag | Environment variable | | --- | --- | | `--permission-mode ` | `CUA_DRIVER_PERMISSION_MODE=` | | `--capability-manifest ` | `CUA_DRIVER_CAPABILITY_MANIFEST_FILE=` | | `--approve-capability-manifest` | `CUA_DRIVER_CAPABILITY_MANIFEST_APPROVED=1` | | `--dangerously-bypass-approvals` | `CUA_DRIVER_DANGEROUSLY_BYPASS_APPROVALS=1` | The variables accept `1`, `true`, `yes`, or `on` for their boolean form. The CLI flag normalizes mode plus acknowledgement in one step; the environment form does not, so `unrestricted` requires `CUA_DRIVER_PERMISSION_MODE=unrestricted` **and** `CUA_DRIVER_DANGEROUSLY_BYPASS_APPROVALS=1`, and `bounded` requires both the capability manifest and its approval. Missing halves fail startup instead of downgrading to `standard`. `CUA_DRIVER_SESSION_POLICY_FILE`, `CUA_DRIVER_SESSION_POLICY_APPROVED`, `--session-policy`, and `--approve-session-policy` remain deprecated aliases for one compatibility window. New integrations should use the capability manifest names. Conflicting old and new paths fail closed. These variables are trusted launch configuration on the same footing as the flags. Whoever can write them for the daemon's environment sets its mode, so treat them like the plist, unit file, or Scheduled Task that carries them. An agent tool call can never set them. A running daemon's mode cannot be changed. Stop it and start it again with the configuration you want. ## What standard allows Standard mode is designed to preserve practical autonomous workflows. It does not open a Cua confirmation card for routine actions. | Operation | Standard behavior | | --- | --- | | Observe windows, applications, and the desktop | Allow | | Click, type, scroll, drag, and focus | Allow | | Create and use a driver-owned isolated browser | Allow | | Read visible page content through typed browser tools | Allow | | Upload, download, screenshot, record, and replay validated paths | Allow | | Change agent-adjustable cursor and image settings | Allow and audit | | Terminate a process proven to have been launched by this runtime | Allow after process fingerprint revalidation | | Terminate a foreign process | Deny | | Run unbounded legacy page mutation scripts | Deny | | Raise an operating-system permission prompt from an agent tool call | Deny | | Attach to an existing logged-in Chromium profile | Require an explicit launch grant or trusted host authorization | | Invoke an unknown risk-bearing operation | Deny | Path canonicalization, browser-origin validation, process identity checks, and managed or user policy still apply when standard permits an operation. Driver-owned isolated browser preparation follows the selected mode and any optional capability manifest. Standard allows it as a routine operation, bounded requires it to match the manifest, and unrestricted relies on the launcher's dangerous acknowledgement. ## Existing logged-in Chromium profiles An existing profile can contain live cookies and authenticated sites. Standard mode therefore keeps attachment as an explicit boundary. Authorize it for a newly launched MCP runtime: ```bash cua-driver mcp --grant existing-profile ``` Authorize it for a daemon: ```bash cua-driver serve --grant existing-profile ``` `--grant` is repeatable and is trusted launch configuration. It cannot modify an already-running daemon. Restart that daemon with the same grant when needed. Other valid authorization paths are: - A capability manifest containing the exact profile scope, combined with the profile's normal approval behavior. In `bounded` it is unattended; in `standard` a launch grant or trusted host decision is still required. - A `DriverAuthorizationHost` callback installed by an embedding application. - `unrestricted` mode with its dangerous acknowledgement. An MCP transport marker, model-supplied Boolean, environment variable, or ordinary tool argument is never an attachment grant. One approved attachment covers the resulting browser binding. Tab changes and reconnection do not create repeated Cua authorization requests. Any configured origin scope still applies to live navigation and input. See [Browser Profile Attachment]() for the platform and CDP boundary. ## Capability manifests A configured capability manifest is deny by default in every profile. A tool must appear in `allow.tools`, and every resource crossed by that call must match the manifest. Approval is considered only after those checks pass. Manifest version 3 removes profile behavior from the file. It supports the same application identities, browser profile kinds, and directory roots as version 2: ```yaml version: 3 expires_after: 8h idle_timeout: 30m allow: tools: - start_session - end_session - launch_app - list_windows - browser_prepare - get_browser_state - browser_navigate - browser_click - browser_type - browser_download - kill_app resources: apps: - executable: /usr/bin/example-editor launch: true windows: all terminate: driver_launched browser: profiles: - kind: isolated - kind: existing_profile origins: - https://app.example.com files: read: - dir: /data/input recursive: true write: - dir: /data/output recursive: true desktop: display: false ``` On macOS, use `bundle_id` for an application identity. On Windows and Linux, use a canonical absolute `executable` path. `windows: all` allows windows owned by the matching application. `terminate: driver_launched` permits termination only when Cua proved that the current runtime launched that exact process instance. Browser origins match exact scheme, host, and port. File roots are canonicalized and compared by path component, so a shared string prefix or a symlink escape does not grant access. ### Origin scope excludes generic input `resources.browser.origins` binds the typed browser adapter only. A manifest that declares origins therefore cannot also allow a tool that reaches a page around that adapter, and the runtime refuses to start if it does: ```text authorization startup error: origin-scoped capability manifests cannot allow 'click' because it bypasses the typed browser origin adapter ``` The excluded tools are `click`, `double_click`, `right_click`, `drag`, `scroll`, `type_text`, `press_key`, `hotkey`, `set_value`, `mouse_button_down`, `mouse_button_up`, `mouse_drag`, `parallel_mouse_drag`, `get_accessibility_tree`, `get_window_state`, `verify_state`, `get_desktop_state`, and `page`. The exclusion covers observation as well as input. A window screenshot or accessibility tree of a browser window exposes whichever tab is open, regardless of the origin allow-list, so where a browser window is in reach these tools bound neither what an agent may change nor what it may read. The check is evaluated at load time against `allow.tools` alone. It does not analyze whether the manifest's application scope actually puts a browser window in reach, so a manifest that names only a non-browser application is refused on the same terms. This is deliberate: deciding reachability would require the runtime to keep a list of browser identities, and a stale list would void the origin boundary silently instead of refusing loudly. Every navigation is checked against the origin set, so a manifest that allows `browser_navigate` with no origins refuses every navigation. Browser access and generic desktop input consequently belong to separate manifests and separate runtimes. Use `list_windows` for the `window_id` a browser call needs. An origin scope binds one runtime's tool surface. It is not a property of the browser, the profile, or the machine. A second runtime that can deliver generic input to a browser window — including a standard-mode runtime without a capability manifest, which allows input against every application — voids the first runtime's origin scope without violating it. Scope a companion runtime to non-browser applications with `desktop.display: false`, and treat same-user processes outside Cua Driver as outside the boundary entirely. Version 1 and 2 manifests remain loadable. Their `mode` field is retained for compatibility, and legacy `ask.tools` entries are treated as deny. Version 3 removes `mode` and `ask.tools`. `expires_after` and `idle_timeout` are optional in version 3, but any declared lifetime is enforced in every profile. Bounded mode requires both fields. For a complete tested example, see [Write a capability manifest](). ## Authorization stack Every call must pass all applicable layers: 1. Hard invariants, including self-targeting and protected-host checks. 2. The reviewed built-in tool and risk map. 3. Administrator policy from `CUA_DRIVER_MANAGED_POLICY_FILE`, when set. 4. User policy from `CUA_DRIVER_POLICY_FILE`, when set. 5. The selected profile's capability and approval behavior. 6. The capability manifest, when configured. 7. A launch grant or trusted host authorization for a residual standard-mode boundary. Each layer can narrow access. No manifest, grant, approval bypass, or permission profile can widen a hard invariant or managed/user policy. The `tools/list`, `manifest`, and status surfaces expose content-free enforcement descriptors. Each descriptor states its adapter ID, risk class, resource scope, mode behavior, authorization source, revocation triggers, and stable refusal code. ## Host authorization and activity An embedding application may install `DriverAuthorizationHost` when it wants to decide a residual standard-mode request itself. Cua supplies an attested, request-bound resource and digest. The host returns allow, deny, or cancel with that exact digest. Cua does not render or prescribe the host's consent experience. The host may use product UI, administrator policy, a second device, or another trusted decision source. An optional `DriverActivityObserver` receives content-free action, authorization, grant, and session events. Events never contain page text, typed input, file contents, screenshots, or raw resource identities. When no launch grant or host callback exists, a residual boundary returns a structured `authorization_required` refusal. It does not open a Cua-owned modal. ## Revocation End one public session: ```bash cua-driver revoke --session research-1 ``` Suspend the complete runtime generation: ```bash cua-driver revoke --all ``` Session end removes its grants, browser bindings, launch provenance, recording state, and cursor. The ended session label remains tombstoned until an explicit `start_session` re-declares it. `revoke --all` is terminal for that runtime generation. Later calls, including anonymous calls, return `authorization_suspended`. Restart the runtime to create a fresh authorization generation. ## Cursor and session badge The semantic cursor is optional activity feedback. It initializes on the first cursor-bearing action even when the transport uses an implicit session. A publicly named session displays its sanitized label in a badge below the cursor. The badge uses the session color, strips control characters, collapses whitespace, and truncates long labels. The cursor and badge are not authorization signals. Hiding the cursor also hides the badge. Headless sessions do not require either. ## Security boundary The host owns the permission profile, capability manifest, launch grants, and any human consent UX. Agent tools cannot change them. Same-user native code, malware, and automation with control of the same desktop are outside Cua Driver's security boundary. Cua can prevent an agent from forging its own tool arguments, but it cannot turn an ordinary desktop window into a secure desktop. ## Related - [Permission policies]() - [Browser Profile Attachment]() - [Write a capability manifest]() - [Restrict tool access]() --- # Permission policies YAML and Rego permission policy schema, environment variable, evaluation rules, and Rego input interface for Cua Driver. Cua Driver evaluates every same-process SDK and daemon tool call against the configured policy stack at the native registry boundary. The runtime loads policies once at process startup; transport adapters may repeat an early policy check as defense in depth. SDK, CLI, MCP, and raw-socket calls cannot bypass the native enforcement point. This page describes the user and managed policy formats. [Permission modes]() select the default autonomy model, while policies can only narrow its capability ceiling. --- ## Environment variable | Variable | Type | Default | |---|---|---| | `CUA_DRIVER_POLICY_FILE` | path to a `.yaml`, `.yml`, `.rego` file, or a directory | not set (policy disabled) | | `CUA_DRIVER_MANAGED_POLICY_FILE` | path to an administrator-owned policy with the same formats | not set | When the variable is absent the policy engine is disabled and every call is allowed. When either variable is explicitly set, a missing, unreadable, empty, or invalid path prevents the daemon from binding its action endpoint. An unset variable still means that layer is absent. When both layers are present, a call must pass both. The user policy can narrow the managed policy but cannot widen it. `cua-driver status` reports a SHA-256 content hash for each loaded layer without exposing policy contents. --- ## YAML policy format ### Top-level structure ```yaml allow: # what to permit (default: nothing allowed) tools: [] # tool names allowed unconditionally rules: [] # rules with argument constraints deny: # hard overrides (checked first) tools: [] # tool names always blocked ``` Both `allow` and `deny` are optional and default to empty lists. A policy with neither section blocks every call. `deny.tools` is checked before `allow.tools` and `allow.rules`. A tool in `deny.tools` is rejected even if it also appears in an allow section. ### Shorthand forms The `allow` and `deny` fields also accept a flat list of strings as a shorthand for the `tools` sub-key: ```yaml allow: - screenshot - wait deny: - shell_execute ``` ### `allow.tools` A list of tool names that are permitted for any argument values. Order does not matter. ```yaml allow: tools: - screenshot - get_window_state - list_apps ``` ### `allow.rules` A list of tool rules. Each rule names a tool and an optional set of argument constraints. A call matches a rule when all constraints for that rule pass. If the same tool appears in multiple rules, the call is allowed when **any** rule matches. ```yaml allow: rules: - tool: type_text constraints: text: max_length: 200 - tool: scroll constraints: amount: min: -10 max: 10 ``` The `constraints` map keys are JSON argument names (snake_case, matching the names in [MCP tools]()). ### Constraint fields Each constraint object may include one or more of the following checks. At least one check is required. | Field | Type | Applies to | Meaning | |---|---|---|---| | `min` | number | numeric argument | argument value must be ≥ this | | `max` | number | numeric argument | argument value must be ≤ this | | `max_length` | integer | string argument | Unicode codepoint count must not exceed this | | `pattern` | string (regex) | string argument | argument must match this regular expression | | `allowed` | list of JSON values | any | argument must equal one of the listed values | When both `min` and `max` are set, `min` must be ≤ `max` or the policy fails to load. The `pattern` field uses the [Rust regex crate](https://docs.rs/regex) syntax (RE2-compatible; no backtracking, no lookahead). ### `deny.tools` A list of tool names that are unconditionally blocked, regardless of allow rules. ```yaml deny: tools: - shell_execute - run_javascript ``` ### Full YAML example ```yaml allow: tools: - screenshot - get_window_state - list_apps - list_windows - click - press_key - scroll - hotkey - wait rules: - tool: type_text constraints: text: max_length: 500 pattern: "^[\\x20-\\x7E\\n\\t]*$" - tool: launch_app constraints: bundle_id: allowed: - "com.apple.Safari" - "com.google.Chrome" - "com.microsoft.VSCode" - tool: scroll constraints: amount: min: -20 max: 20 deny: tools: - shell_execute - run_javascript ``` --- ## Rego policy format Rego policies must define a rule named `data.cua.policy.allow`. The engine uses [Regorus](https://github.com/microsoft/regorus) 0.10.x, which implements the OPA policy language. ### Package declaration ```rego package cua.policy ``` ### `allow` rule The rule must evaluate to a `boolean`. `undefined` is treated as `false` (deny). ```rego package cua.policy import rego.v1 allow if { # rule body } ``` ### Input object The engine sets `input` to the following object for each call: | Field | Type | Description | |---|---|---| | `input.server` | string | Always `"cua-driver"` | | `input.tool` | string | Canonical tool name (e.g. `"type_text"`) | | `input.arguments` | object | Tool arguments with `_session_id` removed | `input.tool` is normalized before evaluation: `type_text_chars` is mapped to `type_text`. ### Minimal Rego example ```rego package cua.policy import rego.v1 safe_tools := { "screenshot", "get_window_state", "list_apps", "list_windows", "click", "press_key", "scroll", "wait", } allow if { input.tool in safe_tools } allow if { input.tool == "type_text" count(input.arguments.text) <= 500 } ``` ### Loading multiple Rego files Point `CUA_DRIVER_POLICY_FILE` at a directory to load all `.rego` files in that directory. Files are loaded in lexicographic order (sort by filename), which controls rule merge order across files. ``` ~/.cua-driver/policies/ 00_base.rego 10_type_text.rego 20_launch_app.rego ``` ```bash export CUA_DRIVER_POLICY_FILE=~/.cua-driver/policies ``` --- ## Evaluation order The daemon first checks the managed policy and then the user policy. A denial or evaluation error in either layer stops the call. Within each YAML layer, evaluation follows this order: For YAML policies: 1. Check `deny.tools` — if the tool is listed, **deny** immediately. 2. Check `allow.tools` — if the tool is listed, **allow** immediately. 3. Evaluate `allow.rules` for matching rules — if any rule's constraints pass, **allow**. 4. No match — **deny** with `"tool 'X' is not allowed by the YAML policy"`. For Rego policies: 1. Evaluate `data.cua.policy.allow` with the call's input. 2. `true` → **allow**; `false` or `undefined` → **deny**; evaluation error → **error** (the call is blocked with the engine error message). --- ## Tool name normalization Before any policy evaluation the driver canonicalizes deprecated aliases: | Alias | Canonical name | |---|---| | `type_text_chars` | `type_text` | Write policies against the canonical name; the alias is accepted automatically. --- ## Denial response When a call is blocked, the MCP response is a tool error with a human-readable message: ``` Permission denied: tool 'shell_execute' is explicitly denied Permission denied: tool 'launch_app' is not allowed by the YAML policy Permission denied: tool 'type_text' argument constraints were not satisfied: argument 'text' must be at most 500 characters Permission denied: tool 'launch_app' is not allowed by the Rego policy ``` --- ## Related - [Restrict tool access with permission policies](): step-by-step setup guide - [How permission policies work](): evaluation engine internals and trust model - [Permission profiles and capability manifests](): startup profiles, narrow-only manifests, launch grants, host callbacks, and revocation - [MCP tools](): tool names and argument schemas --- # Interface Contracts The contracts behind the CLI and MCP surfaces: lifecycle sessions, per-call targets, config persistence, and action routing. The CLI (`cua-driver call …`) and MCP (`cua-driver mcp`) run the same runtime contract, but they differ in **what state survives between calls**, **where configuration lands**, and **which parameters a call must carry**. MCP may own that runtime directly or select `cua-driver serve` explicitly. --- ## CLI versus MCP at a glance | Dimension | CLI (`call`) | MCP (`mcp`) | |---|---|---| | Lifetime | each one-shot call gets a disposable implicit session; the selected service may own other durable state | one implicit session per authenticated transport connection | | `element_index` cache | owned by the selected service; survives until invalidation or service restart | owned by the MCP runtime; lives across calls | | Public label | optional `session`; it does not carry authority | optional `session`; it does not carry authority | | Where `set_config` lands | the persisted global default when no public session is supplied | an in-memory override when a public session is supplied | | Agent cursor | initialized for the call and cleaned up with its disposable session | initialized on the first cursor-bearing action and reused on later unnamed calls | **Warning** `cua-driver call` requires a service at the resolved default endpoint or at the explicit `--socket` endpoint. A freshly built binary's behavior appears through the CLI only after that service restarts, because the action runs in the service process. Integration tests avoid this ambiguity by starting an isolated runtime. Bare MCP owns its runtime on Windows/Linux and uses the signed app service on macOS. `mcp --socket ` explicitly selects a service on every platform; `mcp --direct` explicitly selects process ownership and is mutually exclusive with `--socket`. ## Lifecycle sessions `start_session` is optional. The first admitted stateful call creates one implicit session for the authenticated transport lease. Later unnamed calls on that transport reuse the same identity, so several clicks do not create several sessions. The runtime does not expose its private implicit ID. The default session idle TTL is five minutes. This timer controls lifecycle and resource cleanup. It is separate from the cursor idle-hide timer, which controls visibility only. A call in flight cannot expire, and a completed call refreshes the lifecycle timer once. Transport close, explicit end, revocation, and idle expiry use the same cleanup hooks. For multi-call work, prefer a short public `session` label and pass the same value on every call that accepts it. Passing it once is not sticky: a later call that omits the field uses the transport's implicit session instead. Use `start_session` to name or configure a run before acting, or to revive a public name after it has ended. For one-off or deliberately unlabeled work, omitting `session` is valid. `get_session` reads one visible session, and `list_sessions` returns content-free summaries scoped to the caller's transport. Trusted host code has a separate operator listing. `get_session_state` and `escalate_session` remain deprecated for legacy capture-scope sessions. There is no `deescalate_session`. --- ## Where settings live `set_config` resolves *where* a setting is written from whether a public `session` is declared. Runtime and transport adapters derive their reserved internal session fields; caller-supplied reserved values do not grant authority. | Caller | Public `session` | Effect | |---|---|---| | `cua-driver config set …`, anonymous one-shot `cua-driver call` | absent | writes the global `DriverConfig` and persists to `~/.cua-driver/config.json` | | MCP or CLI call with a `session` | present | in-memory override for that session only; no disk write, no clobber of the default | Every tool then reads the **effective** value with this precedence: ``` effective = call-argument > session override > global default (disk) ``` `max_image_dimension` follows this precedence. The retired `capture_scope` configuration key returns a migration error that points callers to the per-action `target`; only the deprecated `start_session.capture_scope` field remains for legacy session calls. `capture_mode` is deprecated and ignored. `get_window_state` always returns both the tree and a screenshot. See the [`set_config` reference]() for the per-session isolation details. --- ## The per-call target contract Capture modality belongs to each observation or action. A session records the modalities used, but it does not lock the caller into a window or desktop mode. The input tools `move_cursor`, `click`, `drag`, `scroll`, `type_text`, `press_key`, and `hotkey` accept one exact tagged target: ```jsonc { "target": { "kind": "window", "pid": 4711, "window_id": 22 } } { "target": { "kind": "desktop", "display_id": "primary" } } ``` `display_id:"primary"` is the portable desktop target in this release. An unsupported display ID returns `invalid_action_target`. The target cannot be combined with legacy `scope`, `pid`, or `window_id` fields, so the driver never has to guess which coordinate space the caller intended. | | Window target | Desktop target | |---|---|---| | Coordinate space | window-local coordinates from `get_window_state` | screen coordinates from `get_desktop_state` | | Identity | exact `pid` and `window_id` | exact `display_id`, currently `primary` | | Delivery | background by default, or foreground per call | foreground | | Action rung | accessibility element or window-local pixel | screen pixel | Legacy callers may keep using flat `pid`, `window_id`, and `scope` fields during the compatibility window. An explicit legacy `capture_scope` still invokes the old session gate. New callers should use `target` and should not call `escalate_session`. --- ## How an action is routed Every input tool chooses its route from the target and action arguments: | Arguments | Path | Behavior | |---|---|---| | window target plus `element_index` | accessibility action | UIA Invoke, `AXPerformAction`, or AT-SPI. Background, no cursor move, no focus steal. | | window target plus `x`, `y` | window-local pixel | coordinates are relative to that window's screenshot. For keyboard tools, this form can click the coordinate to establish renderer focus before sending keys. | | desktop target plus `x`, `y` | screen-absolute | true screen pixels on the primary display, with foreground delivery | The keyboard family's coordinate form is mutually exclusive with `element_index`. A malformed, mixed, or ambiguous target fails before authorization and platform dispatch. --- ## Valid combinations `get_window_state` returns both the accessibility tree and a screenshot by default. The enforced combinations are target modality, action rung, and delivery mode: | Target | Action rung | `delivery_mode` | Valid? | Why | |---|---|---|---|---| | window | `ax` (`element_index`) | `background` | ✅ | Semantic action on a background-capable window. | | window | `ax` (`element_index`) | `foreground` | ✅ | Activate, then act by element. | | window | `px` (`x`, `y`) | `background` | ✅ | Act at a coordinate from the window screenshot without raising the target when supported. | | window | `px` (`x`, `y`) | `foreground` | ✅ | Activate, then act by coordinate. | | desktop | `px` (`x`, `y`) | `foreground` | ✅ | Act at screen coordinates on the primary display. | | desktop | `ax` (`element_index`) | any | ❌ | A desktop target has no window element tree. | | desktop | any | `background` | ❌ | Screen input lands on the active desktop and has no per-process background route. | The driver enforces these rejections before it sends input. --- ## What a tool returns An action tool answers with one of two payloads, and MCP holds **both** to the tool's advertised `outputSchema` — the schema covers every `structuredContent` a tool emits, refusals included. So `outputSchema` is an `anyOf` of two variants. **Success variant.** Closed: exactly these keys, nothing else. | Field | Required | Values | |---|---|---| | `effect` | ✅ | `confirmed`, `partial`, `unverifiable`, `suspected_noop`, `refused` | | `route` | ✅ | `accessibility`, `synthetic_events`, `global_input`, `system_api`, `dom`, `trusted_input` | | `delivery` | — | `{mode: background \| foreground \| not_applicable \| unknown}` | | `evidence` | — | what was read back to justify `effect` | | `escalation` | — | which rung to try next, and why this one fell short | **Refusal variant.** Accompanies `isError: true` and carries diagnostics instead of an effect. Two shapes are in service: ```json {"status": "refused", "refusal": {"code": "stale_element_token", "message": "element_token is stale; call get_window_state again to refresh"}} ``` ```json {"code": "window_target_not_found", "effect": "refused", "candidates": [], "pid": 4711} ``` This variant is deliberately open — refusals carry tool-specific diagnostic keys (`candidates`, `detail`, `pid`) that help a caller recover. The marker keys `refusal`, `status`, and `code` are what identify a payload as a refusal rather than a malformed success. **Warning** Branch on the refusal `code`, and surface the `content` text — it states the recovery step in plain language. `stale_element_token` means re-run `get_window_state` and retry with fresh indices; it does **not** mean the accessibility route is unavailable. Agents that treat a refusal as a dead route tend to abandon the element path and fall back to blind pixel clicking, which is both slower and unverifiable. Success payloads stay strictly validated: an unknown key on the success shape is still a contract violation and will not be quietly accepted through the refusal variant. --- ## Platform support | Capability | Windows | macOS | Linux | |---|---|---|---| | `get_window_state` returns both tree + screenshot (element ax / px actions) | ✅ | ✅ | ✅ | | `delivery_mode: "background"` (best-effort background) | ✅ | ✅ | ✅ for semantic AT-SPI actions on X11 and Wayland; native-Wayland raw keyboard input remains limited | | `delivery_mode: "foreground"` (action-scoped activation and restore) | ✅ | ✅ | ✅ X11; Wayland activation is compositor-constrained | | `bring_to_front` (persistent activation for focus-proxy surfaces) | ✅ | ✅, chiefly for RDP | ✅ X11 EWMH activation (`_NET_ACTIVE_WINDOW` + input focus); Wayland raise is compositor-constrained | | `get_desktop_state` (desktop capture) | ✅ | ✅ | ✅ | | Per-call window and primary-display targets | ✅ | ✅ | ✅ | --- ## Mental model ```text window target = exact pid + window_id; background by default desktop target = exact display_id; screen coordinates and foreground delivery session = lifecycle, cursor, recording, cleanup, and telemetry ownership target = the window or display selected for this call authority = the independent permission and policy stack CLI call = one disposable implicit session MCP = one reusable implicit session per authenticated transport ``` --- # Platform Support Current operating-system and window-system support, proven surfaces, and known capability boundaries. cua-driver supports Windows, macOS, and Linux. Support is defined by observed behavior in a real application, not by whether a tool call returned success. The exact delivery route depends on the operating system, window system, application toolkit, action, and whether the target may be brought forward. For definitions of AX, PX, foreground, background, window scope, and desktop scope, see [Capture and delivery modalities](). ## Support levels | Level | Meaning | | --------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------- | | Supported | A canonical Rust harness test proves the result against application-owned or desktop-owned state. | | Supported with limits | Common paths are proven, but the platform or window system cannot safely provide every delivery shape. Unsupported paths return a structured refusal. | | Experimental | The backend exists, but representative coverage is incomplete. Do not assume unlisted actions work. | ## Platform overview | Platform | Window system and automation APIs | Current state | | ------------- | -------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Windows | Win32, UI Automation (UIA), native input, and targeted window messages | Supported. Canonical coverage includes Electron, Tauri, WPF, WinUI 3, and WebView2. Some background Chromium gestures and elevated-integrity boundaries remain unavailable or unproven. | | macOS | AppKit, Accessibility (AX), Quartz/HID, and ScreenCaptureKit | Supported. Canonical coverage includes Electron, Tauri, AppKit, SwiftUI, and WKWebView. Accessibility and screen-recording permissions are required. Some background scroll and drag shapes return structured refusals. | | Linux X11 | X11/EWMH, XTest, AT-SPI, and toolkit accessibility bridges | Supported with toolkit-specific limits. Foreground input and semantic background actions are broadly covered. Toolkits that reject synthetic background events receive an explicit refusal instead of a silent success. | | Linux Wayland | AT-SPI plus compositor-specific discovery, capture, activation, and portal input | Supported with compositor-specific limits. Semantic background actions work where the application exposes them. Raw input cannot generally be sent to an arbitrary occluded surface. | Wayland portal grants belong to the compositor/runtime scope that issued them. A direct runtime or private worker reports its resolved display and portal scope, and a replacement runtime may prompt again. Do not treat a successful portal grant as a durable credential that migrates to a later process generation. ## Browser-tool routes Browser mutation always starts from an exact native `(pid, window_id)` binding. The table records the strongest browser route currently proven by the Rust harnesses; an unlisted trust class or host shape is not implied. | Surface | Proven page routes | Explicit limit | | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | | Standalone Chrome and Edge on Windows | Snapshot, navigation, ref-bound typing, trusted background click, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Elevated-integrity and unsupported native-host relationships still refuse | | Standalone Chrome on macOS | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, JavaScript dialogs, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input returns `browser_input_trust_unavailable` before dispatch | | Standalone Chrome and Edge on Linux X11 | Snapshot, navigation, ref-bound typing, explicit DOM click and pointer actions, foreground JavaScript-dialog resolution, file assignment, approved downloads, frames, multi-tab, and exact ambiguity refusal | Trusted CDP pointer input and background JavaScript-dialog resolution return `browser_input_trust_unavailable` before dispatch | | Electron on Windows, macOS, X11, and validated Sway | Typed mutation while one proven native window maps to one CDP page | A second page or native window invalidates the bounded exact route | | Tauri, WKWebView, WebKitGTK, and common split-process WebView2 | Browser identity plus a side-effect-free structured refusal | No exact native-host-to-engine relationship is currently proven | | Safari and Firefox | Native discovery and native AX/PX fallbacks | No typed browser mutation engine is currently advertised | `browser_prepare` may launch a separate driver-owned Chromium profile after an approved destructive call. `isolated_new` processes and their complete child trees are reaped and the marked profile is removed when the owning session ends. Existing-profile attachment is proven for Chrome and Edge on macOS and Windows, for Chrome on Linux X11, and for Chrome in the native Sway lane. The same exact descriptor route covers Chromium and Edge on Linux, subject to a complete AT-SPI renderer tree and product-specific acceptance evidence. The harness proves exact interactive approval, setup-control identity, browser-owned consent, PID-owned loopback endpoint discovery, temporary-tab cleanup, focus restoration, and an exact native/CDP rebind without restarting, copying, or editing the profile. Unsupported products and unproven Wayland compositors return a structured refusal. ## Linux window systems Linux support is recorded per window system and Wayland compositor. "Wayland" is not one uniform automation API: compositors expose different discovery, capture, activation, and input protocols. For the relationship between distributions, desktops, and compositors, read [Linux desktops and computer use](). | Environment | State | What is proven | Main limits | | ------------------------------- | --------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | X11/Xorg | Supported | Window discovery and capture, AT-SPI trees and actions, foreground pointer and keyboard input, semantic background delivery, desktop scope, and structured refusals | Raw background delivery depends on the target toolkit. X11 accepting an event does not prove the application handled it. | | Sway (wlroots reference lane) | Supported with limits | The complete typed Electron, Tauri, GTK, capture, and desktop-scope catalog, including foreground/background and AX/PX outcomes | Focus-bound and raw background input shapes without a target-addressed protocol return structured refusals. Other wlroots compositors are expected to share protocol support but are not yet proven. | | Hyprland / Omarchy | Experimental | Separate [driver-branch and plugin-foundation evidence](#hyprland-and-omarchy); neither establishes a released general Hyprland support claim | Do not inherit Sway coverage. The optional plugin is discovery-only and does not deliver input. Driver capture, activation, semantic actions, and raw background input require separate evidence. | | GNOME/Mutter | Supported with limits | AT-SPI actions, GTK controls, compositor-backed window geometry and capture, verified foreground activation, and portal/libei foreground input | The bundled WinRects Shell helper and one Shell-session restart are prerequisites for authoritative geometry and activation. Portal video recording remains incomplete. | | KDE/KWin | Experimental | Plasma 6 session startup, GTK AT-SPI discovery, generic discovery where exposed, and portal interface availability | The optional KWin helper exposes read-only window identity and state, not activation or input. Raw target-addressed input refuses until a target-bound KWin input path exists; no complete behavioral matrix is accepted. | | `cua-compositor` nested session | Experimental | Native GTK behavior, capture and scope, private route metadata, independent observation, and per-cell video | The complete shared renderer matrix is not accepted. Unicode text and a canonical parallel-drag row remain unproven; this route does not establish a stock-Wayland capability. | | XWayland | Supported with limits | X11 routes are used when the application exposes a real X11 window; native Wayland and AT-SPI fallbacks cover mixed sessions | Capabilities depend on whether the application is actually using X11 or native Wayland. | ### Hyprland and Omarchy Omarchy uses Hyprland; it is not a separate display protocol. Modern Hyprland is not based on wlroots, so its support must be recorded separately from Sway. The following is a source-work status snapshot as of September 5, 2026, not an installer or release-support matrix: | Work | Evidence and boundary | | ------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Driver capture and observation | [Draft PR #3557](https://github.com/trycua/cua/pull/3557) reports scale-1 GTK3 capture, semantic actions, and independent focus/cursor checks at source `b50a4f78f277d05700200e84c5d0f0ea72f10c47`. Its canonical native suite stopped at the foreground-key preflight; no matrix cells ran. This is focused branch evidence, not desktop certification or released support. | | Optional Hyprland plugin | [PR #3547](https://github.com/trycua/cua/pull/3547) adds discovery and liveness only. Its [validation report](https://github.com/trycua/cua/blob/feat/hyprland-plugin-foundation/libs/cua-driver/hyprland-plugin/tests/validation.md) records native build and live lifecycle evidence on Hyprland `0.56.2`. All six input message types return `background_unavailable`; the foundation creates no synthetic seat and changes no Driver delivery route. | | Isolated raw background input | [Draft experiment #3572](https://github.com/trycua/cua/pull/3572) contains a separately opted-in synthetic-seat integration. Its [desktop-transition report](https://github.com/trycua/cua/blob/e8542dece6c8195a8db50ba79aba9615a78340b1/libs/cua-driver/hyprland-plugin/tests/desktop-state-validation.md) records focused Fleet evidence and tested revisions. The experiment is not part of the discovery foundation, a released capability, or an accepted production input/authorization contract. | The plugin is disabled by default and outside the portable driver installation path. Switching driver channels alone does not install or enable it. Its Hyprland ABI and compiler toolchain must match the compositor that loads it. The recorded plugin Fleet environment used Omarchy `4.0.1-1` and Cua Driver `0.22.2`. It does not certify the separate physical-host acceptance target of Omarchy `4.0.2-1` with Cua Driver `0.23.2`, or application control through either driver version. See the [Hyprland roadmap]() for the remaining design and validation gates. ### Wayland background AX and PX **Background AX works when the target exposes a semantic AT-SPI action.** For example, the driver can invoke an accessible button without raising its window. The passing test must also prove that focus and z-order did not change and that input did not leak into the foreground application. **Background PX is proven only for the action/surface cells declared by the catalog.** A caller may address a target by pixel while the safe delivery route hit-tests that point and invokes a semantic AT-SPI action. Renderer and gesture cells without a safe target-addressed route return an exact refusal. Neither outcome establishes arbitrary raw background injection. **Arbitrary raw background PX is not available to an ordinary client on a standard Wayland compositor.** Reconstructing a window's coordinate system tells cua-driver where the target is, but portal/libei and virtual input still deliver through the compositor's active seat. An occluding surface therefore receives a raw event sent at that screen coordinate. cua-driver also contains an opt-in exception: the nested `cua-compositor` backend owns the compositor and can route `wl_pointer` and `wl_keyboard` events directly to a selected client surface through `CUA_INJECT_SOCKET`, without changing seat focus. This implements focus-free raw click, text, named-key, and multi-pointer drag paths. The backend is covered by the converged typed matrix, but remains experimental until the shared renderer matrix also passes. Unicode text, multi-pointer behavior, canvas/game input, and any row without external fixture evidence remain unproven. **Warning** A structured `background_unavailable` or `background_occluded` result is part of the contract. It means cua-driver refused an unsafe or unsupported route before it could disturb the user's active desktop. It is not a silent success. ## Related documentation - [How Cua Driver is validated]() explains why unit tests and application-owned E2E evidence have different roles. - [Platform roadmap]() records the remaining engineering work, evidence gaps, and platform boundaries. - [Known limits]() lists target-specific constraints and available alternatives. - [Development]() provides contributor entry points and canonical validation commands. --- # Platform Roadmap Remaining Cua Driver platform work, evidence gaps, and operating-system boundaries without implied delivery dates. This roadmap records capability and validation work for Cua Driver. It does not promise dates. A capability moves to [Platform Support]() only after a canonical Rust harness observes the required application and desktop state. ## Status vocabulary | Status | Meaning | | ----------------- | -------------------------------------------------------------------------------------------------- | | Proven | Canonical harness evidence supports the current public claim. | | Evidence gap | Code or platform APIs may support the behavior, but representative harness evidence is missing. | | Engineering work | A concrete driver, fixture, observer, or runner change is required. | | Experimental | An opt-in backend or environment has evidence but is not part of the general support contract. | | Platform boundary | The ordinary OS security or windowing model does not expose a safe general route for the behavior. | An evidence gap does not mean a behavior is impossible. A platform boundary does not prevent narrower semantic routes, foreground delivery, or operation inside an environment that owns the compositor. ## Cross-platform priorities | Area | Current state | Next acceptance condition | | --------------------------- | ---------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------- | | Shared application catalog | Electron and Tauri use one typed Rust behavior catalog across operating systems. | Keep the same action, AX/PX, foreground/background, scope, and oracle dimensions on every representative host. | | Native application catalogs | Windows, macOS, and Linux have toolkit-specific source-built harnesses. | Expand native rows toward the shared action cross-product where the toolkit exposes an equivalent behavior. | | Background safety | Background cells attach focus, z-order, cursor, and leaked-input oracles. | Add the same side-effect owners to every new background delivery or refusal row. | | Release evidence | Hosted GUI lanes retain typed results, trajectories, screenshots, logs, and per-cell video. | Require accepted exact-source-SHA runs for affected platforms before changing release support claims. | | Representative environments | Hosted lanes cover Windows, Linux X11, Sway, and the nested compositor; macOS uses a logged-in host. | Close named GNOME, KDE, real-Xorg, and renderer evidence gaps without weakening fixtures or oracles. | ## Windows | Work item | Status | Acceptance condition | | ------------------------------------------ | ---------------- | ----------------------------------------------------------------------------------------------------------------- | | Broader WPF PX gestures and keyboard paths | Evidence gap | Typed foreground/background rows observe native control state and all required desktop side effects. | | WinUI 3 pointer and background coverage | Evidence gap | Right click, double click, drag, scroll, and keyboard contracts are declared and empirically observed or refused. | | WebView2 native-input coverage | Evidence gap | Native pointer and keyboard cells complement the existing page/CDP and background-left-click evidence. | | Elevated-integrity boundary | Engineering work | A controlled fixture proves the `background_uipi_blocked` contract across process integrity levels. | Windows integrity isolation is a platform boundary: a lower-integrity process cannot generally inject input into a higher-integrity target. The roadmap item is to detect and prove that refusal, not bypass the operating-system boundary. ## macOS | Work item | Status | Acceptance condition | | ---------------------------------- | ---------------- | ------------------------------------------------------------------------------------------------------------------------------- | | Native AppKit action cross-product | Evidence gap | Press key, hotkey, AX-addressed pointer gestures, and additional controls have application-owned evidence. | | SwiftUI transient-window discovery | Engineering work | The opened popover or panel is independently visible through targeted window and AX enumeration. | | Installed-app confidence checks | Proven | Keep the typed Calculator and TextEdit rows as supporting evidence; repo-local fixtures remain the primary behavioral contract. | Accessibility and Screen Recording consent remain platform prerequisites. Off-Space SwiftUI tree stripping, minimized keyboard commits, and applications that accept only HID-tap input are platform or target boundaries; their safe alternatives remain documented in [Known Limits](). ## Linux X11 | Work item | Status | Acceptance condition | | --------------------------------- | ------------ | ----------------------------------------------------------------------------------------------------- | | Hosted Openbox/Xvfb catalog | Proven | Preserve the complete shared and GTK catalog with exact delivery or refusal outcomes. | | Real-Xorg MPX and uinput behavior | Evidence gap | A maintainer lane proves routes that Xvfb cannot represent, including multi-pointer desktop behavior. | | Additional toolkit confidence | Evidence gap | Add a surface only when it represents a materially different input or accessibility contract. | X11 permits more synthetic-input routes than Wayland, but an X server accepting an event does not prove the target toolkit handled it. Application-owned state remains required. ## Linux Wayland | Environment | Work item | Status | Acceptance condition | | ----------------------- | ----------------------------------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Sway/wlroots | Preserve the accepted shared and native catalog | Proven | Keep every declared Electron, Tauri, and GTK cell delivering or returning its exact refusal through unchanged oracles. | | Hyprland / Omarchy | Driver integration and isolated-input plugin | Experimental | Accept integrated driver evidence separately from plugin transport tests; complete the design and behavior gates below before claiming isolated input. | | GNOME/Mutter | Shared renderer catalog and portal video | Evidence gap | Electron and representative WebKitGTK rows pass with reporter-owned per-cell recordings. | | KDE/KWin | Target-bound input and full matrix | Engineering work | Bind the input operation itself to the selected target and verify application state. The existing read-only helper and a pre-dispatch focus check cannot make later focus-bound portal input target-safe. | | Nested `cua-compositor` | Complete shared catalog and protocol coverage | Experimental | One accepted full catalog proves renderer behavior, Unicode text, and canonical parallel drag. | | Other wlroots | Representative compatibility | Evidence gap | Add a lane only after a user report demonstrates behavior that the Sway lane does not represent. | ### Hyprland and Omarchy The [support reference]() records the dated driver-branch and plugin evidence. The discovery-only plugin is not an input implementation, and its lifecycle tests do not replace the driver's representative desktop catalog. Promotion requires distinct design, implementation, and validation work: - Accept an RFC for the input and permission contract, including operator authorization, revocation, target identity, and compositor restart behavior. Same-UID socket access does not identify or authorize a particular agent. - Review the nonshipping [input experiment #3572](https://github.com/trycua/cua/pull/3572) before adopting a production integration. Its independent synthetic seat and test-only authorization grants provide focused evidence, not a production approval mechanism or general client compatibility. A second cursor image or a successful transport request is not sufficient. - Prove packaged installation, upgrade refusal, restart, and ABI compatibility on the exact Hyprland/plugin/toolchain combination. Preserve discovery and lifecycle checks as a separate gate. - Run the representative application catalog at the integrated candidate SHA, with fixture-owned state plus focus, z-order, cursor, and leaked-input oracles. Include concurrent foreground grabs, keyboard state, native Wayland and XWayland clients, and explicit unsupported-action refusals. - Complete the plugin's documented physical Omarchy acceptance baseline after Fleet validation. A VM run supplements that gate; it does not certify a bare-metal host or a different package combination. These are acceptance conditions, not an approved protocol or a delivery date. The plugin is Hyprland-specific rather than Omarchy-specific, and does not add the same capability to Sway, GNOME, or KDE. ### Standard Wayland boundary An ordinary client on a standard Wayland compositor cannot generally send raw pointer or keyboard input to an arbitrary occluded, unfocused surface. Reconstructed coordinates solve target geometry, but they do not change which surface receives compositor-seat input. Semantic AT-SPI actions and PX-addressed hit-testing can still operate in the background when a coordinate resolves to an actionable accessible element. The optional nested `cua-compositor` has a different capability because it owns the compositor and can route input directly to a selected client surface. Work in that lane can improve controlled nested sessions, but cannot establish a general Sway, GNOME, KDE, or stock-Wayland claim. ## Promotion rules A roadmap item becomes supported only when all of the following are true: - the behavior is a typed Rust case rather than a second OS-specific matrix; - delivery changes fixture-owned state, or refusal returns the exact declared code; - every required focus, z-order, cursor, leaked-input, capture, and scope oracle passes; - the result identifies the operating system, window system, surface, delivery route, and source commit; - representative evidence is retained and the public support reference is updated. For the reasoning behind these requirements, see [How Cua Driver is validated](). --- # Process model How Cua Driver maps one typed surface onto same-process SDK and daemon-backed execution. Cua Driver has three local execution families around one typed driver surface: list windows, read accessibility, capture the screen, click, type, record, configure, and report state. - `CuaDriver.create()` runs the Rust driver in the importing Python, TypeScript, or Rust process. It requires no daemon, executable, or IPC. - `CuaDriver.create_private_worker()` / `CuaDriver.createPrivateWorker()` directly supervises one child runtime over inherited pipes. The child has no listener, discovery, or reconnect path. - `cua-driver serve` runs the same SDK contract in a long-lived daemon. MCP stdio, one-shot CLI calls, and `CuaDriver.connect()` send requests to it. The MCP server is a downstream consumer of the public typed SDK contract, not a parallel desktop implementation. See [SDK, MCP, and process hosting](). ## Daemon-backed process roles The MCP stdio process is *client-owned*. The parent starts `cua-driver mcp`, keeps stdin and stdout connected, and sends MCP tool calls over that transport. On Windows and Linux it owns the runtime directly unless `--socket` is specified. On macOS it proxies to the permission-owning app daemon unless the caller explicitly passes `--direct` and accepts the spawning host's TCC attribution. `--direct` and `--socket` are mutually exclusive. The daemon shape is *machine-owned*. A single `cua-driver serve` process listens on a local IPC endpoint, such as a mode-`0600` Unix socket or same-user-authenticated named pipe, and keeps driver state in memory while it lives. The one-shot CLI adapter is *call-owned*. `cua-driver call ` connects to the daemon, prints one result, and exits. If the daemon is unavailable, the command fails instead of executing the tool in the CLI process. Finite inspection commands—`list-tools`, `describe`, and `dump-docs`—read the canonical SDK tool inventory without creating an action-capable runtime. They therefore remain available in non-interactive environments such as Windows Session 0. Desktop-owning entry points (`serve`, direct MCP, and `CuaDriver.create()`) still refuse before accepting actions when no interactive desktop is attached. These are transport roles around the same typed runtime that applications can create in process or in a directly supervised private worker. ## Why a daemon proxy exists The **daemon-proxy pattern** separates the process that speaks to the caller from the process that has the right desktop authority. The proxy handles the client protocol. The daemon performs the GUI work. That matters when the caller can start a shell process but cannot correctly operate the user's desktop. On macOS, the root issue is **TCC**, Transparency Consent and Control. Accessibility and Screen Recording grants are attached to a specific app identity, represented by a bundle ID. Grants to `CuaDriver.app` do not automatically cover every subprocess named `cua-driver`. ## Supported macOS identities macOS supports three intentional identities: - **Same-process SDK:** import `CuaDriver.create()` in the signed app that owns the grants. Desktop operations inherit that host process's identity. - **Standalone daemon:** grant permissions to the installed `CuaDriver.app` and launch its daemon through LaunchServices. See [macOS permissions](). - **App-hosted daemon:** have the macOS app that owns the grants spawn `cua-driver serve --embedded` directly, then connect an MCP proxy to its socket. See [Embedding](). A raw daemon launched outside `CuaDriver.app` without embedded mode has no stable TCC identity and is unsupported. Do not grant permissions to arbitrary binary paths or use that configuration in production. If an IDE terminal starts `cua-driver` directly, macOS attributes that subprocess to the terminal app's bundle, not to `CuaDriver.app`. The binary is right, but the privacy identity is wrong. The standalone daemon path fixes attribution for external callers. It launches through LaunchServices with `open -n -g -a CuaDriver`, so macOS treats it as part of `CuaDriver.app`. The MCP stdio process remains where the assistant spawned it, but becomes a thin proxy: it forwards tool calls to the daemon over a Unix socket and returns the daemon's responses. There are two processes, but one tool surface. ## Windows has the same shape for a different reason On Windows, the daemon solves a session problem rather than a bundle-identity problem. When `cua-driver` is reached through SSH, the SSH-side process typically lands in Session 0, the non-interactive service session. Session 0 is not the logged-in user's GUI desktop, so it cannot see or operate those windows. The daemon belongs in the interactive user session instead. It may be kept there by platform autostart machinery such as a Scheduled Task. An SSH-side client can then proxy requests to it. The shape is the same as on macOS: a caller-side process speaks the protocol, while a daemon-side process owns desktop access. The root cause is different. ## Session identity and shared state A daemon drives one physical machine. Multiple MCP clients can connect at the same time, and a trusted SDK host may create multiple direct runtimes, but they still share the same screen, keyboard, pointer, accessibility tree, and OS focus. Session or runtime identity does not create an independent desktop. Native pointer, keyboard, value-setting, and focus calls are admitted one at a time inside a process so their input delivery does not overlap. Post-action recording and PiP capture happen after that admission is released and may observe later desktop activity. Application lifecycle operations and browser/CDP mutations are not covered by the input gate. Higher-level sequences can still interleave unless the host schedules them; in particular, do not split a button-down/drag/up gesture across independently scheduled runtime calls. On Windows and Linux, bare MCP processes each own a separate runtime. To deliberately share one daemon across several MCP clients, start the daemon and give every client the same explicit `cua-driver mcp --socket ` command. Do not rely on ambient daemon discovery. Session identity solves a narrower state problem. Each authenticated transport gets one private implicit lifecycle identity. The daemon uses it to scope mutable state to one client lifetime. Recording ownership, per-session config overrides, and the agent-cursor overlay are keyed by that lifecycle session. Callers may add a public label, but the label does not carry authority. For same-process SDK runtimes, Cua adds a private runtime generation behind the public session label. Two runtimes may therefore both use `research-1` without sharing lifecycle state, tombstones, cursor ownership, recording teardown, browser refs, or element tokens. The private generation is never serialized and is not a credential or same-process security boundary. Repeated unnamed calls on one transport reuse its implicit session. A different transport gets a different private identity. Callers need an explicit public session only when they want a stable human-readable label or explicit lifecycle control. [Four agent sessions share one Windows desktop]() Four Hermes agents operate four app windows. Each session has its own cursor overlay, while all four sessions still share the same desktop. Cleanup follows the proxy connection as well as explicit lifecycle calls. The proxy keeps a long-lived control connection open to the daemon. When the proxy exits, even from an ungraceful kill, the kernel closes that connection. The daemon sees EOF and runs the same cleanup hooks used by `end_session`. The five-minute idle TTL provides the same cleanup for abandoned live transports. ## Lifetimes and memory The proxy and one-shot CLI processes may come and go, but the daemon owns element-index caches, active recordings, configuration, policy, and cursor state. This makes the execution identity and permission-policy boundary stable across client reconnects. If the daemon disappears, daemon-backed clients fail closed. They do not construct a fresh tool registry or continue with partial state. For a same-process SDK runtime, the importing application owns the equivalent lifetime. End every session, await `shutdown()`, and release the generated binding handle during orderly teardown. Shutting down one direct runtime does not stop or revoke another direct runtime in the same process. Before each daemon-backed SDK action, the client reads the daemon metadata and verifies contract, tool-schema, capability, and MCP protocol versions. Incompatible processes refuse before platform dispatch. Remote Driver carriers perform the equivalent capability-range negotiation and must support request cancellation. --- # Known Limits Documented behavioral limits of Cua Driver and available alternatives Cua Driver attempts background delivery only when the operating system and target expose a route that can be addressed safely. Unsupported shapes return a structured refusal instead of reporting a silent success. This page lists target-specific constraints and available alternatives; see [Platform Support]() for the broader operating-system matrix. --- ## Browser tools target exactly bound Chromium pages The typed browser mutation tools currently target Chromium-family browsers and Electron through an owned Chrome DevTools Protocol endpoint. Safari and Firefox may still be inspected through existing accessibility or legacy page routes, but they do not receive typed browser mutation capabilities. | Surface | Typed browser identity | Typed mutation | Available fallback | | --- | --- | --- | --- | | Chrome, Edge, Chromium, Electron | Chromium | Exact CDP binding or structured refusal | Native `get_window_state` and AX/PX actions | | Firefox | Gecko | `browser_route_unavailable` in the current release | Native accessibility and legacy `page` routes where supported | | Safari | WebKit | `browser_route_unavailable` in the current release | Native accessibility and explicit legacy Apple Events routes | | Tauri, WKWebView, and WebKitGTK hosts | WebKit where the host can be identified | Structured refusal until an exact engine/native-window binding exists | Native accessibility and pixel actions | | WebView2 hosts | Chromium renderer in a separate process | Structured refusal for the common split-process shape | Native UIA and pixel actions | Firefox is classified consistently as a browser on macOS, Windows, and Linux, but Cua Driver does not currently advertise a WebDriver BiDi route. Safari Apple Events JavaScript is not equivalent to trusted browser input, and mutable tab ordinals cannot satisfy exact window targeting. These surfaces therefore remain capability-visible but mutation-unavailable instead of being guessed. The typed browser surface is preview-grade before 1.0. Product support means that the exact binding, supported mutations, declared background posture, and structured refusals have release evidence. Recognition alone is not a support claim. | Platform and surface | Current typed-browser status | | --- | --- | | Windows Chrome and Edge | Validated. Trusted background pointer and synthetic DOM routes are covered. An interactive desktop is required. | | macOS Chrome and Edge | Existing-profile attachment is validated. Synthetic DOM actions can remain fully background; trusted standalone pointer routes refuse when Chromium would activate. | | Linux X11 Chrome and Edge | Validated for exact binding and synthetic DOM background routes. Trusted standalone pointer routes refuse when activation is unavoidable. | | Linux Sway with Chrome | Validated only when compositor identity, process, window, and geometry are exact. | | Generic GNOME or KDE Wayland | Read-only discovery or structured refusal; mutation is not claimed. | | Electron | Validated only for the bounded one-native-window to one-CDP-page shape. | | Brave, Vivaldi, Opera, Arc, and other detected Chromium derivatives | Detected but not release-accepted; use at your own risk. | | Safari, Firefox, WebView2, Tauri, WKWebView, and WebKitGTK | Typed mutation unsupported. Use documented native AX/PX fallbacks where available. | Page refs traverse open shadow roots and same-process frames and are capped to the first 300 interactive elements. Out-of-process frames are included only when the runtime exposes a capability-tested CDP session for that frame; otherwise the omitted frame is reported as a limitation instead of being flattened into the main document. A newer snapshot or navigation invalidates previous refs. `browser_prepare` can either create a separate driver-owned isolated profile or attach to an approved existing Chrome, Edge, or Chromium profile on a proven platform. Existing-profile setup may enable that browser instance's own remote-debugging switch through one exact accessibility action. When current macOS Chrome withholds the internal page's web accessibility tree, Cua Driver instead creates and navigates a temporary tab, proves the committed fixed address and expected selected-tab title with no active omnibox edit, and requires one unique checkbox-shaped control in a bounded setup-page region. It revalidates the unchanged target window, PID-routes the click, and verifies the visual state transition. Unsupported appearance, scale, or zoom geometry is refused without a click. Unsupported window sizes or toolbar layouts, including a bookmarks bar that moves the control outside the bounded region, are refused the same way. It never copies profile data, edits profile files, restarts, or terminates the selected process. Endpoint ownership must resolve to the approved browser PID; wrapper processes, ambiguous process trees, unsupported products, unrecognized UI locales, and generic Wayland identities are refused. On generic GNOME or KDE Wayland sessions, browser state may be discoverable without enough compositor evidence to correlate native and DevTools geometry exactly. That route stays read-only. X11 and the validated Sway configuration can authorize mutation when their ownership and geometry proofs agree. This does not imply arbitrary raw background PX delivery on Wayland. --- ## Chromium coerces synthetic right-clicks on web content **Symptom:** `right_click({pid, x, y})` on a Chrome, Edge, Brave, or Arc tab's web content fires a left-click instead of opening the context menu. **Cause:** Chromium's renderer-IPC filter drops the right-click subtype bit on events that don't come through the HID tap. Every synthesized-event path on macOS hits this wall. **Workarounds, in order of preference:** 1. Use `right_click({pid, element_index})` on AX-addressable targets (links, buttons, toolbar items). AX delivery sidesteps the renderer filter entirely. 2. For context menus on pure web content (nothing in the AX tree), activate Chrome briefly and fall back to a HID-tap right-click. This interrupts best-effort background behavior for that one click. **Note** Element-indexed right-click (`right_click` with `element_index`) works fine. The limit is specifically pixel right-click on non-AX Chromium web content. --- ## Canvas apps need brief frontmost activation **Affected:** Blender (GHOST event source), Unity editor / Unity games, most native games, some WebGL-heavy Electron apps. **Symptom:** `click({pid, x, y})` on a Blender viewport silently no-ops. The window is visible and `launch_app` works, but clicks vanish. **Cause:** These apps only accept events from `cghidEventTap` with a leading `mouseMoved`. They explicitly filter out per-pid-routed events, which is the path Cua Driver uses for background delivery. There is no per-pid recipe that reaches them. **Workaround:** Bring the app to the foreground before clicking, then use pixel `click({pid, x, y})`. Where the target exposes AX-addressable controls, prefer `right_click` or element actions, which sidestep the renderer filter without foregrounding. **Warning** When automating Blender or a native game, best-effort background delivery does not apply: these apps must be foregrounded to receive clicks, so do this only when the user is not actively working on the machine. --- ## Off-Space SwiftUI windows strip their AX tree **Symptom:** `get_window_state({pid, window_id})` on a window on a different Space (e.g. System Settings parked on Space 2 while on Space 1) returns a minimal tree that contains only the menu bar, or just the `AXApplication` root. **Cause:** macOS 14+ strips AX detail from non-current-Space SwiftUI windows as a privacy/performance tradeoff. AppKit apps are not affected. There is no workaround that keeps the window off-Space. **Response shape:** Every `get_window_state` response on an off-current-Space window carries `off_space: true`, so callers can decide to switch Space, pick a different window, or skip the turn. **Workarounds:** 1. Switch the user to the target's Space first. This breaks the no-Space-bounce promise. 2. Target an AppKit equivalent of the app if one exists. 3. Limit off-Space automation to AppKit apps where the tree stays populated. --- ## Minimized windows silently drop keyboard commits **Symptom:** `press_key({pid, element_index, key: "return"})` on a text field in a minimized window returns success, but the field doesn't commit. The macOS system-alert beep fires, or nothing happens. **Cause:** AX reads and AX clicks propagate through to minimized windows normally, but keyboard-commit events (Return, Space, Tab) require renderer focus, which AX focus does not confer on a minimized window. This is a macOS-wide behavior. **Workarounds:** 1. Use `set_value({pid, element_index, value: "..."})` to write the field's value directly. No keyboard event involved; no focus handoff required. 2. AX-click a commit-equivalent button (Go, Submit, Send, OK) rather than relying on Return. 3. Restore the window through the Dock or an application-specific Window menu command. This interrupts best-effort background behavior for that window. (`Cmd+M` minimizes a window; it does not restore one.) **Note** `set_value` is the correct approach 90% of the time. It sidesteps both the minimized-focus issue and the general "which event commits this field" ambiguity. --- ## Native Wayland background keyboard input is focus-bound **Affected:** Unfocused GTK/Qt apps running as native clients on standard Wayland compositors, including Sway/wlroots, GNOME/Mutter, and KDE/KWin (no X11 surface). **Symptom:** A background `press_key`, `hotkey`, or `type_text` request for a field that is not AT-SPI-editable returns structured `background_unavailable`. Foreground delivery may also refuse when the desktop has no target-addressable activation or raw-input backend. **Cause:** Wayland blocks one ordinary client from targeting another client's surface with synthetic input. Portal/libei input on GNOME and KDE follows the compositor's active seat and requires a RemoteDesktop grant. The virtual keyboard used on wlroots compositors is also focus-bound, even though it does not require the same portal route. **Workarounds:** 1. Type into accessible text fields with `type_text`. AT-SPI `insertText` writes the field directly, with no synthetic key event involved. 2. Drive controls by `element_index` (`click`, `set_value`) instead of keyboard shortcuts where an equivalent control exists. 3. Retry with `delivery_mode:"foreground"` when the desktop exposes a verified activation and input adapter. 4. Run the app under XWayland (`GDK_BACKEND=x11` / `QT_QPA_PLATFORM=xcb`). It then exposes an X11 surface and the X11 keyboard paths apply. **Note** Background element actions can land through AT-SPI. A coordinate left click can also land when its point resolves to an actionable AT-SPI element. This does not establish arbitrary raw background PX delivery. The opt-in nested `cua-compositor` is a separate, compositor-owned environment. Its private per-surface injection protocol is not constrained like an ordinary client on Sway, GNOME, or KDE, and its capabilities are documented separately. --- ## GTK4 reports (0,0) screen coordinates over AT-SPI (handled) **Symptom:** none in normal use. Element `frame`s, the agent cursor, and vision clicks are correct on GTK4. Documented here because the underlying toolkit bug is real and visible in raw AT-SPI. **Cause:** GTK4's AT-SPI bridge returns `Component.GetExtents(SCREEN)` as `(0,0)` for every widget (GNOME/gtk issues #1564 / #1739). A naive consumer would collapse every element to the window's top-left corner. **How Cua Driver handles it:** it queries `CoordType::Window` (which GTK4 *does* report correctly per-widget) and adds the window's screen origin from `_GTK_FRAME_EXTENTS` on X11 or the `org.cua.WinRects` shell helper on Wayland. That reconstructs true screen coordinates, so no caller action is required. --- ## `elements[].frame` is screen-absolute, but pixel actions use screenshot pixels **Symptom:** a caller reads an element's `frame` from `get_window_state` and passes its center straight to `click({pid, window_id, x, y})`. The click lands somewhere other than the element — displaced by the window's screen origin, and possibly scaled by the capture ratio. Nothing refuses, because the coordinates were structurally valid; they simply addressed a different point. **Cause:** element geometry and pixel action arguments are expressed in two different coordinate spaces, and neither field restates its own space. | Surface | Space | Units | | ----------------------------------------------------------- | ---------------------------------------------------- | ------------------------------------- | | `elements[].frame` on macOS | Screen-absolute, top-left origin | Logical points | | `elements[].frame` on Windows and Linux | Screen-absolute, top-left origin | Physical pixels | | `x`, `y` with `pid` and `window_id` | Window-local, top-left of the `get_window_state` PNG | Screenshot pixels | | `x`, `y` with `scope:"desktop"` and no `pid` or `window_id` | Screen-absolute | Pixels in the `get_desktop_state` PNG | With `pid` and `window_id`, the pixel forms of `click`, `right_click`, `double_click`, `drag`, `scroll`, `type_text`, `press_key`, and `hotkey` read the window-local screenshot space. `double_click` currently labels these as screen coordinates in its generated argument reference, but its implementation uses the same window-local conversion as the other pixel actions. **What to do:** prefer `element_token`, or `element_index` with the matching `snapshot_id`. The accessibility path carries no coordinates, so it is unaffected by this mismatch, and it also works on backgrounded windows. Reach for a pixel action only on canvas, WebGL, or custom-drawn surfaces that expose no element. When a pixel action is unavoidable, match the scope to the space: - **Window scope.** Rebase onto the window origin, then rescale to the capture. Take the origin and size from `list_windows` (`windows[].bounds`), which is reported in the same units as `frame`, and take the capture size from the snapshot's `screenshot_width` and `screenshot_height`: ```text scale_x = screenshot_width / bounds.width scale_y = screenshot_height / bounds.height x = (frame.x + frame.w / 2 - bounds.x) * scale_x y = (frame.y + frame.h / 2 - bounds.y) * scale_y ``` - **Desktop scope.** If the element is on the captured primary display, rescale the frame center into the full-display PNG returned by `get_desktop_state`, then call `click` with `scope:"desktop"` and no `pid` or `window_id`: ```text scale_x = desktop.screenshot_width / desktop.screen_width scale_y = desktop.screenshot_height / desktop.screen_height x = (frame.x + frame.w / 2) * scale_x y = (frame.y + frame.h / 2) * scale_y ``` These ratios are normally 1 on Windows and Linux. On macOS Retina displays, the frame is in logical points while the desktop PNG is in native pixels, so the ratio is normally 2. **Note** Do not assume a window-scope scale is 1. It is normally 2 on a Retina display, and may be a fraction whenever a large window was downscaled to `max_image_dimension`. Re-derive it from each snapshot rather than caching it across calls. On macOS, `get_window_state` also reports `window_bounds`, which may be used in place of `list_windows` there. Windows and Linux report `screenshot_width` and `screenshot_height` only. --- ## Cinnamon disables accessibility advertisement On Cinnamon (`Cinnamon` or `X-Cinnamon`), Cua Driver leaves the session accessibility flags unchanged. Setting `ScreenReaderEnabled=true` launches Orca, while setting only `IsEnabled=true` can make the Cinnamon and GNOME settings schemas repeatedly overwrite each other. Applications that rely on Cua Driver to enable their accessibility bridge may therefore return a degraded or empty AT-SPI tree. Screenshots and available CDP browser routes continue to work. Do not set `CUA_DRIVER_RS_A11Y_ADVERTISE_MODE` to `all` or `is_enabled_only` in a normal Cinnamon session. --- ## Permission boundaries Cua Driver is constrained by the macOS permission model. Two relevant grants: | Grant | Required for | | ----- | ------------ | | **Accessibility** (System Settings → Privacy & Security → Accessibility) | Every AX read, every element-indexed click, every keyboard/text primitive. Without it, `check_permissions` returns `accessibility: false` and every tool returns a structured error. | | **Screen Recording** | Screenshots. `get_window_state` returns both the accessibility tree and a screenshot by default; without this grant it returns the tree only (no PNG). The tree path still works. | Grants are tied to the `CuaDriver.app` bundle identity (`com.trycua.driver`). The Rust build and installer preserve this identity, so TCC grants persist across rebuilds and updates. --- # Development Source layout, harnesses, and test entry points for contributors working on cua-driver. cua-driver lives under `libs/cua-driver` in the repository. The hosted docs describe the public interface; contributor details live next to the code so they stay close to implementation changes. ## Source Map | Path | Purpose | |---|---| | `libs/cua-driver/rust` | Rust workspace for the daemon, platform crates, testkit, and helper crates | | `libs/cua-driver/typescript` | Rust-backed TypeScript SDK plus Node and Electron embedded-host adapters | | `libs/cua-driver/python` | Python package wrapper around the driver binary | | `libs/cua-driver/tests/fixtures` | Source-built GUI harness apps and shared fixtures | | `libs/cua-driver/rust/crates/cua-driver/tests` | Rust integration tests for the driver and GUI harnesses | | `libs/cua-driver/scripts` | Install, uninstall, local build, and VM sync helpers | ## Contributor Entry Points - `libs/cua-driver/README.md` gives the repo-local overview. - `libs/cua-driver/rust/README.md` maps the Cargo workspace and test classes. - `libs/cua-driver/rust/crates/cua-driver/tests/README.md` explains Rust test naming and ignored GUI lanes. - `libs/cua-driver/tests/fixtures/README.md` explains source-built harness apps and staged outputs. - `libs/cua-driver/scripts/README.md` explains install and VM sync helpers. ## Desktop Validation Entry Points The canonical GUI runners execute the complete Rust harness catalog. Their internal CI lanes may split shared, native, and capture owners for reporting, but contributors do not select those partitions directly. ```text Linux X11/session: scripts/ci/linux/run-rust-e2e.sh Linux Sway: scripts/ci/linux/run-rust-e2e-wayland.sh Linux nested: scripts/ci/linux/run-rust-e2e-inject.sh Linux GNOME/KDE: scripts/ci/linux/run-rust-e2e-desktop.sh Linux real Xorg: scripts/ci/linux/run-rust-e2e-desktop.sh xorg Windows: .\scripts\ci\windows\run-rust-e2e.ps1 -RequireGui macOS: scripts/ci/macos/run-rust-e2e.sh ``` The Sway runner creates a controlled stock-wlroots session. Nix owns the Linux source/package gate and provides the optional nested `cua-compositor` session; it is not the wrapper for every Linux E2E environment. GNOME, KDE, real Xorg, Windows, and macOS use an existing graphical login. Windows needs an interactive console or RDP session, and macOS needs a logged-in session with the required Accessibility and Screen Recording permissions. See [Platform support]() for current capability boundaries, [How Cua Driver is validated]() for the evidence model, and [Platform roadmap]() for remaining work. Build artifacts, VM logs, and local verification journals should stay out of git unless they have been promoted into a stable fixture or contributor doc. --- # CLI Reference Command Line Interface reference for Lume A lightweight CLI and local API server to build, run and manage macOS VMs. Documented against Lume **0.5.3**. Run `lume --version` for your installed version. For installation steps, see [Install Lume](). ## VM Management ### lume create Create a new virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name for the virtual machine | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--os` | String | macOS | Operating system to install (macOS or linux) | | `--cpu` | Int | 4 | Number of CPU cores | | `--memory` | String | 8GB | Memory size (e.g., 8GB) | | `--disk-size` | String | 100GB for macOS; 50GB for Linux | Disk size (e.g., 100GB) | | `--display` | String | 1024x768 | Display resolution (e.g., 1024x768) | | `--ipsw` | String | - | Path to IPSW file or 'latest' for macOS VMs | | `--storage` | String | - | VM storage location to use | | `--unattended` | String | - | Prepare macOS unattended setup offline after install. Preset name or YAML path is accepted for compatibility. Built-in presets: sequoia, tahoe. Only supported for macOS VMs. | | `--debug-dir` | String | - | Compatibility option; ignored by offline setup. | | `--vnc-port` | Int | 0 | Port to use for the temporary verification VNC server. Defaults to 0 (auto-assign). | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--debug` | false | Compatibility flag; ignored by offline setup. | | `--no-display` | false | Compatibility flag; offline setup verifies headlessly. | ### lume run Run a virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the VM or image to run (format: name or name:tag) | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--shared-dir` | [String] | - | Directory to share with the VM (format: path or path:ro or path:rw) | | `--mount` | String | - | For Linux VMs only, attach a read-only disk image | | `--usb-storage` | [String] | - | Disk image to attach as USB mass storage device | | `--disk` | [String] | - | Disk image to attach as a read-write virtio-blk device | | `--registry` | String | ghcr.io | Container registry URL | | `--organization` | String | trycua | Organization to pull from | | `--display` | DisplayMode | native | Local viewer to open: vnc, native, or none. The VNC server remains available in every mode | | `--log-file` | String | ~/Library/Logs/lume/{vm}.log | Log path for --detach | | `--vnc` | VNCPolicy | enabled | VNC server policy: enabled or disabled. disabled starts the VM with no VNC listener and reports a null vncUrl | | `--vnc-port` | Int | 0 | Port for VNC server (0 for auto-assign) | | `--recovery-mode` | Bool | false | For macOS VMs only, boot in recovery mode | | `--storage` | String | - | VM storage location to use | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `-d, --no-display` | false | Compatibility alias for --display none | | `--detach` | false | Run the VM in the background and return immediately | | `--clipboard` | false | Enable bidirectional clipboard sync via SSH. Automatic for native macOS display | ### lume attach Open a viewer for a running virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the virtual machine | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--display` | AttachDisplayMode | native with VNC fallback | Viewer to open: native or vnc | | `--storage` | String | - | VM storage location to use | ### lume shutdown Gracefully shut down a virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the virtual machine | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `-u, --user` | String | lume | SSH username | | `-p, --password` | String | lume | SSH and sudo password | | `--storage` | String | - | VM storage location to use | | `-t, --timeout` | Int | 30 | SSH command timeout in seconds | ### lume restart Gracefully restart a virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the virtual machine | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `-u, --user` | String | lume | SSH username | | `-p, --password` | String | lume | SSH and sudo password | | `--storage` | String | - | VM storage location to use | | `-t, --timeout` | Int | 30 | SSH command timeout in seconds | ### lume stop Stop a virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the VM to stop | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--storage` | String | - | VM storage location to use | ### lume delete Delete a virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the VM to delete | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--storage` | String | - | VM storage location to use | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--force` | false | Force deletion without confirmation | ### lume clone Clone an existing virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the source VM | | `` | String | Yes | Name for the cloned VM | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--source-storage` | String | - | Source VM storage location | | `--dest-storage` | String | - | Destination VM storage location | ## VM Information and Configuration ### lume ls List virtual machines **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `-f, --format` | String | text | Output format (json or text) | | `--storage` | String | - | Filter by storage location name | ### lume get Get detailed information about a virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the VM | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `-f, --format` | String | text | Output format | | `--storage` | String | - | VM storage location to use | ### lume set Set new values for CPU, memory, and disk size of a virtual machine **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the VM | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--cpu` | Int | - | New number of CPU cores | | `--memory` | String | - | New memory size (e.g., 8GB) | | `--disk-size` | String | - | New total disk size. Increase only; macOS resizing preserves RecoveryOS and grows APFS. | | `--display` | String | - | New display resolution | | `--storage` | String | - | VM storage location to use | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--no-backup` | false | Skip the macOS rollback backup | | `--keep-backup` | false | Keep rollback files after a successful macOS resize | | `--dry-run` | false | Validate the resize plan without modifying the disk | ## Image Management ### lume images List available macOS images from local cache **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--organization` | String | trycua | Organization to list images for | ### lume pull Pull a prebuilt or custom macOS image from an OCI-compatible registry **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Image to pull (format: name:tag) | | `` | String | No | Name for the resulting VM | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--registry` | String | ghcr.io | Container registry URL | | `--organization` | String | trycua | Organization to pull from | | `--storage` | String | - | VM storage location to use | ### lume push Push a macOS VM to an OCI-compatible registry **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of VM to push | | `` | String | Yes | Image tag (format: name:tag) | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--additional-tags` | [String] | - | Additional tags to push | | `--registry` | String | ghcr.io | Container registry URL | | `--organization` | String | trycua | Organization to push to | | `--storage` | String | - | VM storage location to use | | `--chunk-size-mb` | Int | 512 | Chunk size for upload in MB | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--verbose` | false | Enable verbose logging | | `--dry-run` | false | Prepare files without uploading | | `--reassemble` | true | Verify integrity by reassembling chunks | ### lume convert Convert a legacy Lume image to OCI-compliant format **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Source image to convert (legacy format, for example macos-tahoe:latest) | | `` | String | Yes | Target image to push in OCI format (name:tag) | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--additional-tags` | [String] | - | Additional tags for the OCI image | | `--registry` | String | ghcr.io | Registry to pull from and push to | | `--organization` | String | trycua | Registry organization | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--verbose` | false | Enable verbose logging | | `--dry-run` | false | Prepare files without uploading | | `--single-layer` | false | Push one kubelet-compatible disk layer | ### lume ipsw Get macOS restore image IPSW URL ### lume prune Remove cached images ## Guest Access and Security ### lume ssh Connect to a VM via SSH or execute commands remotely **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the virtual machine | | `` | [String] | No | Command to execute (omit for interactive shell) | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `-u, --user` | String | lume | SSH username | | `-p, --password` | String | lume | SSH password | | `--storage` | String | - | Storage location name or path | | `-t, --timeout` | Int | 60 | Command timeout in seconds (0 for no timeout) | ### lume setup Prepare unattended macOS setup **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Name of the virtual machine | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--unattended` | String | tahoe | Defaults to tahoe. Preset name or YAML path for compatibility and optional post-SSH commands. Built-in presets: sequoia, tahoe. | | `--storage` | String | - | VM storage location to use or direct path to VM location | | `--vnc-port` | Int | 0 | Port to use for the temporary verification VNC server. Defaults to 0 (auto-assign) | | `--debug-dir` | String | - | Compatibility option; ignored by offline setup | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--no-display` | false | Compatibility flag; offline setup verifies headlessly | | `--debug` | false | Compatibility flag; ignored by offline setup | ### lume sip Enable or disable System Integrity Protection on a macOS VM **Arguments:** | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | `` | String | Yes | Desired SIP state: on or off | | `` | String | Yes | Name of the virtual machine | **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--admin-user` | String | lume | Administrator username in the guest | | `--admin-password` | String | - | Administrator password in the guest; prefer --admin-password-stdin | | `--screenshot-dir` | String | - | Directory for Recovery framebuffer screenshots | | `--vnc-port` | Int | 5999 | Port for the temporary Recovery VNC server | | `--storage` | String | - | VM storage location | | `--timeout` | Int | 900 | Overall timeout in seconds | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `-y, --yes` | false | Skip the interactive confirmation prompt | | `--admin-password-stdin` | false | Read one administrator-password line from standard input without echo | ## Configuration and Server ### lume config Get or set lume configuration **Subcommands:** - `lume config get` - Get current configuration - `lume config storage` - Manage VM storage locations - `lume config storage add` - Add a new VM storage location - `lume config storage remove` - Remove a VM storage location - `lume config storage list` - List all VM storage locations - `lume config storage default` - Set the default VM storage location - `lume config cache` - Manage image cache settings - `lume config cache status` - Show cache status and directory - `lume config cache dir` - Get or set cache directory - `lume config cache enable` - Enable image caching - `lume config cache disable` - Disable image caching - `lume config telemetry` - Manage pseudonymous telemetry settings - `lume config telemetry status` - Show current telemetry status - `lume config telemetry enable` - Enable pseudonymous telemetry - `lume config telemetry disable` - Disable pseudonymous telemetry - `lume config telemetry reset-id` - Delete the pseudonymous installation ID and registration markers ### lume serve Start the VM management server **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--port` | Int | 7777 | Port to listen on | ### lume logs View lume serve logs **Subcommands:** - `lume logs info` - View info logs from the daemon - `-n, --lines` - Number of lines to display - `lume logs error` - View error logs from the daemon - `-n, --lines` - Number of lines to display - `lume logs all` - View both info and error logs - `-n, --lines` - Number of lines to display ### lume check-update Check whether a newer Lume release is available **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--json` | false | Emit the structured update-state payload as JSON | | `--no-cache` | false | Bypass the local update-check cache | ### lume update Check for a Lume update and optionally apply it **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--apply` | false | Apply the update by re-running the official installer | | `--json` | false | Emit the structured update-state payload as JSON | ### lume channel Inspect or change the stable/nightly update channel; selection does not install **Subcommands:** - `lume channel status` - Show selected and current release channels - `lume channel set` - Save stable or nightly as the update channel - `` - Release channel: stable or nightly ## Developer Tools ### lume dump-docs Output CLI and API documentation as JSON for tooling and integrations **Options:** | Name | Type | Default | Description | | ---- | ---- | ------- | ----------- | | `--type` | String | cli | Documentation type: cli, api, or all | **Flags:** | Name | Default | Description | | ---- | ------- | ----------- | | `--pretty` | false | Pretty-print JSON output | ## Global Options These options are available for all commands: - `--help` - Show help information - `--version` - Show version number --- # API Reference HTTP API reference for Lume server HTTP API for managing macOS and Linux virtual machines Documented against Lume **0.5.3**. Run `lume --version` for your installed version. ## Default URL ``` http://localhost:7777 ``` Start the server with `lume serve` or specify a custom port with `lume serve --port `. ## VM Management ### List all virtual machines List all virtual machines `GET: /lume/vms` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | storage | string | No | Filter by storage location name | #### Example Request **Curl** ```bash curl "http://localhost:7777/lume/vms" ``` **Python** ```python import requests response = requests.get("http://localhost:7777/lume/vms") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms`); const data = await response.json(); ``` #### Response - **200**: Success - **400**: Bad request --- ### Get detailed information about a specific virtual machine Get detailed information about a specific virtual machine `GET: /lume/vms/:name` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the VM | | storage | string | No | VM storage location to use | #### Example Request **Curl** ```bash curl "http://localhost:7777/lume/vms/my-vm" ``` **Python** ```python import requests response = requests.get("http://localhost:7777/lume/vms/my-vm") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms/my-vm`); const data = await response.json(); ``` #### Response - **200**: Success - **400**: VM not found or invalid request --- ### Create a new virtual machine Create a new virtual machine `POST: /lume/vms` #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name for the virtual machine | | os | string | Yes | Operating system to install (macOS or linux) | | cpu | integer | Yes | Number of CPU cores | | memory | string | Yes | Memory size (e.g., 8GB) | | diskSize | string | Yes | Disk size (e.g., 50GB) | | display | string | Yes | Display resolution (e.g., 1024x768) | | ipsw | string | No | Path to IPSW file or 'latest' for macOS VMs | | storage | string | No | VM storage location to use | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/vms" \ -H "Content-Type: application/json" \ -d '{ "name": "my-vm", "os": "macOS", "cpu": 4, "memory": "8GB", "diskSize": "50GB", "display": "1024x768" }' ``` **Python** ```python import requests data = { "name": "my-vm", "os": "macOS", "cpu": 4, "memory": "8GB", "diskSize": "50GB", "display": "1024x768", } response = requests.post("http://localhost:7777/lume/vms", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "my-vm", os: "macOS", cpu: 4, memory: "8GB", diskSize: "50GB", display: "1024x768", }), }); const data = await response.json(); ``` #### Response - **200**: VM created successfully - **400**: Invalid request body or VM creation failed --- ### Delete a virtual machine and its associated files Delete a virtual machine and its associated files `DELETE: /lume/vms/:name` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the VM to delete | | storage | string | No | VM storage location | #### Example Request **Curl** ```bash curl -X DELETE "http://localhost:7777/lume/vms/my-vm" ``` **Python** ```python import requests response = requests.delete("http://localhost:7777/lume/vms/my-vm") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms/my-vm`, { method: "DELETE", }); const data = await response.json(); ``` #### Response - **200**: VM deleted successfully - **400**: VM not found or deletion failed --- ### Create a copy of an existing virtual machine Create a copy of an existing virtual machine `POST: /lume/vms/clone` #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the source VM | | newName | string | Yes | Name for the cloned VM | | sourceLocation | string | No | Source VM storage location | | destLocation | string | No | Destination VM storage location | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/vms/clone" \ -H "Content-Type: application/json" \ -d '{ "name": "my-vm", "newName": "example" }' ``` **Python** ```python import requests data = { "name": "my-vm", "newName": "example", } response = requests.post("http://localhost:7777/lume/vms/clone", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms/clone`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "my-vm", newName: "example", }), }); const data = await response.json(); ``` #### Response - **200**: VM cloned successfully - **400**: Clone operation failed --- ### Update virtual machine configuration settings Update virtual machine configuration settings `PATCH: /lume/vms/:name` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the VM to update | #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | cpu | integer | No | New number of CPU cores | | memory | string | No | New memory size (e.g., 16GB) | | diskSize | string | No | New total disk size (increase only) | | display | string | No | New display resolution | | storage | string | No | VM storage location | | noBackup | boolean | No | Skip the macOS rollback backup (default: false) | | keepBackup | boolean | No | Keep rollback files after success (default: false) | | dryRun | boolean | No | Validate the resize plan without modifying the disk (default: false) | #### Example Request **Curl** ```bash curl -X PATCH "http://localhost:7777/lume/vms/my-vm" \ -H "Content-Type: application/json" \ -d '{}' ``` **Python** ```python import requests data = { } response = requests.patch("http://localhost:7777/lume/vms/my-vm", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms/my-vm`, { method: "PATCH", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ }), }); const data = await response.json(); ``` #### Response - **200**: Settings updated successfully - **400**: Invalid settings or update failed --- ### Start a virtual machine Start a virtual machine `POST: /lume/vms/:name/run` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the VM to start | #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | noDisplay | boolean | No | Run without VNC display (default: false) | | sharedDirectories | array | No | Directories to share with the VM | | recoveryMode | boolean | No | Boot macOS VM in recovery mode (default: false) | | storage | string | No | VM storage location | | clipboard | boolean | No | Enable bidirectional clipboard sync via SSH (experimental) (default: false) | | vnc | string | No | VNC server policy: 'enabled' or 'disabled'. 'disabled' requires noDisplay=true, starts no VNC listener, and reports a null vncUrl (default: enabled) | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/vms/my-vm/run" \ -H "Content-Type: application/json" \ -d '{}' ``` **Python** ```python import requests data = { } response = requests.post("http://localhost:7777/lume/vms/my-vm/run", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms/my-vm/run`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ }), }); const data = await response.json(); ``` #### Response - **202**: VM start initiated (async operation) - **400**: Invalid request or VM not found --- ### Stop a running virtual machine Stop a running virtual machine `POST: /lume/vms/:name/stop` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the VM to stop | #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | storage | string | No | VM storage location | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/vms/my-vm/stop" \ -H "Content-Type: application/json" \ -d '{}' ``` **Python** ```python import requests data = { } response = requests.post("http://localhost:7777/lume/vms/my-vm/stop", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms/my-vm/stop`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ }), }); const data = await response.json(); ``` #### Response - **200**: VM stopped successfully - **400**: Stop operation failed --- ## Image Management ### List available images from local cache List available images from local cache `GET: /lume/images` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | organization | string | No | Organization to list images for (default: trycua) | #### Example Request **Curl** ```bash curl "http://localhost:7777/lume/images" ``` **Python** ```python import requests response = requests.get("http://localhost:7777/lume/images") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/images`); const data = await response.json(); ``` #### Response - **200**: Success - **400**: Failed to list images --- ### Get the latest macOS restore image (IPSW) URL Get the latest macOS restore image (IPSW) URL `GET: /lume/ipsw` #### Example Request **Curl** ```bash curl "http://localhost:7777/lume/ipsw" ``` **Python** ```python import requests response = requests.get("http://localhost:7777/lume/ipsw") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/ipsw`); const data = await response.json(); ``` #### Response - **200**: Success - **400**: Failed to get IPSW URL --- ### Pull a VM image from a container registry Pull a VM image from a container registry `POST: /lume/pull` #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | image | string | Yes | Image to pull (format: name:tag) | | name | string | No | Name for the resulting VM | | registry | string | No | Container registry URL (default: ghcr.io) | | organization | string | No | Organization to pull from (default: trycua) | | storage | string | No | VM storage location | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/pull" \ -H "Content-Type: application/json" \ -d '{ "image": "macos-tahoe-vanilla:latest" }' ``` **Python** ```python import requests data = { "image": "macos-tahoe-vanilla:latest", } response = requests.post("http://localhost:7777/lume/pull", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/pull`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ image: "macos-tahoe-vanilla:latest", }), }); const data = await response.json(); ``` #### Response - **200**: Image pulled successfully - **400**: Pull operation failed --- ### Push a VM image to a container registry Push a VM image to a container registry `POST: /lume/vms/push` #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the local VM to push | | imageName | string | Yes | Base name for the image in the registry | | tags | array | Yes | List of tags to push | | registry | string | No | Container registry URL (default: ghcr.io) | | organization | string | No | Organization to push to (default: trycua) | | storage | string | No | VM storage location | | chunkSizeMb | integer | No | Chunk size for upload in MB (default: 512) | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/vms/push" \ -H "Content-Type: application/json" \ -d '{ "name": "my-vm", "imageName": "example", "tags": [ "latest" ] }' ``` **Python** ```python import requests data = { "name": "my-vm", "imageName": "example", "tags": ["latest"], } response = requests.post("http://localhost:7777/lume/vms/push", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/vms/push`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "my-vm", imageName: "example", tags: ["latest"], }), }); const data = await response.json(); ``` #### Response - **202**: Push initiated (async operation) - **400**: Invalid request --- ### Remove cached images to free up disk space Remove cached images to free up disk space `POST: /lume/prune` #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/prune" ``` **Python** ```python import requests response = requests.post("http://localhost:7777/lume/prune") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/prune`); const data = await response.json(); ``` #### Response - **200**: Images pruned successfully - **400**: Prune operation failed --- ## Configuration ### Get current Lume configuration settings Get current Lume configuration settings `GET: /lume/config` #### Example Request **Curl** ```bash curl "http://localhost:7777/lume/config" ``` **Python** ```python import requests response = requests.get("http://localhost:7777/lume/config") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/config`); const data = await response.json(); ``` #### Response - **200**: Success - **400**: Failed to get config --- ### Update Lume configuration settings Update Lume configuration settings `POST: /lume/config` #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | homeDirectory | string | No | VM home directory path | | cacheDirectory | string | No | Cache directory path | | cachingEnabled | boolean | No | Enable or disable image caching | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/config" \ -H "Content-Type: application/json" \ -d '{}' ``` **Python** ```python import requests data = { } response = requests.post("http://localhost:7777/lume/config", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/config`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ }), }); const data = await response.json(); ``` #### Response - **200**: Configuration updated successfully - **400**: Invalid request --- ### List all VM storage locations List all VM storage locations `GET: /lume/config/locations` #### Example Request **Curl** ```bash curl "http://localhost:7777/lume/config/locations" ``` **Python** ```python import requests response = requests.get("http://localhost:7777/lume/config/locations") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/config/locations`); const data = await response.json(); ``` #### Response - **200**: Success - **400**: Failed to get locations --- ### Add a new VM storage location Add a new VM storage location `POST: /lume/config/locations` #### Request Body | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Storage location name | | path | string | Yes | Path to storage directory | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/config/locations" \ -H "Content-Type: application/json" \ -d '{ "name": "my-vm", "path": "/path/to/storage" }' ``` **Python** ```python import requests data = { "name": "my-vm", "path": "/path/to/storage", } response = requests.post("http://localhost:7777/lume/config/locations", json=data) print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/config/locations`, { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify({ name: "my-vm", path: "/path/to/storage", }), }); const data = await response.json(); ``` #### Response - **200**: Location added successfully - **400**: Invalid request or location already exists --- ### Remove a VM storage location Remove a VM storage location `DELETE: /lume/config/locations/:name` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the location to remove | #### Example Request **Curl** ```bash curl -X DELETE "http://localhost:7777/lume/config/locations/my-vm" ``` **Python** ```python import requests response = requests.delete("http://localhost:7777/lume/config/locations/my-vm") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/config/locations/my-vm`, { method: "DELETE", }); const data = await response.json(); ``` #### Response - **200**: Location removed successfully - **400**: Location not found or cannot be removed --- ### Set the default VM storage location Set the default VM storage location `POST: /lume/config/locations/default/:name` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | name | string | Yes | Name of the location to set as default | #### Example Request **Curl** ```bash curl -X POST "http://localhost:7777/lume/config/locations/default/my-vm" ``` **Python** ```python import requests response = requests.post("http://localhost:7777/lume/config/locations/default/my-vm") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/config/locations/default/my-vm`); const data = await response.json(); ``` #### Response - **200**: Default location set successfully - **400**: Location not found --- ## Logs ### Retrieve Lume server logs Retrieve Lume server logs `GET: /lume/logs` #### Parameters | Name | Type | Required | Description | | ---- | ---- | -------- | ----------- | | type | string | No | Log type: 'info', 'error', or 'all' (default: all) | | lines | integer | No | Number of lines to return from end of log | #### Example Request **Curl** ```bash curl "http://localhost:7777/lume/logs" ``` **Python** ```python import requests response = requests.get("http://localhost:7777/lume/logs") print(response.json()) ``` **TypeScript** ```typescript const response = await fetch(`http://localhost:7777/lume/logs`); const data = await response.json(); ``` #### Response - **200**: Success - **400**: Failed to read logs --- --- # MCP tools Tool reference for Lume's stdio MCP server. Start the server with `lume serve --mcp`. Tool names use `snake_case`. ## `lume_resize_disk` Grow a stopped VM disk to a new total size. | Parameter | Type | Required | Description | | --- | --- | --- | --- | | `name` | string | yes | VM name. | | `disk_size` | string | yes | New total size, such as `120GB`. Must exceed the current size. | | `storage` | string | no | Named storage location or direct path. | | `no_backup` | boolean | no | Skip the macOS rollback copy. Defaults to `false`. | | `keep_backup` | boolean | no | Keep rollback files after success. Defaults to `false`. | | `dry_run` | boolean | no | Validate and print the plan without changing the disk. Defaults to `false`. | `no_backup` and `keep_backup` cannot both be true. macOS resizing preserves the paired RecoveryOS partition and expands the main APFS container. Linux resizing increases the image only. The MCP server also exposes tools for listing, creating, running, stopping, cloning, deleting, and accessing VMs. Clients should discover the live tool list from MCP so their schemas match the installed Lume version. --- # Telemetry and privacy Exact Lume telemetry controls, schema, and privacy boundaries. Lume sends content-free, pseudonymous product telemetry to the PostHog EU ingest endpoint. Telemetry is enabled by default. The first-run notice appears before the first event, and the preference persists across upgrades. Installation and release delivery is serialized across Lume processes. A success marker is written only after PostHog accepts the event. Failed lifecycle delivery is retried after a 15-minute backoff. ## Control telemetry ```bash lume config telemetry status lume config telemetry disable lume config telemetry enable lume config telemetry reset-id ``` `LUME_TELEMETRY_ENABLED` overrides the saved preference for the current process. Disabling telemetry stops every telemetry request but retains the installation ID. `reset-id` deletes the local pseudonymous ID and lifecycle markers while preserving the enabled or disabled preference. ## Separate update check The telemetry setting does not control the read-only request to GitHub Releases made by `lume check-update`, `lume update`, or the MCP `check_for_update` tool. The update response is cached for up to 20 hours and does not include the telemetry installation ID. Disable that request separately: ```bash LUME_UPDATE_CHECK=false lume check-update ``` ## Schema-v3 envelope Every schema-v3 event has these fixed fields: | Property | Meaning | | --- | --- | | `telemetry_schema_version` | Fixed integer `3` | | `product_version` | Installed Lume version | | `os_family` | Host family, currently `macos` | | `os_major` | Host major operating-system version only | | `arch` | Bounded host CPU architecture | | `is_ci` | Whether a recognized CI environment is present | | `is_synthetic` | Explicit Cua-owned test marker; `false` by default | | `transport` | `cli`, `http`, or `mcp_stdio` | | `process_session_id` | Random ID created once per process and never persisted | | `id_persisted` | Whether the installation ID is durable | Lume sets `$process_person_profile` to `false` and `$geoip_disable` to `true`. The installation UUID is used only as PostHog's `distinct_id`. Lume does not retain location derived from the request IP. ## Fixed events | Event | Additional bounded properties | | --- | --- | | `lume_install` | install channel and product version | | `lume_release_installed` | install channel and product version | | `lume_operation_completed` | allowlisted operation, success, fixed error class, duration bucket, `accepted` or `completed` phase, and optional guest OS | | `lume_vm_started` | guest OS and successful start signal | | `lume_mcp_session_started` | fixed MCP protocol and stdio transport | | `lume_mcp_tool_completed` | allowlisted tool and operation, success, fixed error class, and duration bucket | | `lume_provisioning_started` | guest OS and whether provisioning is asynchronous | | `lume_provisioning_completed` | guest OS, asynchronous flag, success, fixed error class, and coarse provisioning-duration bucket | | `lume_update_checked` | fixed source and outcome, strict public target version, and cache-hit flag | | `lume_update_apply_started` | strict public target version and fixed daemon-not-applicable state | | `lume_update_apply_completed` | strict public target version, fixed outcome and failure class, and duration bucket | Legacy per-command and `lume_api_*` attempt events remain during the schema migration. Analytics should use schema-v3 completion and lifecycle events for success, reliability, and active-installation metrics. Routine telemetry is capped at 1,000 events per process per hour. One successful VM-start value event remains eligible after the cap so a noisy process cannot erase the active-installation signal. ## Data that is never collected Telemetry builders accept only fixed event names and bounded properties. They do not accept VM or image names, registry or organization names, file paths, URLs, command arguments, MCP arguments or results, SSH commands or output, request or response bodies, unattended configuration, VNC credentials, VM contents, screenshots, or raw error messages. `LUME_TELEMETRY_SYNTHETIC=true` marks Cua-owned certification traffic so production dashboards can exclude it. It does not enable telemetry when the saved preference or `LUME_TELEMETRY_ENABLED` disables telemetry. --- # Limits and VM states Host requirements, VM states, storage behavior, and common Lume failures. This page records constraints that affect Lume workflows. Use the [CLI reference]() for command syntax. ## Host requirements - Lume runs on Apple Silicon Macs. - macOS 13 or later is required. - Lume needs at least 8GB of available memory. 16GB is a better starting point for a guest with 8GB allocated. - Leave at least 50GB of free disk space for a macOS guest and its restore image. ## macOS VM concurrency Apple's Virtualization framework allows Lume to run up to two macOS guests on a host. Memory, disk, and CPU capacity can reduce the practical limit. ## VM states `lume ls` can report these states: | State | Meaning | | --- | --- | | `stopped` | The VM is ready to start. | | `running` | The VM is booted and has a live virtualization session. | | `provisioning` | Lume is creating the VM or completing an asynchronous operation. | | `provisioning (stale)` | A provisioning marker remains after its operation stopped unexpectedly. | Wait for a provisioning operation to finish before running the VM. If a VM is stale, inspect the Lume logs and remove the incomplete VM before recreating it. ## Common failures ### Auxiliary storage is locked Only one process can run a VM at a time. Check the VM list and stop the process that owns the VM before retrying: ```bash lume ls lume stop macos-tahoe ``` ### `sudo lume` cannot find a VM Lume stores VMs under the current user's home directory by default. Running with `sudo` changes `HOME`, so the root process looks in a different VM store. Run Lume as the account that created the VM. ### SSH is unavailable VMs created with `--unattended tahoe` enable SSH and use `lume` / `lume` for the initial account. Check that the VM is running, then inspect its details: ```bash lume run macos-tahoe --no-display lume get macos-tahoe lume ssh macos-tahoe 'id -un' ``` ## SIP automation - `lume sip` supports stopped macOS VMs only. - The VM must have a working administrator account and Remote Login enabled. - The optional `vncdotool` package provides Recovery VNC input. - Recovery input accepts administrator passwords made from lowercase ASCII letters, digits, and hyphens. - The account named by `--admin-user` must match the Recovery password prompt. - Tahoe is the currently verified Recovery workflow. - The command leaves the VM stopped after a successful policy change. - If Recovery does not halt during its grace period, Lume stops that exact run process and requires a normal-boot `csrutil status` verification. - Customized SIP policies are rejected because they do not have one canonical enabled or disabled state. ## Disk allocation macOS VM disks use sparse files. The configured disk capacity describes the guest's logical disk. The host allocation grows as the guest writes data. ## Disk resizing - Disk resizing is increase-only and requires a stopped VM. - macOS resizing supports the standard Lume layout: ISC, main APFS, then paired RecoveryOS. - FileVault-encrypted macOS guests are unsupported. - The host must run the same or a newer macOS version than the guest. - The default rollback copy requires enough host capacity for the allocated RecoveryOS data and copy-on-write metadata. - The first macOS increase must leave enough room to relocate RecoveryOS without overlapping its old location. Lume reports the minimum accepted total size. - Linux resizing increases the image only. Grow the partition and filesystem in the guest. --- # Sandbox SDK reference Packages, configuration, and Sandbox lifecycle APIs for Python and TypeScript. ## Packages This reference describes the published `cua-sandbox` 0.4.3 Python package and `@trycua/fleet` 0.1.1 TypeScript package. | Distribution | Import | Requirement | API | | --- | --- | --- | --- | | [`cua-sandbox` 0.4.3](https://pypi.org/project/cua-sandbox/0.4.3/) | `cua_sandbox` | Python `>=3.11,<3.14` | `Sandbox`, `Image`, `Pool`, `Template`, and interfaces | | [`cua` 0.1.6](https://pypi.org/project/cua/0.1.6/) | `cua` | Python `>=3.12,<3.14` | Umbrella package; reexports `Sandbox` and `Image`, but not `Pool` | | [`@trycua/fleet` 0.1.1]() | `@trycua/fleet/node` or `@trycua/fleet/browser` | Environment-specific WebAssembly entry point | Fleet client, resource types, and builders | The Python Sandbox distribution can be installed independently: ```bash pip install cua-sandbox==0.4.3 ``` Its primary imports are: ```python from cua_sandbox import Image, Pool, Sandbox, Template ``` See [Pool and claim reference](), [Image reference](), [OS and image catalog](), [Sandbox interfaces](), and [TypeScript Fleet reference](). ## Configuration and authentication `configure()` updates process-wide settings. All parameters are optional, keyword-only strings; `None` leaves a setting unchanged. ```python def configure( *, api_key=None, base_url=None, fleet_base_url=None, token_url=None, client_id=None, client_secret=None, fleet_token=None, ) -> None: ... ``` Fleet authentication uses a configured `fleet_token`, then `FLEETS_TOKEN`. If neither supplies a nonempty token, it uses `client_id` and `client_secret` from `configure()`, falling back to `CUA_CLIENT_ID` and `CUA_CLIENT_SECRET`. | Environment variable | Purpose | Default | | --- | --- | --- | | `FLEETS_TOKEN` | Fleet bearer token; takes precedence over OAuth client credentials | None | | `CUA_CLIENT_ID`, `CUA_CLIENT_SECRET` | Fleet OAuth client credentials | None | | `CUA_TOKEN_URL` | OAuth token endpoint | `https://auth.cua.ai/realms/cyclops-cs/protocol/openid-connect/token` | | `CUA_FLEET_BASE_URL` | Fleet API endpoint | `https://run.cua.ai` | | `CUA_API_KEY` | Legacy VM API key | None | | `CUA_BASE_URL` | Legacy VM API endpoint | `https://api.cua.ai` | Endpoint environment variables override the corresponding `configure()` values. Legacy API keys resolve from the per-call `api_key`, global configuration, `~/.cua/credentials`, then `CUA_API_KEY`. Credential acquisition is documented in [Fleet pools need their own credentials](). `login(base_url=None)` and `whoami(api_key=None)` belong to the legacy API-key flow; they do not configure Fleet OAuth credentials. ## Sandbox creation `Sandbox.create()` returns a connected `Sandbox`. `Sandbox.ephemeral()` is an async context manager that performs cleanup on exit. The complete `create()` signature is: ```python async def create( image=None, *, pool=None, name=None, replicas=1, service="server", claim_spec=None, keep_alive_minutes=None, api_key=None, local=False, runtime=None, cpu=None, memory_mb=None, disk_gb=None, region="us-east-1", time_to_start=None, request_timeout=None, server_port=8000, telemetry_enabled=True, ) -> Sandbox: ... ``` `ephemeral()` accepts the same parameters plus `keep_pool=False`. | Parameter | Type | Contract | | --- | --- | --- | | `image` | `Image or None` | Required when `pool` is omitted; mutually exclusive with `pool` | | `pool` | `Pool, str, or None` | Existing Fleet pool or pool name | | `name` | `str or None` | Claim name with `pool`; pool name for Fleet image-based `ephemeral`; sandbox name for local/legacy creation | | `replicas` | `int` | Pool replicas for Fleet image-based `ephemeral`; default `1` | | `service` | `str` | Fleet service to connect to; default `"server"` | | `claim_spec` | `ClaimSpec or None` | Explicit Fleet claim specification | | `keep_alive_minutes` | `float or None` | Renews an acquired Fleet claim's shutdown deadline; positive minutes | | `local` | `bool` | Selects local provisioning; default `False` | | `runtime` | Runtime instance or `None` | Explicit local runtime adapter | | `cpu`, `memory_mb`, `disk_gb` | `int or None` | Resource overrides where the selected backend accepts them | | `time_to_start` | `float or None` | Service startup timeout in seconds | | `request_timeout` | `float or None` | Transport request timeout where supported | | `server_port` | `int` | Server port, `1`–`65535`; default `8000` | | `api_key`, `region` | `str or None`, `str` | Legacy cloud options; region defaults to `"us-east-1"` | | `telemetry_enabled` | `bool` | Creation telemetry option; default `True` | | `keep_pool` | `bool` | `ephemeral()` only: retain the image-created Fleet pool after releasing the claim; requires `name` | ### Fleet selection and constraints `pool=` explicitly selects Fleet. It rejects an image, resource overrides, `replicas != 1`, `local=True`, an explicit runtime or API key, a nondefault region, `request_timeout`, or a nondefault `server_port`. Configure the pool through `Pool.apply()` or `Pool.reconcile()` instead. For image-based calls, Fleet is selected when Fleet authentication is configured, no per-call `api_key` is supplied, the image resolves to a Fleet registry image, and neither `local=True` nor a runtime is supplied. Persistent Fleet creation requires an explicitly named pool: `Sandbox.create(image)` rejects this path. `Sandbox.ephemeral(image)` can create a disposable pool and claim. Fleet image-based provisioning rejects `disk_gb`, a nondefault region, and `request_timeout`. The presence of an image constructor does not establish backend support. See the [Fleet image constraints](). ## Lifecycle and ownership Fleet pool configuration and claim lifetime are separate resources. | Operation | Fleet effect | | --- | --- | | `await pool.claim()` or `await Sandbox.create(pool=pool)` | Acquires a claim; caller owns its release | | `async with pool.claim()` | Acquires a claim and calls `close()` on exit | | `async with Sandbox.ephemeral(pool=pool)` | Releases the claim on exit; retains the supplied pool | | `async with Sandbox.ephemeral(image)` | Releases the claim and deletes its created pool/template on exit, unless `keep_pool=True` | | `await sb.disconnect()` | Closes the connection; does not release the claim | | `await sb.close()` | Releases a Fleet claim and disconnects; repeated calls are safe | | `await pool.delete()` | Deletes the pool; also deletes the template owned by that `Pool.apply()` result | `destroy()` is the local/legacy transport cleanup method. It does not release a claim obtained through `Pool.claim()`; use `close()` for that claim. Local and legacy `ephemeral()` call `destroy()` on exit, or suspend when the instance has snapshots. Cleanup can fail; it is not a guarantee of deletion. ### Reconnection and renewal These methods apply to Fleet claim-backed instances: | Member | Signature or type | Contract | | --- | --- | --- | | `claim_name` | `str or None` | Claim identity, distinct from the bound sandbox's `name` | | `pool_name` | `str or None` | Pool owning the claim | | `to_dict` | `to_dict() -> dict[str, Any]` | Serializes claim identity and service, not credentials or a snapshot | | `from_dict` | `Sandbox.from_dict(data)` | Awaitable/context manager that reconnects; context exit disconnects without release | | `keep_alive` | `async keep_alive(*, minutes: float) -> None` | Moves the shutdown deadline to now plus positive minutes; caller must renew before expiry | | `close` | `async close() -> None` | Releases the claim and disconnects | The serialized shape is: ```python { "version": 1, "provider": "fleet", "namespace": "example-pool", "pool": "example-pool", "claim": "example-claim", "service": "server", } ``` `Sandbox.from_dict()` requires the namespace to match the pool. It still requires Fleet credentials and a claim that exists. Serialization, renewal, and `close()` raise `NotImplementedError` on instances without a Fleet claim handle. ## Other Sandbox methods `Sandbox.connect()` supports `await` and `async with`; context exit disconnects. For durable Fleet claim reconnection, use the serialized reference above. ```python def connect( name: str, *, api_key=None, local=False, ws_url=None, http_url=None, container_name=None, cpu=None, memory_mb=None, disk_gb=None, region="us-east-1", telemetry_enabled=True, ): ... ``` The lifecycle class methods accept keyword-only `local=False, api_key=None`: | Method | Return type | Purpose | | --- | --- | --- | | `await Sandbox.list(...)` | `list[SandboxInfo]` | Lists local sandboxes, legacy VMs, or Fleet pools according to backend selection | | `await Sandbox.get_info(name, ...)` | `SandboxInfo` | Gets metadata | | `await Sandbox.suspend(name, ...)` | `None` | Requests backend-specific suspension | | `await Sandbox.resume(name, ...)` | `Sandbox` | Requests resume and connects | | `await Sandbox.restart(name, ...)` | `Sandbox` | Requests restart and connects | | `await Sandbox.delete(name, ...)` | `None` | Requests backend-specific deletion | These name-based operations are backend-specific. For an explicitly acquired Fleet claim, use `close()` to release it and `Pool.delete()` to remove its pool. The connected instance also provides these methods: | Method | Return type | Purpose | | --- | --- | --- | | `await sb.screenshot(text=None, format="png", quality=95)` | `bytes` | Captures PNG or JPEG bytes; `text` is unused in this release | | `await sb.screenshot_base64(text=None, format="png", quality=95)` | `str` | Captures a base64 screenshot | | `await sb.get_environment()` | `str` | Reports the transport's environment | | `await sb.get_dimensions()` | `tuple[int, int]` | Returns width and height in pixels | | `await sb.get_display_url(share=False)` | `str` | Returns a display URL where the transport supports it; shared links may embed credentials | | `await sb.snapshot(name=None, stateful=False)` | `Image` | Backend-specific snapshot operation; not implemented for Fleet claim-backed instances | `SandboxInfo` has required string fields `name`, `status`, and `source`, plus optional string fields `os_type`, `host`, `vnc_url`, `api_url`, and `created_at`. ## Interfaces and Localhost `Sandbox` exposes `shell`, `files`, `mouse`, `keyboard`, `screen`, `clipboard`, `tunnel`, `terminal`, `window`, `mobile`, `services`, and `apps`. Availability depends on the connected transport. `exposed_ports` maps local guest ports to host ports; Fleet publishes named services instead. `Localhost.connect()` supports `await` and `async with` and controls the host through `cua_auto`, without a sandbox. Context exit disconnects. Its interfaces include `screen`, `mouse`, `keyboard`, `clipboard`, `shell`, `window`, and `terminal`. See [Sandbox interfaces](). --- # Pool and claim reference Python Fleet pool configuration, claim acquisition, cleanup, and resource types in cua-sandbox 0.4.3. `Pool` and `Template` are exported by `cua_sandbox` 0.4.3. These APIs use Fleet credentials independently of the legacy `CUA_API_KEY` flow. See [configuration and authentication](). ## Pool.apply Reconciles a named Fleet pool and its template from an `Image`: ```python async def apply( image: Image, *, name: str, replicas: int = 1, cpu: int | None = None, memory_mb: int | None = None, services: dict[str, int] | None = None, autoscaling: WarmPoolAutoscaling | None = None, ttl_seconds_after_created: int | None = None, ) -> Pool: ... ``` | Parameter | Contract | | --- | --- | | `image` | Built-in or explicit registry image accepted by the [Fleet image constraints]() | | `name` | Required nonempty pool name; pool names are globally unique across accounts | | `replicas` | Desired pool replica count; default `1` | | `cpu`, `memory_mb` | Optional template CPU and memory overrides | | `services` | Mapping from service name to target port. If omitted or empty, uses `server: 8000` and `port-N: N` for exposed image ports other than 8000 | | `autoscaling` | Optional `WarmPoolAutoscaling` resource | | `ttl_seconds_after_created` | Optional pool expiry in seconds from creation | The method reconciles the pool, then its template. If template reconciliation fails, it attempts to delete the reconciled pool. A returned `Pool` holds the template reference so that `delete()` can remove both resources. ## Pool lookup, reconciliation, and deletion These methods use the typed Fleet requests for explicit configuration: ```python async def get(name: str) -> Pool: ... async def reconcile(request: CreatePoolRequest) -> Pool: ... async def delete(self) -> None: ... ``` `Pool.get()` retrieves an existing pool without reconciling its configuration. `Pool.reconcile()` requires a `CreatePoolRequest` instance and creates or reconciles the pool described by it. Both return a wrapper with `name: str` and `resource`, the underlying Fleet pool resource. `delete()` deletes the pool. It also deletes the owned template when called on the result of `Pool.apply()`. A wrapper returned by `get()` or `reconcile()` does not record template ownership; manage that template separately. `PoolAccessDeniedError` reports Fleet pool access denial. Choose a pool name owned by the authenticated account; an inaccessible name is not an invitation to overwrite another account's pool. ## Pool.claim Returns an awaitable that also supports an async context manager: ```python def claim( self, *, spec: ClaimSpec | None = None, name: str | None = None, service: str = "server", time_to_start: float | None = None, ttl_seconds_after_created: int | None = None, ): ... ``` | Parameter | Contract | | --- | --- | | `spec` | Optional typed claim specification | | `name` | Optional claim identity. If a matching claim exists in the pool namespace, reconnects to it; otherwise creates it | | `service` | Named service to connect to; default `"server"` | | `time_to_start` | Optional service readiness timeout in seconds, after claim binding | | `ttl_seconds_after_created` | Optional claim expiry in seconds from creation; cannot be combined with an explicit `spec` | `await pool.claim()` waits for binding and service readiness, then returns a connected `Sandbox`. Its caller must call `close()` to release the claim. `async with pool.claim()` calls `close()` on exit, including when it reuses a named claim. If acquisition fails, the SDK attempts to release a claim created by that acquisition; it does not release a preexisting claim on that path. `disconnect()` only drops the connection. It does not release the claim. See [Sandbox lifecycle and ownership](). ## Pool.create_claim Creates a claim without waiting for binding or connecting a service: ```python async def create_claim( self, *, spec: ClaimSpec | None = None, name: str | None = None, ttl_seconds_after_created: int | None = None, ): ... ``` The returned handle has these members. Its concrete class is private; obtain it through `create_claim()` rather than importing the class. | Member | Contract | | --- | --- | | `namespace`, `name`, `pool_name`, `service` | Claim identity and selected service; default service is `"server"` | | `to_dict() -> dict[str, Any]` | Serializes identity for `Sandbox.from_dict()` | | `await wait(service=None, time_to_start=None) -> Sandbox` | Waits for binding and service readiness and connects | | `await renew(shutdown_time: str) -> None` | Updates the absolute shutdown timestamp | | `await release() -> None` | Deletes the claim; an already-missing claim is accepted | The caller owns cleanup after `create_claim()`, including when a later `wait()` fails. After connecting, `Sandbox.close()` releases the claim and disconnects. ## Expiry and renewal The `ttl_seconds_after_created` convenience parameters accept integer values from `0` through `4294967295`, excluding booleans. `None` omits the field. The schema describes the field as a creation-age TTL; accepting a value at the client does not establish the server's expiry policy. Pool and claim TTLs are configured separately. When supplying an explicit `ClaimSpec`, set the TTL in that spec instead of passing the convenience parameter. `Sandbox.keep_alive(minutes=...)` updates the claim's absolute shutdown time. It does not reset the resource's creation timestamp. See [Expire pools and claims]() for expiry configuration. ## Template and resource types `Template.reconcile(request: CreateTemplateRequest)` is async and returns a wrapper with `name` and `resource`. `TemplateResource` is the underlying Fleet template type, distinct from the `Template` wrapper. The following types and their named builders are reexported by `cua_sandbox`: | Type | Main fields or purpose | Builder | | --- | --- | --- | | `CreatePoolRequest` | `namespace`, `spec` | `CreatePoolRequestBuilder` | | `CreateTemplateRequest` | `namespace`, `name`, `spec` | `CreateTemplateRequestBuilder` | | `SandboxTemplateRef` | Template `name` | `SandboxTemplateRefBuilder` | | `OsGymSandboxWarmPoolSpec` | `replicas`, `sandbox_template_ref`, `autoscaling`, `ttl_seconds_after_created` | `OsGymSandboxWarmPoolSpecBuilder` | | `WarmPoolAutoscaling` | `min_pool_size`, `initial_pool_size`, `max_pool_size` | `WarmPoolAutoscalingBuilder` | | `OsGymSandboxTemplateSpec` | `vm_template` | `OsGymSandboxTemplateSpecBuilder` | | `VmTemplate` | Container disk image, runtime, resources, firmware, and services | `VmTemplateBuilder` | | `SandboxService` | `name`, `target_port`, optional `protocol` | `SandboxServiceBuilder` | `ClaimSpec`, `RuntimeKind`, `Firmware`, and `ServiceProtocol` are also reexported. A `ClaimSpec` constructor requires keyword arguments `sandbox_template_ref`, `warmpool`, `bind_deadline`, and `lifecycle`, even when the latter three are `None`; `ttl_seconds_after_created` is optional. Builders provide named setters and `build()` for the types listed above. These bindings come from the published `cua-fleet` 0.1.14 dependency, imported as `fleet_sdk`. Type availability describes the client schema; it does not establish runtime or image support on a deployment. --- # Sandbox interfaces reference Python Sandbox computer-control, file, service, tunnel, and application interfaces in cua-sandbox 0.4.3. These interfaces are exposed by `Sandbox` in the published `cua-sandbox` 0.4.3 package. A method's presence does not guarantee support by every transport. Fleet claims route computer-server operations through the selected named service; local and legacy transports have different capabilities. | Attribute | Class | Purpose | |-----------|-------|---------| | sb.shell | Shell | Run shell commands | | sb.files | Files | Read, write, and transfer files | | sb.mouse | Mouse | Mouse control | | sb.keyboard | Keyboard | Keyboard control | | sb.screen | Screen | Screenshots and screen info | | sb.clipboard | Clipboard | Clipboard read/write | | sb.tunnel | Tunnel | Port forwarding | | sb.terminal | Terminal | PTY terminal sessions | | sb.window | Window | Window management | | sb.mobile | Mobile | Mobile (Android) touch and hardware-key control | | sb.services | Services | Send requests to named services | | sb.apps | Apps | Install and launch catalog applications | ## Shell ```python async def run(command: str, timeout: int = 30, background: bool = False) -> CommandResult ``` ### CommandResult | Field | Type | Description | |-------|------|-------------| | stdout | str | Standard output | | stderr | str | Standard error | | returncode | int | Exit code | | success | bool (property) | True if returncode == 0 | ## Mouse ```python async def click(x: int, y: int, button: str = 'left') -> None async def right_click(x: int, y: int) -> None async def double_click(x: int, y: int) -> None async def move(x: int, y: int) -> None async def scroll(x: int, y: int, scroll_x: int = 0, scroll_y: int = 3) -> None async def mouse_down(x: int, y: int, button: str = 'left') -> None async def mouse_up(x: int, y: int, button: str = 'left') -> None async def drag(start_x: int, start_y: int, end_x: int, end_y: int, button: str = 'left') -> None ``` ## Keyboard ```python async def type(text: str) -> None # Type a string of text. async def keypress(keys: Union[List[str], str]) -> None # Press a key combination, e.g. ['ctrl', 'c'] or 'enter'. async def key_down(key: str) -> None async def key_up(key: str) -> None ``` ## Screen ```python async def screenshot(format: str = 'png', quality: int = 95) -> bytes # format: 'png' (lossless) or 'jpeg' (lossy). quality: 1-95, ignored for PNG. async def screenshot_base64(format: str = 'png', quality: int = 95) -> str # Screenshot as base64-encoded string. async def size() -> Tuple[int, int] # Returns (width, height) in pixels. ``` ## Clipboard ```python async def get() -> str # Returns current clipboard text. async def set(text: str) -> None # Sets clipboard text. ``` ## Tunnel ```python def forward(*ports) -> _TunnelContext ``` Requests forwarding for one or more sandbox ports or Android abstract sockets from the connected transport. Supports `await` and `async with`; context exit closes the forwards. In 0.4.3, the `FleetTransport` used by `Pool.claim()` does not implement `tunnel.forward()`. Use `sb.services.request()` for authenticated HTTP requests to a Fleet service. Local HTTP transports also do not implement this tunnel API; use `sb.exposed_ports` for ports forwarded at local startup. *ports* may be int (TCP port) or str (Android abstract socket name, e.g. 'chrome_devtools_remote'). Returns: - single TunnelInfo when one target is given - dict[sandbox_port, TunnelInfo] when multiple targets are given ### TunnelInfo | Field | Type | Description | |-------|------|-------------| | host | str | Host chosen by the transport | | port | int | Port chosen by the transport | | sandbox_port | int or str | Original port/socket inside the sandbox | | url | str (property) | Explicit transport URL, or `http://{host}:{port}` when none is supplied | ```python async def close() -> None # Close this tunnel. No-op if already closed or inside a context manager. ``` ## Services Sends an authenticated request to a named service on the connected sandbox: ```python async def request( name: str, *, method: str, path: str, json: Any = None, headers: dict[str, str] | None = None, ) -> Any: ... ``` For Fleet claims, `name` must be in the bound sandbox's service list. The return value is an `httpx.Response`. `method` is the HTTP method, `path` is the service request path, `json` is an optional JSON body, and `headers` adds request headers. Transports without named service support raise `NotImplementedError`. The released 0.4.3 `Services` interface has `request()` only. Signed service URL creation is available in the [TypeScript Fleet client](); `create_signed_url()`, `list_signed_urls()`, and `revoke_signed_url()` are not part of this Python release. ## Files Paths refer to the sandbox unless a parameter explicitly names a local path. These async methods use the connected computer-server file operations: | Method | Return type | Purpose | | --- | --- | --- | | `exists(path: str)` | `bool` | Tests whether a file exists | | `is_dir(path: str)` | `bool` | Tests whether a directory exists | | `size(path: str)` | `int` | Gets file size in bytes | | `list(path: str)` | `list[FileEntry]` | Lists directory entries | | `make_dir(path: str)` | `None` | Creates a directory | | `remove_dir(path: str)` | `None` | Removes a directory through the server | | `remove(path: str)` | `None` | Removes a file | | `read_text(path: str)` | `str` | Reads text | | `write_text(path: str, content: str)` | `None` | Writes text | | `read_bytes(path: str, offset=0, length=None)` | `bytes` | Reads bytes, with an optional byte range | | `write_bytes(path: str, content: bytes)` | `None` | Writes bytes | | `upload(local_path, remote_path: str)` | `None` | Copies a host file into the sandbox | | `download(remote_path: str, local_path)` | `None` | Copies a sandbox file to the host | Local paths accept `str` or `pathlib.Path`. `FileEntry` has `name: str`, `path: str`, `is_dir: bool`, and `size: int or None`. ## Apps `await sb.apps.install(app_id: str)` and `await sb.apps.launch(app_id: str)` return `CommandResult`. They use the application catalog and guest installer; an application ID does not establish availability on every guest OS. ## Terminal ```python async def create(command: Optional[str] = None, cols: int = 80, rows: int = 24) -> dict # Create a PTY session (default command is the login shell). Returns {'pid': int, 'cols': int, 'rows': int}. async def send_input(pid: int, data: str) -> None # Send input to a PTY session. async def info(pid: int) -> Optional[dict] # Return PTY session info, or None if the session is gone. async def close(pid: int) -> bool # Kill a PTY session. Returns True on success. ``` ## Window ```python async def get_active_title() -> str # Returns the title of the currently focused window. ``` ## Mobile Mobile (Android) touch and hardware-key control. Coordinates are in screen pixels. Single-touch methods use input tap/swipe via adb shell. Multi-touch gestures use adb root + MT Protocol B sendevent. ```python async def tap(x: int, y: int) -> None async def long_press(x: int, y: int, duration_ms: int = 1000) -> None async def double_tap(x: int, y: int, delay: float = 0.1) -> None async def type_text(text: str) -> None async def swipe(x1: int, y1: int, x2: int, y2: int, duration_ms: int = 300) -> None async def scroll_up(x: int, y: int, distance: int = 600, duration_ms: int = 400) -> None async def scroll_down(x: int, y: int, distance: int = 600, duration_ms: int = 400) -> None async def scroll_left(x: int, y: int, distance: int = 400, duration_ms: int = 300) -> None async def scroll_right(x: int, y: int, distance: int = 400, duration_ms: int = 300) -> None async def fling(x1: int, y1: int, x2: int, y2: int) -> None async def gesture(*finger_paths: tuple[int, int], duration_ms: int = 400, steps: int = 0) -> None # N-finger gesture via MT Protocol B sendevent. Each positional arg is (x,y) waypoints for one finger. # steps=0 = auto (duration_ms // 20, min 5). async def pinch_in(cx: int, cy: int, spread: int = 300, duration_ms: int = 400) -> None # Pinch-in (zoom out), two simultaneous fingers. async def pinch_out(cx: int, cy: int, spread: int = 300, duration_ms: int = 400) -> None # Pinch-out (zoom in), two simultaneous fingers. async def key(keycode: int) -> None async def home() -> None async def back() -> None async def recents() -> None async def power() -> None async def volume_up() -> None async def volume_down() -> None async def enter() -> None async def backspace() -> None async def wake() -> None async def notifications() -> None async def close_notifications() -> None ``` --- # Image reference API reference for the Image class: the immutable, chainable image specification used to configure sandbox environments. `Image` is an immutable, chainable image specification in `cua-sandbox` 0.4.3. Each builder method returns a new `Image` instance; the original is unchanged. Import it from `cua_sandbox`: ```python from cua_sandbox import Image ``` ## Fleet image constraints `Pool.apply()` accepts an explicit registry reference or a built-in descriptor that resolves to a registry image. Fleet rejects images with setup layers, environment variables, copied files, snapshot sources, or local disk paths. Exposed service ports are accepted. An OCI reference for Fleet must identify a compatible container disk; an arbitrary application container is not a VM disk. In 0.4.3, the built-in registry mappings are `Image.linux()` with Ubuntu 24.04 and `kind='vm'`, and `Image.windows()` with version `'2022'` and `kind='vm'`. Other descriptors require an explicit compatible registry image for Fleet. These mappings describe client image selection, not deployment certification. Exact artifact references and per-image evidence are listed in the [OS and image catalog](). The builder methods below describe image specifications. They do not establish that every runtime, operating system, or cloud deployment executes those specifications. Local runtime selection and legacy cloud provisioning are separate from Fleet. See [Pool reference]() for Fleet configuration and [Sandbox creation]() for backend selection. ## Constructors ### Image.linux ```python @classmethod def linux(cls, distro: str = 'ubuntu', version: str = '24.04', kind: str = 'vm') -> Image ``` Linux image specification. Defaults to `kind='vm'`; `kind='container'` describes a container. Local runtime selection depends on the host and installed runtime. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `distro` | `str` | `'ubuntu'` | Linux distribution | | `version` | `str` | `'24.04'` | Distribution version | | `kind` | `str` | `'vm'` | `'vm'` or `'container'` | ### Image.macos ```python @classmethod def macos(cls, version: str = '26', kind: str = 'vm') -> Image ``` macOS image specification. Version aliases include `'15'` / `'sequoia'` and `'26'` / `'tahoe'`. The constructor does not establish Fleet availability. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `version` | `str` | `'26'` | macOS version string or name | | `kind` | `str` | `'vm'` | VM specification; constructor does not validate runtime availability | ### Image.windows ```python @classmethod def windows(cls, version: str = '2022', kind: str = 'vm') -> Image ``` Windows image specification. The released default is Windows Server `2022`. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `version` | `str` | `'2022'` | Windows version | | `kind` | `str` | `'vm'` | VM specification; constructor does not validate runtime availability | ### Image.android ```python @classmethod def android(cls, version: str = '14', kind: str = 'vm') -> Image ``` Android image specification. This constructor does not establish Fleet support. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `version` | `str` | `'14'` | Android version | | `kind` | `str` | `'vm'` | VM specification; constructor does not validate runtime availability | ### Image.from_registry ```python @classmethod def from_registry(cls, ref: str, *, os_type: str = 'linux', kind: Optional[str] = None) -> Image ``` Image from an OCI registry reference. `os_type` identifies the guest operating system; it is not inferred from a tag. When omitted, `kind` remains unresolved until image resolution. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `ref` | `str` | required | OCI reference for an image appropriate to the selected backend | | `os_type` | `str` | `'linux'` | Guest OS; use `'windows'` for a Windows disk | | `kind` | `str or None` | `None` | Explicit `'vm'` or `'container'` kind | ### Image.from_file ```python @classmethod def from_file(cls, path: str, *, os_type: str = 'windows', kind: str = 'vm', agent_type: Optional[str] = None) -> Image ``` Local-runtime image from a disk file, ISO, or URL. Disk handling depends on the runtime. HTTP/HTTPS images are downloaded and cached under `~/.cua/cua-sandbox/image-cache/`. Fleet rejects `from_file()` specifications. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `path` | `str` | required | Local file path or `http`/`https` URL | | `os_type` | `str` | `'windows'` | `'linux'`, `'windows'`, `'macos'`, or `'android'` | | `kind` | `str` | `'vm'` | `'vm'` or `'container'` | | `agent_type` | `str or None` | `None` | e.g. `'osworld'` for OSWorld Flask server | ### Image.from_dict ```python @classmethod def from_dict(cls, data: Dict[str, Any]) -> Image ``` Reconstruct an `Image` from a serialized spec dict (e.g. the output of `to_dict()`). --- ## Builder methods Each method returns a new `Image`. These methods add setup instructions for a runtime that implements them. Fleet rejects setup layers; use an image with the required software already installed. ### Image.apt_install ```python def apt_install(self, *packages: str) -> Image ``` Install packages via `apt`. Linux only. ### Image.brew_install ```python def brew_install(self, *packages: str) -> Image ``` Install packages via Homebrew. macOS only. ### Image.choco_install ```python def choco_install(self, *packages: str) -> Image ``` Install packages via Chocolatey. Windows only. ### Image.winget_install ```python def winget_install(self, *packages: str) -> Image ``` Install packages via `winget`. Windows only. ### Image.apk_install ```python def apk_install(self, *apk_paths: str) -> Image ``` Install APK files via `adb`. Android only. ### Image.pwa_install ```python def pwa_install( self, manifest_url: str, package_name: Optional[str] = None, keystore: Optional[str] = None, keystore_alias: str = 'android', keystore_password: str = 'android', builder: str = 'pwa2apk', push_timeout: Optional[float] = None, ) -> Image ``` Build an APK from a PWA manifest URL and install it via `adb`. Android only. The default `builder='pwa2apk'` produces a lightweight WebView APK. With `builder='bubblewrap'` it builds a Trusted Web Activity (TWA), which additionally requires the keystore SHA-256 fingerprint to match the server's `/.well-known/assetlinks.json` so Chrome opens the PWA without browser chrome. | Parameter | Type | Default | Description | |-----------|------|---------|-------------| | `manifest_url` | `str` | required | Full URL to the PWA's `manifest.json` | | `package_name` | `str or None` | `None` | Android package ID; derived from hostname if omitted | | `keystore` | `str or None` | `None` | Path to `.keystore`/`.jks` file; auto-generated and cached if omitted | | `keystore_alias` | `str` | `'android'` | Key alias inside the keystore | | `keystore_password` | `str` | `'android'` | Password for both the store and the key | | `builder` | `str` | `'pwa2apk'` | `'pwa2apk'` (WebView APK) or `'bubblewrap'` (TWA) | | `push_timeout` | `float or None` | `None` | Timeout for the `adb install` push | Built APKs are cached by `(manifest_url, package_name)`. Requirements: Node.js >= 18, Java <= 21 on `PATH` (Gradle does not support Java 22+). ### Image.pip_install ```python def pip_install(self, *packages: str) -> Image ``` Install Python packages via `pip`. ### Image.uv_install ```python def uv_install(self, *packages: str) -> Image ``` Adds a `uv add` package-installation layer for the cua-server project. ### Image.run ```python def run(self, command: str) -> Image ``` Run an arbitrary shell command during image setup. ### Image.env ```python def env(self, **variables: str) -> Image ``` Set environment variables. Values are stored in the `Image` spec in plaintext. Do not use for secrets. ### Image.copy ```python def copy(self, src: str, dst: str) -> Image ``` Copy a local file into the image at the specified destination path. ### Image.expose ```python def expose(self, port: int) -> Image ``` Marks a port the sandbox will serve on. Fleet's default service mapping names it `port-N`, where `N` is the port, except for the server port. Local forwarding is reported by `sb.exposed_ports` where the runtime provides it. ### Image.app_install ```python def app_install(self, app_id: str) -> Image ``` Adds an application-installation layer by app catalog ID. Requires the runtime and application installer to support the selected app; Fleet rejects this layer. --- ## Serialization ### Image.to_dict ```python def to_dict(self) -> Dict[str, Any] ``` Serialize to a plain dict suitable for JSON or the cloud API. Example output: ```python { 'os_type': 'linux', 'distro': 'ubuntu', 'version': '24.04', 'kind': 'container', 'layers': [ {'type': 'apt_install', 'packages': ['curl']}, {'type': 'pip_install', 'packages': ['requests']}, ] } ``` ### Image.to_cloud_init ```python def to_cloud_init(self) -> str ``` Generate a cloud-init user-data script from the image layers. --- ## Attributes | Name | Type | Description | |------|------|-------------| | `os_type` | `str` | `'linux'`, `'macos'`, `'windows'`, or `'android'` | | `distro` | `str` | Distribution name (e.g. `'ubuntu'`) | | `version` | `str` | Version string (e.g. `'24.04'`) | | `kind` | `str or None` | `'container'` or `'vm'` | --- # OS and image catalog Sandbox operating systems, Linux distributions, desktop environments, window managers, and image registries. Windows images do not include a Windows license. Bring your own valid licenses covering your intended use, including the applicable hosting and virtualization rights. An image being available or booting successfully does not establish licensing eligibility. Bundled applications have their own license terms. | OS | Distribution / version / architecture | Desktop / window manager | Image kind / runtime | Registry image or source | Evidence / limits | | ------- | ----------------------------------------------------------------------- | ------------------------------------------------ | ------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Linux | Ubuntu 24.04; artifact architecture not established by SDK mapping | Artifact-dependent; not specified by SDK mapping | Fleet containerDisk; local bare-metal QEMU | `public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04:main-38352d34` | `Image.linux()` mapping in `cua-sandbox` 0.4.3; versioned tag, not a digest pin. [SDK checks]() establish selection, not a live pull, boot, or guest-service check. | | Linux | Ubuntu 24.04 | XFCE runtime mapping | Local Docker container | `public.ecr.aws/k5j5w0x5/cua-ubuntu-24.04:docker-latest` | Local container selector; mutable tag, not the Fleet VM disk | | Linux | Omarchy (Arch Linux-based), amd64 | Hyprland / Wayland | Fleet containerDisk | `public.ecr.aws/k5j5w0x5/cua-omarchy-workspace@sha256:d9b7be06beac425084eaa99eb912589b38b5cc86ae3e3ec45c9c5d59d4b3a7ab` | Digest-pinned explicit registry image. The [Fleet recipe]() reports boot verification and defines `server: 8000` (computer-server) and `mcp: 3000` (Cua Driver); it does not certify every operation or Hyprland configuration. | | Linux | NixOS; release and architecture not recorded in the publication summary | StumpWM | Published candidate image | `public.ecr.aws/k5j5w0x5/cua-nixos-stumpwm:git-ee87f38043c043831a8761016015d6ef5ae783d9` | [Publication summary](https://github.com/trycua/cua/pull/3585) reports computer-server and Cua Driver included, guest tests and CI anonymous-pull checks passed. Component versions, service ports, and Fleet claim checks are not recorded there. Commit-named candidate tag, not a digest pin or `latest`; no built-in SDK mapping. | | Linux | NixOS; release and architecture not recorded in the publication summary | Xfce / Xfwm4 | Published candidate image | `public.ecr.aws/k5j5w0x5/cua-nixos-xfce:git-ee87f38043c043831a8761016015d6ef5ae783d9` | [Publication summary](https://github.com/trycua/cua/pull/3585) reports computer-server and Cua Driver included, guest tests and CI anonymous-pull checks passed. Component versions, service ports, and Fleet claim checks are not recorded there. Commit-named candidate tag, not a digest pin or `latest`; no built-in SDK mapping. | | Windows | Server 2022; artifact architecture not established by SDK mapping | Native Windows desktop | Fleet containerDisk; local Hyper-V or QEMU | `public.ecr.aws/k5j5w0x5/cua-windows-2022:main-bac7daa3` | `Image.windows()` mapping in `cua-sandbox` 0.4.3; Fleet selects EFI. Versioned tag, not a digest pin. [SDK checks]() establish selection and firmware, not a live pull, boot, or guest-service check. | | macOS | Sequoia 15 | Native macOS desktop | Local Lume VM | `ghcr.io/trycua/macos-sequoia-cua:latest` | Compatible Apple silicon host required; mutable tag; no built-in Fleet mapping | | macOS | Tahoe 26 | Native macOS desktop | Local Lume VM | `ghcr.io/trycua/macos-tahoe-cua:latest` | Default `Image.macos()` version; compatible Apple silicon host required; mutable tag; no built-in Fleet mapping | --- # TypeScript Fleet reference Entry points, client methods, resource types, and lifecycle contracts for @trycua/fleet 0.1.1. This page describes the published `@trycua/fleet` 0.1.1 package ([npm version metadata](https://registry.npmjs.org/@trycua/fleet/0.1.1)). It provides Fleet resource operations through WebAssembly bindings. Its `Pool` and `Sandbox` exports are resource records, not the Python `Pool.apply()` and computer-control `Sandbox` classes. ## Package and entry points The package is installed with: ```bash npm install @trycua/fleet@0.1.1 ``` Choose the entry point for the execution environment: | Import | Contract | | --- | --- | | `@trycua/fleet/node` | Loads the bundled WebAssembly file from disk; exports `FetchHttpClient` | | `@trycua/fleet/browser` | Loads WebAssembly in a browser; uses browser-compatible client construction | | `@trycua/fleet` | Package root has conditional exports; explicit `/node` or `/browser` avoids depending on resolver condition ordering | The package does not declare a Node.js `engines` range. Its Node HTTP adapter uses global `fetch` and `AbortSignal.timeout`; import availability alone does not establish compatibility with every Node.js release or browser bundler. ## Initialization and authentication `uniffiInitAsync(): Promise` initializes the WebAssembly bindings. Await it before using generated builders or client constructors. Repeated calls reuse the initialization promise. The Node entry point provides this helper: ```ts declare function createFleetClient( configuration: CyclopsTokenProviderConfiguration, accessToken: string, ): Promise; ``` This helper initializes the bindings and constructs a client with a supplied bearer token. It does not read Python SDK environment variables or acquire a token. The browser entry point instead exposes `CyclopsClient.connectBrowserWithAccessToken(configuration, accessToken)` after `uniffiInitAsync()`. For credential acquisition, see [Fleet pools need their own credentials](). The token-based configuration has these fields: | Field | Type | Purpose | | --- | --- | --- | | `baseUrl` | `string` | Fleet API base URL | | `poolPollIntervalMs` | `bigint` | Pool polling interval in milliseconds | | `poolPollLimit` | `number` | Pool polling attempt limit | | `claimPollIntervalMs` | `bigint` | Claim polling interval in milliseconds | | `claimPollLimit` | `number` | Claim polling attempt limit | For explicit OAuth client credentials in Node, initialize the bindings and use `CyclopsClient.connect(configuration, new FetchHttpClient())`. Its `CyclopsConfiguration` adds `tokenUrl: string` and `credentials: CyclopsCredentials` to the polling and API fields above. `CyclopsCredentials` accepts the client ID and secret. Keep client secrets out of browser bundles; the browser entry point accepts a bearer token obtained by the application's authentication flow. ## Client methods These methods belong to `CyclopsClientLike`, the interface implemented by the client. Each async method accepts a final optional `{ signal: AbortSignal }` argument. Types below use the exported TypeScript names. | Method | Return type | Contract | | --- | --- | --- | | `createPool(request: CreatePoolRequest)` | `Promise` | Creates a pool | | `reconcilePool(request: CreatePoolRequest)` | `Promise` | Creates or reconciles a pool | | `getPool(name: string)` | `Promise` | Gets a named pool | | `listPools(namespace: string)` | `Promise` | Lists pools in a namespace | | `updatePool(pool: Pool)` | `Promise` | Updates a pool resource | | `deletePool(pool: Pool)` | `Promise` | Deletes a pool | | `createTemplate(request: CreateTemplateRequest)` | `Promise