cb CLI
cua-bench: run computer-use tasks and benchmark datasets on sandboxes.
cb <COMMAND> [OPTIONS]Run cb <command> --help for the build you have.
Commands#
| Command | Description |
|---|---|
cb run | Run a task or dataset, and manage runs. |
cb run task | Run one task variant. |
cb run dataset | Run every task and variant of a dataset in parallel. |
cb run list | List all runs with status. |
cb run info | Show detailed info about a run. |
cb run watch | Watch a run in real-time with live updates. |
cb run stop | Stop a run and release its sandboxes. |
cb run logs | View combined logs from a run or session. |
cb interact | Run a task's setup with its desktop visible, then evaluate. |
cb task | Inspect and manage task environments. |
cb task info | Show detailed information about a task. |
cb task list | List tasks in a directory. |
cb task create | Scaffold a new task environment. |
cb task generate | Generate a task from a prompt using Claude. |
cb dataset | Manage datasets and build from outputs. |
cb dataset list | List available datasets from registry. |
cb dataset build | Build a dataset from batch outputs. |
cb trace | View and analyze trace datasets. |
cb trace view | View a single trace in browser. |
cb trace traj | View agent trajectories from a run in cua.ai/trajectory-viewer. |
cb env | List local sandboxes. |
cb env ls | List local sandboxes. |
cb status | Show system status dashboard. |
cb prune | Clean up cua-bench run data (and 0.2.x image leftovers). |
cb agent | Scaffold custom agents. |
cb agent init | Create a new agent from template. |
cb agent build | Deprecated: cb run executes agents in-process. |
cb image | Show the canonical images (the local image store is deprecated). |
cb image list | List all images. |
cb image info | Show image details. |
cb image create | Deprecated: create an image from a platform. |
cb image delete | Deprecated: delete a stored image. |
cb image clone | Deprecated: clone a stored image. |
cb image shell | Deprecated: interactive shell into a stored image. |
cb platform | Show available platform configurations. |
cb platform list | List all available platforms. |
cb platform info | Show platform details. |
cb run#
Run a task or dataset, and manage runs.
Run a task (a directory with main.py) or a dataset, with an agent or the task's oracle. cb run <path> is shorthand for cb run task|dataset <path>.
cb run [COMMAND]Examples
# Run a task with its oracle solution
cb run ./tasks/hello_file_env
# Run a dataset with an agent, four variants at once
cb run cua-bench-basic --agent cua-agent --model anthropic/claude-sonnet-4-20250514 -j 4
# List runs
cb run listcb run task#
Run one task variant.
cb run task [OPTIONS] <task_path>| Argument | Type | Default | Description |
|---|---|---|---|
<task_path> | string | required | Path to task directory (containing main.py). |
| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
--variant-id | integer | 0 | Task variant index (default: 0). | |
--agent | string | Agent to use for evaluation (e.g., cua-agent). | ||
--agent-import-path | string | Import path for custom agent (e.g., "path.to.agent:MyCustomAgent"). | ||
--model | string | Model to use with the agent (e.g., anthropic/claude-sonnet-4-20250514). | ||
--max-steps | integer | 100 | Maximum number of steps for agent execution (default: 100). | |
--on | local | cloud | e2b | daytona | modal | cloudflare | vercel | morph | runloop | fly | blaxel | codesandbox | northflank | aws | gcp | Where sandboxes run: local (this machine), your own cloud account (aws, gcp, modal; connect it first with cua cloud connect <provider>), or a contrib provider (e2b, daytona; needs a cua SDK built with contrib and the provider's key). Default: CUA_DEFAULT_ON, else cua config set default.on, else local. | ||
--kind | auto | container | vm | What kind of sandbox: container or vm. Default auto: Linux as a container unless the task or image says VM; Windows, macOS and Android are VM-only. Default: CUA_DEFAULT_KIND, else cua config set default.kind. | ||
--runtime | auto | gvisor | runc | qemu | lume | Which engine: gvisor or runc (containers), qemu or lume (VMs). Implies its kind. Default auto: the SDK picks (CUA_DEFAULT_RUNTIME, else cua config set default.runtime). | ||
--image | string | Image for every task (overrides the task's setup_config.image): an OS alias (linux, windows, macos:tahoe), ghcr.io/trycua/<os>, any registry ref or a digest. Env: CUA_BENCH_IMAGE. | ||
--cpu | integer | vCPUs per sandbox. | ||
--memory | string | Memory per sandbox (e.g. 4096, 4G). | ||
--output-dir | string | Output directory for session results. | ||
--attempts | integer | 1 | Run every variant this many times (fresh sandbox each); summary.json reports pass@k (default: 1). | |
--retries | integer | 0 | Retry a variant that failed before evaluation (sandbox start, connection) up to this many times, with backoff (default: 0). | |
--keep-runs | integer | After the run, delete all but the N newest runs in the runs directory (default: keep everything; env CUA_BENCH_KEEP_RUNS). | ||
--max-age | number | After the run, delete runs older than DAYS (env CUA_BENCH_MAX_AGE_DAYS). | ||
--max-results-size | string | After the run, delete the oldest runs until the runs directory fits in SIZE, e.g. 20G (env CUA_BENCH_MAX_RESULTS_SIZE). | ||
--oracle | boolean | false | Run the oracle solution (the default when no agent is given). | |
--noop | boolean | false | Set up and evaluate with no actions (the null baseline for eval parity). | |
--detach | -d | boolean | false | Run in the background and return (follow with cb run watch <id>). |
--dry-run | boolean | false | Print each variant's image, kind, variant and backend, then exit without starting a sandbox. |
Examples
# Run variant 0 with the oracle solution
cb run task ./tasks/hello_file_env
# Run variant 2 with an agent on this machine
cb run task ./tasks/hello_file_env --variant-id 2 --agent cua-agent --model anthropic/claude-sonnet-4-20250514 --on local
# Show what would run without starting a sandbox
cb run ./tasks/hello_file_env --dry-runcb run dataset#
Run every task and variant of a dataset in parallel.
cb run dataset [OPTIONS] <dataset_path>| Argument | Type | Default | Description |
|---|---|---|---|
<dataset_path> | string | required | Path to dataset directory, or dataset name from registry. |
| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
--max-parallel | -j | integer | 4 | Variants running at once (default: 4). |
--max-variants | integer | Maximum number of variants to run per task (default: all). | ||
--task-filter | string | Filter tasks by name pattern (glob). | ||
--agent | string | Agent to use for evaluation (e.g., cua-agent). | ||
--agent-import-path | string | Import path for custom agent (e.g., "path.to.agent:MyCustomAgent"). | ||
--model | string | Model to use with the agent (e.g., anthropic/claude-sonnet-4-20250514). | ||
--max-steps | integer | 100 | Maximum number of steps for agent execution (default: 100). | |
--on | local | cloud | e2b | daytona | modal | cloudflare | vercel | morph | runloop | fly | blaxel | codesandbox | northflank | aws | gcp | Where sandboxes run: local (this machine), your own cloud account (aws, gcp, modal; connect it first with cua cloud connect <provider>), or a contrib provider (e2b, daytona; needs a cua SDK built with contrib and the provider's key). Default: CUA_DEFAULT_ON, else cua config set default.on, else local. | ||
--kind | auto | container | vm | What kind of sandbox: container or vm. Default auto: Linux as a container unless the task or image says VM; Windows, macOS and Android are VM-only. Default: CUA_DEFAULT_KIND, else cua config set default.kind. | ||
--runtime | auto | gvisor | runc | qemu | lume | Which engine: gvisor or runc (containers), qemu or lume (VMs). Implies its kind. Default auto: the SDK picks (CUA_DEFAULT_RUNTIME, else cua config set default.runtime). | ||
--image | string | Image for every task (overrides the task's setup_config.image): an OS alias (linux, windows, macos:tahoe), ghcr.io/trycua/<os>, any registry ref or a digest. Env: CUA_BENCH_IMAGE. | ||
--cpu | integer | vCPUs per sandbox. | ||
--memory | string | Memory per sandbox (e.g. 4096, 4G). | ||
--output-dir | string | Output directory for session results. | ||
--attempts | integer | 1 | Run every variant this many times (fresh sandbox each); summary.json reports pass@k (default: 1). | |
--retries | integer | 0 | Retry a variant that failed before evaluation (sandbox start, connection) up to this many times, with backoff (default: 0). | |
--keep-runs | integer | After the run, delete all but the N newest runs in the runs directory (default: keep everything; env CUA_BENCH_KEEP_RUNS). | ||
--max-age | number | After the run, delete runs older than DAYS (env CUA_BENCH_MAX_AGE_DAYS). | ||
--max-results-size | string | After the run, delete the oldest runs until the runs directory fits in SIZE, e.g. 20G (env CUA_BENCH_MAX_RESULTS_SIZE). | ||
--oracle | boolean | false | Run the oracle solution (the default when no agent is given). | |
--noop | boolean | false | Set up and evaluate with no actions (the null baseline for eval parity). | |
--detach | -d | boolean | false | Run in the background and return (follow with cb run watch <id>). |
--dry-run | boolean | false | Print each variant's image, kind, variant and backend, then exit without starting a sandbox. |
Examples
# Run a registry dataset with its oracle solutions
cb run dataset cua-bench-basic
# Run one variant of each matching task, eight at once, in the background
cb run ./tasks --task-filter 'click*' --max-variants 1 -j 8 --detach
# Run every variant three times and report pass@k
cb run dataset cua-bench-basic --agent cua-agent --attempts 3cb run list#
List all runs with status.
cb run list [OPTIONS]| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
--verbose | -v | boolean | false | Show verbose debugging information. |
Examples
# List runs
cb run list
# Include debugging details
cb run list --verbosecb run info#
Show detailed info about a run.
cb run info <run_id>| Argument | Type | Default | Description |
|---|---|---|---|
<run_id> | string | required | Run ID to show info for. |
Examples
# Show one run
cb run info 30c12572cb run watch#
Watch a run in real-time with live updates.
cb run watch <run_id>| Argument | Type | Default | Description |
|---|---|---|---|
<run_id> | string | required | Run ID to watch. |
Examples
# Follow a detached run
cb run watch 30c12572cb run stop#
Stop a run and release its sandboxes.
cb run stop <run_id>| Argument | Type | Default | Description |
|---|---|---|---|
<run_id> | string | required | Run ID to stop. |
Examples
# Stop a run
cb run stop 30c12572cb run logs#
View combined logs from a run or session.
cb run logs [OPTIONS] <identifier>| Argument | Type | Default | Description |
|---|---|---|---|
<identifier> | string | required | Run ID or Session ID to view logs for. |
| Flag | Type | Default | Description |
|---|---|---|---|
--tail | integer | Show only the last N lines. |
Examples
# Print a run's logs
cb run logs 30c12572
# Print the last 50 lines
cb run logs 30c12572 --tail 50cb interact#
Run a task's setup with its desktop visible, then evaluate.
cb interact [OPTIONS] <env_path>| Argument | Type | Default | Description |
|---|---|---|---|
<env_path> | string | required | Path to the task directory, or a task name with --dataset or --dataset-path. |
| Flag | Type | Default | Description |
|---|---|---|---|
--variant-id | integer | 0 | Task variant index (default: 0). |
--dataset | string | Registry dataset to resolve the task from (CUA_REGISTRY_HOME). | |
--dataset-path | string | Path to dataset directory containing multiple tasks. | |
--max-steps | integer | Maximum number of env.step() calls before stopping. | |
--screenshot | string | Save a screenshot to this file. | |
--trace-out | string | Record a trace and save it as a dataset to this path on exit. | |
--on | local | cloud | e2b | daytona | modal | cloudflare | vercel | morph | runloop | fly | blaxel | codesandbox | northflank | aws | gcp | Where sandboxes run: local (this machine), your own cloud account (aws, gcp, modal; connect it first with cua cloud connect <provider>), or a contrib provider (e2b, daytona; needs a cua SDK built with contrib and the provider's key). Default: CUA_DEFAULT_ON, else cua config set default.on, else local. | |
--kind | auto | container | vm | What kind of sandbox: container or vm. Default auto: Linux as a container unless the task or image says VM; Windows, macOS and Android are VM-only. Default: CUA_DEFAULT_KIND, else cua config set default.kind. | |
--runtime | auto | gvisor | runc | qemu | lume | Which engine: gvisor or runc (containers), qemu or lume (VMs). Implies its kind. Default auto: the SDK picks (CUA_DEFAULT_RUNTIME, else cua config set default.runtime). | |
--image | string | Image for every task (overrides the task's setup_config.image): an OS alias (linux, windows, macos:tahoe), ghcr.io/trycua/<os>, any registry ref or a digest. Env: CUA_BENCH_IMAGE. | |
--cpu | integer | vCPUs per sandbox. | |
--memory | string | Memory per sandbox (e.g. 4096, 4G). | |
--oracle | boolean | false | Run the solution after setup. |
--view | boolean | false | Open the trace viewer when done. |
--no-wait | boolean | false | Skip the interactive prompt (useful for SSH/CI testing). |
--no-browser | boolean | false | Print the Display: URL without opening it (also CUA_BENCH_NO_BROWSER=1). |
Examples
# Open a task and its display, wait for Enter, then evaluate
cb interact ./tasks/hello_file_env
# Run variant 1 with its oracle and save a screenshot
cb interact ./tasks/hello_file_env --variant-id 1 --oracle --screenshot shot.png
# Resolve a task from a registry dataset
cb interact click-button --dataset cua-bench-basiccb task#
Inspect and manage task environments.
cb task [COMMAND]Examples
# List the tasks in a directory
cb task list ./tasks
# Inspect one task
cb task info ./tasks/hello_file_envcb task info#
Show detailed information about a task.
cb task info <path>| Argument | Type | Default | Description |
|---|---|---|---|
<path> | string | required | Path to task directory (containing main.py). |
Examples
# Inspect a task's variants and lifecycle functions
cb task info ./tasks/hello_file_envcb task list#
List tasks in a directory.
cb task list [<path>]| Argument | Type | Default | Description |
|---|---|---|---|
<path> | string | . | Path to directory containing tasks (default: current directory). |
Examples
# List tasks here
cb task list
# List tasks in a directory
cb task list ./taskscb task create#
Scaffold a new task environment.
cb task create [<path>]| Argument | Type | Default | Description |
|---|---|---|---|
<path> | string | . | Target directory to create the task in (default: current directory). |
Examples
# Scaffold a task interactively under ./tasks
cb task create ./taskscb task generate#
Generate a task from a prompt using Claude.
cb task generate [OPTIONS] <prompt> [<output>]| Argument | Type | Default | Description |
|---|---|---|---|
<prompt> | string | required | Natural language description of the task to generate (e.g., "2048 game"). |
<output> | string | optional | Output directory (default: named after the prompt). |
| Flag | Type | Default | Description |
|---|---|---|---|
--no-interaction | boolean | false | Skip interactive prompts and run Claude non-interactively. |
Examples
# Generate a task
cb task generate '2048 game'
# Generate into a directory without prompts
cb task generate 'Sort a spreadsheet column' ./tasks/sort_column --no-interactioncb dataset#
Manage datasets and build from outputs.
cb dataset [COMMAND]Examples
# List registry datasets
cb dataset list
# Build a training dataset from run outputs
cb dataset build ./outputscb dataset list#
List available datasets from registry.
cb dataset listExamples
# List registry datasets
cb dataset listcb dataset build#
Build a dataset from batch outputs.
cb dataset build [OPTIONS] <outputs_path> [<max_samples>]| Argument | Type | Default | Description |
|---|---|---|---|
<outputs_path> | string | required | Path to outputs folder from batch dump (e.g., ./outputs or /tmp/td_output). |
<max_samples> | integer | optional | Limit number of samples processed (useful for testing). |
| Flag | Type | Default | Description |
|---|---|---|---|
--mode | string | aguvis-stage-1 | Processing mode: aguvis-stage-1 (action augmentation) or gui-r1 (low-level click). |
--dataset-name | string | Dataset name when saving to disk (default: td_<mode>_dataset). | |
--save-dir | string | Directory to save a JSONL dataset (default: <outputs>/processed if not pushing). | |
--repo-id | string | HF Hub repository ID (e.g., username/repo). Required with --push-to-hub. | |
--push-to-hub | boolean | false | Push the dataset to the Hugging Face Hub. |
--private | boolean | false | When pushing to hub, create/update the repo as private. |
Examples
# Build an aguvis-stage-1 dataset from run outputs
cb dataset build ./outputs
# Build 100 gui-r1 samples
cb dataset build ./outputs 100 --mode gui-r1
# Push a private dataset to the Hugging Face Hub
cb dataset build ./outputs --push-to-hub --repo-id me/cua-clicks --privatecb trace#
View and analyze trace datasets.
cb trace [COMMAND]Examples
# Open a run's trace in the browser
cb trace view 30c12572
# Open a run's agent trajectories
cb trace traj 30c12572cb trace view#
View a single trace in browser.
cb trace view <identifier>| Argument | Type | Default | Description |
|---|---|---|---|
<identifier> | string | required | Run ID or session ID (e.g., "30c12572" or "task-30c12572-click-button-v0"). |
Examples
# Open a run's trace
cb trace view 30c12572
# Open one session's trace
cb trace view task-30c12572-click-button-v0cb trace traj#
View agent trajectories from a run in cua.ai/trajectory-viewer.
cb trace traj [OPTIONS] <identifier>| Argument | Type | Default | Description |
|---|---|---|---|
<identifier> | string | required | Run ID (e.g., "30c12572"). |
| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
--port | -p | integer | 8090 | Local file server port (default: 8090). |
Examples
# Open a run's trajectories
cb trace traj 30c12572
# Serve the files on another port
cb trace traj 30c12572 --port 9000Sandboxes and account#
Local sandboxes, status and cleanup.
cb env#
List local sandboxes.
cb env [COMMAND]Examples
# List local sandboxes
cb env lscb env ls#
List local sandboxes.
cb env ls [OPTIONS]Alias: cb env list.
| Flag | Type | Default | Description |
|---|---|---|---|
--local | boolean | false | List local sandboxes (the default; kept as a no-op). |
Examples
# List local sandboxes
cb env lscb status#
Show system status dashboard.
cb statusExamples
# Show runs, sandboxes and credentials at a glance
cb statuscb prune#
Clean up cua-bench run data (and 0.2.x image leftovers).
cb prune [OPTIONS]| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
--keep | integer | With --runs: keep the N newest runs and remove only older ones. | ||
--older-than | number | With --runs: remove only runs older than DAYS. | ||
--all | -a | boolean | false | Remove runs and 0.2.x image leftovers. |
--images | boolean | false | Remove 0.2.x stored images (deprecated store). | |
--overlays | boolean | false | Remove 0.2.x task overlays (deprecated store). | |
--runs | boolean | false | Remove run logs and registry. | |
--docker | boolean | false | Deprecated: cua-bench no longer creates Docker resources. | |
--dry-run | boolean | false | Show what would be deleted without deleting. | |
--force | -f | boolean | false | Skip confirmation prompts. |
Examples
# Preview what would be removed
cb prune --all --dry-run
# Remove run logs without asking
cb prune --runs --force
# Keep the 20 newest runs, delete the rest
cb prune --runs --keep 20
# Delete runs older than 30 days
cb prune --runs --older-than 30cb agent#
Scaffold custom agents.
Create and manage custom agents for cua-bench.
cb agent [COMMAND]Examples
# Scaffold an agent in ./my-agent
cb agent init my-agentcb agent init#
Create a new agent from template.
Scaffold a new custom agent with working sample code.
cb agent init [OPTIONS] <name>| Argument | Type | Default | Description |
|---|---|---|---|
<name> | string | required | Agent name (e.g., 'my-agent'). |
| Flag | Short | Type | Default | Description |
|---|---|---|---|---|
--output-dir | -o | string | Output directory (default: ./<name>). |
Examples
# Scaffold an agent in ./my-agent
cb agent init my-agent
# Scaffold it into another directory
cb agent init my-agent --output-dir ./agents/my-agentcb agent build#
Deprecated: cb run executes agents in-process.
Build a Docker image for an agent directory containing a Dockerfile.
cb agent build [OPTIONS] <path>Examples
# Build an agent image (deprecated)
cb agent build ./my-agent --tag my-agent:devImages and platforms#
The canonical images and platform configurations.
cb image#
Show the canonical images (the local image store is deprecated).
cb image [COMMAND]Examples
# List the canonical images
cb image
# Show one image
cb image info linuxcb image list#
List all images.
cb image list [OPTIONS]| Flag | Type | Default | Description |
|---|---|---|---|
--platform | string | Filter by platform. | |
--format | table | json | table | Output format. |
Examples
# List images
cb image list
# List one platform's images as JSON
cb image list --platform linux-docker --format jsoncb image info#
Show image details.
cb image info <name>| Argument | Type | Default | Description |
|---|---|---|---|
<name> | string | required | Image name. |
Examples
# Show one image
cb image info linuxcb image create#
Deprecated: create an image from a platform.
cb image create [OPTIONS] <platform>Examples
# Create a Linux image (deprecated)
cb image create linux-docker --name my-linuxcb image delete#
Deprecated: delete a stored image.
cb image delete [OPTIONS] <name>Examples
# Delete a stored image (deprecated)
cb image delete my-linux --forcecb image clone#
Deprecated: clone a stored image.
cb image clone [OPTIONS] <source> <target>Examples
# Clone a stored image (deprecated)
cb image clone my-linux my-linux-2cb image shell#
Deprecated: interactive shell into a stored image.
cb image shell [OPTIONS] <name>Examples
# Open a shell in a stored image (deprecated)
cb image shell my-linuxcb platform#
Show available platform configurations.
cb platform [COMMAND]Examples
# List platforms
cb platform
# Show one platform
cb platform info linux-dockercb platform list#
List all available platforms.
cb platform list [OPTIONS]| Flag | Type | Default | Description |
|---|---|---|---|
--format | table | json | table | Output format. |
Examples
# List platforms
cb platform list
# As JSON
cb platform list --format jsoncb platform info#
Show platform details.
cb platform info <platform>| Argument | Type | Default | Description |
|---|---|---|---|
<platform> | string | required | Platform name (e.g., linux-docker, windows-qemu). |
Examples
# Show one platform
cb platform info windows-qemuGlobal options#
--help prints help for any command; --version prints the version.
Exit codes#
| Code | Meaning |
|---|---|
0 | Success: every variant completed. |
1 | Failure: a variant failed or was cancelled, or the command could not run. |
2 | Usage error: an unknown command, flag or value. |
130 | Interrupted (Ctrl-C); running variants are cancelled. |