Run a benchmark

Pick a taskset from the Cua Bench registry, run it with one cb run command on a desktop sandbox on your machine, and read the results.

A benchmark in Cua Bench is a registry taskset: a versioned set of desktop tasks, each with a setup, an oracle (a built-in solution) and an evaluator. Browse them at cua.ai/cuabench/registry and run one by name.

1. Pick a taskset#

cb dataset list

Every task runs on a full desktop sandbox. Tasks that open their own app (the cua-bench-basic widgets) start it as a pywebview window on that desktop; the agent sees the screen and drives mouse and keyboard, never a DOM. A taskset name resolves through the index bundled with cb to a pinned git commit, fetched once into ~/.cua/cbregistry.

cua-bench-kicad and cua-bench-workflows install their app with sudo apt during setup, which a local gVisor sandbox does not allow: run them locally with --runtime runc. The workflows have no oracle; check them with --noop or an agent.

cb run cua-bench-kicad --task-filter 154d0750 --runtime runc

2. Run it#

cb run cua-bench-basic --task-filter click-button --max-variants 1

With no agent, cb run runs each task's oracle: a quick check that the taskset works on your machine (reward=1). Then give it an agent:

cb run cua-bench-basic --agent cua-agent --model anthropic/claude-sonnet-4-20250514 -j 4

--on local (the default) runs sandboxes on this machine (Docker, gVisor when installed; check with cua runtime doctor).

Agents: cua-agent (any model the Cua agent supports) or your own with --agent-import-path module:Class. --model needs that provider's API key in the environment.

3. Read the results#

cb run list
cb run info <run-id>

Each run writes summary.json (mean reward, pass@k) and one directory per variant with result.json, run.log and the agent's trajectory.json (ATIF, with screenshots), under ~/.local/share/cua-bench/runs. Every key is in Results.

Try a task by hand#

cb interact opens one task's desktop, waits while you solve it, then scores it:

cb interact click-button --dataset cua-bench-basic

Your own registry#

CUA_BENCH_REGISTRY points cb at another index (a path or URL). Each entry pins a directory of tasks to a commit; image names the desktop image for tasks that set none:

{
  "datasets": [
    {
      "name": "my-tasks",
      "version": "1.0",
      "description": "Our internal desktop tasks",
      "git_url": "https://github.com/acme/bench-tasks.git",
      "git_commit_id": "0123456789abcdef0123456789abcdef01234567",
      "path": "tasks",
      "image": "BENCH_WEB"
    }
  ]
}

cb dataset list shows its entries; Harbor-format entries (a tasks list pinned per task) are fetched as-is. To write tasks for it, see Write a task.

Build a benchmark image#

The adapter benchmarks run on converted images: an upstream benchmark's environment, rebuilt as one Cua image with a container variant and a VM variant. The pipeline lives in scripts/bench-images/. Each benchmark is described once in libs/images/bench/<id>/bench.json (pinned sources, license, redistribution class, server and variants). The pipeline fetches the sources, builds both variants, smoke-tests them and publishes them.

python3 scripts/bench-images/bench.py validate
python3 scripts/bench-images/bench.py fetch osworld      # sha256-checked downloads
scripts/bench-images/build-qcow2.sh osworld              # rootfs + containerdisk variants
scripts/bench-images/publish.sh web                      # stage immutable pins, then --promote

--promote moves <ver> and <ver>-disk only after cua-spacesd doctor passes on every staged digest, architecture and variant.

On an arm64 host, amd64 images run under x86 emulation, where Mesa's llvmpipe draws a black screen. Server, file and spacesd checks still work, but doctor and screen evidence for amd64 need a native amd64 host.

Next#

• Adapter benchmarks: OSWorld-Verified, MiniWoB++, WebVoyager, Online-Mind2Web, WebGym and the grounding sets, through adapters.
• cb run reference