Run OSWorld on Fleet

Turn the official OSWorld image into a Fleet containerDisk with Cua Driver inside, claim it from the Sandbox SDK, and run OSWorld tasks against it.

You start from the official OSWorld Ubuntu disk, add the Cua Driver MCP server to it, publish it as a Fleet containerDisk, and drive it with the Sandbox SDK. The same script runs the disk locally under QEMU.

Before you start#

• Python >=3.11,<3.14, uv, and Docker.
• About 60 GB of free disk.
• Fleet credentials exported as CUA_CLIENT_ID, CUA_CLIENT_SECRET, and CUA_TOKEN_URL.
• A checkout of the Cua repository, for samples/python/osworld-image/ and samples/python/3_osworld_fleet.py.

1. Get the OSWorld image#

mkdir -p ~/osworld-cua && cd ~/osworld-cua
curl -fSL --retry 5 -o Ubuntu.qcow2.zip \
  https://huggingface.co/datasets/xlangai/ubuntu_osworld/resolve/main/Ubuntu.qcow2.zip
unzip Ubuntu.qcow2.zip && mv Ubuntu.qcow2 base.qcow2

The disk ships Ubuntu 22.04, the auto-login user user (password password), the OSWorld control server on port 5000, and the benchmark applications. The recipe below does not change any of that.

2. Add Cua Driver to the image#

Download the Linux driver release and copy the osworld-image/ folder next to it:

V=0.26.1
curl -fsSL -o cua-driver-rs-$V-linux-x86_64-binary.tar.gz \
  https://github.com/trycua/cua/releases/download/cua-driver-rs-v$V/cua-driver-rs-$V-linux-x86_64-binary.tar.gz
cp -r /path/to/cua/samples/python/osworld-image .

The folder holds a systemd unit that runs cua-driver mcp behind supergateway on 0.0.0.0:3000/mcp, a netplan file that gives the guest a DHCP lease under KubeVirt, a udev rule that makes /dev/uinput writable (needed by the driver's isolated-input route), the virt-customize recipe, and the containerDisk Dockerfile.

Build the helper container and run the recipe. It works on an overlay, so base.qcow2 is never modified, and it needs no network inside the guest:

docker build -t osworld-guestfs:local osworld-image/guestfs
docker run --rm --device /dev/kvm -v "$PWD:/work" osworld-guestfs:local \
  bash /work/osworld-image/build-osworld-cua.sh

Drop --device /dev/kvm on hosts without KVM. The result is disk.img, a compressed qcow2.

3. Publish the containerDisk#

Fleet boots an OCI image whose only content is the guest disk at /disk/disk.img. Push it and keep the digest:

mkdir ctx && cp disk.img osworld-image/Dockerfile ctx/
REPO=<registry>/<repo>
docker build -t $REPO:osworld-cua-driver -f ctx/Dockerfile ctx
docker push $REPO:osworld-cua-driver
docker inspect --format '{{index .RepoDigests 0}}' $REPO:osworld-cua-driver

Fleet pulls public registries anonymously and private ECR only for allowlisted repositories; a repository outside the allowlist fails at Pool.apply with PoolAccessDenied ... k8s request is not allowed. See Prepare and reference a Fleet image.

4. Claim it#

3_osworld_fleet.py claims the image, takes a screenshot through the OSWorld server, and calls a few Cua Driver MCP tools:

export OSWORLD_IMAGE="<registry>/<repo>@sha256:<digest>"
uv run samples/python/3_osworld_fleet.py

The image is declared once; everything else is the normal Sandbox surface. agent_type="osworld" makes screenshot(), get_dimensions(), and shell.run() talk to the OSWorld server on 5000, and defaults the Fleet server service to that port. Expose the ports OSWorld's own tooling uses:

image = Image.from_registry(IMAGE_REF, os_type="linux", kind="vm", agent_type="osworld")
image = image.expose(5000).expose(3000).expose(8080).expose(9222)  # OSWorld, MCP, VLC, Chrome CDP
 
async with Sandbox.ephemeral(image, cpu=4, memory_mb=8192) as sb:
    png = await sb.screenshot()
    r = await sb.services.request("port-3000", method="POST", path="/mcp", json=..., headers=...)

Sandbox.ephemeral creates a pool and a claim, then deletes both on exit. For a loop over many tasks, create one pool with Pool.apply(image, name=..., replicas=N, services={...}) and claim from it per task instead; a fresh claim is how you get a clean desktop, since Fleet has no snapshot revert.

To connect an MCP client (for example the Claude Agent SDK) rather than sending single requests, create a signed URL for the port-3000 service (Share a service with a signed URL) and pass it as an HTTP MCP server.

Run an OSWorld task#

OSWorld's SetupController and evaluators only need an HTTP endpoint for the control server plus the two application ports, so point them at the claim:

from types import SimpleNamespace
from desktop_env.controllers.python import PythonController
from desktop_env.controllers.setup import SetupController
from desktop_env.evaluators import getters, metrics
 
host, port = "127.0.0.1", sb.exposed_ports[5000]      # local; on Fleet use a signed service URL
env = SimpleNamespace(
    vm_ip=host, server_port=port, chromium_port=sb.exposed_ports[9222], vlc_port=sb.exposed_ports[8080],
    cache_dir="cache/" + task["id"], vm_platform="Ubuntu", vm_machine="amd", current_use_proxy=False,
    getters=getters, action_history=[],
)
env.controller = PythonController(host, port)
env.setup_controller = SetupController(vm_ip=host, server_port=port, chromium_port=env.chromium_port,
                                       vlc_port=env.vlc_port, cache_dir=env.cache_dir, client_password="password")
env.setup_controller.setup(task["config"], False)   # downloads happen on your machine and are uploaded
... run your agent ...
score = metrics_func(getters_func(env, task["evaluator"]["result"]), ...)  # mirror DesktopEnv.evaluate

Wrap setup and evaluation in timeouts: the OSWorld controller has none, and a hung desktop otherwise hangs your loop. Agents that work from screenshots should downscale the 1920x1080 capture to about 1280 px and scale coordinates back, or the model's own image resizing will skew clicks.

Run it locally#

Needs qemu-system-x86_64 with KVM (on Windows, inside WSL2). Boot an overlay so disk.img stays clean:

qemu-img create -f qcow2 -F qcow2 -b "$PWD/disk.img" osworld-local.qcow2
OSWORLD_LOCAL=1 OSWORLD_QCOW2="$PWD/osworld-local.qcow2" uv run samples/python/3_osworld_fleet.py

Locally the only code difference is Image.from_file(QCOW2, os_type="linux", agent_type="osworld"); exposed guest ports appear in sb.exposed_ports. The local guest has no outbound network, so tasks that need internet inside the VM (Chrome, Thunderbird, anything that apt-gets during evaluation) only run on Fleet.

Iterate on the image without rebuilding#

Upload a new file through the OSWorld server and install it with sudo; the driver's MCP service restarts in seconds:

requests.post(f"http://{host}:{port}/setup/upload",
              data={"file_path": "/home/user/cua-driver.new"}, files={"file_data": open("cua-driver", "rb")})
await sb.shell.run("echo password | sudo -S bash -c 'install -m755 /home/user/cua-driver.new "
                   "/usr/local/bin/cua-driver && systemctl restart cua-driver-mcp'")

Troubleshooting#

• Claim never binds, pool stays at ready_replicas: 0. The probe is a TCP check on 5000. The image is still pulling, or the guest has no lease (the netplan file is missing). Poll list_pools/list_claims for the namespace; Fleet does not surface VM events. To separate a pull problem from a boot problem, push a known-good containerDisk to the same repository and claim it.
• Fleet cloud supports registry images with optional exposed services only. Builder layers are local-only; bake changes into the disk in step 2.
• Local boot times out. The local runtime waits 120 s; the QEMU process is left running, so kill it before retrying.
• Port 3000 answers 502. The MCP unit waits for the X session; read journalctl -u cua-driver-mcp through sb.shell.run().