Control the desktop and the browser

Take screenshots, click and type, call typed Cua Driver tools, open a shell or a browser view, give an agent a browser, read Space thumbnails, and attach to any cua-spacesd.

Images that ship cua-spacesd (port 3211, service env), such as Image.linux(), expose the desktop. spacesd delegates pointer and keyboard input to Cua Driver in the guest. On an image without spacesd, sb.spacesd() and the interfaces below raise SpacesdNotAvailable.

To drive a sandbox from your own agent loop or bot, with a complete program in Python, TypeScript and Rust, see Use Cua in your own harness.

Screenshots and input#

The window manager can start after the sandbox is ready, so the example waits until the terminal really has focus before it types.

from pathlib import Path
from cua_sandbox import Image, Sandbox
 
FOCUS_TERMINAL = (
    "export DISPLAY=:1; setsid -f xfce4-terminal -T demo >/dev/null 2>&1; "
    "end=$(($(date +%s) + 20)); "
    "while [ $(date +%s) -lt $end ]; do "
    "w=$(xdotool search --onlyvisible --limit 1 --name ^demo$) && "
    "xprop -id $w WM_STATE | grep -q window.state && "
    "timeout 5 xdotool windowactivate --sync $w && exit 0; "
    "sleep 0.2; done; "
    "echo demo terminal never got keyboard focus >&2; exit 1"
)
 
async with Sandbox.ephemeral(Image.linux()) as sb:
    width, height = await sb.get_dimensions()
    await sb.mouse.click(width // 2, height // 2)
    # Keys go to the focused window, so open one.
    focus = await sb.shell.run(FOCUS_TERMINAL, timeout=40)
    assert focus.success, f"terminal focus failed ({focus.returncode}): {focus.stderr}"
    await sb.keyboard.type("hello")
    await sb.keyboard.keypress(["ctrl", "a"])
    Path("desk.png").write_bytes(await sb.screenshot())

Typed Cua Driver#

pip install 'cua-sandbox[driver]==0.11.0'

from cua_driver import GetScreenSizeInput
 
async with Sandbox.ephemeral(image) as sb:
    async with sb.driver.connect() as driver:
        result = await driver.get_screen_size(GetScreenSizeInput(session=None))
        print(result.text)

connect() goes through spacesd's /mcp route with the sandbox's own credentials; connect(service="mcp", transport="mcp") selects a named MCP service instead.

cua mcp is a stdio MCP server for agents:

cua mcp --sandbox desk

Shell#

A shell is a PTY served by spacesd; no SSH needed.

cua sb shell dev                    # interactive login shell
cua sb shell dev python3            # a program in a PTY
cua sb exec dev ls -la /tmp         # one command, no PTY

from cua_sandbox import Sandbox, Image
 
async with Sandbox.ephemeral(Image.linux()) as sb:
    session = await sb.terminal.create(cols=80, rows=24)   # command='python3' for a program
    pid = session['pid']
    await sb.terminal.send_input(pid, 'echo hello\n')
    print(await sb.terminal.info(pid))
    print('closed:', await sb.terminal.close(pid))

Browser view#

Images with cua-spacesd, such as Image.linux(), serve an HTML5 viewer at /viewer/ on the env service (port 3211). It streams H.264 video (WebCodecs, PNG fallback) and Opus audio, and sends keyboard, mouse and touch input through Cua Driver.

cua sb view desk                        # open the viewer
cua sb view desk --view-only --no-open  # print a watch-only link
cua sb view desk --files none --ttl 30m # no file access, link valid 30 minutes
cua sb create linux --name desk2 --view # open it when the sandbox is ready
cua viewer                              # one local page for every sandbox (Cua Spaces)

from cua_sandbox import Image, Sandbox
 
async with Sandbox.ephemeral(Image.linux()) as sb:
    print(await sb.viewer_url())

The cua SDK's sb.viewer_url(options) (Rust, Python, TypeScript, Swift, Kotlin) returns {url, expires_at_unix}; options are ttl_seconds, view_only, clipboard, files_root and microphone. The MCP tool is sandbox_view.

The link's fragment carries a viewer ticket that authorizes only the viewer's calls (never processes or tokens), expires (default 1 hour, at most 24) and is revoked when the root token rotates.

The toolbar has full screen, scaling, a keys menu, two-way clipboard, file upload (to ~/Downloads), folder sharing, microphone and stats.

Folder sharing picks a local folder and keeps it in sync with ~/Shared/<name> both ways. On a conflict both are kept (the sandbox one as name (sandbox copy YYYY-MM-DD HHMMSS).ext). Sharing stops when you close the page or click Stop.

Images without cua-spacesd that serve their own web display page open with cua sb view NAME --service <service>.

Give an agent a browser#

A coding agent can browse the web with the cua MCP server and nothing else. The browser runs inside a sandbox, never on your machine: in a local container (gVisor when installed) or a VM. The agent drives it through the sandbox's own Cua Driver, and the cua MCP server reaches Cua Driver's typed browser tools with call_tool.

Set up your agent#

cua connect setup installs the Cua skills (the cua-sandboxes skill carries the browsing recipe) and adds the cua MCP server to every agent it finds.

cua connect setup --agents claude,codex,cursor -y

Restart the agent. The rest of this section is what the agent does with the cua MCP tools.

Pick an image#

images_list returns the image catalog that the image catalog details page is generated from: each image's OS, container or VM form, runtimes, whether it ships cua-spacesd, and its browsers. browser_tools: true marks the images whose browser the Cua Driver browser tools drive.

cat = await agent.ok("images_list", {"browser": True})
images = [i for i in cat["images"] if i["browser_tools"]]

The same list from a terminal:

cua images ls --browser

Create a sandbox with a browser#

sandbox_create with browser: true starts the canonical Linux image (ghcr.io/trycua/linux:24.04) on this machine, launches Chromium with a fresh throwaway profile, and binds the window for the browser tools. With url it also opens a first page. It returns the sandbox id and the session, target_id and tab_id that every browser call takes.

sb = await agent.ok("sandbox_create", create)
space = sb["id"]
ids = {
    "session": sb["session"],
    "target_id": sb["browser"]["target_id"],
    "tab_id": sb["browser"]["tab_id"],
}

From a terminal, create the same sandbox and watch it with cua sb view:

cua sb create --browser --open https://example.com --name web

Read, click and type#

Every browser action is a call_tool on the sandbox with the ids above plus the tool's own arguments:

A small helper keeps the calls short:

async def driver(tool: str, **args) -> dict:
    return await agent.ok(
        "call_tool", {"space": space, "tool": tool, "arguments": {**ids, **args}}
    )
 
async def read() -> dict:
    return await driver("get_browser_state", snapshot_format="semantic_v2")
 
async def click(ref: str) -> None:
    # In a sandbox nobody else uses the browser window, so the
    # trusted route may activate it (Linux Chromium needs that).
    await driver("browser_click", ref=ref, delivery_mode="foreground")

Refs belong to the read that returned them and stay valid until the page changes. Read again after a navigation or a submit.

The sandbox is a whole computer, so you can also serve your app in it (space_write, space_bash) and browse http://127.0.0.1:<port>. This fills and submits a form:

await driver("browser_navigate", url=f"http://127.0.0.1:{FORM_PORT}/")
# One read; its refs stay valid until the page changes.
page = await read()
name_ref = find_ref(page, "textbox", "Name")
agree_ref = find_ref(page, "checkbox", "I agree")
submit_ref = find_ref(page, "button", "Submit")
await driver("browser_type", ref=name_ref, text="Ada Lovelace", replace=True)
await click(agree_ref)
await click(submit_ref)

Sign in with your own session#

To browse as you on a site where you are signed in, the agent can teleport that site's session from your browser (Firefox or Chrome) into the sandbox with teleport_browser_session. It names the exact sites. Nothing is selected by default, and there is no "all sessions" option. The first call moves nothing: it returns what would move and a consent requirement, which the agent shows you. Only your approval moves those sites' sessions into the sandbox. Delete the sandbox, or sign out there, to end them.

Clean up#

await agent.call("sandbox_delete", {"name": space})

What the browser tools cover#

Reading pages (outline, refs, text, a scoped query), navigation, clicks, typing and filling, Enter, hover, scroll, drag, uploads (browser_set_input_files), downloads (browser_download), page dialogs (browser_dialog) and tab screenshots. They drive Chromium-family browsers over the Chrome DevTools Protocol. Firefox ships in the image too: drive it with the desktop tools (computer_screenshot, computer_click). Waiting is a bounded loop of reads. There is no JavaScript eval, network interception or cookie editing tool.

Thumbnails#

For a small preview of a Space (a tile, a list row, a placeholder while a stream connects), use Space.thumbnail instead of screenshot. It returns a small JPEG of the primary display and when it was captured, from a cache that every client on the machine shares: the Spaces app, the CLI and your scripts.

A cached image younger than max_age_ms comes back at once. Otherwise the SDK captures a fresh one through cua-spacesd and keeps it for the next caller. If that capture fails, you get the older image, and captured_at_ms says how old it is.

import cua
 
space = await cua.connect().spaces().space("local:dev")
thumb = await space.thumbnail(max_age_ms=120_000)  # None: any age
print(thumb.width, thumb.height, thumb.captured_at_ms)

The cua daemon keeps the cache under ~/.cua/cache/thumbnails: the latest image per Space, dropped when the Space is deleted, and capped at 32 MB (the oldest images go first). While any client has asked for a thumbnail in the last 10 minutes, the daemon captures a new one for each running Space about every 90 seconds. When nobody asks, it captures nothing and puts no load on the guests.

Attach to or run spacesd#

To use any machine that runs cua-spacesd, or to run it yourself, see Attach to or run cua-spacesd.