Verify a desktop action

Check a Python or TypeScript desktop action against an independent postcondition.

Use this guide to capture the desktop, enter a unique value in a local browser fixture, and prove that the page received it. A loopback state endpoint provides the proof independently from the action response.

Get the example#

Clone Cua and enter the executable example directory:

git clone https://github.com/trycua/cua.git
cd cua/libs/cua-driver/examples/agent-sdks

The fixture listens only on 127.0.0.1. Start it in one terminal:

python3 fixture_server.py

Your browser opens a page containing one autofocus input. Keep that window focused. On macOS, grant Screen Recording and Accessibility permission to the Python or Node process that imports Cua Driver.

Run the native SDK loop#

python3.12 -m venv .venv
.venv/bin/pip install -r requirements.txt
.venv/bin/python native_driver.py

The program performs four observable steps:

1. CuaDriver.create() loads the Rust runtime in the importing process.
2. get_desktop_state returns the initial whole-desktop image.
3. type_text and press_key submit a randomly generated value.
4. The program polls the fixture's /state endpoint and captures the desktop again only after the exact value appears.

Successful output resembles:

perceived desktop: image/png
verified submitted value: cua-2ca7d7cc81

See how uncertain actions are handled#

An input action can reach the OS even if the caller times out before receiving the response. The examples catch that condition but do not repeat the action. They check the independent fixture first. If the value is present, the postcondition resolves the unknown response; if it is absent, the run fails without silently replaying a mutation.

Shut down the fixture#

Press Ctrl+C in the fixture terminal. The SDK examples already end their session in finally and await runtime shutdown.

Next, connect the same driver to Claude Agent SDK or Codex SDK.