Computer use with Gemini

This guide shows how to build a Gemini agent that operates real desktop apps on your computer while you keep working. Cua Driver reads each app through its accessibility tree and screenshots, and sends clicks and keystrokes to that app in the background, so the agent works while you keep working.

We had early access to Gemini 4 Argon (gemini-4-argon) and recorded the demos below on the public model with Cua Driver 0.35.2.

You can connect Gemini to Cua Driver in two ways:

• Path A: Antigravity. Add Cua Driver as an MCP server and let the Antigravity agent use it. No code.
• Path B: your own agent loop. Use the google-genai SDK and the Cua Driver SDK to turn Cua Driver's tools into Gemini function declarations. About 150 lines of Python or TypeScript.

See it work#

In each recording, Gemini works in the Mac apps on the left while a person types a long message into a chat app on the right with a real keyboard. The bar at the bottom counts the keystrokes the chat app received against the ones that were typed. The middle of each clip is sped up.

Before you begin#

You need:

• A Mac with macOS 14 or later. For Windows and Linux, see Next steps.
• For Path A, Antigravity. Gemini 4 Argon in Antigravity is available to Google AI Ultra subscribers.
• For Path B, a Gemini API key from Google AI Studio on a paid tier (Gemini 4 Argon is not on the free tier), and Python 3.10 or later or Node.js 20 or later.

Install Cua Driver#

1. Install Cua Driver and start its background service:

/bin/bash -c "$(curl -fsSL https://cua.ai/driver/install.sh)"
open -n -g -a CuaDriver --args serve
cua-driver permissions grant

When macOS asks, choose Open System Settings and turn on CuaDriver under Accessibility and Screen & System Audio Recording.

2. Check the result:

cua-driver --version
cua-driver permissions status

The version should be 0.35.2 or later, and both Accessibility and Screen Recording should show as granted.

Path A: Antigravity with Cua Driver#

Antigravity runs agents on Gemini and supports local MCP servers. These steps use the Antigravity CLI, agy. The Antigravity desktop app and IDE read the same MCP configuration.

1. Install the Antigravity CLI:

curl -fsSL https://antigravity.google/cli/install.sh | bash

2. Add Cua Driver as an MCP server:

agy mcp add cua-driver "$(command -v cua-driver)" mcp
agy mcp list

NAME        TYPE   STATUS   COMMAND/URL
cua-driver  stdio  enabled  /Users/you/.local/bin/cua-driver mcp

This writes the server to ~/.gemini/config/mcp_config.json. To edit that file yourself instead, cua-driver mcp-config --client antigravity prints the entry to paste.

3. Let the agent use Cua Driver's tools without asking each time. Add an allow rule to ~/.gemini/antigravity-cli/settings.json:

{
  "permissions": {
    "allow": ["mcp(cua-driver/*)"]
  }
}

Without the rule, Antigravity asks you to approve each tool call.

4. Start agy, choose Gemini 4 Argon as the model, and give it a task. Name the tools you want it to use, and say that someone is using the computer:

Use only the cua-driver tools. In Contacts, open Daniel Reyes and add this note
to his card: "Met at the Design Systems Meetup booth." Save the card. Work in
the background; I'm using the computer.

To run a task without the interactive interface, use print mode:

agy -p "Use only the cua-driver tools. List the windows that are open right now."

In print mode, Antigravity can't ask you to approve a tool, so it denies any tool your allow list doesn't cover, such as shell commands.

Path B: build your own loop#

To control the prompt, the tool set, and the loop yourself, call the Gemini Interactions API and connect to Cua Driver from your own process.

1. Install the two SDKs and export your key:

pip install google-genai "cua-driver>=0.35.2"
export GEMINI_API_KEY="YOUR_API_KEY"

2. Write the agent. The loop has three parts:

• Tool declarations. list_tools_json() returns each Cua Driver tool with a JSON Schema, which Gemini accepts as a function declaration unchanged.
• Tool calls. call_tool(name, arguments_json) runs one tool and returns a summary, structured details such as window IDs and element tokens, and screenshots.
• Results. Each result goes back to Gemini as a function_result with text and image parts. Include the structured details: the window IDs and element tokens the model needs for its next call are there, not in the summary.

The script connects to the Cua Driver service you started with CuaDriver.connect(None). The service holds the macOS permissions, so the terminal you run the script from doesn't need them, and it draws the agent's own cursor on screen so you can follow what the agent does.

"""A Gemini agent that operates desktop apps in the background with Cua Driver.
 
    pip install google-genai "cua-driver>=0.35.2"
    export GEMINI_API_KEY=...
    python gemini_cua_agent.py "Add each person in signups.pdf (open in Preview) to Contacts"
"""
 
import argparse
import asyncio
import json
import os
import time
 
from cua_driver import CuaDriver
from google import genai
 
MODEL = os.environ.get("GEMINI_MODEL", "gemini-4-argon")
 
# A focused subset of Cua Driver's tools. launch_app and invoke_menu are left
# out because both can bring an app to the front. Batch tools are added when
# the installed Cua Driver offers them.
TOOLS = [
    "list_apps",
    "list_windows",
    "get_window_state",
    "click",
    "type_text",
    "set_value",
    "press_key",
    "hotkey",
    "scroll",
    "verify_state",
    "run_steps",
    "run_script",
]
 
SYSTEM_INSTRUCTION = """\
You operate macOS apps through Cua Driver tools while the user keeps working
on the same computer.
 
- Stay in the background. Don't bring windows to the front, and keep the
  default background delivery for every action.
- Find the apps you need with list_apps and list_windows.
- Before acting on a window, call get_window_state(pid, window_id) and use the
  element_token values from that snapshot. Take a fresh snapshot after any
  change; old tokens go stale.
- Prefer accessibility actions (element_token, set_value) over pixel
  coordinates.
- Check the result of each change in a fresh snapshot before moving on.
- When the task is done, reply with a short summary and call no more tools.
"""
 
TIMEOUT_S = 20 * 60  # One Gemini call can take several minutes.
 
 
def to_function(tool: dict) -> dict:
    """Map one Cua Driver tool to a Gemini function declaration."""
    return {
        "type": "function",
        "name": tool["name"],
        "description": tool["description"],
        "parameters": tool["inputSchema"],
    }
 
 
def to_result(call, result) -> dict:
    """Map a Cua Driver ToolResult to a Gemini function_result input."""
    content = [{"type": "text", "text": result.text or "ok"}]
    if result.structured_json:  # window IDs, element tokens and other details
        content.append({"type": "text", "text": result.structured_json})
    content += [
        {"type": "image", "mime_type": image.mime_type, "data": image.data_base64}
        for image in result.images
    ]
    return {
        "type": "function_result",
        "name": call.name,
        "call_id": call.id,
        "result": content,
        "is_error": result.is_error,
    }
 
 
async def run(task: str, max_turns: int, thinking_level: str) -> str:
    client = genai.Client()
    driver = CuaDriver.connect(None)  # the Cua Driver service started earlier
    session = f"gemini-{int(time.time())}"
 
    async def call_tool(name: str, args: dict):
        return await driver.call_tool(name, json.dumps(args))
 
    try:
        # Show this session's agent cursor while it works.
        await call_tool("start_session", {"session": session})
        await call_tool("set_agent_cursor_enabled", {"session": session, "enabled": True})
 
        catalog = json.loads(await driver.list_tools_json())["tools"]
        tools = [to_function(t) for t in catalog if t["name"] in TOOLS]
        takes_session = {
            t["name"] for t in catalog if "session" in t["inputSchema"].get("properties", {})
        }
 
        request = dict(
            model=MODEL,
            tools=tools,
            system_instruction=SYSTEM_INSTRUCTION,
            generation_config={"thinking_level": thinking_level, "max_output_tokens": 65536},
            timeout=TIMEOUT_S,
        )
        interaction = await client.aio.interactions.create(input=task, **request)
 
        for turn in range(1, max_turns + 1):
            calls = [s for s in interaction.steps or [] if s.type == "function_call"]
            if not calls:
                return interaction.output_text or ""
 
            results = []
            for call in calls:
                args = dict(call.arguments)
                if call.name in takes_session:
                    args["session"] = session
                started = time.monotonic()
                result = await call_tool(call.name, args)
                status = "error" if result.is_error else "ok"
                print(f"[{turn}] {call.name} {json.dumps(call.arguments)[:100]} -> {status} "
                      f"({time.monotonic() - started:.1f}s)")
                results.append(to_result(call, result))
 
            interaction = await client.aio.interactions.create(
                input=results, previous_interaction_id=interaction.id, **request
            )
        return f"Stopped after {max_turns} turns."
    finally:
        await call_tool("end_session", {"session": session})
        await driver.shutdown()
 
 
def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
    parser.add_argument("task")
    parser.add_argument("--max-turns", type=int, default=200)
    parser.add_argument("--thinking-level", default="low", choices=["low", "medium", "high"])
    args = parser.parse_args()
    print(asyncio.run(run(args.task, args.max_turns, args.thinking_level)))
 
 
if __name__ == "__main__":
    main()

3. Open the apps and run it. The script expects the apps it works in to be open already. For example, with a PDF of sign-ups open in Preview and Contacts open:

python gemini_cua_agent.py "signups.pdf is open in Preview. Add each person in \
its table to Contacts as a new contact with their name, company and email. \
Contacts is already open. Work in the background: I'm typing in another app, so \
don't bring any window to the front. When you're done, tell me what you added."

Keep working in another app while it runs. Each line the script prints is one Cua Driver call. These lines are from the Ledger run above:

[1] list_windows {"app": "Ledger"} -> ok (0.0s)
[2] get_window_state {"window_id": 842, "pid": 16196} -> ok (0.2s)
[3] run_steps {"window_id": 842, "observe": {"since": null, "include_screenshot": false}, "steps": [{"tool": "set_ -> ok (5.8s)
[4] run_steps {"window_id": 842, "steps": [{"tool": "set_value", "args": {"element_token": "s00000002:1", "value": -> ok (5.8s)

When Gemini answers without calling a tool, the script prints that answer and stops.

Settings for Gemini 4 Argon#

These follow Google's Gemini 4 Argon guidance. For Google's built-in computer use tool, see the computer use reference.

• Use thinking_level, not thinking_budget. Argon accepts low, medium, and high. Most UI steps are routine, so low keeps the loop responsive; raise it when a task needs real planning, such as reconciling two documents.
• Set max_output_tokens to 65,536. A low limit can stop the model before it finishes thinking. For outputs past 64K tokens, see Long Decode.
• Allow long HTTP timeouts. Google recommends at least 15 minutes per request; the examples use 20. Give the whole task its own wall-clock budget instead of a short per-call limit.
• Leave out sampling parameters. Argon ignores temperature, top_p, and top_k.
• Pass thought signatures back unchanged. With previous_interaction_id, the server keeps the conversation, including thoughts and their signatures. If you keep history yourself (store=False), resend every model step exactly as you received it, thought_signature included.

Tips#

• Watch context on long tasks. With previous_interaction_id, every snapshot stays in the conversation. A large window such as a Keynote document can return tens of thousands of tokens per snapshot, so a long task can reach the 1M-token input limit. For long tasks, keep history yourself, drop old snapshots and screenshots, and keep the model's steps unchanged.
• Keep observations lean. When the agent only needs fresh element tokens, it can call get_window_state with include_screenshot: false.
• Prefer accessibility actions. set_value and clicks by element_token work on background windows and don't depend on pixel coordinates.
• Batch steps. run_steps and run_script run several actions in one tool call, which cuts round trips to Gemini.
• Offer a focused tool set. Cua Driver has more than 50 tools. The examples offer about a dozen for desktop work. Add the browser_* tools when the task is in a web page.
• Keep risky actions supervised. Run unattended agents in a separate macOS user account or a virtual machine, and keep purchases, messages, deletions, and credentials out of unattended tasks.

Next steps#

• Windows and Linux. Cua Driver runs on Windows 10 and 11 and on x86_64 Linux desktops (X11 or XWayland, with AT-SPI 2), with the same MCP server and SDKs. See the Quickstart for the install steps, and change "macOS apps" in the system instruction to your platform.
• Shared computers for teams. Cua Spaces gives your team and its agents shared computers that everyone can watch, so an agent can work on a computer other than your own.
• Reference.
  • Use the SDK and the MCP tool reference
  • Antigravity MCP configuration
  • Gemini Interactions API, function calling, and computer use