Run Cua Driver with local models

Drive apps with a model on your own machine through Ollama or llama.cpp, and keep the tool list small enough for its context window.

Tested: Meta's Muse Glimmer 30B with Claude Code as the harness, served by Ollama or by llama.cpp with Unsloth's UD-Q4_K_XL GGUF. Every tool schema costs context, so hide unused tools with the filter:

curl -fsSL https://cua.ai/docs-assets/examples/local-models/cua-mcp-filter.py \
  -o "$HOME/.local/bin/cua-mcp-filter" && chmod +x "$HOME/.local/bin/cua-mcp-filter"

Save as muse-cua-mcp.json (the filter hides tools; it does not authorize):

{
  "mcpServers": {
    "cua-computer-use": {
      "command": "cua-mcp-filter",
      "args": [
        "--allow",
        "start_session,end_session,launch_app,list_apps,list_windows,get_window_state,get_accessibility_tree,move_cursor,click,type_text,press_key,hotkey,invoke_menu"
      ]
    }
  }
}

ollama pull muse-glimmer:30b-mlx
CLAUDE_CODE_MAX_CONTEXT_TOKENS=131072 \
ollama launch claude --model muse-glimmer:30b-mlx --yes -- \
  --bare --strict-mcp-config --mcp-config ./muse-cua-mcp.json --tools ""

Keep context small: bound get_window_state with max_elements and max_depth, reuse pid and window_id, and batch text into one type_text. In the tested Calculator runs this cut uncached input from 71,088 to 12,251 tokens. Use at least a 64K context window. To target a macOS VM, pass the guest's SSH MCP command (Run in a VM) to cua-mcp-filter --driver.

Text-only models#

For a model that cannot accept images, such as iFlytek Spark-X2.5 on the Astron Token Plan, add --text-only to the filter's args. The filter then requests get_window_state without a screenshot and replaces any remaining image content with a short text note, so the model works through the accessibility tree and its element_token values. Leave pixel-only tools such as screenshot and zoom out of the allowlist. Without --text-only, a provider that rejects images fails every request once a screenshot reaches the conversation. Apps that expose no usable accessibility elements (canvas, video, custom-drawn surfaces) need a vision model instead.