Cua Docs

Agent action policy

Behavior an agent should follow when choosing element, pixel, page, and foreground actions in Cua Driver.

Agent action policy

This policy describes how an agent should choose Cua Driver action parameters. It is meant for agent prompts, wrappers, and evaluators. It is not a user walkthrough.

Use it when the agent must decide what to pass to click, type_text, get_window_state, and related tools: element_index versus x, y, delivery_mode: background versus foreground, and window versus desktop scope.

Preconditions#

The agent should already be connected to the driver and able to launch an app and read a window. Perception is no longer a mode to pick: get_window_state returns both the accessibility tree and a screenshot in one call by default, and the agent chooses the rung at action time. For the concepts behind the axes, see Capture and delivery modalities.

Start on the accessibility path (the default)#

The agent should default to the element ax action with background delivery. Act by element_index. It is the only rung the driver can verify, it avoids foregrounding when the target surface supports it, and it works on Windows, macOS, and Linux (X11 and Wayland).

Read the window once, then act on an element from that snapshot. The snapshot already carries the screenshot too, so the agent does not re-capture to switch how it addresses the target:

// 1. snapshot: returns the accessibility tree AND a screenshot by default
get_window_state({ pid, window_id })
// → elements[], each with an element_index and a frame, plus a grounding screenshot
 
// 2. act by element_index: the element ax action, background by default
click({ pid, window_id, element_index: 12 })
type_text({ pid, text: "hello" })

When the agent is only re-indexing before an element ax action and does not need fresh pixels, pass include_screenshot: false to skip the screen grab and get the tree alone. The ax versus px decision still happens at action time, by how the agent addresses the target. To pin the rendered frame to disk instead of inlining it, set screenshot_out_file.

Follow the escalation ladder#

Every action response carries the signals that tell the agent the next rung. Walk the ladder in order, and only step down when the response says to:

  1. Element ax action, background (the default). Act by element_index. If the response shows effect: "confirmed", the driver read the result back. If get_window_state came back degraded (empty AX tree), an action returns effect: "suspected_noop" (the AX action ran but likely no-op'd), an action returns effect: "unverifiable" on an echo-prone surface, or the tree disagrees with the screenshot (an h:1 or off-viewport row), follow escalation.recommended.

  2. Element px action, background. When escalation.recommended is "px", pick the target pixel from the screenshot already in the get_window_state response and click it. Coordinates are window-relative for a windowed target.

    // the screenshot is already in the snapshot above; read a pixel off it
click({ pid, window_id, x: 320, y: 210 }) // → { path: "cgevent", effect: "unverifiable" }

    Use the same px form for keyboard fallback. If AX type_text, press_key, or hotkey returns effect: "unverifiable" on Electron/Chromium, retry with x, y: the tool pixel-clicks to focus, then sends the keys.

    type_text({ pid, window_id, element_index: 18, text: "hello" })
// → { effect: "unverifiable", escalation: { recommended: "px", reason: "..." } }
 
type_text({ pid, window_id, x: 320, y: 210, text: "hello" })
press_key({ pid, window_id, x: 320, y: 210, key: "return" })
hotkey({ pid, window_id, x: 320, y: 210, keys: ["cmd", "a"] })

    On Linux this still avoids synthetic input where it can: the driver resolves the pixel to the element under it and fires that element's action via AT-SPI doAction at that point. See Known limits for the remaining Wayland keyboard gap.

  3. Browser-tab DOM. When escalation.recommended is "page", switch to the page tool for browser-tab DOM work instead of retrying the AX write.

    type_text({ pid, window_id, element_index: 18, text: "hello" })
// → { effect: "unverifiable", escalation: { recommended: "page", reason: "..." } }
 
page({
  pid,
  window_id,
  action: "execute_javascript",
  javascript: "document.querySelector('#search').value = 'hello'"
})

  4. Foreground. If the response recommends "foreground" or the pixel click still does not land, retry with delivery_mode: "foreground". This activates the window first. Common cases include DirectInput games, raw-input canvases (Blender, Unity), and focus-polling apps.

    click({ pid, window_id, x: 320, y: 210, delivery_mode: "foreground" })
// → { path: "cgevent_fg", effect: "unverifiable" }

    Use foreground only for the action that needs it, and only when the user is not actively working on the machine. It raises the window. See Known limits for the specific apps.

The escalation signal on the response#

Two additive fields make the ladder explicit, so the agent escalates from data rather than a hunch:

  • effect: "confirmed" (the driver verified the result through AX read-back), "unverifiable" (the rung fired but only the caller can confirm), or "suspected_noop" (an AX action ran but almost certainly did nothing).
  • escalation: present when there is a next rung: { recommended: "px" | "foreground" | "page", reason }. A degraded get_window_state carries the same hint (recommending px).
click({ pid, window_id, element_index: 12 })
// → { effect: "suspected_noop", escalation: { recommended: "px", reason: "..." } }

Wayland exception. On a native Wayland session, raw keyboard input has no universal background path. When an AX keyboard action no-ops there, prefer an accessible field action or run the app under XWayland; otherwise escalate only that action to foreground. See Known limits.

Switch to desktop scope only for screen-absolute work#

Reach for desktop scope only when the action has no single window, such as dragging between windows or clicking absolute screen coordinates. It foregrounds and works on pixels, so it cannot provide best-effort background behavior.

set_config({ capture_scope: "desktop" })
get_desktop_state()                       // full-screen screenshot
click({ x: 1280, y: 40 })                 // screen-absolute, no window_id

A window-less click while scope is still window is rejected with desktop_scope_disabled. That error is the prompt to switch scope.

Confirm the action landed#

Only AX read-back can produce verified: true (the driver read the result back). Echo-prone AX surfaces, pixel actions, and foreground actions return verified: false or omit it; use effect and escalation to decide the next call. After an unverifiable action, re-read and check (the re-read returns both the tree and the screenshot):

click({ pid, window_id, x: 320, y: 210 })   // → { verified: false, path: "cgevent", effect: "unverifiable" }
get_window_state({ pid, window_id })         // confirm the change against tree + screenshot

Troubleshooting#

Problem: the call returned success but nothing changed (false success). Do not trust the status code on a verified: false action. Re-read the window. The snapshot carries both the tree and the screenshot. Confirm the effect; if it did not land, switch rung (element ax action -> element px action off the same screenshot) or escalate delivery (background -> foreground).

Problem: desktop_scope_disabled on a window-less click. The agent is in window scope. Either pass a window_id (+ element_index or x, y), or set_config({ capture_scope: "desktop" }) for genuinely screen-absolute work.

Problem: keystrokes don't land on a Linux app. On native Wayland, raw keys have no background path. Type into accessible fields with type_text, drive controls by element_index, or run the app under XWayland. See Known limits.

Related#