Known Limits

Documented behavioral limits of Cua Driver and available alternatives

Cua Driver attempts background delivery only when the operating system and target expose a route that can be addressed safely. Unsupported shapes return a structured refusal instead of reporting a silent success. This page lists target-specific constraints and available alternatives; see Platform Support for the broader operating-system matrix.

Browser tools target exactly bound Chromium pages#

The typed browser mutation tools currently target Chromium-family browsers and Electron through an owned Chrome DevTools Protocol endpoint. Safari and Firefox may still be inspected through existing accessibility or legacy page routes, but they do not receive typed browser mutation capabilities.

Firefox is classified consistently as a browser on macOS, Windows, and Linux, but Cua Driver does not currently advertise a WebDriver BiDi route. Safari Apple Events JavaScript is not equivalent to trusted browser input, and mutable tab ordinals cannot satisfy exact window targeting. These surfaces therefore remain capability-visible but mutation-unavailable instead of being guessed.

The typed browser surface is preview-grade before 1.0. Product support means that the exact binding, supported mutations, declared background posture, and structured refusals have release evidence. Recognition alone is not a support claim.

Page refs traverse open shadow roots and same-process frames and are capped to the first 300 interactive elements. Out-of-process frames are included only when the runtime exposes a capability-tested CDP session for that frame; otherwise the omitted frame is reported as a limitation instead of being flattened into the main document. A newer snapshot or navigation invalidates previous refs.

browser_prepare can either create a separate driver-owned isolated profile or attach to an approved existing Chrome, Edge, or Chromium profile on a proven platform. Existing-profile setup may enable that browser instance's own remote-debugging switch through one exact accessibility action. When current macOS Chrome withholds the internal page's web accessibility tree, Cua Driver instead creates and navigates a temporary tab, proves the committed fixed address and expected selected-tab title with no active omnibox edit, and requires one unique checkbox-shaped control in a bounded setup-page region. It revalidates the unchanged target window, PID-routes the click, and verifies the visual state transition. Unsupported appearance, scale, or zoom geometry is refused without a click. Unsupported window sizes or toolbar layouts, including a bookmarks bar that moves the control outside the bounded region, are refused the same way. It never copies profile data, edits profile files, restarts, or terminates the selected process. Endpoint ownership must resolve to the approved browser PID; wrapper processes, ambiguous process trees, unsupported products, unrecognized UI locales, and generic Wayland identities are refused.

On generic GNOME or KDE Wayland sessions, browser state may be discoverable without enough compositor evidence to correlate native and DevTools geometry exactly. That route stays read-only. X11 and the validated Sway configuration can authorize mutation when their ownership and geometry proofs agree. This does not imply arbitrary raw background PX delivery on Wayland.

Chromium coerces synthetic right-clicks on web content#

Symptom: right_click({pid, x, y}) on a Chrome, Edge, Brave, or Arc tab's web content fires a left-click instead of opening the context menu.

Cause: Chromium's renderer-IPC filter drops the right-click subtype bit on events that don't come through the HID tap. Every synthesized-event path on macOS hits this wall.

Workarounds, in order of preference:

1. Use right_click({pid, element_index}) on AX-addressable targets (links, buttons, toolbar items). AX delivery sidesteps the renderer filter entirely.
2. For context menus on pure web content (nothing in the AX tree), activate Chrome briefly and fall back to a HID-tap right-click. This interrupts best-effort background behavior for that one click.

Canvas apps need brief frontmost activation#

Affected: Blender (GHOST event source), Unity editor / Unity games, most native games, some WebGL-heavy Electron apps.

Symptom: click({pid, x, y}) on a Blender viewport silently no-ops. The window is visible and launch_app works, but clicks vanish.

Cause: These apps only accept events from cghidEventTap with a leading mouseMoved. They explicitly filter out per-pid-routed events, which is the path Cua Driver uses for background delivery. There is no per-pid recipe that reaches them.

Workaround: Bring the app to the foreground before clicking, then use pixel click({pid, x, y}). Where the target exposes AX-addressable controls, prefer right_click or element actions, which sidestep the renderer filter without foregrounding.

Off-Space SwiftUI windows strip their AX tree#

Symptom: get_window_state({pid, window_id}) on a window on a different Space (e.g. System Settings parked on Space 2 while on Space 1) returns a minimal tree that contains only the menu bar, or just the AXApplication root.

Cause: macOS 14+ strips AX detail from non-current-Space SwiftUI windows as a privacy/performance tradeoff. AppKit apps are not affected. There is no workaround that keeps the window off-Space.

Response shape: Every get_window_state response on an off-current-Space window carries off_space: true, so callers can decide to switch Space, pick a different window, or skip the turn.

Workarounds:

1. Switch the user to the target's Space first. This breaks the no-Space-bounce promise.
2. Target an AppKit equivalent of the app if one exists.
3. Limit off-Space automation to AppKit apps where the tree stays populated.

Minimized windows silently drop keyboard commits#

Symptom: press_key({pid, element_index, key: "return"}) on a text field in a minimized window returns success, but the field doesn't commit. The macOS system-alert beep fires, or nothing happens.

Cause: AX reads and AX clicks propagate through to minimized windows normally, but keyboard-commit events (Return, Space, Tab) require renderer focus, which AX focus does not confer on a minimized window. This is a macOS-wide behavior.

Workarounds:

1. Use set_value({pid, element_index, value: "..."}) to write the field's value directly. No keyboard event involved; no focus handoff required.
2. AX-click a commit-equivalent button (Go, Submit, Send, OK) rather than relying on Return.
3. Restore the window through the Dock or an application-specific Window menu command. This interrupts best-effort background behavior for that window. (Cmd+M minimizes a window; it does not restore one.)

Native Wayland background keyboard input is focus-bound#

Affected: Unfocused GTK/Qt apps running as native clients on standard Wayland compositors, including Sway/wlroots, GNOME/Mutter, and KDE/KWin (no X11 surface).

Symptom: A background press_key, hotkey, or type_text request for a field that is not AT-SPI-editable returns structured background_unavailable. Foreground delivery may also refuse when the desktop has no target-addressable activation or raw-input backend.

Cause: Wayland blocks one ordinary client from targeting another client's surface with synthetic input. Portal/libei input on GNOME and KDE follows the compositor's active seat and requires a RemoteDesktop grant. The virtual keyboard used on wlroots compositors is also focus-bound, even though it does not require the same portal route.

Workarounds:

1. Type into accessible text fields with type_text. AT-SPI insertText writes the field directly, with no synthetic key event involved.
2. Drive controls by element_index (click, set_value) instead of keyboard shortcuts where an equivalent control exists.
3. Retry with delivery_mode:"foreground" when the desktop exposes a verified activation and input adapter.
4. Run the app under XWayland (GDK_BACKEND=x11 / QT_QPA_PLATFORM=xcb). It then exposes an X11 surface and the X11 keyboard paths apply.

The opt-in nested cua-compositor is a separate, compositor-owned environment. Its private per-surface injection protocol is not constrained like an ordinary client on Sway, GNOME, or KDE, and its capabilities are documented separately.

GTK4 reports (0,0) screen coordinates over AT-SPI (handled)#

Symptom: none in normal use. Element frames, the agent cursor, and vision clicks are correct on GTK4. Documented here because the underlying toolkit bug is real and visible in raw AT-SPI.

Cause: GTK4's AT-SPI bridge returns Component.GetExtents(SCREEN) as (0,0) for every widget (GNOME/gtk issues #1564 / #1739). A naive consumer would collapse every element to the window's top-left corner.

How Cua Driver handles it: it queries CoordType::Window (which GTK4 does report correctly per-widget) and adds the window's screen origin from _GTK_FRAME_EXTENTS on X11 or the org.cua.WinRects shell helper on Wayland. That reconstructs true screen coordinates, so no caller action is required.

elements[].frame is screen-absolute, but pixel actions use screenshot pixels#

Symptom: a caller reads an element's frame from get_window_state and passes its center straight to click({pid, window_id, x, y}). The click lands somewhere other than the element — displaced by the window's screen origin, and possibly scaled by the capture ratio. Nothing refuses, because the coordinates were structurally valid; they simply addressed a different point.

Cause: element geometry and pixel action arguments are expressed in two different coordinate spaces, and neither field restates its own space.

With pid and window_id, the pixel forms of click, right_click, double_click, drag, scroll, type_text, press_key, and hotkey read the window-local screenshot space. double_click currently labels these as screen coordinates in its generated argument reference, but its implementation uses the same window-local conversion as the other pixel actions.

What to do: prefer element_token, or element_index with the matching snapshot_id. The accessibility path carries no coordinates, so it is unaffected by this mismatch, and it also works on backgrounded windows. Reach for a pixel action only on canvas, WebGL, or custom-drawn surfaces that expose no element.

When a pixel action is unavoidable, match the scope to the space:

• Window scope. Rebase onto the window origin, then rescale to the capture. Take the origin and size from list_windows (windows[].bounds), which is reported in the same units as frame, and take the capture size from the snapshot's screenshot_width and screenshot_height:

  scale_x = screenshot_width  / bounds.width
  scale_y = screenshot_height / bounds.height
   
  x = (frame.x + frame.w / 2 - bounds.x) * scale_x
  y = (frame.y + frame.h / 2 - bounds.y) * scale_y

• Desktop scope. If the element is on the captured primary display, rescale the frame center into the full-display PNG returned by get_desktop_state, then call click with scope:"desktop" and no pid or window_id:

  scale_x = desktop.screenshot_width  / desktop.screen_width
  scale_y = desktop.screenshot_height / desktop.screen_height
   
  x = (frame.x + frame.w / 2) * scale_x
  y = (frame.y + frame.h / 2) * scale_y

  These ratios are normally 1 on Windows and Linux. On macOS Retina displays, the frame is in logical points while the desktop PNG is in native pixels, so the ratio is normally 2.

On macOS, get_window_state also reports window_bounds, which may be used in place of list_windows there. Windows and Linux report screenshot_width and screenshot_height only.

Permission boundaries#

Cua Driver is constrained by the macOS permission model. Two relevant grants:

Grants are tied to the CuaDriver.app bundle identity (com.trycua.driver). The Rust build and installer preserve this identity, so TCC grants persist across rebuilds and updates.