Click and type

Click a target pid. Prefer element_token (from get_window_state): it identifies one snapshot element, works on backgrounded, minimized or off-Space windows, and moves no cursor or focus. Use x, y only for canvas / video / WebGL / custom-drawn surfaces missing from the AX tree.

• element_token: AX action path. Tokens are scoped to (pid, window_id) and stale once the window is snapshotted again.
• x, y: window-local pixels of the get_window_state screenshot, sent as CGEvent mouse events. Needs a visible window. modifier takes cmd/shift/option/ctrl.

button defaults to left; on the AX path "right" maps to AXShowMenu and "middle" falls back to a pixel click at the element center. action: press (default), show_menu, pick, confirm, cancel, open, show_alternate_ui, show_default_ui; an element listing showalternateui reveals its hover-only controls with show_alternate_ui, in the background and without the pointer. from_zoom:true translates zoom-image pixels after a zoom call. Pressing a popup button opens its menu and leaves it open until an item is chosen or it is dismissed; in the background the user's front window keeps key focus meanwhile. To choose an option in one call use set_value. In the background a web <select> (Chrome, Electron) is not opened, because its menu would raise the browser window: the click is refused with select_menu_would_raise_window; use set_value.

macOS parameters

Double-click at (x, y) or on an AX element identified by element_token.

AX path (element_token provided): performs AXOpen when the element advertises it (Finder items, openable list rows/cells); otherwise resolves the element's on-screen center and falls back to a pixel double-click there.

Pixel path (x, y provided): two down/up pairs ~80 ms apart at the given coordinates.

macOS parameters

Right-click against a target pid. Two addressing modes:

• element_token (from the last get_window_state snapshot): performs AXShowMenu on the cached element. Pure AX RPC, works on backgrounded / hidden windows, no cursor move or focus steal. Requires a prior get_window_state(pid, window_id) in this turn.

• x, y: synthesizes rightMouseDown / rightMouseUp CGEvent pair posted to the pid. Driver converts image-pixel → screen-point internally. modifier forces the CGEvent path (AX actions don't propagate modifier keys).

Exactly one of element_token or (x AND y) must be provided. pid is required with x, y; an element_token carries its own pid.

macOS parameters

Insert text via AXSetAttribute(kAXSelectedText) into the element given by element_token, or the pid's focused element. Does not press keys: use press_key / hotkey for Return, Escape, arrows. If AX insertion is unsupported (Chromium/Electron), it falls back to CGEvent typing when the route fits the transport budget; longer routes are refused with a safe chunk size.

Web content (browser tabs, Slack, VS Code): AXValue is not proof the DOM saw the text, so the result is effect:"unverifiable" and Electron web content refuses background delivery. For a browser tab use the page tool. For an embedded web view pass x, y (no element_token) to pixel-click the field then type in one call; a pixel click will not reliably open a closed control, so AX-press it open first. Confirm from the screenshot, and use delivery_mode:"foreground" only if background still drops the text.

macOS parameters

Press and release one key. delivery_mode as in click/type_text.

• background (default): posts to the pid without raising the window; with element_token it focuses that element first.
• foreground: fronts the exact window (needs window_id), sends a real HID key transition (Chromium content, inline editors, menu equivalents), then restores the prior app.

An element target whose focus cannot be confirmed is refused with target_focus_unconfirmed, effect "refused", and no key dispatched. Confirmed only when a native AX value/selection read-back changes; otherwise effect:"unverifiable" (not a delivery failure). Keys: return, tab, escape, up/down/left/right, space, delete, home, end, pageup, pagedown, f1-f12, any letter or digit.

macOS parameters

Press a key combination: e.g. ["cmd", "c"] for Copy, ["cmd", "shift", "4"] for screenshot selection. Follows the same delivery_mode ladder as click/type_text: it does NOT raise the window by default: • background (default): post the combo to the target pid WITHOUT fronting or raising it; uses the macOS 14+ auth-message envelope so Chromium/Electron accept it as trusted live input. With an AX target, focus that exact element first. No top-level focus steal. window_id here only targets the combo; it does not raise. Background key events never reach a native app's menus, so when the combo is a menu item's key equivalent (cmd+s, cmd+w, ...) and the app is not in front, that menu item runs through accessibility instead (a disabled Save is saved through the app's scripting interface and verified on disk); a menu item macOS keeps disabled in the background is refused with background_unavailable, and no keys are sent. • foreground: briefly front the window (NSMenu path, < 1 ms via SLPSSetFrontProcessWithOptions) so native menu key-equivalents (Cmd+Z, Cmd+W) dispatch, then restore the prior frontmost; the explicit escalation for menu-bar shortcuts on non-Chromium apps that ignore a background combo. With an AX target or x,y, the focused field receives the chord through the foreground HID queue (needed by native Chromium fields such as the omnibox). Requires window_id.

A combo is never driver-verifiable (no read-back) → effect:"unverifiable"; confirm via screenshot. NOTE: a keyboard combo does NOT focus a text field; to type into a backgrounded Electron input, establish real renderer focus with a PIXEL click first, then type_text. If an app only accepts paste, call clipboard_write, then clipboard_read and verify its types (and text when applicable) before selecting or replacing editor content; only then send Cmd+V.

Recognized modifiers: cmd/command, shift, option/alt, ctrl/control, fn. Non-modifier keys use the same vocabulary as press_key. Order: modifiers first, one non-modifier last.

macOS parameters

Set an element's value by element_token. A popup button / select dropdown: selects the menu item whose title or value matches value (case-insensitive). Safari <select> is set through the DOM; AppKit and Chromium popups list items only while open, so the menu is opened, the item pressed and the menu closed again (in the background the user's front window keeps key focus). An unknown value lists the options. Any other element: writes AXValue (sliders, steppers, date pickers, native text fields).

A Finder file name cell (list or icon view, not being edited) is renamed through Finder's scripting interface with Finder left in the background, and the new name is read back from disk; the value is the full new name with its extension. macOS asks the user once to allow Cua Driver to control Finder. Finder's Get Info Name field is refused with file_name_needs_rename. For free-form text in web inputs use type_text_chars: WebKit ignores AXValue writes.

macOS parameters