Window state tools
Screenshots, accessibility trees, verification and visual regions of a window.
Screenshots, accessibility trees, verification and visual regions of a window.
| Tool | Description | Platforms |
|---|---|---|
get_window_state | Walk a running app's AX tree and return BOTH a structured elements array (preferred) AND a Markdown rendering of the same tree (back-compat). | macOS, Linux, Windows |
get_accessibility_tree | Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid. | macOS, Linux, Windows |
verify_state | Deterministically verify bounded predicates against one exact window. | macOS, Linux, Windows |
parse_visual_regions | Parse one immutable registered capture into model-neutral text and icon regions. | macOS, Linux, Windows |
Served by cua-driver mcp; see MCP tools for every tool.
get_window_state#Effect: read-only. Platforms: macOS, Linux, Windows.
Walk a running app's AX tree and return BOTH a structured elements array (preferred) AND a Markdown rendering of the same tree (back-compat). Every actionable element is tagged with [element_index N] in the markdown and as element_index in the structured array; pass each element's element_token to click, type_text, press_key, etc.
INVARIANT: call get_window_state once per turn per (pid, window_id) before any element action. The next snapshot of the window replaces this one, stales its element tokens, and lists the replaced ids in invalidated_snapshot_ids.
PREFERRED CONSUMERS read structuredContent.elements (one entry per indexed row with element_index, role, label, value (the element's text/AXValue when present: use it to verify what a field holds), actions (names of AX actions exposed by the element, omitted when empty), frame: {x,y,w,h}, parent_index, depth). The markdown tree_markdown stays available and unchanged in shape for existing text-parsing callers: but new fields will only be added to the structured side.
Always returns BOTH the element tree AND a screenshot: ground on both and cross-check (the tree lies on some surfaces: Electron echo-confirms, Catalyst null values, virtualized off-viewport rows with h:1 frames). You choose the modality at ACTION time, not here: an element ax action (pass element_token → the accessibility rung) or an element px action (pass x,y → the pixel rung, read straight off this screenshot). capture_mode is deprecated and ignored. Pass include_screenshot:false to skip the grab and get the tree only: the cheap path when you're just re-indexing before an element ax action.
The mirror image: pass include_accessibility_tree:false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata, window_bounds, screenshot_scale, screenshot_width/screenshot_height, app_name, and window_title, the capture-only path for rendering a live window preview / picture-in-picture without paying for perception. Setting BOTH include_accessibility_tree:false and include_screenshot:false is an error (nothing to return). Optional max_image_dimension overrides the configured screenshot long-edge limit for this call; use 0 for native resolution. The legacy max_dimension remains a tighter cap for compatibility.
The snapshot is SCOPED to window_id: a window_id that no longer exists is refused with window_id_not_found, and one owned by another process is refused with window_owner_pid_mismatch naming the real owner_pid to retry with (macOS hosts a sandboxed app's Open/Save panel out-of-process, so its window belongs to the panel service, not the app). If the window is live under this pid but its accessibility surface can't be resolved, the tree comes back EMPTY with degraded_reason: ax_window_unresolved and the screenshot of the requested window; background input is refused until it resolves, so re-snapshot or act with delivery_mode:"foreground". When that pid is an app still launching (its window exists before it answers accessibility), the walk first waits up to timeout_ms for it; if it never answers, the tree comes back EMPTY with degraded_reason: ax_app_launching, truncated: true and truncation_reason: app_lookup_timeout. A window on another Space still resolves by its exact CGWindowID. This tool never returns another surface's elements under your window_id. Before exposing a screenshot, its raw dimensions are validated as a coherent 1x/2x representation of the requested WindowServer bounds. px_frame_mismatch or px_capture_unavailable omits an unprovable screenshot/pixel frame instead of guessing a transform; the truthful AX payload remains available.
Optional query projects both tree_markdown and structured elements to matching lines plus their ancestor chain (case-insensitive substring). The element_index values are unchanged, the complete snapshot remains actionable, and element_count continues to report its total size; filtered_element_count reports the projected response size.
Optional max_elements / max_depth bound the AX walk to mitigate context-window blow-up on Electron / Obsidian / large web apps that produce 10k+ element trees. When applied, BOTH the markdown and the structured elements are truncated identically. Omit both for current default behaviour (≤2 000 elements, depth ≤25).
macOS parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
capture_mode | "ax" | "vision" | DEPRECATED and ignored. get_window_state always returns BOTH the element tree and a screenshot: ground on both. The modality is chosen at action time by how you address the target: an element ax action (element_token) or an element px action (x,y). Any value (including the old "som"/"screenshot" aliases) is accepted but has no effect. | |
include_accessibility_tree | boolean | Default true: walk the AX tree and return elements + tree_markdown alongside the screenshot. Set false to SKIP the AX walk entirely (the expensive part, bounded by timeout_ms) and return just the screenshot plus window metadata (bounds, scale, app_name, window_title): the capture-only path for rendering a live window preview / picture-in-picture. Mirrors include_screenshot. Setting BOTH include_accessibility_tree:false AND include_screenshot:false is an error (nothing to return). | |
include_screenshot | boolean | Default true: returns a grounding screenshot alongside the tree. Set false to skip the grab and return the tree only (the cheap path when you're just re-indexing before an element ax action; saves the image tokens + screen-grab latency). screenshot_out_file still forces a capture to disk. | |
max_depth | integer | Cap on the AX-tree walk depth. Nodes whose rendered indent would exceed this are omitted. Omit for the default (25). Lower this for deep menu/Electron trees. Minimum: 1. | |
max_dimension | integer | Optional cap on the returned screenshot's long edge, in pixels (aspect ratio preserved): the cheap path for a small preview / thumbnail. Applied on top of the session/global max_image_dimension ceiling; the tighter of the two wins. Omit for the configured default. Minimum: 1. | |
max_elements | integer | Cap on the total number of AX nodes walked. Truncates depth-first; markdown and structured elements truncate together. Omit for the default (2 000). Lower this for Electron / Obsidian / large web apps that produce 10k+ element trees and blow context windows. Minimum: 1. | |
max_image_dimension | integer | Per-call override for the returned screenshot's long edge in pixels. An explicit value wins over the session/global setting; 0 returns native resolution. Omit to preserve configured behavior. Minimum: 0. | |
pid | integer | required | Target process ID. |
query | string | Case-insensitive filter for tree_markdown and structured elements. Returns matching actionable rows plus their actionable ancestors without renumbering element_index values. | |
screenshot_out_file | string | When set, write the PNG to this file path (~ expanded) instead of embedding base64 in the response. The structured output will contain screenshot_file_path instead. | |
window_id | integer | required | Target window ID from list_windows. |
Parameters on every platform
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. | |
timeout_ms | integer | 1000 | Wall-clock budget in milliseconds for the accessibility-tree walk (default 1000, min 100, max 120000). Bounds the WHOLE walk. When the budget runs out the tool returns the PARTIAL tree it has, flagged with truncated: true, truncation_reason, nodes_visited, nodes_pending and elements_complete: false; retry with a larger value (e.g. 5000) or narrow with query / max_depth. Range: 100 to 120000. |
Example arguments
{"pid":1,"window_id":1}get_accessibility_tree#Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
Return a lightweight snapshot of the desktop: running regular apps and on-screen visible windows with their bounds, z-order, and owner pid.
For the full AX subtree of a single window (with interactive element indices you can click by), use get_window_state instead: that's the heavy per-window tool. This one is a fast discovery read that needs no TCC grants.
Parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
verify_state#Deterministically verify bounded predicates against one exact window. The driver evaluates structured window/accessibility state and may return the final screenshot as uninterpreted visual evidence for a multimodal caller. Predicate results are satisfied, unsatisfied, or unknown; unknown never implies success. Accessibility projections are conservative: absence remains unknown unless the observed search domain is proven exhaustive.
Effect: read-only. Platforms: macOS, Linux, Windows.
| Parameter | Type | Default | Description |
|---|---|---|---|
expect | object[] | required | One to eight predicates, combined with logical AND. Items: 1 to 8. |
include_screenshot | boolean | Return the final window screenshot as image content for a multimodal caller. The driver does not interpret that image. | |
pid | integer | required | Exact process whose window may be observed. Minimum: 1. |
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. This field never selects capture modality or authorization. | |
stable_samples | integer | 2 | Consecutive satisfied samples required before returning success. Range: 1 to 5. |
timeout_ms | integer | 5000 | Bounded wait. Zero performs one sample. Range: 0 to 10000. |
window_id | integer | required | Exact native window identifier. |
Example arguments
{"pid":1,"window_id":1,"expect":[{}]}parse_visual_regions#Effect: read-only, idempotent. Platforms: macOS, Linux, Windows.
Parse one immutable registered capture into model-neutral text and icon regions.
macOS parameters
| Parameter | Type | Default | Description |
|---|---|---|---|
session | string | For multi-call work, prefer a short public session label and repeat it on every call that accepts it. Omit it to use the authenticated transport's implicit lifecycle session. |
Parameters on every platform
| Parameter | Type | Default | Description |
|---|---|---|---|
capture_id | string | required | capture_id of a screenshot returned by get_window_state or get_desktop_state. |
options | object | {} | Optional, model-neutral controls for one bounded parse. |
options.kinds | "text" | "icon"[] | Region kinds to return (text, icon). Omit for all kinds. | |
options.max_regions | integer | Return at most this many regions. Minimum: 1. | |
options.min_confidence | number | Drop regions below this confidence (0 to 1). Range: 0 to 1. |
Example arguments
{"capture_id":"<capture_id>"}