Windows and accessibility
cua.env.v1 WindowsService and AccessibilityService: list, watch and arrange windows, launch apps, and the accessibility tree.
cua.env.v1 WindowsService and AccessibilityService: list, watch and arrange windows, launch apps, and the accessibility tree.
Windows, app launch and the accessibility tree. These services need the desktop provider.
Source: libs/cua/proto/cua/env/v1/windows.proto, libs/cua/proto/cua/env/v1/accessibility.proto.
cua.env.v1.WindowsService
Window enumeration, window management and app launching.
Windows are addressed with opaque WindowRef handles (id + epoch). Native
handles never cross the wire. Nothing here goes through a shell: apps are
launched by identifier with an explicit argument vector.
/cua.env.v1.WindowsService/ListWindows, unary: ListWindowsRequest to ListWindowsResponse.
Lists windows matching a filter.
/cua.env.v1.WindowsService/WatchWindows, server stream: WatchWindowsRequest to WatchWindowsResponse.
Streams window lifecycle changes. Starts with a snapshot of all matching windows so clients never race between a list and a watch.
/cua.env.v1.WindowsService/GetWindow, unary: GetWindowRequest to GetWindowResponse.
Returns one window.
/cua.env.v1.WindowsService/ActivateWindow, unary: ActivateWindowRequest to ActivateWindowResponse.
Brings a window to the front and gives it keyboard focus.
/cua.env.v1.WindowsService/SetWindowBounds, unary: SetWindowBoundsRequest to SetWindowBoundsResponse.
Moves and/or resizes a window.
/cua.env.v1.WindowsService/MinimizeWindow, unary: MinimizeWindowRequest to MinimizeWindowResponse.
Minimizes (iconifies) a window.
/cua.env.v1.WindowsService/MaximizeWindow, unary: MaximizeWindowRequest to MaximizeWindowResponse.
Maximizes (zooms) a window.
/cua.env.v1.WindowsService/RestoreWindow, unary: RestoreWindowRequest to RestoreWindowResponse.
Restores a minimized or maximized window to its normal state.
/cua.env.v1.WindowsService/CloseWindow, unary: CloseWindowRequest to CloseWindowResponse.
Asks a window to close (like clicking its close button), or force-closes it.
/cua.env.v1.WindowsService/LaunchApp, unary: LaunchAppRequest to LaunchAppResponse.
Launches an application.
/cua.env.v1.WindowsService/Open, unary: OpenRequest to OpenResponse.
Opens a URL or path with its default (or a given) application.
cua.env.v1.AccessibilityService
The guest accessibility tree (macOS AX, Windows UI Automation, Linux AT-SPI via cua-driver platform-linux).
Every read returns a snapshot id. Element ids are only meaningful together
with the snapshot id they came from; acting on an element from an older
snapshot fails with ERROR_REASON_STALE_SNAPSHOT unless the server can
prove the element is unchanged. Requires capability "a11y".
| RPC | Request | Response | Kind |
|---|---|---|---|
GetTree | GetTreeRequest | GetTreeResponse | unary |
Find | FindRequest | FindResponse | unary |
Act | ActRequest | ActResponse | unary |
/cua.env.v1.AccessibilityService/GetTree, unary: GetTreeRequest to GetTreeResponse.
Returns the accessibility tree of a window (or the focused window).
/cua.env.v1.AccessibilityService/Find, unary: FindRequest to FindResponse.
Finds elements matching a query.
/cua.env.v1.AccessibilityService/Act, unary: ActRequest to ActResponse.
Performs an accessibility action on an element.
cua/env/v1/windows.proto#Application that owns a window.
| Field | # | Type | Description |
|---|---|---|---|
name | 1 | string | Display name, for example "Firefox". |
app_id | 2 | string | Platform app identifier: bundle id on macOS, .desktop id on Linux, AppUserModelID or executable name on Windows. Empty when unknown. |
pid | 3 | uint32 | Guest process id of the owning process. Guest-local; never a host id. |
Description of a window.
| Field | # | Type | Description |
|---|---|---|---|
ref | 1 | WindowRef | Handle. |
title | 2 | string | Title. |
app | 3 | AppInfo | Owning application. |
bounds | 4 | Rect | Frame in global logical points. Advisory: it is a snapshot and may be stale by the time it is used. Media sessions and screenshots report their own authoritative geometry. |
display_id | 5 | string | Display containing most of the window. |
state | 6 | WindowState | State. |
kind | 7 | WindowKind | Role. |
focused | 8 | bool | True if this window has keyboard focus. |
on_screen | 9 | bool | True if any part is on screen (not minimized, hidden or off-desktop). |
z_order | 10 | uint32 | Front-to-back stacking order among listed windows, 0 is frontmost. |
Filter for window listing and watching. All set fields must match.
| Field | # | Type | Description |
|---|---|---|---|
title_contains | 1 | string | Case-insensitive substring of the title. |
app_name_contains | 2 | string | Case-insensitive substring of the app name. |
app_id | 3 | string | Exact AppInfo.app_id. |
pid | 4 | uint32 | Owning process id. |
display_id | 5 | string | Only windows on this display. |
on_screen_only | 6 | bool | Only on-screen windows. |
include_system | 7 | bool | Include WINDOW_KIND_SYSTEM and WINDOW_KIND_PHANTOM windows, which are omitted by default. |
Request for WindowsService.ListWindows.
| Field | # | Type | Description |
|---|---|---|---|
filter | 1 | WindowFilter | Filter. |
Response for WindowsService.ListWindows.
| Field | # | Type | Description |
|---|---|---|---|
windows | 1 | repeated WindowInfo | Matching windows, front to back. |
Request for WindowsService.WatchWindows.
| Field | # | Type | Description |
|---|---|---|---|
filter | 1 | WindowFilter | Filter applied to every event. |
keepalive_interval | 2 | google.protobuf.Duration | Interval between keepalive messages. Unset means 30 seconds. |
Initial state of a window watch.
| Field | # | Type | Description |
|---|---|---|---|
windows | 1 | repeated WindowInfo | All matching windows, front to back. |
A window was closed or no longer matches the filter.
| Field | # | Type | Description |
|---|---|---|---|
ref | 1 | WindowRef | The window. |
One message of the WindowsService.WatchWindows stream.
| Field | # | Type | Description |
|---|---|---|---|
snapshot | 1 | WindowSnapshot (oneof event) | First message: current matching windows. |
created | 2 | WindowInfo (oneof event) | A new window appeared (or started matching). |
updated | 3 | WindowInfo (oneof event) | A window changed title, bounds, state, focus or z-order. Carries the full new description. |
closed | 4 | WindowClosed (oneof event) | A window closed. |
keepalive | 5 | KeepAlive (oneof event) | Idle stream heartbeat. |
Request for WindowsService.GetWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
Response for WindowsService.GetWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window. |
Request for WindowsService.ActivateWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
Response for WindowsService.ActivateWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
Request for WindowsService.SetWindowBounds.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
position | 2 | Point | New top-left corner in global logical points. Unset keeps the position. |
width | 3 | optional double | New width in logical points. Unset keeps the width. |
height | 4 | optional double | New height in logical points. Unset keeps the height. |
Response for WindowsService.SetWindowBounds.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. Its bounds may differ from the request when the window manager or app constrains them. |
Request for WindowsService.MinimizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
Response for WindowsService.MinimizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
Request for WindowsService.MaximizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
Response for WindowsService.MaximizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
Request for WindowsService.RestoreWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
Response for WindowsService.RestoreWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
Request for WindowsService.CloseWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
force | 2 | bool | Destroy the window without giving the app a chance to object (for example to an unsaved-changes prompt). On most platforms this kills the owning process. |
Response for WindowsService.CloseWindow.
| Field | # | Type | Description |
|---|---|---|---|
closed | 1 | bool | True if the window is gone. False when the app kept it open (for example it showed a save prompt). |
Identifies an application to launch.
| Field | # | Type | Description |
|---|---|---|---|
app_id | 1 | string (oneof app) | Platform app identifier: macOS bundle id ("org.mozilla.firefox"), Linux .desktop id ("firefox.desktop"), Windows AppUserModelID. |
executable | 2 | string (oneof app) | Executable name (resolved on PATH) or absolute path. |
name | 3 | string (oneof app) | Display name, resolved by the platform (for example "Firefox"). The least precise option. |
Request for WindowsService.LaunchApp.
| Field | # | Type | Description |
|---|---|---|---|
app | 1 | AppSpec | What to launch. |
args | 2 | repeated string | Arguments passed verbatim, never through a shell. |
env | 3 | map of string to string | Extra environment variables. |
cwd | 4 | string | Working directory. Empty means the user's home. |
delivery | 5 | Delivery | DELIVERY_BACKGROUND launches without activating the app or stealing focus (where the platform allows). DELIVERY_FOREGROUND activates it. |
wait_for_window | 6 | google.protobuf.Duration | Wait up to this long for the app's first window and return it. Unset returns immediately after the process starts. |
Response for WindowsService.LaunchApp.
| Field | # | Type | Description |
|---|---|---|---|
pid | 1 | uint32 | Guest process id of the launched (or already running, for single- instance apps) process. 0 when the platform does not report it. |
windows | 2 | repeated WindowInfo | Windows that appeared within wait_for_window. |
Request for WindowsService.Open.
| Field | # | Type | Description |
|---|---|---|---|
url | 1 | string (oneof target) | A URL, for example "https://example.com" or "mailto:a@b". |
path | 2 | string (oneof target) | A guest file or directory path. |
with_app | 3 | AppSpec | Open with this app instead of the default handler. |
delivery | 4 | Delivery | Delivery, as for LaunchAppRequest.delivery. |
Response for WindowsService.Open.
| Field | # | Type | Description |
|---|---|---|---|
pid | 1 | uint32 | Guest process id of the handling app, 0 when unknown. |
Window state.
| Value | # | Description |
|---|---|---|
WINDOW_STATE_UNSPECIFIED | 0 | Not reported. |
WINDOW_STATE_NORMAL | 1 | Normal (neither minimized, maximized nor fullscreen). |
WINDOW_STATE_MINIMIZED | 2 | Minimized / iconified. |
WINDOW_STATE_MAXIMIZED | 3 | Maximized / zoomed. |
WINDOW_STATE_FULLSCREEN | 4 | Fullscreen. |
WINDOW_STATE_HIDDEN | 5 | Hidden (app hidden, withdrawn, or on another virtual desktop). |
Broad window role.
| Value | # | Description |
|---|---|---|
WINDOW_KIND_UNSPECIFIED | 0 | Not reported. |
WINDOW_KIND_STANDARD | 1 | A normal application window. |
WINDOW_KIND_DIALOG | 2 | A dialog or sheet. |
WINDOW_KIND_PANEL | 3 | A floating panel or utility window. |
WINDOW_KIND_MENU | 4 | A menu or popup. |
WINDOW_KIND_TOOLTIP | 5 | A tooltip. |
WINDOW_KIND_SYSTEM | 6 | Desktop chrome: dock, menu bar strip, panel, wallpaper. |
WINDOW_KIND_PHANTOM | 7 | Invisible or indicator surfaces (for example the macOS screen-recording indicator, 1x1 helper windows). Hidden unless include_system is set. |
cua/env/v1/accessibility.proto#One accessibility element. Trees are returned flattened in pre-order (parents before children) to avoid unbounded message nesting.
| Field | # | Type | Description |
|---|---|---|---|
element_id | 1 | string | Element id, unique within its snapshot. |
parent_id | 2 | string | Parent element id, empty for the root. |
depth | 3 | uint32 | Depth below the root (root is 0). |
role | 4 | string | Normalized role, for example "button", "text_field", "menu_item", "window". Platform roles are mapped to one vocabulary. |
native_role | 5 | string | The platform's native role string (for example "AXButton", "ControlType.Button", "push button"), for debugging. |
name | 6 | string | Accessible name / label. |
value | 7 | string | Current value (text content, slider value), when it has one. |
description | 8 | string | Accessible description / help text. |
bounds | 9 | Rect | Frame in global logical points, when known. |
states | 10 | repeated string | Normalized states, for example "focused", "enabled", "selected", "checked", "expanded", "editable", "offscreen". |
actions | 11 | repeated AccessibilityAction | Actions the element supports (see AccessibilityAction). |
attributes | 12 | map of string to string | Other attributes the platform exposes, stringified. |
Request for AccessibilityService.GetTree.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Window whose tree to read. Unset means the focused window. |
max_depth | 2 | uint32 | Maximum depth below the root. 0 means unlimited (subject to the server's node limit). |
include_hidden | 3 | bool | Include elements that are offscreen or not visible. |
max_nodes | 4 | uint32 | Stop after this many nodes. 0 means the server limit (normally 5000). |
Response for AccessibilityService.GetTree.
| Field | # | Type | Description |
|---|---|---|---|
snapshot_id | 1 | string | Snapshot id for later Act calls. |
window | 2 | WindowRef | The window the tree belongs to. |
nodes | 3 | repeated AccessibilityNode | Nodes in pre-order. |
truncated | 4 | bool | True if max_depth or max_nodes cut the tree short. |
Query for AccessibilityService.Find. All set fields must match.
| Field | # | Type | Description |
|---|---|---|---|
role | 1 | string | Normalized role. |
name | 2 | string | Exact accessible name. |
name_contains | 3 | string | Case-insensitive substring of the accessible name. |
value_contains | 4 | string | Case-insensitive substring of the value. |
states | 5 | repeated string | Required states. |
Request for AccessibilityService.Find.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Window to search. Unset means the focused window. |
query | 2 | AccessibilityQuery | What to look for. |
max_results | 3 | uint32 | Maximum matches. 0 means 50. |
Response for AccessibilityService.Find.
| Field | # | Type | Description |
|---|---|---|---|
snapshot_id | 1 | string | Snapshot id for later Act calls. |
nodes | 2 | repeated AccessibilityNode | Matching elements (their parent_id refers to elements not necessarily included). |
Reference to an element within a snapshot.
| Field | # | Type | Description |
|---|---|---|---|
snapshot_id | 1 | string | Snapshot the element id came from. |
element_id | 2 | string | Element id. |
Request for AccessibilityService.Act.
| Field | # | Type | Description |
|---|---|---|---|
element | 1 | ElementRef | Which element. |
action | 2 | AccessibilityAction | Which action. |
value | 3 | string | New value for ACCESSIBILITY_ACTION_SET_VALUE. |
custom_action | 4 | string | Native action name for ACCESSIBILITY_ACTION_CUSTOM. |
delivery | 5 | Delivery | Delivery mode. Accessibility actions are background-safe on most platforms; DELIVERY_FOREGROUND activates the window first. |
Response for AccessibilityService.Act.
| Field | # | Type | Description |
|---|---|---|---|
report | 1 | DeliveryReport | What actually happened. |
Accessibility action.
| Value | # | Description |
|---|---|---|
ACCESSIBILITY_ACTION_UNSPECIFIED | 0 | Not set. Rejected. |
ACCESSIBILITY_ACTION_PRESS | 1 | Press / invoke / click. |
ACCESSIBILITY_ACTION_FOCUS | 2 | Move keyboard focus to the element. |
ACCESSIBILITY_ACTION_SET_VALUE | 3 | Replace the element's value with ActRequest.value. |
ACCESSIBILITY_ACTION_INCREMENT | 4 | Increment (sliders, steppers). |
ACCESSIBILITY_ACTION_DECREMENT | 5 | Decrement. |
ACCESSIBILITY_ACTION_SHOW_MENU | 6 | Open the element's context menu. |
ACCESSIBILITY_ACTION_EXPAND | 7 | Expand (tree items, disclosure triangles, combo boxes). |
ACCESSIBILITY_ACTION_COLLAPSE | 8 | Collapse. |
ACCESSIBILITY_ACTION_SELECT | 9 | Select (list items, tabs). |
ACCESSIBILITY_ACTION_SCROLL_INTO_VIEW | 10 | Scroll the element into view. |
ACCESSIBILITY_ACTION_CUSTOM | 11 | A platform-specific action named by ActRequest.custom_action. |