Computer, windows and accessibility
cua.env.v1 ComputerService, DriverService, WindowsService and AccessibilityService: screenshots, input, clipboard, displays, Cua Driver tools, windows and the accessibility tree.
Desktop control. These services need the desktop provider; input goes through Cua Driver, which DriverService also exposes tool by tool.
Source: libs/cua/proto/cua/env/v1/computer.proto, libs/cua/proto/cua/env/v1/driver.proto, libs/cua/proto/cua/env/v1/windows.proto, libs/cua/proto/cua/env/v1/accessibility.proto.
ComputerService#
cua.env.v1.ComputerService
Screen capture and synthetic input.
Implemented by calling the cua-driver-core tool registry in-process (not
over MCP). Every input RPC states its delivery mode explicitly and reports
what actually happened in a DeliveryReport.
Coordinates: screenshots report both their pixel size and the logical
region they cover, so clients never guess a Retina scale. Input can be
expressed in screen points, window points, normalized units or the pixels
of a specific screenshot (see CoordinateSpace); the server performs the
single conversion to native coordinates.
ComputerService.Screenshot#
/cua.env.v1.ComputerService/Screenshot, unary: ScreenshotRequest to ScreenshotResponse.
Captures a display, a window or a region of either.
ComputerService.Pointer#
/cua.env.v1.ComputerService/Pointer, unary: PointerRequest to PointerResponse.
Performs one pointer (mouse) operation.
ComputerService.Keyboard#
/cua.env.v1.ComputerService/Keyboard, unary: KeyboardRequest to KeyboardResponse.
Performs one keyboard operation.
ComputerService.GetClipboard#
/cua.env.v1.ComputerService/GetClipboard, unary: GetClipboardRequest to GetClipboardResponse.
Reads the clipboard.
ComputerService.SetClipboard#
/cua.env.v1.ComputerService/SetClipboard, unary: SetClipboardRequest to SetClipboardResponse.
Replaces the clipboard.
ComputerService.GetCursorPosition#
/cua.env.v1.ComputerService/GetCursorPosition, unary: GetCursorPositionRequest to GetCursorPositionResponse.
Returns the user-visible pointer position.
ComputerService.ListDisplays#
/cua.env.v1.ComputerService/ListDisplays, unary: ListDisplaysRequest to ListDisplaysResponse.
Lists displays attached to the guest desktop.
DriverService#
cua.env.v1.DriverService
Passthrough to the full cua-driver tool registry.
Every cua-driver tool is reachable here without new protos. The same
registry is exposed to agents as streamable-HTTP MCP at /mcp on the same
port. Arguments and structured results are JSON text so that tool schemas
evolve with the cua-driver contract, not with this package.
| RPC | Request | Response | Kind |
|---|---|---|---|
ListTools | ListToolsRequest | ListToolsResponse | unary |
CallTool | CallToolRequest | CallToolResponse | unary |
DriverService.ListTools#
/cua.env.v1.DriverService/ListTools, unary: ListToolsRequest to ListToolsResponse.
Lists the tools the linked cua-driver registry exposes on this platform.
DriverService.CallTool#
/cua.env.v1.DriverService/CallTool, unary: CallToolRequest to CallToolResponse.
Invokes one tool.
WindowsService#
cua.env.v1.WindowsService
Window enumeration, window management and app launching.
Windows are addressed with opaque WindowRef handles (id + epoch). Native
handles never cross the wire. Nothing here goes through a shell: apps are
launched by identifier with an explicit argument vector.
WindowsService.ListWindows#
/cua.env.v1.WindowsService/ListWindows, unary: ListWindowsRequest to ListWindowsResponse.
Lists windows matching a filter.
WindowsService.WatchWindows#
/cua.env.v1.WindowsService/WatchWindows, server stream: WatchWindowsRequest to WatchWindowsResponse.
Streams window lifecycle changes. Starts with a snapshot of all matching windows so clients never race between a list and a watch.
WindowsService.GetWindow#
/cua.env.v1.WindowsService/GetWindow, unary: GetWindowRequest to GetWindowResponse.
Returns one window.
WindowsService.ActivateWindow#
/cua.env.v1.WindowsService/ActivateWindow, unary: ActivateWindowRequest to ActivateWindowResponse.
Brings a window to the front and gives it keyboard focus.
WindowsService.SetWindowBounds#
/cua.env.v1.WindowsService/SetWindowBounds, unary: SetWindowBoundsRequest to SetWindowBoundsResponse.
Moves and/or resizes a window.
WindowsService.MinimizeWindow#
/cua.env.v1.WindowsService/MinimizeWindow, unary: MinimizeWindowRequest to MinimizeWindowResponse.
Minimizes (iconifies) a window.
WindowsService.MaximizeWindow#
/cua.env.v1.WindowsService/MaximizeWindow, unary: MaximizeWindowRequest to MaximizeWindowResponse.
Maximizes (zooms) a window.
WindowsService.RestoreWindow#
/cua.env.v1.WindowsService/RestoreWindow, unary: RestoreWindowRequest to RestoreWindowResponse.
Restores a minimized or maximized window to its normal state.
WindowsService.CloseWindow#
/cua.env.v1.WindowsService/CloseWindow, unary: CloseWindowRequest to CloseWindowResponse.
Asks a window to close (like clicking its close button), or force-closes it.
WindowsService.LaunchApp#
/cua.env.v1.WindowsService/LaunchApp, unary: LaunchAppRequest to LaunchAppResponse.
Launches an application.
WindowsService.Open#
/cua.env.v1.WindowsService/Open, unary: OpenRequest to OpenResponse.
Opens a URL or path with its default (or a given) application.
AccessibilityService#
cua.env.v1.AccessibilityService
The guest accessibility tree (macOS AX, Windows UI Automation, Linux AT-SPI via cua-driver platform-linux).
Every read returns a snapshot id. Element ids are only meaningful together
with the snapshot id they came from; acting on an element from an older
snapshot fails with ERROR_REASON_STALE_SNAPSHOT unless the server can
prove the element is unchanged. Requires capability "a11y".
| RPC | Request | Response | Kind |
|---|---|---|---|
GetTree | GetTreeRequest | GetTreeResponse | unary |
Find | FindRequest | FindResponse | unary |
Act | ActRequest | ActResponse | unary |
AccessibilityService.GetTree#
/cua.env.v1.AccessibilityService/GetTree, unary: GetTreeRequest to GetTreeResponse.
Returns the accessibility tree of a window (or the focused window).
AccessibilityService.Find#
/cua.env.v1.AccessibilityService/Find, unary: FindRequest to FindResponse.
Finds elements matching a query.
AccessibilityService.Act#
/cua.env.v1.AccessibilityService/Act, unary: ActRequest to ActResponse.
Performs an accessibility action on an element.
Types in cua/env/v1/computer.proto#
ScreenshotRequest#
Request for ComputerService.Screenshot.
| Field | # | Type | Description |
|---|---|---|---|
display_id | 1 | string (oneof source) | A display id, or "primary". |
window | 2 | WindowRef (oneof source) | A single window. Captured without occluding windows where the platform supports it (see capability "window_stream"). |
region | 3 | Rect | Sub-region in logical points, relative to the source's top-left corner. Unset captures the whole source. |
format | 4 | ImageFormat | Output format. |
quality | 5 | uint32 | Quality 1 to 100 for lossy formats. 0 means 80. |
max_dimension | 6 | uint32 | Downscale so the longer edge is at most this many pixels, preserving aspect ratio. 0 means no limit (native resolution). |
include_cursor | 7 | bool | Draw the pointer into the image. |
ScreenshotResponse#
Response for ComputerService.Screenshot.
| Field | # | Type | Description |
|---|---|---|---|
image | 1 | bytes | Encoded image bytes. |
format | 2 | ImageFormat | Format of image. |
image_size | 3 | PixelSize | Pixel size of image (after any downscale). |
native_size | 4 | PixelSize | Pixel size of the captured area at native (physical) resolution. |
scale | 5 | double | Image pixels per logical point. To convert an image pixel to a screen point: screen = logical_bounds.origin + pixel / scale. |
logical_bounds | 6 | Rect | The captured area in global logical points (COORDINATE_SPACE_SCREEN). |
screenshot_id | 7 | string | Id to reference this capture's pixel space in later input requests (COORDINATE_SPACE_SCREENSHOT). Valid until the geometry of the source changes; the server keeps at least the 16 most recent ids. |
display_id | 8 | string | Display the capture came from (for a window capture, the display containing most of the window). |
captured_at | 9 | google.protobuf.Timestamp | When the frame was captured. |
InputTarget#
Where and how an input request is delivered.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Window to deliver to. Unset delivers to whatever is under the pointer (pointer) or focused (keyboard). Required for DELIVERY_BACKGROUND on most platforms and for COORDINATE_SPACE_WINDOW. |
display_id | 2 | string | Display for COORDINATE_SPACE_NORMALIZED without a window. Empty means the primary display. |
delivery | 3 | Delivery | Delivery mode. |
space | 4 | CoordinateSpace | Coordinate space of every point in the request. |
screenshot_id | 5 | string | Required with COORDINATE_SPACE_SCREENSHOT: which screenshot's pixel space the points refer to. |
PointerClick#
Click (press and release) one or more times.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to click. Unset clicks at the current pointer position. |
button | 2 | MouseButton | Which button. |
count | 3 | uint32 | Number of clicks: 1 single, 2 double, 3 triple. 0 means 1. |
modifiers | 4 | repeated Key | Modifier keys held during the click. |
PointerMove#
Move the pointer.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Destination. |
duration | 2 | google.protobuf.Duration | Animate the move over this duration. Unset jumps instantly. |
PointerDown#
Press a button without releasing it.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to press. Unset presses at the current position. |
button | 2 | MouseButton | Which button. |
PointerUp#
Release a pressed button.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to release. Unset releases at the current position. |
button | 2 | MouseButton | Which button. |
PointerDrag#
Press at from, move along path to to, release.
| Field | # | Type | Description |
|---|---|---|---|
from | 1 | Point | Start point. |
to | 2 | Point | End point. |
path | 3 | repeated Point | Optional intermediate points, in order. |
button | 4 | MouseButton | Which button. |
duration | 5 | google.protobuf.Duration | Total drag duration. Unset means 300 ms. |
modifiers | 6 | repeated Key | Modifier keys held during the drag. |
PointerScroll#
Scroll by a delta.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to scroll. Unset scrolls at the current pointer position. The pointer is not moved for background delivery. |
delta_x | 2 | double | Horizontal delta; positive scrolls content right. |
delta_y | 3 | double | Vertical delta; positive scrolls content down. |
unit | 4 | ScrollUnit | Unit of the deltas. |
PointerRequest#
Request for ComputerService.Pointer.
| Field | # | Type | Description |
|---|---|---|---|
target | 1 | InputTarget | Target, delivery and coordinate space. |
click | 2 | PointerClick (oneof action) | Click. |
move | 3 | PointerMove (oneof action) | Move. |
down | 4 | PointerDown (oneof action) | Button down. |
up | 5 | PointerUp (oneof action) | Button up. |
drag | 6 | PointerDrag (oneof action) | Drag. |
scroll | 7 | PointerScroll (oneof action) | Scroll. |
PointerResponse#
Response for ComputerService.Pointer.
| Field | # | Type | Description |
|---|---|---|---|
report | 1 | DeliveryReport | What actually happened. |
cursor_position | 2 | Point | User-visible pointer position afterwards, in screen points. |
KeyInput#
One key, either named or as a character.
| Field | # | Type | Description |
|---|---|---|---|
named | 1 | Key (oneof key) | A named key. |
character | 2 | string (oneof key) | A single Unicode scalar value, mapped through the guest's current keyboard layout to the key (and Shift/AltGr) that produces it. Fails with ERROR_REASON_DELIVERY_FAILED if no key produces it; use KeyboardType for arbitrary text. |
KeyboardType#
Type text.
| Field | # | Type | Description |
|---|---|---|---|
text | 1 | string | UTF-8 text. Newlines are typed as Enter. |
mode | 2 | TextEntryMode | Entry mode. |
delay | 3 | google.protobuf.Duration | Delay between characters for TEXT_ENTRY_MODE_KEYSTROKES. Unset means no deliberate delay. |
KeyboardPress#
Press and release one key, optionally with modifiers held.
| Field | # | Type | Description |
|---|---|---|---|
key | 1 | KeyInput | The key. |
modifiers | 2 | repeated Key | Modifiers held while the key is pressed. |
repeat | 3 | uint32 | Number of presses. 0 means 1. |
KeyboardHotkey#
Press a chord: keys go down in order and up in reverse order (for example Control, Shift, T).
| Field | # | Type | Description |
|---|---|---|---|
keys | 1 | repeated KeyInput | Keys in press order. At least one. |
KeyboardDown#
Press a key without releasing it.
| Field | # | Type | Description |
|---|---|---|---|
key | 1 | KeyInput | The key. |
KeyboardUp#
Release a pressed key.
| Field | # | Type | Description |
|---|---|---|---|
key | 1 | KeyInput | The key. |
KeyboardRequest#
Request for ComputerService.Keyboard.
| Field | # | Type | Description |
|---|---|---|---|
target | 1 | InputTarget | Target and delivery. space and points are ignored. |
type | 2 | KeyboardType (oneof action) | Type text. |
press | 3 | KeyboardPress (oneof action) | Press one key. |
hotkey | 4 | KeyboardHotkey (oneof action) | Press a chord. |
down | 5 | KeyboardDown (oneof action) | Key down. |
up | 6 | KeyboardUp (oneof action) | Key up. |
KeyboardResponse#
Response for ComputerService.Keyboard.
| Field | # | Type | Description |
|---|---|---|---|
report | 1 | DeliveryReport | What actually happened. |
ClipboardContent#
Clipboard content.
| Field | # | Type | Description |
|---|---|---|---|
text | 1 | optional string | Plain UTF-8 text. At most 1 MiB. Unset means no text flavor. |
file_paths | 2 | repeated string | Absolute guest paths of files on the clipboard (file copy/paste). To put host files on the guest clipboard, upload them first with FilesystemService, then set their paths here. Requires capability "clipboard.files". |
image_png | 3 | optional bytes | A PNG image (for example a copied screenshot). At most 16 MiB. Requires capability "clipboard.image". |
GetClipboardRequest#
Request for ComputerService.GetClipboard.
No fields.
GetClipboardResponse#
Response for ComputerService.GetClipboard.
| Field | # | Type | Description |
|---|---|---|---|
content | 1 | ClipboardContent | Current content. Only flavors the driver understands are reported. |
generation | 2 | uint64 | Clipboard generation counter; changes whenever the content changes. |
SetClipboardRequest#
Request for ComputerService.SetClipboard.
| Field | # | Type | Description |
|---|---|---|---|
content | 1 | ClipboardContent | New content. Replaces every flavor. |
SetClipboardResponse#
Response for ComputerService.SetClipboard.
| Field | # | Type | Description |
|---|---|---|---|
generation | 1 | uint64 | Generation after the write. |
GetCursorPositionRequest#
Request for ComputerService.GetCursorPosition.
No fields.
GetCursorPositionResponse#
Response for ComputerService.GetCursorPosition.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Position in global logical points (COORDINATE_SPACE_SCREEN). |
display_id | 2 | string | Display containing the pointer. |
ListDisplaysRequest#
Request for ComputerService.ListDisplays.
No fields.
ListDisplaysResponse#
Response for ComputerService.ListDisplays.
| Field | # | Type | Description |
|---|---|---|---|
displays | 1 | repeated Display | Attached displays, primary first. |
ImageFormat#
Encoded image format.
| Value | # | Description |
|---|---|---|
IMAGE_FORMAT_UNSPECIFIED | 0 | Not set. Treated as IMAGE_FORMAT_PNG. |
IMAGE_FORMAT_PNG | 1 | PNG (lossless). |
IMAGE_FORMAT_JPEG | 2 | JPEG (lossy, honours quality). |
IMAGE_FORMAT_WEBP | 3 | WebP (lossy, honours quality). |
MouseButton#
Mouse button.
| Value | # | Description |
|---|---|---|
MOUSE_BUTTON_UNSPECIFIED | 0 | Not set. Treated as MOUSE_BUTTON_LEFT. |
MOUSE_BUTTON_LEFT | 1 | Primary button. |
MOUSE_BUTTON_RIGHT | 2 | Secondary button. |
MOUSE_BUTTON_MIDDLE | 3 | Middle button / wheel press. |
MOUSE_BUTTON_BACK | 4 | Back (button 4). |
MOUSE_BUTTON_FORWARD | 5 | Forward (button 5). |
ScrollUnit#
Unit of scroll deltas.
| Value | # | Description |
|---|---|---|
SCROLL_UNIT_UNSPECIFIED | 0 | Not set. Treated as SCROLL_UNIT_LINE. |
SCROLL_UNIT_LINE | 1 | Wheel notches / text lines. |
SCROLL_UNIT_PIXEL | 2 | Logical points (precise, trackpad-style scrolling). |
Key#
Named keys, independent of keyboard layout. Printable characters that are
not listed here are sent with KeyInput.character.
Values are grouped in numeric blocks with gaps left for additions within each block: modifiers 1-19, editing and navigation 20-59, function keys 60-99, letters 100-129, digits 130-149, punctuation 150-179, numeric keypad 180-219, system and lock keys 220-239, media and hardware keys 240-269, browser and launcher keys 270-299.
| Value | # | Description |
|---|---|---|
KEY_UNSPECIFIED | 0 | Not set. Rejected. |
KEY_SHIFT | 1 | Shift (either side). |
KEY_SHIFT_LEFT | 2 | Left Shift. |
KEY_SHIFT_RIGHT | 3 | Right Shift. |
KEY_CONTROL | 4 | Control (either side). |
KEY_CONTROL_LEFT | 5 | Left Control. |
KEY_CONTROL_RIGHT | 6 | Right Control. |
KEY_ALT | 7 | Alt / Option (either side). |
KEY_ALT_LEFT | 8 | Left Alt / Option. |
KEY_ALT_RIGHT | 9 | Right Alt / Option / AltGr. |
KEY_META | 10 | Meta: Command on macOS, Windows key on Windows, Super on Linux (either side). |
KEY_META_LEFT | 11 | Left Meta. |
KEY_META_RIGHT | 12 | Right Meta. |
KEY_FN | 13 | Fn (where the platform exposes it). |
KEY_CAPS_LOCK | 14 | Caps Lock. |
KEY_ENTER | 20 | Enter / Return. |
KEY_TAB | 21 | Tab. |
KEY_SPACE | 22 | Space bar. |
KEY_BACKSPACE | 23 | Backspace (Delete on macOS keyboards). |
KEY_DELETE | 24 | Forward delete. |
KEY_ESCAPE | 25 | Escape. |
KEY_INSERT | 26 | Insert. |
KEY_HOME | 27 | Home. |
KEY_END | 28 | End. |
KEY_PAGE_UP | 29 | Page Up. |
KEY_PAGE_DOWN | 30 | Page Down. |
KEY_ARROW_UP | 31 | Up arrow. |
KEY_ARROW_DOWN | 32 | Down arrow. |
KEY_ARROW_LEFT | 33 | Left arrow. |
KEY_ARROW_RIGHT | 34 | Right arrow. |
KEY_CONTEXT_MENU | 35 | Context menu / Application key. |
KEY_HELP | 36 | Help. |
KEY_F1 | 60 | F1. |
KEY_F2 | 61 | F2. |
KEY_F3 | 62 | F3. |
KEY_F4 | 63 | F4. |
KEY_F5 | 64 | F5. |
KEY_F6 | 65 | F6. |
KEY_F7 | 66 | F7. |
KEY_F8 | 67 | F8. |
KEY_F9 | 68 | F9. |
KEY_F10 | 69 | F10. |
KEY_F11 | 70 | F11. |
KEY_F12 | 71 | F12. |
KEY_F13 | 72 | F13. |
KEY_F14 | 73 | F14. |
KEY_F15 | 74 | F15. |
KEY_F16 | 75 | F16. |
KEY_F17 | 76 | F17. |
KEY_F18 | 77 | F18. |
KEY_F19 | 78 | F19. |
KEY_F20 | 79 | F20. |
KEY_F21 | 80 | F21. |
KEY_F22 | 81 | F22. |
KEY_F23 | 82 | F23. |
KEY_F24 | 83 | F24. |
KEY_A | 100 | Letter A (physical key position on a US layout). |
KEY_B | 101 | Letter B. |
KEY_C | 102 | Letter C. |
KEY_D | 103 | Letter D. |
KEY_E | 104 | Letter E. |
KEY_F | 105 | Letter F. |
KEY_G | 106 | Letter G. |
KEY_H | 107 | Letter H. |
KEY_I | 108 | Letter I. |
KEY_J | 109 | Letter J. |
KEY_K | 110 | Letter K. |
KEY_L | 111 | Letter L. |
KEY_M | 112 | Letter M. |
KEY_N | 113 | Letter N. |
KEY_O | 114 | Letter O. |
KEY_P | 115 | Letter P. |
KEY_Q | 116 | Letter Q. |
KEY_R | 117 | Letter R. |
KEY_S | 118 | Letter S. |
KEY_T | 119 | Letter T. |
KEY_U | 120 | Letter U. |
KEY_V | 121 | Letter V. |
KEY_W | 122 | Letter W. |
KEY_X | 123 | Letter X. |
KEY_Y | 124 | Letter Y. |
KEY_Z | 125 | Letter Z. |
KEY_DIGIT_0 | 130 | Digit 0 (top row). |
KEY_DIGIT_1 | 131 | Digit 1. |
KEY_DIGIT_2 | 132 | Digit 2. |
KEY_DIGIT_3 | 133 | Digit 3. |
KEY_DIGIT_4 | 134 | Digit 4. |
KEY_DIGIT_5 | 135 | Digit 5. |
KEY_DIGIT_6 | 136 | Digit 6. |
KEY_DIGIT_7 | 137 | Digit 7. |
KEY_DIGIT_8 | 138 | Digit 8. |
KEY_DIGIT_9 | 139 | Digit 9. |
KEY_MINUS | 150 | Minus / underscore. |
KEY_EQUAL | 151 | Equal / plus. |
KEY_BRACKET_LEFT | 152 | Left bracket / brace. |
KEY_BRACKET_RIGHT | 153 | Right bracket / brace. |
KEY_BACKSLASH | 154 | Backslash / pipe. |
KEY_SEMICOLON | 155 | Semicolon / colon. |
KEY_QUOTE | 156 | Quote / double quote. |
KEY_BACKQUOTE | 157 | Backquote / tilde. |
KEY_COMMA | 158 | Comma / less-than. |
KEY_PERIOD | 159 | Period / greater-than. |
KEY_SLASH | 160 | Slash / question mark. |
KEY_INTL_BACKSLASH | 161 | The extra key left of Z on ISO keyboards. |
KEY_NUMPAD_0 | 180 | Keypad 0. |
KEY_NUMPAD_1 | 181 | Keypad 1. |
KEY_NUMPAD_2 | 182 | Keypad 2. |
KEY_NUMPAD_3 | 183 | Keypad 3. |
KEY_NUMPAD_4 | 184 | Keypad 4. |
KEY_NUMPAD_5 | 185 | Keypad 5. |
KEY_NUMPAD_6 | 186 | Keypad 6. |
KEY_NUMPAD_7 | 187 | Keypad 7. |
KEY_NUMPAD_8 | 188 | Keypad 8. |
KEY_NUMPAD_9 | 189 | Keypad 9. |
KEY_NUMPAD_ADD | 190 | Keypad +. |
KEY_NUMPAD_SUBTRACT | 191 | Keypad -. |
KEY_NUMPAD_MULTIPLY | 192 | Keypad *. |
KEY_NUMPAD_DIVIDE | 193 | Keypad /. |
KEY_NUMPAD_DECIMAL | 194 | Keypad decimal point. |
KEY_NUMPAD_ENTER | 195 | Keypad Enter. |
KEY_NUMPAD_EQUAL | 196 | Keypad =. |
KEY_NUM_LOCK | 197 | Num Lock / Clear. |
KEY_PRINT_SCREEN | 220 | Print Screen / SysRq. |
KEY_SCROLL_LOCK | 221 | Scroll Lock. |
KEY_PAUSE | 222 | Pause / Break. |
KEY_MEDIA_PLAY_PAUSE | 240 | Play / pause toggle. |
KEY_MEDIA_STOP | 241 | Stop. |
KEY_MEDIA_NEXT | 242 | Next track. |
KEY_MEDIA_PREVIOUS | 243 | Previous track. |
KEY_VOLUME_UP | 244 | Volume up. |
KEY_VOLUME_DOWN | 245 | Volume down. |
KEY_VOLUME_MUTE | 246 | Mute toggle. |
KEY_BRIGHTNESS_UP | 247 | Brightness up. |
KEY_BRIGHTNESS_DOWN | 248 | Brightness down. |
KEY_EJECT | 249 | Eject. |
KEY_POWER | 250 | Power. |
KEY_SLEEP | 251 | Sleep. |
KEY_BROWSER_BACK | 270 | Browser back. |
KEY_BROWSER_FORWARD | 271 | Browser forward. |
KEY_BROWSER_REFRESH | 272 | Browser refresh. |
KEY_BROWSER_HOME | 273 | Browser home. |
KEY_BROWSER_SEARCH | 274 | Browser search. |
KEY_LAUNCH_MAIL | 275 | Launch mail client. |
KEY_LAUNCH_APP1 | 276 | Launch application 1 (often "My Computer" / Mission Control). |
KEY_LAUNCH_APP2 | 277 | Launch application 2 (often calculator / Launchpad). |
TextEntryMode#
How KeyboardType enters text.
| Value | # | Description |
|---|---|---|
TEXT_ENTRY_MODE_UNSPECIFIED | 0 | Not set. Treated as TEXT_ENTRY_MODE_KEYSTROKES. |
TEXT_ENTRY_MODE_KEYSTROKES | 1 | Synthesize a key press per character (Unicode injection for characters without a key). Honors the caret position and triggers key handlers. |
TEXT_ENTRY_MODE_INSERT | 2 | Insert the text in one operation (accessibility value insertion or paste). Fast, but inserts relative to the element's own insertion point and may not fire per-key handlers. |
Types in cua/env/v1/driver.proto#
ListToolsRequest#
Request for DriverService.ListTools.
No fields.
ToolInfo#
A tool description, mirroring the MCP tool shape.
| Field | # | Type | Description |
|---|---|---|---|
name | 1 | string | Tool name, for example "click" or "get_window_state". |
description | 2 | string | Human-readable description. |
input_schema_json | 3 | string | JSON Schema of the arguments object, as JSON text. |
output_schema_json | 4 | string | JSON Schema of the structured result, as JSON text. Empty when the tool declares none. |
read_only | 5 | bool | True if the tool does not modify guest state. |
destructive | 6 | bool | True if the tool may perform destructive changes. |
ListToolsResponse#
Response for DriverService.ListTools.
| Field | # | Type | Description |
|---|---|---|---|
tools | 1 | repeated ToolInfo | Available tools. |
contract_version | 2 | string | cua-driver contract version the registry implements. |
CallToolRequest#
Request for DriverService.CallTool.
| Field | # | Type | Description |
|---|---|---|---|
name | 1 | string | Tool name. |
arguments_json | 2 | string | Arguments object as JSON text. Empty means {}. |
timeout | 3 | google.protobuf.Duration | Abort the call after this long. Unset means the tool's own default. |
ToolImage#
An image produced by a tool.
| Field | # | Type | Description |
|---|---|---|---|
data | 1 | bytes | Encoded bytes. |
mime_type | 2 | string | Media type, for example "image/png". |
ToolContent#
One part of a tool result, mirroring MCP content parts.
| Field | # | Type | Description |
|---|---|---|---|
text | 1 | string (oneof content) | Text. |
image | 2 | ToolImage (oneof content) | An image. |
json | 3 | string (oneof content) | JSON text. |
CallToolResponse#
Response for DriverService.CallTool. Tool-level failures are reported
with is_error, not as gRPC errors; gRPC errors mean the call could not be
dispatched (unknown tool, invalid JSON, timeout).
| Field | # | Type | Description |
|---|---|---|---|
content | 1 | repeated ToolContent | Result parts in order. |
structured_json | 2 | string | Structured result as JSON text, when the tool declares an output schema. |
is_error | 3 | bool | True if the tool reported a failure. |
Types in cua/env/v1/windows.proto#
AppInfo#
Application that owns a window.
| Field | # | Type | Description |
|---|---|---|---|
name | 1 | string | Display name, for example "Firefox". |
app_id | 2 | string | Platform app identifier: bundle id on macOS, .desktop id on Linux, AppUserModelID or executable name on Windows. Empty when unknown. |
pid | 3 | uint32 | Guest process id of the owning process. Guest-local; never a host id. |
WindowInfo#
Description of a window.
| Field | # | Type | Description |
|---|---|---|---|
ref | 1 | WindowRef | Handle. |
title | 2 | string | Title. |
app | 3 | AppInfo | Owning application. |
bounds | 4 | Rect | Frame in global logical points. Advisory: it is a snapshot and may be stale by the time it is used. Media sessions and screenshots report their own authoritative geometry. |
display_id | 5 | string | Display containing most of the window. |
state | 6 | WindowState | State. |
kind | 7 | WindowKind | Role. |
focused | 8 | bool | True if this window has keyboard focus. |
on_screen | 9 | bool | True if any part is on screen (not minimized, hidden or off-desktop). |
z_order | 10 | uint32 | Front-to-back stacking order among listed windows, 0 is frontmost. |
WindowFilter#
Filter for window listing and watching. All set fields must match.
| Field | # | Type | Description |
|---|---|---|---|
title_contains | 1 | string | Case-insensitive substring of the title. |
app_name_contains | 2 | string | Case-insensitive substring of the app name. |
app_id | 3 | string | Exact AppInfo.app_id. |
pid | 4 | uint32 | Owning process id. |
display_id | 5 | string | Only windows on this display. |
on_screen_only | 6 | bool | Only on-screen windows. |
include_system | 7 | bool | Include WINDOW_KIND_SYSTEM and WINDOW_KIND_PHANTOM windows, which are omitted by default. |
ListWindowsRequest#
Request for WindowsService.ListWindows.
| Field | # | Type | Description |
|---|---|---|---|
filter | 1 | WindowFilter | Filter. |
ListWindowsResponse#
Response for WindowsService.ListWindows.
| Field | # | Type | Description |
|---|---|---|---|
windows | 1 | repeated WindowInfo | Matching windows, front to back. |
WatchWindowsRequest#
Request for WindowsService.WatchWindows.
| Field | # | Type | Description |
|---|---|---|---|
filter | 1 | WindowFilter | Filter applied to every event. |
keepalive_interval | 2 | google.protobuf.Duration | Interval between keepalive messages. Unset means 30 seconds. |
WindowSnapshot#
Initial state of a window watch.
| Field | # | Type | Description |
|---|---|---|---|
windows | 1 | repeated WindowInfo | All matching windows, front to back. |
WindowClosed#
A window was closed or no longer matches the filter.
| Field | # | Type | Description |
|---|---|---|---|
ref | 1 | WindowRef | The window. |
WatchWindowsResponse#
One message of the WindowsService.WatchWindows stream.
| Field | # | Type | Description |
|---|---|---|---|
snapshot | 1 | WindowSnapshot (oneof event) | First message: current matching windows. |
created | 2 | WindowInfo (oneof event) | A new window appeared (or started matching). |
updated | 3 | WindowInfo (oneof event) | A window changed title, bounds, state, focus or z-order. Carries the full new description. |
closed | 4 | WindowClosed (oneof event) | A window closed. |
keepalive | 5 | KeepAlive (oneof event) | Idle stream heartbeat. |
GetWindowRequest#
Request for WindowsService.GetWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
GetWindowResponse#
Response for WindowsService.GetWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window. |
ActivateWindowRequest#
Request for WindowsService.ActivateWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
ActivateWindowResponse#
Response for WindowsService.ActivateWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
SetWindowBoundsRequest#
Request for WindowsService.SetWindowBounds.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
position | 2 | Point | New top-left corner in global logical points. Unset keeps the position. |
width | 3 | optional double | New width in logical points. Unset keeps the width. |
height | 4 | optional double | New height in logical points. Unset keeps the height. |
SetWindowBoundsResponse#
Response for WindowsService.SetWindowBounds.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. Its bounds may differ from the request when the window manager or app constrains them. |
MinimizeWindowRequest#
Request for WindowsService.MinimizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
MinimizeWindowResponse#
Response for WindowsService.MinimizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
MaximizeWindowRequest#
Request for WindowsService.MaximizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
MaximizeWindowResponse#
Response for WindowsService.MaximizeWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
RestoreWindowRequest#
Request for WindowsService.RestoreWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
RestoreWindowResponse#
Response for WindowsService.RestoreWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowInfo | The window afterwards. |
CloseWindowRequest#
Request for WindowsService.CloseWindow.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Which window. |
force | 2 | bool | Destroy the window without giving the app a chance to object (for example to an unsaved-changes prompt). On most platforms this kills the owning process. |
CloseWindowResponse#
Response for WindowsService.CloseWindow.
| Field | # | Type | Description |
|---|---|---|---|
closed | 1 | bool | True if the window is gone. False when the app kept it open (for example it showed a save prompt). |
AppSpec#
Identifies an application to launch.
| Field | # | Type | Description |
|---|---|---|---|
app_id | 1 | string (oneof app) | Platform app identifier: macOS bundle id ("org.mozilla.firefox"), Linux .desktop id ("firefox.desktop"), Windows AppUserModelID. |
executable | 2 | string (oneof app) | Executable name (resolved on PATH) or absolute path. |
name | 3 | string (oneof app) | Display name, resolved by the platform (for example "Firefox"). The least precise option. |
LaunchAppRequest#
Request for WindowsService.LaunchApp.
| Field | # | Type | Description |
|---|---|---|---|
app | 1 | AppSpec | What to launch. |
args | 2 | repeated string | Arguments passed verbatim, never through a shell. |
env | 3 | map of string to string | Extra environment variables. |
cwd | 4 | string | Working directory. Empty means the user's home. |
delivery | 5 | Delivery | DELIVERY_BACKGROUND launches without activating the app or stealing focus (where the platform allows). DELIVERY_FOREGROUND activates it. |
wait_for_window | 6 | google.protobuf.Duration | Wait up to this long for the app's first window and return it. Unset returns immediately after the process starts. |
LaunchAppResponse#
Response for WindowsService.LaunchApp.
| Field | # | Type | Description |
|---|---|---|---|
pid | 1 | uint32 | Guest process id of the launched (or already running, for single- instance apps) process. 0 when the platform does not report it. |
windows | 2 | repeated WindowInfo | Windows that appeared within wait_for_window. |
OpenRequest#
Request for WindowsService.Open.
| Field | # | Type | Description |
|---|---|---|---|
url | 1 | string (oneof target) | A URL, for example "https://example.com" or "mailto:a@b". |
path | 2 | string (oneof target) | A guest file or directory path. |
with_app | 3 | AppSpec | Open with this app instead of the default handler. |
delivery | 4 | Delivery | Delivery, as for LaunchAppRequest.delivery. |
OpenResponse#
Response for WindowsService.Open.
| Field | # | Type | Description |
|---|---|---|---|
pid | 1 | uint32 | Guest process id of the handling app, 0 when unknown. |
WindowState#
Window state.
| Value | # | Description |
|---|---|---|
WINDOW_STATE_UNSPECIFIED | 0 | Not reported. |
WINDOW_STATE_NORMAL | 1 | Normal (neither minimized, maximized nor fullscreen). |
WINDOW_STATE_MINIMIZED | 2 | Minimized / iconified. |
WINDOW_STATE_MAXIMIZED | 3 | Maximized / zoomed. |
WINDOW_STATE_FULLSCREEN | 4 | Fullscreen. |
WINDOW_STATE_HIDDEN | 5 | Hidden (app hidden, withdrawn, or on another virtual desktop). |
WindowKind#
Broad window role.
| Value | # | Description |
|---|---|---|
WINDOW_KIND_UNSPECIFIED | 0 | Not reported. |
WINDOW_KIND_STANDARD | 1 | A normal application window. |
WINDOW_KIND_DIALOG | 2 | A dialog or sheet. |
WINDOW_KIND_PANEL | 3 | A floating panel or utility window. |
WINDOW_KIND_MENU | 4 | A menu or popup. |
WINDOW_KIND_TOOLTIP | 5 | A tooltip. |
WINDOW_KIND_SYSTEM | 6 | Desktop chrome: dock, menu bar strip, panel, wallpaper. |
WINDOW_KIND_PHANTOM | 7 | Invisible or indicator surfaces (for example the macOS screen-recording indicator, 1x1 helper windows). Hidden unless include_system is set. |
Types in cua/env/v1/accessibility.proto#
AccessibilityNode#
One accessibility element. Trees are returned flattened in pre-order (parents before children) to avoid unbounded message nesting.
| Field | # | Type | Description |
|---|---|---|---|
element_id | 1 | string | Element id, unique within its snapshot. |
parent_id | 2 | string | Parent element id, empty for the root. |
depth | 3 | uint32 | Depth below the root (root is 0). |
role | 4 | string | Normalized role, for example "button", "text_field", "menu_item", "window". Platform roles are mapped to one vocabulary. |
native_role | 5 | string | The platform's native role string (for example "AXButton", "ControlType.Button", "push button"), for debugging. |
name | 6 | string | Accessible name / label. |
value | 7 | string | Current value (text content, slider value), when it has one. |
description | 8 | string | Accessible description / help text. |
bounds | 9 | Rect | Frame in global logical points, when known. |
states | 10 | repeated string | Normalized states, for example "focused", "enabled", "selected", "checked", "expanded", "editable", "offscreen". |
actions | 11 | repeated AccessibilityAction | Actions the element supports (see AccessibilityAction). |
attributes | 12 | map of string to string | Other attributes the platform exposes, stringified. |
GetTreeRequest#
Request for AccessibilityService.GetTree.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Window whose tree to read. Unset means the focused window. |
max_depth | 2 | uint32 | Maximum depth below the root. 0 means unlimited (subject to the server's node limit). |
include_hidden | 3 | bool | Include elements that are offscreen or not visible. |
max_nodes | 4 | uint32 | Stop after this many nodes. 0 means the server limit (normally 5000). |
GetTreeResponse#
Response for AccessibilityService.GetTree.
| Field | # | Type | Description |
|---|---|---|---|
snapshot_id | 1 | string | Snapshot id for later Act calls. |
window | 2 | WindowRef | The window the tree belongs to. |
nodes | 3 | repeated AccessibilityNode | Nodes in pre-order. |
truncated | 4 | bool | True if max_depth or max_nodes cut the tree short. |
AccessibilityQuery#
Query for AccessibilityService.Find. All set fields must match.
| Field | # | Type | Description |
|---|---|---|---|
role | 1 | string | Normalized role. |
name | 2 | string | Exact accessible name. |
name_contains | 3 | string | Case-insensitive substring of the accessible name. |
value_contains | 4 | string | Case-insensitive substring of the value. |
states | 5 | repeated string | Required states. |
FindRequest#
Request for AccessibilityService.Find.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Window to search. Unset means the focused window. |
query | 2 | AccessibilityQuery | What to look for. |
max_results | 3 | uint32 | Maximum matches. 0 means 50. |
FindResponse#
Response for AccessibilityService.Find.
| Field | # | Type | Description |
|---|---|---|---|
snapshot_id | 1 | string | Snapshot id for later Act calls. |
nodes | 2 | repeated AccessibilityNode | Matching elements (their parent_id refers to elements not necessarily included). |
ElementRef#
Reference to an element within a snapshot.
| Field | # | Type | Description |
|---|---|---|---|
snapshot_id | 1 | string | Snapshot the element id came from. |
element_id | 2 | string | Element id. |
ActRequest#
Request for AccessibilityService.Act.
| Field | # | Type | Description |
|---|---|---|---|
element | 1 | ElementRef | Which element. |
action | 2 | AccessibilityAction | Which action. |
value | 3 | string | New value for ACCESSIBILITY_ACTION_SET_VALUE. |
custom_action | 4 | string | Native action name for ACCESSIBILITY_ACTION_CUSTOM. |
delivery | 5 | Delivery | Delivery mode. Accessibility actions are background-safe on most platforms; DELIVERY_FOREGROUND activates the window first. |
ActResponse#
Response for AccessibilityService.Act.
| Field | # | Type | Description |
|---|---|---|---|
report | 1 | DeliveryReport | What actually happened. |
AccessibilityAction#
Accessibility action.
| Value | # | Description |
|---|---|---|
ACCESSIBILITY_ACTION_UNSPECIFIED | 0 | Not set. Rejected. |
ACCESSIBILITY_ACTION_PRESS | 1 | Press / invoke / click. |
ACCESSIBILITY_ACTION_FOCUS | 2 | Move keyboard focus to the element. |
ACCESSIBILITY_ACTION_SET_VALUE | 3 | Replace the element's value with ActRequest.value. |
ACCESSIBILITY_ACTION_INCREMENT | 4 | Increment (sliders, steppers). |
ACCESSIBILITY_ACTION_DECREMENT | 5 | Decrement. |
ACCESSIBILITY_ACTION_SHOW_MENU | 6 | Open the element's context menu. |
ACCESSIBILITY_ACTION_EXPAND | 7 | Expand (tree items, disclosure triangles, combo boxes). |
ACCESSIBILITY_ACTION_COLLAPSE | 8 | Collapse. |
ACCESSIBILITY_ACTION_SELECT | 9 | Select (list items, tabs). |
ACCESSIBILITY_ACTION_SCROLL_INTO_VIEW | 10 | Scroll the element into view. |
ACCESSIBILITY_ACTION_CUSTOM | 11 | A platform-specific action named by ActRequest.custom_action. |