Computer and driver
cua.env.v1 ComputerService and DriverService: screenshots, pointer, keyboard, clipboard, displays and Cua Driver tools.
cua.env.v1 ComputerService and DriverService: screenshots, pointer, keyboard, clipboard, displays and Cua Driver tools.
Desktop control. These services need the desktop provider; input goes through Cua Driver, which DriverService also exposes tool by tool.
Source: libs/cua/proto/cua/env/v1/computer.proto, libs/cua/proto/cua/env/v1/driver.proto.
cua.env.v1.ComputerService
Screen capture and synthetic input.
Implemented by calling the cua-driver-core tool registry in-process (not
over MCP). Every input RPC states its delivery mode explicitly and reports
what actually happened in a DeliveryReport.
Coordinates: screenshots report both their pixel size and the logical
region they cover, so clients never guess a Retina scale. Input can be
expressed in screen points, window points, normalized units or the pixels
of a specific screenshot (see CoordinateSpace); the server performs the
single conversion to native coordinates.
/cua.env.v1.ComputerService/Screenshot, unary: ScreenshotRequest to ScreenshotResponse.
Captures a display, a window or a region of either.
/cua.env.v1.ComputerService/Pointer, unary: PointerRequest to PointerResponse.
Performs one pointer (mouse) operation.
/cua.env.v1.ComputerService/Keyboard, unary: KeyboardRequest to KeyboardResponse.
Performs one keyboard operation.
/cua.env.v1.ComputerService/GetClipboard, unary: GetClipboardRequest to GetClipboardResponse.
Reads the clipboard.
/cua.env.v1.ComputerService/SetClipboard, unary: SetClipboardRequest to SetClipboardResponse.
Replaces the clipboard.
/cua.env.v1.ComputerService/GetCursorPosition, unary: GetCursorPositionRequest to GetCursorPositionResponse.
Returns the user-visible pointer position.
/cua.env.v1.ComputerService/ListDisplays, unary: ListDisplaysRequest to ListDisplaysResponse.
Lists displays attached to the guest desktop.
cua.env.v1.DriverService
Passthrough to the full cua-driver tool registry.
Every cua-driver tool is reachable here without new protos. The same
registry is exposed to agents as streamable-HTTP MCP at /mcp on the same
port. Arguments and structured results are JSON text so that tool schemas
evolve with the cua-driver contract, not with this package.
| RPC | Request | Response | Kind |
|---|---|---|---|
ListTools | ListToolsRequest | ListToolsResponse | unary |
CallTool | CallToolRequest | CallToolResponse | unary |
/cua.env.v1.DriverService/ListTools, unary: ListToolsRequest to ListToolsResponse.
Lists the tools the linked cua-driver registry exposes on this platform.
/cua.env.v1.DriverService/CallTool, unary: CallToolRequest to CallToolResponse.
Invokes one tool.
cua/env/v1/computer.proto#Request for ComputerService.Screenshot.
| Field | # | Type | Description |
|---|---|---|---|
display_id | 1 | string (oneof source) | A display id, or "primary". |
window | 2 | WindowRef (oneof source) | A single window. Captured without occluding windows where the platform supports it (see capability "window_stream"). |
region | 3 | Rect | Sub-region in logical points, relative to the source's top-left corner. Unset captures the whole source. |
format | 4 | ImageFormat | Output format. |
quality | 5 | uint32 | Quality 1 to 100 for lossy formats. 0 means 80. |
max_dimension | 6 | uint32 | Downscale so the longer edge is at most this many pixels, preserving aspect ratio. 0 means no limit (native resolution). |
include_cursor | 7 | bool | Draw the pointer into the image. |
Response for ComputerService.Screenshot.
| Field | # | Type | Description |
|---|---|---|---|
image | 1 | bytes | Encoded image bytes. |
format | 2 | ImageFormat | Format of image. |
image_size | 3 | PixelSize | Pixel size of image (after any downscale). |
native_size | 4 | PixelSize | Pixel size of the captured area at native (physical) resolution. |
scale | 5 | double | Image pixels per logical point. To convert an image pixel to a screen point: screen = logical_bounds.origin + pixel / scale. |
logical_bounds | 6 | Rect | The captured area in global logical points (COORDINATE_SPACE_SCREEN). |
screenshot_id | 7 | string | Id to reference this capture's pixel space in later input requests (COORDINATE_SPACE_SCREENSHOT). Valid until the geometry of the source changes; the server keeps at least the 16 most recent ids. |
display_id | 8 | string | Display the capture came from (for a window capture, the display containing most of the window). |
captured_at | 9 | google.protobuf.Timestamp | When the frame was captured. |
Where and how an input request is delivered.
| Field | # | Type | Description |
|---|---|---|---|
window | 1 | WindowRef | Window to deliver to. Unset delivers to whatever is under the pointer (pointer) or focused (keyboard). Required for DELIVERY_BACKGROUND on most platforms and for COORDINATE_SPACE_WINDOW. |
display_id | 2 | string | Display for COORDINATE_SPACE_NORMALIZED without a window. Empty means the primary display. |
delivery | 3 | Delivery | Delivery mode. |
space | 4 | CoordinateSpace | Coordinate space of every point in the request. |
screenshot_id | 5 | string | Required with COORDINATE_SPACE_SCREENSHOT: which screenshot's pixel space the points refer to. |
Click (press and release) one or more times.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to click. Unset clicks at the current pointer position. |
button | 2 | MouseButton | Which button. |
count | 3 | uint32 | Number of clicks: 1 single, 2 double, 3 triple. 0 means 1. |
modifiers | 4 | repeated Key | Modifier keys held during the click. |
Move the pointer.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Destination. |
duration | 2 | google.protobuf.Duration | Animate the move over this duration. Unset jumps instantly. |
Press a button without releasing it.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to press. Unset presses at the current position. |
button | 2 | MouseButton | Which button. |
Release a pressed button.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to release. Unset releases at the current position. |
button | 2 | MouseButton | Which button. |
Press at from, move along path to to, release.
| Field | # | Type | Description |
|---|---|---|---|
from | 1 | Point | Start point. |
to | 2 | Point | End point. |
path | 3 | repeated Point | Optional intermediate points, in order. |
button | 4 | MouseButton | Which button. |
duration | 5 | google.protobuf.Duration | Total drag duration. Unset means 300 ms. |
modifiers | 6 | repeated Key | Modifier keys held during the drag. |
Scroll by a delta.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Where to scroll. Unset scrolls at the current pointer position. The pointer is not moved for background delivery. |
delta_x | 2 | double | Horizontal delta; positive scrolls content right. |
delta_y | 3 | double | Vertical delta; positive scrolls content down. |
unit | 4 | ScrollUnit | Unit of the deltas. |
Request for ComputerService.Pointer.
| Field | # | Type | Description |
|---|---|---|---|
target | 1 | InputTarget | Target, delivery and coordinate space. |
click | 2 | PointerClick (oneof action) | Click. |
move | 3 | PointerMove (oneof action) | Move. |
down | 4 | PointerDown (oneof action) | Button down. |
up | 5 | PointerUp (oneof action) | Button up. |
drag | 6 | PointerDrag (oneof action) | Drag. |
scroll | 7 | PointerScroll (oneof action) | Scroll. |
Response for ComputerService.Pointer.
| Field | # | Type | Description |
|---|---|---|---|
report | 1 | DeliveryReport | What actually happened. |
cursor_position | 2 | Point | User-visible pointer position afterwards, in screen points. |
One key, either named or as a character.
| Field | # | Type | Description |
|---|---|---|---|
named | 1 | Key (oneof key) | A named key. |
character | 2 | string (oneof key) | A single Unicode scalar value, mapped through the guest's current keyboard layout to the key (and Shift/AltGr) that produces it. Fails with ERROR_REASON_DELIVERY_FAILED if no key produces it; use KeyboardType for arbitrary text. |
Type text.
| Field | # | Type | Description |
|---|---|---|---|
text | 1 | string | UTF-8 text. Newlines are typed as Enter. |
mode | 2 | TextEntryMode | Entry mode. |
delay | 3 | google.protobuf.Duration | Delay between characters for TEXT_ENTRY_MODE_KEYSTROKES. Unset means no deliberate delay. |
Press and release one key, optionally with modifiers held.
| Field | # | Type | Description |
|---|---|---|---|
key | 1 | KeyInput | The key. |
modifiers | 2 | repeated Key | Modifiers held while the key is pressed. |
repeat | 3 | uint32 | Number of presses. 0 means 1. |
Press a chord: keys go down in order and up in reverse order (for example Control, Shift, T).
| Field | # | Type | Description |
|---|---|---|---|
keys | 1 | repeated KeyInput | Keys in press order. At least one. |
Press a key without releasing it.
| Field | # | Type | Description |
|---|---|---|---|
key | 1 | KeyInput | The key. |
Release a pressed key.
| Field | # | Type | Description |
|---|---|---|---|
key | 1 | KeyInput | The key. |
Request for ComputerService.Keyboard.
| Field | # | Type | Description |
|---|---|---|---|
target | 1 | InputTarget | Target and delivery. space and points are ignored. |
type | 2 | KeyboardType (oneof action) | Type text. |
press | 3 | KeyboardPress (oneof action) | Press one key. |
hotkey | 4 | KeyboardHotkey (oneof action) | Press a chord. |
down | 5 | KeyboardDown (oneof action) | Key down. |
up | 6 | KeyboardUp (oneof action) | Key up. |
Response for ComputerService.Keyboard.
| Field | # | Type | Description |
|---|---|---|---|
report | 1 | DeliveryReport | What actually happened. |
Clipboard content.
| Field | # | Type | Description |
|---|---|---|---|
text | 1 | optional string | Plain UTF-8 text. At most 1 MiB. Unset means no text flavor. |
file_paths | 2 | repeated string | Absolute guest paths of files on the clipboard (file copy/paste). To put host files on the guest clipboard, upload them first with FilesystemService, then set their paths here. Requires capability "clipboard.files". |
image_png | 3 | optional bytes | A PNG image (for example a copied screenshot). At most 16 MiB. Requires capability "clipboard.image". |
Request for ComputerService.GetClipboard.
No fields.
Response for ComputerService.GetClipboard.
| Field | # | Type | Description |
|---|---|---|---|
content | 1 | ClipboardContent | Current content. Only flavors the driver understands are reported. |
generation | 2 | uint64 | Clipboard generation counter; changes whenever the content changes. |
Request for ComputerService.SetClipboard.
| Field | # | Type | Description |
|---|---|---|---|
content | 1 | ClipboardContent | New content. Replaces every flavor. |
Response for ComputerService.SetClipboard.
| Field | # | Type | Description |
|---|---|---|---|
generation | 1 | uint64 | Generation after the write. |
Request for ComputerService.GetCursorPosition.
No fields.
Response for ComputerService.GetCursorPosition.
| Field | # | Type | Description |
|---|---|---|---|
position | 1 | Point | Position in global logical points (COORDINATE_SPACE_SCREEN). |
display_id | 2 | string | Display containing the pointer. |
Request for ComputerService.ListDisplays.
No fields.
Response for ComputerService.ListDisplays.
| Field | # | Type | Description |
|---|---|---|---|
displays | 1 | repeated Display | Attached displays, primary first. |
Encoded image format.
| Value | # | Description |
|---|---|---|
IMAGE_FORMAT_UNSPECIFIED | 0 | Not set. Treated as IMAGE_FORMAT_PNG. |
IMAGE_FORMAT_PNG | 1 | PNG (lossless). |
IMAGE_FORMAT_JPEG | 2 | JPEG (lossy, honours quality). |
IMAGE_FORMAT_WEBP | 3 | WebP (lossy, honours quality). |
Mouse button.
| Value | # | Description |
|---|---|---|
MOUSE_BUTTON_UNSPECIFIED | 0 | Not set. Treated as MOUSE_BUTTON_LEFT. |
MOUSE_BUTTON_LEFT | 1 | Primary button. |
MOUSE_BUTTON_RIGHT | 2 | Secondary button. |
MOUSE_BUTTON_MIDDLE | 3 | Middle button / wheel press. |
MOUSE_BUTTON_BACK | 4 | Back (button 4). |
MOUSE_BUTTON_FORWARD | 5 | Forward (button 5). |
Unit of scroll deltas.
| Value | # | Description |
|---|---|---|
SCROLL_UNIT_UNSPECIFIED | 0 | Not set. Treated as SCROLL_UNIT_LINE. |
SCROLL_UNIT_LINE | 1 | Wheel notches / text lines. |
SCROLL_UNIT_PIXEL | 2 | Logical points (precise, trackpad-style scrolling). |
Named keys, independent of keyboard layout. Printable characters that are
not listed here are sent with KeyInput.character.
Values are grouped in numeric blocks with gaps left for additions within each block: modifiers 1-19, editing and navigation 20-59, function keys 60-99, letters 100-129, digits 130-149, punctuation 150-179, numeric keypad 180-219, system and lock keys 220-239, media and hardware keys 240-269, browser and launcher keys 270-299.
| Value | # | Description |
|---|---|---|
KEY_UNSPECIFIED | 0 | Not set. Rejected. |
KEY_SHIFT | 1 | Shift (either side). |
KEY_SHIFT_LEFT | 2 | Left Shift. |
KEY_SHIFT_RIGHT | 3 | Right Shift. |
KEY_CONTROL | 4 | Control (either side). |
KEY_CONTROL_LEFT | 5 | Left Control. |
KEY_CONTROL_RIGHT | 6 | Right Control. |
KEY_ALT | 7 | Alt / Option (either side). |
KEY_ALT_LEFT | 8 | Left Alt / Option. |
KEY_ALT_RIGHT | 9 | Right Alt / Option / AltGr. |
KEY_META | 10 | Meta: Command on macOS, Windows key on Windows, Super on Linux (either side). |
KEY_META_LEFT | 11 | Left Meta. |
KEY_META_RIGHT | 12 | Right Meta. |
KEY_FN | 13 | Fn (where the platform exposes it). |
KEY_CAPS_LOCK | 14 | Caps Lock. |
KEY_ENTER | 20 | Enter / Return. |
KEY_TAB | 21 | Tab. |
KEY_SPACE | 22 | Space bar. |
KEY_BACKSPACE | 23 | Backspace (Delete on macOS keyboards). |
KEY_DELETE | 24 | Forward delete. |
KEY_ESCAPE | 25 | Escape. |
KEY_INSERT | 26 | Insert. |
KEY_HOME | 27 | Home. |
KEY_END | 28 | End. |
KEY_PAGE_UP | 29 | Page Up. |
KEY_PAGE_DOWN | 30 | Page Down. |
KEY_ARROW_UP | 31 | Up arrow. |
KEY_ARROW_DOWN | 32 | Down arrow. |
KEY_ARROW_LEFT | 33 | Left arrow. |
KEY_ARROW_RIGHT | 34 | Right arrow. |
KEY_CONTEXT_MENU | 35 | Context menu / Application key. |
KEY_HELP | 36 | Help. |
KEY_F1 | 60 | F1. |
KEY_F2 | 61 | F2. |
KEY_F3 | 62 | F3. |
KEY_F4 | 63 | F4. |
KEY_F5 | 64 | F5. |
KEY_F6 | 65 | F6. |
KEY_F7 | 66 | F7. |
KEY_F8 | 67 | F8. |
KEY_F9 | 68 | F9. |
KEY_F10 | 69 | F10. |
KEY_F11 | 70 | F11. |
KEY_F12 | 71 | F12. |
KEY_F13 | 72 | F13. |
KEY_F14 | 73 | F14. |
KEY_F15 | 74 | F15. |
KEY_F16 | 75 | F16. |
KEY_F17 | 76 | F17. |
KEY_F18 | 77 | F18. |
KEY_F19 | 78 | F19. |
KEY_F20 | 79 | F20. |
KEY_F21 | 80 | F21. |
KEY_F22 | 81 | F22. |
KEY_F23 | 82 | F23. |
KEY_F24 | 83 | F24. |
KEY_A | 100 | Letter A (physical key position on a US layout). |
KEY_B | 101 | Letter B. |
KEY_C | 102 | Letter C. |
KEY_D | 103 | Letter D. |
KEY_E | 104 | Letter E. |
KEY_F | 105 | Letter F. |
KEY_G | 106 | Letter G. |
KEY_H | 107 | Letter H. |
KEY_I | 108 | Letter I. |
KEY_J | 109 | Letter J. |
KEY_K | 110 | Letter K. |
KEY_L | 111 | Letter L. |
KEY_M | 112 | Letter M. |
KEY_N | 113 | Letter N. |
KEY_O | 114 | Letter O. |
KEY_P | 115 | Letter P. |
KEY_Q | 116 | Letter Q. |
KEY_R | 117 | Letter R. |
KEY_S | 118 | Letter S. |
KEY_T | 119 | Letter T. |
KEY_U | 120 | Letter U. |
KEY_V | 121 | Letter V. |
KEY_W | 122 | Letter W. |
KEY_X | 123 | Letter X. |
KEY_Y | 124 | Letter Y. |
KEY_Z | 125 | Letter Z. |
KEY_DIGIT_0 | 130 | Digit 0 (top row). |
KEY_DIGIT_1 | 131 | Digit 1. |
KEY_DIGIT_2 | 132 | Digit 2. |
KEY_DIGIT_3 | 133 | Digit 3. |
KEY_DIGIT_4 | 134 | Digit 4. |
KEY_DIGIT_5 | 135 | Digit 5. |
KEY_DIGIT_6 | 136 | Digit 6. |
KEY_DIGIT_7 | 137 | Digit 7. |
KEY_DIGIT_8 | 138 | Digit 8. |
KEY_DIGIT_9 | 139 | Digit 9. |
KEY_MINUS | 150 | Minus / underscore. |
KEY_EQUAL | 151 | Equal / plus. |
KEY_BRACKET_LEFT | 152 | Left bracket / brace. |
KEY_BRACKET_RIGHT | 153 | Right bracket / brace. |
KEY_BACKSLASH | 154 | Backslash / pipe. |
KEY_SEMICOLON | 155 | Semicolon / colon. |
KEY_QUOTE | 156 | Quote / double quote. |
KEY_BACKQUOTE | 157 | Backquote / tilde. |
KEY_COMMA | 158 | Comma / less-than. |
KEY_PERIOD | 159 | Period / greater-than. |
KEY_SLASH | 160 | Slash / question mark. |
KEY_INTL_BACKSLASH | 161 | The extra key left of Z on ISO keyboards. |
KEY_NUMPAD_0 | 180 | Keypad 0. |
KEY_NUMPAD_1 | 181 | Keypad 1. |
KEY_NUMPAD_2 | 182 | Keypad 2. |
KEY_NUMPAD_3 | 183 | Keypad 3. |
KEY_NUMPAD_4 | 184 | Keypad 4. |
KEY_NUMPAD_5 | 185 | Keypad 5. |
KEY_NUMPAD_6 | 186 | Keypad 6. |
KEY_NUMPAD_7 | 187 | Keypad 7. |
KEY_NUMPAD_8 | 188 | Keypad 8. |
KEY_NUMPAD_9 | 189 | Keypad 9. |
KEY_NUMPAD_ADD | 190 | Keypad +. |
KEY_NUMPAD_SUBTRACT | 191 | Keypad -. |
KEY_NUMPAD_MULTIPLY | 192 | Keypad *. |
KEY_NUMPAD_DIVIDE | 193 | Keypad /. |
KEY_NUMPAD_DECIMAL | 194 | Keypad decimal point. |
KEY_NUMPAD_ENTER | 195 | Keypad Enter. |
KEY_NUMPAD_EQUAL | 196 | Keypad =. |
KEY_NUM_LOCK | 197 | Num Lock / Clear. |
KEY_PRINT_SCREEN | 220 | Print Screen / SysRq. |
KEY_SCROLL_LOCK | 221 | Scroll Lock. |
KEY_PAUSE | 222 | Pause / Break. |
KEY_MEDIA_PLAY_PAUSE | 240 | Play / pause toggle. |
KEY_MEDIA_STOP | 241 | Stop. |
KEY_MEDIA_NEXT | 242 | Next track. |
KEY_MEDIA_PREVIOUS | 243 | Previous track. |
KEY_VOLUME_UP | 244 | Volume up. |
KEY_VOLUME_DOWN | 245 | Volume down. |
KEY_VOLUME_MUTE | 246 | Mute toggle. |
KEY_BRIGHTNESS_UP | 247 | Brightness up. |
KEY_BRIGHTNESS_DOWN | 248 | Brightness down. |
KEY_EJECT | 249 | Eject. |
KEY_POWER | 250 | Power. |
KEY_SLEEP | 251 | Sleep. |
KEY_BROWSER_BACK | 270 | Browser back. |
KEY_BROWSER_FORWARD | 271 | Browser forward. |
KEY_BROWSER_REFRESH | 272 | Browser refresh. |
KEY_BROWSER_HOME | 273 | Browser home. |
KEY_BROWSER_SEARCH | 274 | Browser search. |
KEY_LAUNCH_MAIL | 275 | Launch mail client. |
KEY_LAUNCH_APP1 | 276 | Launch application 1 (often "My Computer" / Mission Control). |
KEY_LAUNCH_APP2 | 277 | Launch application 2 (often calculator / Launchpad). |
How KeyboardType enters text.
| Value | # | Description |
|---|---|---|
TEXT_ENTRY_MODE_UNSPECIFIED | 0 | Not set. Treated as TEXT_ENTRY_MODE_KEYSTROKES. |
TEXT_ENTRY_MODE_KEYSTROKES | 1 | Synthesize a key press per character (Unicode injection for characters without a key). Honors the caret position and triggers key handlers. |
TEXT_ENTRY_MODE_INSERT | 2 | Insert the text in one operation (accessibility value insertion or paste). Fast, but inserts relative to the element's own insertion point and may not fire per-key handlers. |
cua/env/v1/driver.proto#Request for DriverService.ListTools.
No fields.
A tool description, mirroring the MCP tool shape.
| Field | # | Type | Description |
|---|---|---|---|
name | 1 | string | Tool name, for example "click" or "get_window_state". |
description | 2 | string | Human-readable description. |
input_schema_json | 3 | string | JSON Schema of the arguments object, as JSON text. |
output_schema_json | 4 | string | JSON Schema of the structured result, as JSON text. Empty when the tool declares none. |
read_only | 5 | bool | True if the tool does not modify guest state. |
destructive | 6 | bool | True if the tool may perform destructive changes. |
Response for DriverService.ListTools.
| Field | # | Type | Description |
|---|---|---|---|
tools | 1 | repeated ToolInfo | Available tools. |
contract_version | 2 | string | cua-driver contract version the registry implements. |
Request for DriverService.CallTool.
| Field | # | Type | Description |
|---|---|---|---|
name | 1 | string | Tool name. |
arguments_json | 2 | string | Arguments object as JSON text. Empty means {}. |
timeout | 3 | google.protobuf.Duration | Abort the call after this long. Unset means the tool's own default. |
An image produced by a tool.
| Field | # | Type | Description |
|---|---|---|---|
data | 1 | bytes | Encoded bytes. |
mime_type | 2 | string | Media type, for example "image/png". |
One part of a tool result, mirroring MCP content parts.
| Field | # | Type | Description |
|---|---|---|---|
text | 1 | string (oneof content) | Text. |
image | 2 | ToolImage (oneof content) | An image. |
json | 3 | string (oneof content) | JSON text. |
Response for DriverService.CallTool. Tool-level failures are reported
with is_error, not as gRPC errors; gRPC errors mean the call could not be
dispatched (unknown tool, invalid JSON, timeout).
| Field | # | Type | Description |
|---|---|---|---|
content | 1 | repeated ToolContent | Result parts in order. |
structured_json | 2 | string | Structured result as JSON text, when the tool declares an output schema. |
is_error | 3 | bool | True if the tool reported a failure. |