Computer and media

Screenshots, pointer, keyboard, clipboard and displays of a sandbox desktop, and its video and audio streams.

Coordinates are screen pixels. cua-spacesd delegates input to Cua Driver in the guest; these calls need an image with a desktop.

Python programs usually use the high-level API instead: sb.mouse, sb.keyboard, sb.screen.

Desktop methods#

Methods of SpacesdClient.

SpacesdClient.screenshot#

Captures a display.

async def screenshot(self, options: Optional[ScreenshotOptions]) -> Screenshot

Returns Screenshot · Async · Raises CuaError

Example

import asyncio
 
import cua
 
async def main():
    guest = await cua.embedded().spacesd(URL, TOKEN)
    shot = await guest.screenshot(None)  # PNG by default
    await guest.click(shot.width / 2, shot.height / 2)
    print([shot.width > 0, shot.image[:4]])
    # [True, b'\x89PNG']
 
asyncio.run(main())

SpacesdClient.click#

Left click.

async def click(self, x: float, y: float) -> None

Async · Raises CuaError

SpacesdClient.double_click#

Double click.

async def double_click(self, x: float, y: float) -> None

Async · Raises CuaError

SpacesdClient.right_click#

Right click.

async def right_click(self, x: float, y: float) -> None

Async · Raises CuaError

SpacesdClient.move_to#

Moves the pointer.

async def move_to(self, x: float, y: float) -> None

Async · Raises CuaError

SpacesdClient.drag#

Drags with the left button.

async def drag(self, from_x: float, from_y: float, to_x: float, to_y: float) -> None

Async · Raises CuaError

SpacesdClient.scroll#

Scrolls by line deltas.

async def scroll(self, dx: float, dy: float) -> None

Async · Raises CuaError

SpacesdClient.type_text#

Types text.

async def type_text(self, text: str) -> None

Async · Raises CuaError

SpacesdClient.press#

Presses one key (enter, KEY_ESCAPE, a, ...).

async def press(self, key: str) -> None

Async · Raises CuaError

SpacesdClient.hotkey#

Presses a chord (["ctrl", "c"]).

async def hotkey(self, keys: List[str]) -> None

Async · Raises CuaError

SpacesdClient.pointer_json#

Full ComputerService.Pointer from proto3 JSON.

async def pointer_json(self, request_json: str) -> str

Returns String · Async · Raises CuaError

SpacesdClient.keyboard_json#

Full ComputerService.Keyboard from proto3 JSON.

async def keyboard_json(self, request_json: str) -> str

Returns String · Async · Raises CuaError

SpacesdClient.get_clipboard#

Clipboard text.

async def get_clipboard(self) -> Optional[str]

Returns Option<String> · Async · Raises CuaError

SpacesdClient.set_clipboard#

Sets clipboard text; returns the clipboard generation.

async def set_clipboard(self, text: str) -> int

Returns u64 · Async · Raises CuaError

SpacesdClient.cursor_position#

Pointer position.

async def cursor_position(self) -> Point

Returns Point · Async · Raises CuaError

SpacesdClient.displays#

Displays as proto3 JSON ([Display]).

async def displays(self) -> str

Returns String · Async · Raises CuaError

Screenshot record#

A captured screenshot.

Returned by Sandbox.guest_screenshot, Space.screenshot, SpacesdClient.screenshot.

ScreenshotOptions record#

Screenshot options.

ImageFormat enum#

Screenshot encoding.

ImageFormat.PNG
ImageFormat.JPEG
ImageFormat.WEBP

Point record#

A point.

Returned by SpacesdClient.cursor_position.

Media#

Media sessions deliver encoded frames and audio packets (FrameSink, AudioSink) to callbacks you implement.

Open a session#

Methods of SpacesdClient.

SpacesdClient.open_media#

Opens a media session and starts delivering encoded video frames and control events to frames. Decoding stays in the host language.

async def open_media(self, options: MediaOpenOptions, frames: FrameSink) -> MediaSession

Returns MediaSession · Async · Raises CuaError

SpacesdClient.open_media_with_audio#

SpacesdClient.open_media that also delivers audio packets to audio (set options.audio to negotiate audio tracks).

Two methods instead of an optional sink: optional callback objects do not lower in every binding generator.

async def open_media_with_audio(self, options: MediaOpenOptions, frames: FrameSink, audio: AudioSink) -> MediaSession

Returns MediaSession · Async · Raises CuaError

Media bridge#

Methods of Sandbox.

Sandbox.open_media_bridge#

Opens a media session and returns a loopback WebSocket bridge for a webview (daemon mode). open_media_json is a cua.env.v1.OpenMediaRequest in proto3 JSON (None: primary display).

async def open_media_bridge(self, open_media_json: Optional[str]) -> MediaBridge

Returns MediaBridge · Async · Raises CuaError

MediaSession#

A live media session.

Returned by Space.attach_stream, SpacesdClient.open_media, SpacesdClient.open_media_with_audio.

MediaSession.close#

Closes the socket and the session.

async def close(self) -> None

Async · Raises CuaError

MediaSession.request_keyframe#

StreamService.RequestKeyframe.

async def request_keyframe(self) -> None

Async · Raises CuaError

MediaSession.send_control#

Sends a JSON control message on the media socket (for example {"type":"request_keyframe","payload":{}} or interactive input).

def send_control(self, json: str) -> None

Raises CuaError

MediaSession.set_preferences_json#

StreamService.SetPreferences from proto3 JSON (the session id is filled in).

async def set_preferences_json(self, json: str) -> None

Async · Raises CuaError

MediaOpenOptions record#

Media session options.

MediaBridge record#

A media bridge for webviews (daemon mode).

Returned by Sandbox.open_media_bridge.

AudioSink#

Receives audio packets. Must return quickly.

AudioSink.on_audio#

An audio packet.

def on_audio(self, packet: AudioPacket) -> None

FrameSink#

Receives video frames and control events. Implementations must return quickly and hand off decoding.

FrameSink.on_event#

A control message or the socket close.

def on_event(self, event: MediaEvent) -> None

FrameSink.on_frame#

A video access unit.

def on_frame(self, frame: VideoFrame) -> None

AudioPacket record#

One audio packet (MEDIA.md §12.2).

MediaEvent record#

A control message (hello, session_opened, lifecycle, audio_config, stats, error, ...) or the close of the socket (kind = "closed", json = {"code":…, "reason":…}).

MediaStats record#

Delivery counters.

Returned by MediaSession.stats.

VideoFrame record#

One encoded video access unit.