Stream a desktop

Stream a Space desktop or one window with audio: open a media ticket, pick a codec and transport, and start from the streaming examples.

OpenMedia on the Space's cua-spacesd returns a short-lived ticket bound to one target, codec and policy; the media then flows over a WebSocket or QUIC.

cua spacesd targets http://10.0.0.5:3211 --windows    # displays and windows

import cua
 
opts = cua.SpaceStreamOptions(app_name="Firefox", audio=True, max_fps=30, codecs=["h264"])
ticket = await space.open_stream(opts)        # hand ticket.ws_url to a player
media = await space.attach_stream(ticket, frame_sink, audio_sink)  # encoded frames, as they arrive
await space.close_stream(ticket.media_session_id)

attach_stream delivers the media socket's encoded frames and audio packets as they arrive; keyframe gating, loss recovery and decoding are yours. The streaming client that does them for you, space.stream_session(...) with decoded frames, ships with Cua Spaces (source-available, FSL-1.1-MIT): it runs in the Spaces apps and in processes that register it, and raises HostCapabilityMissing elsewhere. The wire contract (StreamService, the RCDP messages and framing) is open source, so any client can attach to a ticket.

The same calls in TypeScript, Rust and Swift, from the examples in other languages. The first frame is a keyframe.

const ticket = await space.openStream(
  SpaceStreamOptions.create({ maxFps: opts.maxFps, maxDimension: opts.maxDimension }),
)
const session = await space.attachStream(ticket, counter, undefined)

const ticket = await space.openStream(
  SpaceStreamOptions.create({ maxFps: opts.maxFps, maxDimension: opts.maxDimension, codecs: ["h264"] }),
)

Option	Meaning
`window_id`, `app_name`, `display`	Target (none: the primary display).
`codecs`	Preference order: `h264`, `bgra`, `png`.
`max_fps`, `max_dimension`	Limits (`max_fps` 0 means 30).
`audio`	Add an Opus track.
`policy`	`view_only` (default), `background_only`, or `allow_activation` for input over the media socket.
`ticket_ttl_ms`	Ticket lifetime (default 60 s, at most 10 min).

Without the SDK: cua spacesd call <url> StreamService/OpenMedia '{"maxFps": 30}'. Agents use the MCP tool stream_endpoint. A Space without desktop_stream, window_stream or audio.desktop raises CapabilityMissing.

Transport and codecs#

	Details
WebSocket `/media?ticket=...` on 3211	Default. Works through the relay and browsers.
QUIC (ALPN `rcdp/2`) on UDP 3212	Lower latency when you reach the guest directly; certificate pinned from `OpenMedia`.
Video	H.264 (Annex B, SPS/PPS on every keyframe). Encode: VideoToolbox (hardware) on macOS, OpenH264 (software) in Linux Spaces. SDK decode: VideoToolbox (hardware) on macOS, OpenH264 elsewhere. Browser viewers decode with WebCodecs.
Audio	Opus 48 kHz stereo, 20 ms frames, same clock as video (A/V skew within ±40 ms).

NVENC, VA-API, QSV, AMF and Media Foundation are detected but not used yet. Per-app audio and microphone uplink need PipeWire (the Linux Spaces image).

The wire format is in libs/cua/proto/MEDIA.md.

Presence#

The operator and each agent join a Space with an identity and move a shared cursor on its desktop.

const session = await space.joinPresence(
  { id: `openkoalabots-${randomBytes(4).toString("hex")}`, displayName: b.displayName || "Operator", color: "", agent: false },
  10_000n,
)

await s.presence.session.updateCursor({ displayId: "", windowId: undefined, x: b.x ?? 0, y: b.y ?? 0, visible: true, pressed: false, shape: "arrow", shapeSource: "unspecified", atMs: 0, receivedMs: 0 })

Python: space.join_presence(identity, timeout_ms). Rust: space.join_presence(Identity { .. }, timeout). Swift: native.joinPresence(identity:timeoutMs:).

Examples#

Each example streams the desktop and one window with audio, clicks, and verifies, against CUA_ENV_URL and CUA_ENV_TOKEN: Rust, Python, TypeScript (Node), TypeScript (browser), Swift, Kotlin.

libs/cua/bench/streaming/scripts/space.sh start        # a throwaway local Space
eval "$(libs/cua/bench/streaming/scripts/space.sh env)"

libs/cua/bench/streaming benchmarks time to first frame, fps, bandwidth and latency per codec, transport and language against budgets.json.