Media
Video and audio streaming sessions and the sink callbacks you implement.
Video and audio streaming sessions and the sink callbacks you implement.
Media sessions deliver encoded frames and audio packets (FrameSink, AudioSink) to callbacks you implement. Decoded BGRA frames and PCM audio come with Cua Spaces: see Decoded media.
Methods of SpacesdClient.
SpacesdClient.open_media#Opens a media session and starts delivering encoded video frames and
control events to frames. Decoding stays in the host language.
async def open_media(self, options: MediaOpenOptions, frames: FrameSink) -> MediaSession| Parameter | Type | Default |
|---|---|---|
options | MediaOpenOptions | required |
frames | FrameSink | required |
Returns MediaSession · Async · Raises CuaError
SpacesdClient.open_media_with_audio#SpacesdClient.open_media that also delivers audio packets to
audio (set options.audio to negotiate audio tracks).
Two methods instead of an optional sink: optional callback objects do not lower in every binding generator.
async def open_media_with_audio(self, options: MediaOpenOptions, frames: FrameSink, audio: AudioSink) -> MediaSession| Parameter | Type | Default |
|---|---|---|
options | MediaOpenOptions | required |
frames | FrameSink | required |
audio | AudioSink | required |
Returns MediaSession · Async · Raises CuaError
Methods of Sandbox.
Sandbox.open_media_bridge#Opens a media session and returns a loopback WebSocket bridge for a
webview (daemon mode). open_media_json is a
cua.env.v1.OpenMediaRequest in proto3 JSON (None: primary display).
async def open_media_bridge(self, open_media_json: Optional[str]) -> MediaBridge| Parameter | Type | Default |
|---|---|---|
open_media_json | Option<String> | required |
Returns MediaBridge · Async · Raises CuaError
MediaSession#A live media session.
Returned by Space.attach_stream, SpacesdClient.open_media, SpacesdClient.open_media_with_audio.
| Method | Description |
|---|---|
close | Closes the socket and the session. |
request_keyframe | StreamService.RequestKeyframe. |
send_control | Sends a JSON control message on the media socket (for example {"type":"request_keyframe","payload":{}} or interactive input). |
set_preferences_json | StreamService.SetPreferences from proto3 JSON (the session id is filled in). |
| Accessor | Returns | Description |
|---|---|---|
codec() | String | Negotiated video codec (h264, bgra, png). |
is_closed() | bool | Whether the socket has closed. |
open_response_json() | String | OpenMediaResponse as proto3 JSON (ticket removed). |
session_id() | String | OpenMediaResponse.media_session_id. |
stats() | MediaStats | Delivery counters. |
MediaSession.close#Closes the socket and the session.
async def close(self) -> NoneAsync · Raises CuaError
MediaSession.request_keyframe#StreamService.RequestKeyframe.
async def request_keyframe(self) -> NoneAsync · Raises CuaError
MediaSession.send_control#Sends a JSON control message on the media socket (for example
{"type":"request_keyframe","payload":{}} or interactive input).
def send_control(self, json: str) -> None| Parameter | Type | Default |
|---|---|---|
json | String | required |
Raises CuaError
MediaSession.set_preferences_json#StreamService.SetPreferences from proto3 JSON (the session id is
filled in).
async def set_preferences_json(self, json: str) -> None| Parameter | Type | Default |
|---|---|---|
json | String | required |
Async · Raises CuaError
MediaOpenOptions record#Media session options.
| Field | Type | Default | Description |
|---|---|---|---|
display | Option<String> | None | Display id (primary when unset). Ignored when window_handle is set. |
window_handle / windowHandle | Option<String> | None | Window handle (from ListTargets / ListWindows). |
max_fps / maxFps | u32 | 0 | FPS cap (0 = driver default). |
max_dimension / maxDimension | u32 | 0 | Long-edge cap (0 = native). |
audio | bool | false | Also negotiate desktop/app audio (Opus). |
disable_video / disableVideo | bool | false | Audio only. |
request_json / requestJson | Option<String> | None | Extra OpenMediaRequest fields as proto3 JSON, merged over the above (escape hatch). |
MediaBridge record#A media bridge for webviews (daemon mode).
Returned by Sandbox.open_media_bridge.
| Field | Type | Default | Description |
|---|---|---|---|
ws_url / wsUrl | String | Loopback ws:// URL carrying a short-lived bridge ticket. Speaks rcdp wire v2 unchanged; no Fleet bearer or env token is exposed. | |
ticket | String | The bridge ticket (also accepted as subprotocol cua.ticket.<ticket>). | |
open_media_response_json / openMediaResponseJson | String | cua.env.v1.OpenMediaResponse as proto3 JSON (upstream ticket removed). |
AudioSink#Receives audio packets. Must return quickly.
You implement AudioSink and pass it to the SDK (a callback interface): subclass it in Python, implement the interface in TypeScript, the AudioSink protocol in Swift, the interface in Kotlin and the trait in Rust.
| Method | Description |
|---|---|
on_audio | An audio packet. |
AudioSink.on_audio#An audio packet.
def on_audio(self, packet: AudioPacket) -> None| Parameter | Type | Default |
|---|---|---|
packet | AudioPacket | required |
FrameSink#Receives video frames and control events. Implementations must return quickly and hand off decoding.
You implement FrameSink and pass it to the SDK (a callback interface): subclass it in Python, implement the interface in TypeScript, the FrameSink protocol in Swift, the interface in Kotlin and the trait in Rust.
FrameSink.on_event#A control message or the socket close.
def on_event(self, event: MediaEvent) -> None| Parameter | Type | Default |
|---|---|---|
event | MediaEvent | required |
FrameSink.on_frame#A video access unit.
def on_frame(self, frame: VideoFrame) -> None| Parameter | Type | Default |
|---|---|---|
frame | VideoFrame | required |
AudioPacket record#One audio packet (MEDIA.md §12.2).
| Field | Type | Default | Description |
|---|---|---|---|
track_id / trackId | u16 | Track id from NegotiatedAudio. | |
sequence | u32 | Per-track sequence (wraps mod 2³²). | |
pts_us / ptsUs | u64 | Media-clock time of the first sample (µs). | |
frame_samples / frameSamples | u16 | Samples per channel. | |
config_epoch / configEpoch | u8 | Must match the latest audio_config of the track. | |
discontinuity | bool | The previous packet was dropped or the track restarted. | |
dtx | bool | DTX / comfort-noise frame. | |
data | Vec<u8> | One Opus packet or interleaved s16le PCM. |
MediaEvent record#A control message (hello, session_opened, lifecycle,
audio_config, stats, error, ...) or the close of the socket
(kind = "closed", json = {"code":…, "reason":…}).
| Field | Type | Default | Description |
|---|---|---|---|
kind | String | The message type. | |
json | String | The whole message as JSON. |
MediaStats record#Delivery counters.
Returned by MediaSession.stats.
| Field | Type | Default | Description |
|---|---|---|---|
frames | u64 | Video frames delivered. | |
frames_dropped / framesDropped | u64 | Video frames dropped because the sink fell behind. | |
audio_packets / audioPackets | u64 | Audio packets delivered. | |
malformed | u64 | Malformed binary messages ignored. | |
events | u64 | Control events delivered. |
VideoFrame record#One encoded video access unit.
| Field | Type | Default | Description |
|---|---|---|---|
sequence | u64 | Per-target frame sequence (may start anywhere; gaps mean replaced frames). | |
codec | String | h264, bgra or png. | |
keyframe | bool | Random-access point (IDR with SPS/PPS for H.264). | |
width | u32 | Payload width in pixels. | |
height | u32 | Payload height in pixels. | |
capture_timestamp_us / captureTimestampUs | u64 | Media-clock capture time (µs). | |
codec_epoch / codecEpoch | u64 | Decoder configuration generation; reset the decoder when it changes. | |
geometry_epoch / geometryEpoch | u64 | Coordinate generation. | |
data | Vec<u8> | Encoded payload. | |
header_json / headerJson | String | The packet's JSON header, verbatim. |