Decoded media
Media sessions that deliver decoded BGRA frames and PCM audio.
Media sessions that deliver decoded BGRA frames and PCM audio.
These APIs are part of the Cua Spaces app export for building Spaces UIs. They are source-available under FSL-1.1-MIT and ship for Swift only (import CuaSpacesFFI); the Rust tab shows the cua-spaces-ffi crate they come from. The open source SDK packages for Python, TypeScript and Kotlin do not include them.
The streaming decoder ships with Cua Spaces. spacesd_open_media_decoded opens a media session on a SpacesdClient and delivers decoded video (packed BGRA; VideoToolbox on macOS, OpenH264 elsewhere) to a DecodedFrameSink, requesting a keyframe whenever a reference is lost (Swift: client.openMediaDecoded(...)). The open source SDK delivers the encoded frames: see Media.
spacesd_open_media_decoded#Opens a media session on client and delivers decoded video (packed
BGRA) and control events to frames. Lost references trigger a keyframe
request automatically. (Swift: client.openMediaDecoded(...).)
func spacesdOpenMediaDecoded(client: SpacesdClient, options: MediaOpenOptions, frames: DecodedFrameSink) async throws -> MediaSession| Parameter | Type | Default |
|---|---|---|
client | SpacesdClient | required |
options | MediaOpenOptions | required |
frames | DecodedFrameSink | required |
Returns MediaSession · Async · Raises CuaError
spacesd_open_media_decoded_with_audio#spacesd_open_media_decoded that also delivers decoded PCM audio to
pcm (set options.audio to negotiate audio tracks).
func spacesdOpenMediaDecodedWithAudio(client: SpacesdClient, options: MediaOpenOptions, frames: DecodedFrameSink, pcm: PcmSink) async throws -> MediaSession| Parameter | Type | Default |
|---|---|---|
client | SpacesdClient | required |
options | MediaOpenOptions | required |
frames | DecodedFrameSink | required |
pcm | PcmSink | required |
Returns MediaSession · Async · Raises CuaError
DecodedVideoFrame record#One decoded video frame : packed top-down BGRA.
| Field | Type | Default | Description |
|---|---|---|---|
sequence | u64 | Per-target frame sequence. | |
width | u32 | Width in pixels. | |
height | u32 | Height in pixels. | |
stride | u32 | Bytes per row (width * 4). | |
format | String | Pixel format; always bgra. | |
capture_timestamp_us / captureTimestampUs | u64 | Media-clock capture time (µs). | |
codec_epoch / codecEpoch | u64 | Decoder configuration generation of the source frame. | |
geometry_epoch / geometryEpoch | u64 | Coordinate generation. | |
keyframe | bool | The source access unit was a keyframe. | |
encoded_size / encodedSize | u64 | Bytes of the source (encoded) access unit. | |
received_at_us / receivedAtUs | u64 | When the encoded frame reached the decoder (Unix µs, this host). | |
decode_duration_us / decodeDurationUs | u64 | Time spent decoding and converting it (µs). | |
data | Vec<u8> | Pixels. |
PcmAudio record#Decoded audio : interleaved signed 16-bit PCM.
| Field | Type | Default | Description |
|---|---|---|---|
track_id / trackId | u16 | Track id from NegotiatedAudio. | |
sample_rate / sampleRate | u32 | Samples per second per channel. | |
channels | u16 | Interleaved channels. | |
pts_us / ptsUs | u64 | Media-clock time of the first sample (µs). | |
concealed | bool | Synthesized by packet-loss concealment for a lost packet. | |
sequence | u32 | RAU2 sequence of the packet this frame was decoded from (the packet that revealed the loss, for concealed frames). | |
config_epoch / configEpoch | u8 | RAU2 config epoch (the audio codec generation). | |
encoded_size / encodedSize | u64 | Bytes of the encoded packet (0 for concealed frames). | |
received_at_us / receivedAtUs | u64 | When the packet reached the decoder (Unix µs, this host). | |
decode_duration_us / decodeDurationUs | u64 | Time spent decoding the packet (µs). | |
samples | Vec<i16> | Interleaved samples. |
DecodedFrameSink#Receives decoded video frames and control events. The SDK decodes on the session's delivery thread (VideoToolbox on macOS, OpenH264 elsewhere); implementations must return quickly.
You implement DecodedFrameSink and pass it to the SDK (a callback interface): implement the DecodedFrameSink protocol in Swift or the trait in Rust.
| Method | Description |
|---|---|
on_decoded_frame | A decoded frame. |
on_event | A control message, a decode_error, or the socket close. |
DecodedFrameSink.on_decoded_frame#A decoded frame.
func onDecodedFrame(frame: DecodedVideoFrame)| Parameter | Type | Default |
|---|---|---|
frame | DecodedVideoFrame | required |
DecodedFrameSink.on_event#A control message, a decode_error, or the socket close.
func onEvent(event: MediaEvent)| Parameter | Type | Default |
|---|---|---|
event | MediaEvent | required |
PcmSink#Receives decoded audio. Must return quickly.
You implement PcmSink and pass it to the SDK (a callback interface): implement the PcmSink protocol in Swift or the trait in Rust.
| Method | Description |
|---|---|
on_pcm | One decoded (or concealed) audio frame. |
PcmSink.on_pcm#One decoded (or concealed) audio frame.
func onPcm(audio: PcmAudio)| Parameter | Type | Default |
|---|---|---|
audio | PcmAudio | required |