A live signal is not yet a model turn.
Audio and video can keep moving while a conversation system decides what evidence belongs in the next bounded exchange.
The timeline below is invented to teach the layers. No microphone, camera, realtime model, transcription service or latency test was used. Systems may send audio directly to a multimodal model, transcribe it first, combine both, or omit video entirely.
A media connection answers “what samples are arriving?” A conversational turn answers a different question: “which bounded evidence should the system use now?” Treating those as the same thing hides the decisions that shape responsiveness and meaning.
Five boundaries between a signal and a response
One synthetic conversation, several clocks
| Invented time | Media layer | Turn layer | Context decision |
|---|---|---|---|
| 00:00.000 | Audio track is live; video track is live. | No turn exists yet. | Keep track identity and media time; do not call “live” a user request. |
| 00:01.200 | Speech-like audio begins. A frame shows a blue card. | An optional detector opens candidate turn T1. | Retain the relevant frame time and audio interval, not every prior frame. |
| 00:03.900 | The speaker pauses. | A policy closes T1 after its configured evidence—not because transport stopped. | Assemble T1 from selected audio or optional transcript, the blue-card observation, and prior conversation state. |
| 00:04.600 | Tracks continue while output starts. | T1 is committed; response R1 is in progress. | Keep new incoming media separate from T1 unless the product explicitly supports revising an open turn. |
| 00:05.100 | New speech-like audio arrives. | Candidate T2 opens and may interrupt R1. | Record the interruption event; stop, fade or continue R1 according to product policy. |
These numbers are labels, not measurements. A real implementation can have capture time, sender time, receiver time, playout time and application time. Preserve which clock a timestamp belongs to before subtracting two values. WebRTC statistics even distinguish ingest, emit and estimated playout timing; one “latency” number cannot safely stand in for all of them.
Transport, segmentation and understanding are separate jobs
- Continuous media transport
- Moves audio or video tracks and manages their lifecycle. A track can be live, muted or ended independently of whether a conversational turn is open.
- Observation timing
- Associates samples, frames or derived events with a stated clock. Synchronization is an aim, not a promise that every source shares one perfect timeline.
- Turn segmentation
- Chooses a boundary using push-to-talk, a user action, silence/VAD, a fixed window, semantic cues or a combination. VAD is one possible signal, not the definition of a turn.
- Model context assembly
- Selects what the model receives: perhaps audio, perhaps text, perhaps sampled frames or structured observations, plus relevant conversation state. Transcription is optional, not universal.
- Output and interruption
- Schedules a response while input may continue. “Barge-in” requires an explicit policy for cancelling output, retaining partial output and opening the next turn.
Interruption is a state transition, not just louder audio
When new input arrives during output, the system has at least three states to reconcile: the committed input turn, the response already produced, and the candidate interruption. A robust design records which response was interrupted and why. Otherwise the next model call may receive partial assistant text as though it had been fully delivered.
A practical review: ask five questions
- What remains continuous? Name each audio/video track and when it is live, muted or ended.
- Which clock labels each event? Keep capture, receive, playout and application times distinct.
- What closes a turn? Document the user action or segmentation policy and its fallback.
- What actually enters model context? Audio, transcript, frames and conversation state are different representations; list the selected ones.
- What happens on interruption? Define whether output is stopped, what partial response is retained, and how the next turn is assembled.
If those answers are missing, “realtime” describes an aspiration, not an inspectable interaction design.
What the web standards establish
Streams contain tracks
The W3C Media Capture and Streams Candidate Recommendation Draft defines MediaStreamTrack as one media source and MediaStream as a grouping of tracks intended to be synchronized when rendered. It also notes that synchronization can be imperfect when sources use different clocks. MDN’s MediaStreamTrack reference documents the observable live/ended lifecycle and muted state.
Transport timing has multiple landmarks
The W3C WebRTC Statistics specification defines timestamps and metrics such as jitter-buffer ingest/emit timing and estimated playout timestamps. These support measurement vocabulary; they do not prescribe conversational turn segmentation.
Audio processing has its own time coordinate
The W3C Web Audio API specification defines an audio-context time coordinate. It is one useful clock domain, not proof that media, network and application events share a single timestamp.
Current W3C publications and MDN reference checked; Media Capture and Streams was a Candidate Recommendation Draft. The five-layer sequence, synthetic timeline, interruption state and review questions are original synthesis. No vendor capability, device behavior or measured performance is represented.