VoiceBot Web implementation Doc Server–Client Event Handling This document explains how the VoiceBot component communicates with the server via WebSocket using structured event-based messages. It covers: Client-sent events (start, media) Server-sent events (media, mark, clear) Audio processing flow 🔁 Event Flow Summary [Client] └─ ➤ Connect to WebSocket └─ ➤ Send 'start' event └─ ➤ Start recording microphone └─ ➤ Encode PCM(16-bit Mono) (S16LE) → Base64 └─ ➤ Send 'media' chunks [Server] └─ ➤ Receive 'media' chunks └─ ➤ Process + respond with 'media' (Base64) └─ ➤ Optionally send 'mark' or 'clear' events [Client] └─ ➤ Receive events └─ ➤ 'media': decode + play └─ ➤ 'clear': stop playback 󰳕 Client-Sent Events 1. start EventSent once after WebSocket connection is established. Informs server to begin voice session. { "event": "start", "start": { "call_sid": "test", // placeholder or session identifier (for server) "stream_sid": "test" // stream identifier (for client) } } 📌 Sent from: initializeWebSocket 📌 Sent after: 2-second delay post connection 2. media Event Sent continuously during active microphone recording every 200ms of 200ms of audio. { "event": "media", "media": { "chunk": "inbound", // direction of audio "payload": "<base64_PCM>", // audio chunk (16-bit Mono PCM encoded to Base64) "timestamp": 1721047849.123 // Unix timestamp } } 🌐 Server-Sent Events 1. media Event Sent by server as a response audio chunk. Meant to be played by the client. { "event": "media", "media": { "payload": "<base64_PCM>" } } 📌 Handled in: ws.onmessage 📌 Decoded and played in: playAudioResponse() 2. clear Event Instructs client to stop audio playback and clear existing buffer. In this case server has encountered interruption based on voice, this axpect client to stop playing the audio and clear the audio buffer if any for older audio{ "event": "clear" } 3. mark Event (Optional) Typically used as metadata or checkpoints from server. this is sent every time the audio is played till the server mark event and should be returned to server after finishing the audio playback. { "event": "mark", "mark": { "type": "info", "value": "partial_result" } } Timeout if WebSocket doesn't open in 10s 🛠 Developer Notes ● Make sure server is expecting start and media messages ● media event will only be sent after the start even sent by client, all the media event if sent before start will be ignored or may fail ● Audio must be 16-bit linear PCM, sampled at 8000 Hz