LiveKit — WebRTC Media Server

LiveKit — WebRTC Media Server

LiveKit is the SFU media server that carries real-time audio for the Dudoxx voice assistant — the in-clinic STT-bot that doctors and nurses talk to. A Python voice agent runs as a sidecar and handles the STT → LLM → TTS pipeline, while NestJS issues JWT tokens and exposes a tool-bridge that the agent calls back to. LiveKit is for the voice-bot RTC path; full clinical video telemedicine runs on Jitsi (separate stack).

Business Purpose

Dudoxx HMS includes a voice assistant ("Dudoxx Assistant") that doctors invoke at the bedside to dictate notes, book follow-ups, retrieve patient context, or trigger TUCAN tools — hands-free. Doing this over WebRTC instead of long-poll HTTP means sub-second turn-taking, native browser microphone access, and a clean path to clinic-phone integration via SIP. LiveKit gives us the media transport (SFU + TURN + UDP traversal) and a polished Agents SDK; the Python sidecar handles the dual-STT (on-prem Deepgram Nova-2 primary, cloud Nova-3 optional), Dudoxx LLM router, and Kokoro/ElevenLabs/Cartesia TTS. NestJS keeps the contract narrow: mint tokens, terminate sessions, proxy TUCAN tool calls — never touch media itself.

Audiences

• Investor: The voice assistant is a major differentiator versus typing-only EHRs. LiveKit + the Python agent + the on-prem Deepgram + the Dudoxx LLM router give us a fully self-hostable voice pipeline — a property that matters to German clinics that cannot send patient audio to a public cloud.
• Clinical buyer: When you press the voice button, your microphone connects via LiveKit, the Python agent listens, and TUCAN tools (booking, prescriptions, patient lookup) execute against the clinic's database. The 46+ TUCAN tools available to the voice agent are the same tools the chat assistant uses — voice is a different surface, not a different brain.
• Developer/partner: NestJS does not join LiveKit rooms. It mints JWT tokens (LiveKitTokenService), manages session lifecycle (LiveKitAgentService), and exposes a tool-bridge endpoint (VoiceToolProxyService) the Python agent calls via HTTP. The Python agent (in ddx-docker-livekit/agent/) is the only LiveKit room participant on the server side. LIVEKIT_API_KEY / LIVEKIT_API_SECRET must match across ddx-api, the Python agent, and the LiveKit container — mismatch causes a silent "Connecting…" hang.
• Internal (ops/support): Six services with profile-gating. Always-on: livekit-server + dudoxx-agent. Optional: livekit-egress (recording), livekit-sip (telephony), livekit-ingress (RTMP/WHIP), livekit-redis (session storage for multi-agent scaling). UDP port range 15890-15990 (avoids macOS sharingd conflicts in the 50000s).

Architecture

All services sit on the ddx-001-network + livekit compose network.

Always on — livekit-server v1.9.11 port map (host → container):

dudoxx-agent (Python 3.12) capabilities:

NestJS (ddx-api :6100) services in this path:

Profile-gated:

LiveKit + Jitsi serve different use cases. LiveKit = STT-bot voice RTC (this page). Jitsi = full clinical video telemedicine (separate stack, separate page). Do not conflate.

Architecture anchor: coding_context/ddx-hms-context.md for HMS topology. The ddx-docker-livekit/CLAUDE.md is the load-bearing local doc — diagnostic order, profile gating, agent feature flags — and is the right read for anyone debugging an ICE failure.

Tech Stack & Choices

• Server: livekit/livekit-server:v1.9.11 (ddx-docker-livekit/docker-compose.yaml:66). Pinned. WebRTC SFU + TURN + Agents framework.
• Agent runtime: Python 3.12 image built from ddx-docker-livekit/agent/. Uses livekit-agents v1.4-series. Lives inside the LiveKit Docker compose stack so its lifecycle is coupled to the server (docker-compose.yaml:121).
• Config rendering: LiveKit's YAML does not natively expand ${VAR}, so a one-shot init container (livekit-config-init at docker-compose.yaml:35) runs envsubst over livekit.yaml.tpl and writes the rendered file into a shared named volume that livekit-server mounts read-only. UDP port range is the only template substitution; everything else is static.
• STT — Deepgram on-premises Nova-2 is the primary path (DUDOXX_DEEPGRAM_API_URL, DUDOXX_DEEPGRAM_API_KEY); Deepgram Cloud Nova-3 is an optional fallback (DEEPGRAM_CLOUD_API_KEY). On-prem keeps clinic audio on the LAN; cloud is only enabled when the clinic accepts the data residency trade-off.
• LLM: Dudoxx LLM Router (OpenAI-compatible) at https://llm-router.dudoxx.com/v1 — the same router used by every other AI path in the platform.
• TTS: Provider switch via TTS_PROVIDER (the LiveKit compose default is still literally kokoro at docker-compose.yaml:155, with elevenlabs/cartesia cloud options). DEPRECATION NOTE: Kokoro is deprecated platform-wide — DDX CUDA TTS is the canonical provider now (memory [[feedback-kokoro-tts-deprecated]]); ddx-live-tts is a deprecated service pending removal. Treat the kokoro default as legacy: new deployments should point TTS_PROVIDER at the DDX CUDA TTS endpoint.
• Token signing: LiveKitTokenService uses livekit-server-sdk JWT signing. Token grants are role-aware — generateUserToken (livekit-token.service.ts:109) sets per-role permissions, generateAgentToken (:189) gives the Python worker recorder/agent grants, generateWorkerToken (:232) is for backend workers.
• Session lifecycle: LiveKitAgentService (livekit-agent.service.ts:45) wraps the LiveKit Room Service API. NestJS does not join rooms — it queries them and terminates them. Active sessions are listed by polling room state, not by holding local refs.
• Why LiveKit and not Vapi/Pipecat/Twilio: Vapi/Pipecat are SaaS-only (data residency blocker). Twilio is great for telephony but lacks the multi-modal Agents framework. LiveKit's open-source server + Agents SDK is the only stack we found that runs entirely on-prem and ships a polished VAD/STT/LLM/TTS pipeline.

Data Flow

Voice session start:

1. Doctor clicks "voice" in ddx-web. ddx-web POSTs /api/v1/livekit/voice/join → NestJS LiveKitController.joinVoiceSession() (livekit.controller.ts:92-173).
2. NestJS computes role-appropriate grants via LiveKitTokenService.generateUserToken(...) (livekit-token.service.ts:109-179) — the user gets join + microphone permissions for one specific room name (per-clinic, per-user).
3. Browser receives the JWT + LiveKit URL, opens a WebRTC connection to livekit-server (wss://livekit-server:7880 internal, or the public DNS in production).
4. LiveKit dispatches the Python agent (dudoxx-agent) into the same room — agent token is minted via generateAgentToken (livekit-token.service.ts:189-223).
5. The agent connects (typically connectTime < 100 ms), starts the STT pipeline, and the conversation begins.

Tool call (agent → NestJS):

1. Python agent decides to invoke a TUCAN tool (e.g. "list_patient_appointments").
2. Agent issues an HTTP POST to NESTJS_BASE_URL/livekit/voice/tools/execute → LiveKitController.executeVoiceTool() (livekit.controller.ts:737-796). Headers: NESTJS_API_KEY (service identity), NESTJS_CLINIC_ID (slug — tenant scope).
3. VoiceToolProxyService resolves the tool from the TUCAN registry, runs RBAC checks via VoiceAgentFactory, executes, and returns the result.
4. The agent speaks the result via the configured TTS provider.

Session end:

1. Browser disconnects, or LiveKitController.stopSession() (livekit.controller.ts:324-365) is called.
2. NestJS issues DeleteRoom via LiveKitAgentService.stopVoiceSession() (livekit-agent.service.ts:83-119).
3. A voice event is emitted via the SSE event bus (emitVoiceEvent at livekit-agent.service.ts:247-260) so dependent UI surfaces (admin dashboards, audit views) see the session ended in real time. The SSE transport is the same Redis layer the rest of HMS uses.

No direct media in NestJS: NestJS does not touch audio bytes. All media flows browser ↔ LiveKit server ↔ Python agent. NestJS is the control plane (tokens, sessions, tools), not the data plane.

Implicated Code

≥3 file:line citations required. Counts: 11 below.

• ddx-docker-livekit/docker-compose.yaml:65 — livekit-server service (v1.9.11).
• ddx-docker-livekit/docker-compose.yaml:70 — host port mappings (15880 WS, 15881 RTC TCP, 15478 TURN UDP, 15349 TURN TLS, 15890-15990 media UDP).
• ddx-docker-livekit/docker-compose.yaml:84 — LIVEKIT_KEYS env injection — API_KEY: SECRET pair, must match across ddx-api + Python agent.
• ddx-docker-livekit/docker-compose.yaml:121 — dudoxx-agent service (Python STT/LLM/TTS pipeline).
• ddx-docker-livekit/docker-compose.yaml:35 — livekit-config-init one-shot envsubst container that renders livekit.yaml.tpl into a shared volume.
• ddx-api/src/voice-video/livekit/livekit.module.ts:39 — LiveKitModule decorator + provider list (lines 39–77) — LiveKitTokenService, LiveKitAgentService, VoiceAgentFactory, VoiceConfigService, VoiceResultFormatterService, VoiceToolProxyService, KokoroTTSPlugin.
• ddx-api/src/voice-video/livekit/livekit-token.service.ts:58 — LiveKitTokenService class (lines 58–321).
• ddx-api/src/voice-video/livekit/livekit-token.service.ts:109 — generateUserToken() (lines 109–179) — per-role grants for browser clients.
• ddx-api/src/voice-video/livekit/livekit-token.service.ts:189 — generateAgentToken() (lines 189–223) — agent grants for the Python worker.
• ddx-api/src/voice-video/livekit/livekit-agent.service.ts:45 — LiveKitAgentService class (lines 45–261) wraps the LiveKit Room Service API.
• ddx-api/src/voice-video/livekit/livekit-agent.service.ts:83 — stopVoiceSession() (lines 83–119) — terminates the room and emits a voice event.
• ddx-api/src/voice-video/livekit/livekit-agent.service.ts:247 — emitVoiceEvent() (lines 247–260) — publishes lifecycle events via the SSE event bus.
• ddx-api/src/voice-video/livekit/livekit.controller.ts:72 — LiveKitController class (lines 72–875).
• ddx-api/src/voice-video/livekit/livekit.controller.ts:92 — joinVoiceSession() (lines 92–173) — REST entry for browser join flow.
• ddx-api/src/voice-video/livekit/livekit.controller.ts:737 — executeVoiceTool() (lines 737–796) — tool-bridge endpoint called by the Python agent.

Business → Technical match. Business outcome: a doctor says "book Frau Müller for next Tuesday at 10" into the bedside microphone, and the appointment is booked in the clinic database, with the booking event broadcast over SSE to the receptionist's screen. Technical mechanism: the audio is captured by the browser, transported via LiveKit (livekit-server at port 15880), transcribed by the Python agent's Deepgram pipeline, the agent calls the TUCAN tool book_appointment via executeVoiceTool at livekit.controller.ts:737, the tool writes to Prisma and HAPI FHIR, and LiveKitAgentService.emitVoiceEvent() at livekit-agent.service.ts:247 publishes an SSE event over the shared Redis transport — the receptionist's UI re-renders without a manual refresh.

Operational Notes

• LIVEKIT_API_KEY / LIVEKIT_API_SECRET must match across all three places: the LiveKit container (docker-compose.yaml:84), the Python agent (docker-compose.yaml:132-133), and ddx-api (LiveKitTokenService reads from ConfigService). Mismatch causes a silent "Connecting…" hang in ddx-web — the most-reported voice issue in the project pitfall log.
• config/livekit.yaml is host-specific and git-ignored. UDP port ranges, use_external_ip, and node_ip are tuned per machine. For same-host dev, use use_external_ip: false and enable_loopback_candidate: true. See ddx-docker-livekit/CLAUDE.md ("Local Dev Caveats") for the full diagnostic order on ICE failures.
• UDP port range 15890-15990 (LIVEKIT_UDP_START/LIVEKIT_UDP_END; media-publish line docker-compose.yaml:82) is deliberately below the macOS 49152 ephemeral floor (memory [[project-livekit-udp-range-ephemeral]]) — a range in the 50000s collides with macOS sharingd. The range MUST match rtc.port_range_start/end in the rendered livekit.yaml (from livekit.yaml.tpl:37-38, driven by the same two env vars) — drift causes ICE timeouts that pass signaling but fail media. NOTE: the livekit-config-init container carries a stale fallback default 51000-51100 (docker-compose.yaml:40-41) that the media-publish line does NOT use; set LIVEKIT_UDP_START/END explicitly to keep both in lockstep.
• "Could not establish pc connection" diagnostic order (from ddx-docker-livekit/CLAUDE.md):
  1. Try a non-incognito browser (Incognito forces mDNS ICE obfuscation).
  2. Confirm docker port matches rtc.port_range_* in livekit.yaml.
  3. Confirm nodeIP in server startup logs is a Docker bridge or loopback, not WAN.
  4. If the Python agent joins the same room with connectTime<100ms, the server is healthy — fault is browser-side.
• Profile gating — only run what you need. --profile egress for recording compliance, --profile sip for telephony, --profile full for everything. Egress uses Chrome internally → shm_size: "2gb" is required.
• Recording is compliance-grade (livekit-egress profile). Output configurable (local FS, S3, GCS) via config/egress.yaml.
• SIP integration (livekit-sip profile) lets clinic phones reach the voice agent for after-hours triage. UDP+TCP on port 15060.
• Validation commands:
  • cd ddx-docker-livekit && ./scripts/start.sh (server + agent)
  • docker logs -f ddx-001-livekit-1 (server log; look for nodeIP=...)
  • docker logs -f ddx-001-livekit-1-agent (agent log; look for Registered worker)

Related Topics

• Redis — Cache and SSE Transport — voice-session lifecycle events fan out over the shared Redis SSE transport (NOT the optional livekit-redis session-storage container, which is a separate Redis for multi-agent scaling).
• PostgreSQL — Multi-Schema Database Infrastructure — voice sessions, transcripts, and tool-call audit rows persist in ddx_voice and ddx_api_log databases.
• Jitsi (full clinical video telemedicine) is intentionally documented separately — it's a different stack, not a LiveKit feature.
• Memory shards: MEMORY_LIVEKIT.md (FlowAgent v4, suspend/resume), MEMORY_TUCAN_AI.md (the tools the voice agent invokes).