LiveKit FlowAgent TypeScript — Voice Bridge

LiveKit FlowAgent TypeScript — Voice Bridge

Standalone Node.js process (tsx src/index.ts start) living at ddx-livekit-agents-ts/agents/flowagent/ that joins LiveKit rooms tagged mode === 'flowagent', wires on-prem STT (Dudoxx CUDA / Deepgram) + Dudoxx CUDA TTS + Silero VAD into a Mastra-driven AgentSession, and bridges every Mastra tool call back to ddx-api over HTTP — the runtime that turns an authored FlowAgent scenario into an actual voice conversation in the patient's room.

PATH NOTE (2026-07-01 re-verify): this package's real root is ddx-livekit-agents-ts/agents/flowagent/** (a workspace under the shared LiveKit-agents monorepo). The former standalone ddx-livekit-flowagent-ts/** root is GONE — do not chase it. All file:line citations below are relative to the live root.

Business Purpose

We need a LiveKit agent runtime that:

1. Is isolated from ddx-api so a Node-side C++ crash on room teardown (@livekit/rtc-node mutex race) cannot kill the NestJS API.
2. Runs the same Mastra version the rest of the FlowAgent toolchain authors against (@mastra/core v1.20.0 + @mastra/memory + @mastra/libsql).
3. Exposes a provider registry (src/providers/registry.ts) so a tenant can swap STT/TTS providers via voiceConfig with zero code change. Default plugins now: src/plugins/dudoxx-stt.ts + src/plugins/deepgram-stt.ts (STT) and src/plugins/dudoxx-tts.ts (Dudoxx CUDA TTS — Kokoro is deprecated).

This package is the operational heart of the FlowAgent voice product (see flowagent-scenarios.md for the business-of-scenarios angle). It is also a teaching reference for any future Dudoxx LiveKit Agent — the operator/orchestrator patterns live here, just one Mastra notch behind ddx-vivoxx.

Brief drift disclosure: the W4 brief referenced this page as the home of operator.agent.ts under a standalone ddx-livekit-flowagent-ts/ root. Neither exists — the agent here is src/agents/flowagent-voice.ts under ddx-livekit-agents-ts/agents/flowagent/. The "TUCAN Operator" actually lives in ddx-api/src/ai/tucan-operator/; it shares the LiveKit + Mastra pattern documented here but is a separate NestJS-hosted runtime. Documented for honesty; do not chase a missing file.

Audiences

• Investor: Voice agent is "another deployable" — not a NestJS feature. Customers under voice load do not pay NestJS scaling cost.
• Developer/partner: Single entry (tsx src/index.ts start). Room filter (metadata.mode === 'flowagent') lets multiple LiveKit agents (TUCAN Operator, Vivoxx, FlowAgent) coexist on one cluster.
• Internal (ops/support): Logs are LiveKit-style; LibSQL DB (mastra-flowagent.db) is a single file; secrets are .env only.

Architecture

ddx-livekit-agents-ts/agents/flowagent (Node.js process) — entered when LiveKit Cluster (livekit.home.dudoxx.com) sets room.metadata.mode === 'flowagent'.

src/index.ts wiring:

createFlowAgentPlugins provides:

The session terminates at the LiveKit room ↔ patient mic. DataChannel events: state_changed | tool_event | transcript | agent_speech | voice_language_changed.

Tech Stack & Choices

• Runtime: Node.js (no NestJS). Started via tsx src/index.ts start (cli.runApp pattern from @livekit/agents).
• LiveKit SDK: @livekit/agents v1.x with @livekit/agents-plugin-silero for VAD. @livekit/rtc-node@0.13.24 underneath.
• Mastra: @mastra/core v1.20.0 + @mastra/memory + @mastra/libsql. A DIFFERENT Mastra line from ddx-api's embedded @mastra/core (1.37.1 — verify live from node_modules/@mastra/core/package.json, the ddx-api CLAUDE.md "1.5.0" line is stale) — DO NOT cross-import between the two.
• Storage: LibSQL via MASTRA_STORAGE_URL=file:./mastra-flowagent.db. Single-file SQLite-compatible DB.
• Provider registry: pluggable STT/TTS factories (src/providers/registry.ts). Default STT/TTS plugins are registered at module load (src/agents/flowagent-voice.ts); STT = Dudoxx CUDA (src/plugins/dudoxx-stt.ts) + Deepgram (src/plugins/deepgram-stt.ts), TTS = Dudoxx CUDA (src/plugins/dudoxx-tts.ts). Kokoro TTS is deprecated (project memory feedback_kokoro_tts_deprecated).
• Voice config cascade: per-session config wins over package defaults (flowagent-voice.ts:72-77).
• Memory (index.ts:311-326): lastMessages: 20, semanticRecall: false, workingMemory disabled (reasoning model leaks template into TTS), observationalMemory enabled for long sessions.
• Temperature: 0.01 in voice mode (flowagent-voice.ts:129) — deterministic tool calls; instruction echo is prevented by lean prompts.
• Why not embed in NestJS? Native @livekit/rtc-node crashes on disconnect (C++ mutex — MEMORY_LIVEKIT.md). A separate process lets the agent die and respawn without taking ddx-api down.
• Why a custom MastraWorkflowLLM? LiveKit's AgentSession expects an llm.LLM; the workflow IS the LLM here (Mastra owns conversation state). workflow-bridge.ts adapts the workflow to LiveKit's LLM interface so the SDK can drive the turn loop unchanged.

Data Flow

1. Worker boot: cli.runApp registers defineAgent('ddx-flowagent-voice'). JobProcess prewarms Silero VAD.
2. Room joined: JobContext.room.metadata is parsed by isFlowAgentRoom; non-FlowAgent rooms exit immediately.
3. Voice context fetched: HTTP GET against ddx-api/voice/... returns the scenario snapshot + form definition + per-session voiceConfig.
4. Plugins created: createFlowAgentPlugins(vad, metadata, voiceContext) returns {stt, tts, vad, bridge, toolRegistry, locale, greeting, model}.
5. Workflow compiled: compileFlowAgentWorkflow(scenario, formDef) → Mastra workflow chain (greeting → sections → completion).
6. Mastra registered: workflow is registered with a Mastra instance for snapshot persistence (suspend/resume across crashes).
7. AgentSession: voice.AgentSession({ stt, tts, vad, llm: MastraWorkflowLLM }). The LLM adapter calls workflow.start({...}) → suspends → resumes per turn.
8. DataChannel events (registerEventPublishers, index.ts:70): FunctionToolsExecuted → publish tool_event; transcript chunks → publish transcript; state changes → publish state_changed (UI source of truth).
9. Tool bridge: every Mastra tool call (fill_field, skip_field, …) goes through NestJSToolBridge → HTTP POST to ddx-api → DB persistence + audit + SSE → returned to Mastra.

Implicated Code

MANDATORY: ≥3 file:line citations.

• ddx-livekit-agents-ts/agents/flowagent/src/index.ts — package entry: agent name ddx-flowagent-voice, room filter metadata.mode === 'flowagent', Mastra workflow engine bootstrap.
• ddx-livekit-agents-ts/agents/flowagent/src/orchestrator-publishers.ts — DataChannel event publishers (tool_event / transcript / agent_speech / state_changed / voice_language_changed).
• ddx-livekit-agents-ts/agents/flowagent/src/mastra/memory.ts — LibSQLStore + Memory wiring (workingMemory disabled; observationalMemory for long sessions).
• ddx-livekit-agents-ts/agents/flowagent/src/mastra/agent-factory.ts — Mastra agent/instance construction + RequestContext (model, memory, sessionId).
• ddx-livekit-agents-ts/agents/flowagent/src/agents/flowagent-voice.ts — createFlowAgentPlugins (voiceConfig cascade, ProviderRegistry resolution, default STT/TTS registration).
• ddx-livekit-agents-ts/agents/flowagent/src/mastra/model.ts — Dudoxx Mastra model factory (voice temperature ~0.01, thinking disabled — see voxial-llm.md).
• ddx-livekit-agents-ts/agents/flowagent/src/providers/registry.ts — provider registry impl (register / create / fallback).
• ddx-livekit-agents-ts/agents/flowagent/src/plugins/dudoxx-stt.ts · src/plugins/deepgram-stt.ts · src/plugins/dudoxx-tts.ts · src/plugins/tts-prefilter.ts — STT/TTS plugins + TTS sanitisation prefilter.
• ddx-livekit-agents-ts/agents/flowagent/src/tools/nestjs-bridge.ts — NestJSToolBridge (HTTP bridge to ddx-api FlowAgent endpoints).

Operational Notes

• Start command: tsx src/index.ts start (LiveKit Agents CLI runs the worker pool).
• Env vars (key NAMES only — resolve live from src/config.ts / .env): LIVEKIT_URL/LIVEKIT_API_KEY/LIVEKIT_API_SECRET, the STT provider vars (Dudoxx CUDA STT + DEEPGRAM_*), the Dudoxx CUDA TTS provider vars, the Dudoxx LLM vars (DUDOXX_BASE_URL/DUDOXX_API_KEY/DUDOXX_MODEL_NAME), the ddx-api bridge vars (DDX_API_URL/DDX_API_KEY), MASTRA_STORAGE_URL, MASTRA_LAST_MESSAGES, WORKFLOW_MAX_STEPS, WORKFLOW_SUSPEND_TIMEOUT_MS. (Kokoro TTS env vars are removed — TTS is Dudoxx CUDA.)
• Pitfall — LiveKit key drift: API key + secret MUST match across ddx-api + ddx-vivoxx + this package + ddx-mastra-codebase. Mismatch presents as silent "Connecting…" hang at the patient — no error, no timeout. (context/REMINDERS.md voice section; root project memory.)
• Pitfall — workingMemory: enabling it makes DeepSeek R1 (via LiteLLM) read the template aloud. Keep semanticRecall: false + workingMemory undefined.
• Pitfall — cleanup race: ddx-web/src/lib/flowagent-client/hooks/useFlowAgentLiveKit.ts calls stopVoiceSession on Strict Mode double-render → known crash (MEMORY_LIVEKIT.md Known Open Bugs).
• Pitfall — agent model inheritance: NEVER pass an explicit model: to inner Agent() constructions inside steps — use the model wired into the step's RequestContext. (feedback_agent_model_inheritance.md memory shard, root project.)
• Worker lifecycle: LiveKit Agents pool prewarms VAD per process; entry function runs per room.
• Crash recovery: workflow snapshot persisted in mastra-flowagent.db; on respawn the worker resumes from the last suspend point.

Related Topics

• FlowAgent Scenarios — the workflow this package executes
• Voxial LLM — model + middleware used by createDudoxxMastraModel
• Live STT — STT provider plug-in (default deepgram-onprem)
• Live TTS — TTS provider plug-in (default Dudoxx CUDA TTS)
• FlowAgent AG-UI Bridge — the second, text/agentic AG-UI surface for the SAME FlowAgent sessions (staff + guest live-bridge)
• Vivoxx Voice — sister LiveKit Agent runtime (POC + production)
• Voice Commands — how out-of-form intents leave the workflow
• LiveKit Infra (W1) — cluster + auth + URL