/Knowledge Base
GitHub
07-voice-and-telemedwave: W4filled16 citations

Live TTS — Real-Time Text-to-Speech

Audiences: developer, internal

Live TTS — Real-Time Text-to-Speech

⚠️ DEPRECATED (2026-06). The ddx-live-tts (Kokoro-family) FastAPI service documented here has been moved out of this monorepo to dudoxx-omni-intelligence, and Kokoro is fully retired (garaged 2026-06-12). It survives in-tree only until the client migration completes (root CLAUDE.md "Deferred" + ddx-reminders/). The live on-prem TTS is now the Dudoxx CUDA Live TTS stack (ddx-cuda-live-tts, Qwen3+XTTS under the canonical dudoxx-tts provider) served at tts.home.dudoxx.com / tts.forge.dudoxx.com over a streaming SSE contract (GET /v1/speak/sse). Voices are ddx_* (e.g. ddx_heart), never the legacy af_* / dm_* Kokoro ids. See on-prem-stt-tts-llm.md + vivoxx-voice.md for the live path. Do NOT wire NEW consumers to ddx-live-tts or any KOKORO_* env. Historical Kokoro description below.

Multi-engine on-prem speech synthesis service (Kokoro-82M / Qwen3-TTS / Piper / Chatterbox) that turns Mastra-agent text into 24 kHz PCM for LiveKit playback — used by TUCAN Operator, FlowAgent voice, Vivoxx avatar, and stand-alone clinical readers, with a single auto-routing endpoint that hides engine choice from callers.

Business Purpose

Voice features need a TTS partner that (a) speaks 10+ languages with credible quality (clinical Spanish/French/German is non-negotiable in our target markets), (b) supports voice cloning so the Vivoxx avatar can present a consistent "Dudoxx clinician" persona, and (c) runs on-prem. ElevenLabs / OpenAI TTS fail (a) for our EU residency story; Coqui has limited maintenance. ddx-live-tts bundles the best open engines behind one FastAPI surface and is part of the Dudoxx Voice Pack license SKU.

Audiences

  • Investor: Removes per-character SaaS TTS cost (ElevenLabs ~$0.18/1k chars), enables voice cloning as a paid add-on, and unlocks EU/CH/DE accounts that legally cannot egress voice prompts to US TTS providers.
  • Clinical buyer: Voice answers feel natural across DE/FR/EN/ES; can clone a clinic's own voice (e.g. the lead doctor's German persona) so the assistant sounds like a real clinic member.
  • Developer/partner: One endpoint (POST /tts/stream) auto-selects the engine based on voice + language. No engine branching in the consumer.
  • Internal (ops/support): Single systemd unit (kokoro-tts.service), single port, single token. RTF 0.15-0.25 on RTX 4090 → <300 ms first-byte latency for voice agents.

Architecture

diagram
ddx-livekit-flowagent-ts / ddx-api / Vivoxx
   │  POST /tts/stream  { text, voice, language }
   ▼
ddx-live-tts :8890 ──── FastAPI
   routes/v2/speech.py            (unified /v2/speech, /v2/speech:stream, /v2/speech:socket)
   routes/tts.py + routes/agentic.py + routes/realtime.py
        │
        ▼
   utils/pipeline.detect_engine(voice, language)
        ├── qwen3_wrapper  (Qwen3-TTS-0.6B, EN/FR/DE, voice clone, 24 kHz)
        ├── piper_wrapper  (German Thorsten, CPU, fastest path)
        ├── chatterbox_wrapper (English + 23-lang MTL, emotion tags, voice clone)
        └── kokoro KPipeline (54 voices × 10 languages, default)
        │
        ▼
   PCM stream (chunked) → LiveKit room audio out

Engine routing is a single function (pipeline.detect_engine); consumers stay agnostic. The same voice ID flowing into /tts/stream, /tts/agentic/stream, or /v2/speech:socket lands at the right engine.

Tech Stack & Choices

  • Framework: FastAPI + Uvicorn (ddx-live-tts/server/main.py:1-17); systemd unit kokoro-tts.service; Conda env dudoxx-odoo.
  • Default engine: Kokoro-82M — 54 voices, 10 languages, ~150 MB model.
  • Voice cloning: Qwen3-TTS-0.6B (ddx-live-tts/server/utils/qwen3_wrapper.py:21-46) + Chatterbox. Qwen3 voice IDs end in _q (e.g. dm_frenz_q for the Frenz German male clone). Reference WAVs live under server/voices/{lang}/.
  • German fast path: Piper TTS (Thorsten voice) on CPU — fallback when GPU is saturated.
  • Emotion control: Chatterbox supports paralinguistic tags (<laughter>, <sigh>), routed via /tts/emotion.
  • API versions: v1 routes (/tts/stream, /tts/agentic/stream, /tts/multivoice, /tts/clone/*) plus unified v2 (POST /v2/speech, POST /v2/speech:stream, WS /v2/speech:socket — routes/v2/speech.py:50-130).
  • Why not Coqui XTTS? License + maintenance state. Why not Bark? Latency. Why not OpenAI TTS? On-prem requirement.

Data Flow

  1. Caller (FlowAgent step, TUCAN voice extractor, Vivoxx agent) POSTs {text, voice, language} with Authorization: Bearer $DUDOXX_TTS_TOKEN.
  2. detect_engine(voice, language) (utils/pipeline.py:21-31) picks qwen3 | piper | kokoro. Chatterbox is selected explicitly via /tts/chatterbox/* or /tts/emotion.
  3. Engine wrapper streams PCM frames; the route wraps them in StreamingResponse (chunked) or pushes them through a WebSocket (routes/v2/speech.py:97).
  4. Consumer pipes PCM into the LiveKit room (FlowAgent: ddx-livekit-flowagent-ts/src/plugins/kokoro-tts.ts:103, :151, :218 for the chunked + synth-stream sessions).
  5. Before synthesis, the consumer sanitizes text (strips <think> tags, markdown, "section is complete" hallucinations) — see MEMORY_LIVEKIT.md + step-helpers.ts.

Implicated Code

MANDATORY: ≥3 file:line citations.

  • ddx-live-tts/server/main.py:14-32 — config import + router mounts (12 routers: health, config, tts, realtime, multivoice, agentic, fish_speech, chatterbox, voice_clone, emotion, qwen3, v2/speech).
  • ddx-live-tts/server/main.py:43-67 — lifespan; pre-loads default Kokoro pipeline + Qwen3 at boot, cleans up on shutdown.
  • ddx-live-tts/server/utils/pipeline.py:21-31 — detect_engine — single source of truth for engine routing.
  • ddx-live-tts/server/utils/qwen3_wrapper.py:21-46 — QWEN3_VOICES registry (EN/FR/DE clones, including dm_frenz_q).
  • ddx-live-tts/server/routes/v2/speech.py:50-95 — unified v2 buffered + streaming endpoints.
  • ddx-live-tts/server/routes/v2/speech.py:96-130 — WS /v2/speech:socket real-time synthesis.
  • ddx-live-tts/CLAUDE.md:3 — version v6.4.0, port 8890, RTF 0.15-0.25 (GPU).
  • ddx-livekit-flowagent-ts/src/plugins/kokoro-tts.ts:103 — KokoroTTS adapter (LiveKit-side); :151 chunked stream; :218 synthesize stream.
  • ddx-livekit-flowagent-ts/src/agents/flowagent-voice.ts:15 + :30-31 — provider registration; :73-76 voice + base URL resolution.

Operational Notes

  • Port: 8890 (ddx-live-tts/server/config/settings.py:11-12 — HOST/PORT env-driven, default 8890). The W4 brief referenced "6600" — that was a planning-phase placeholder; live deployment listens on 8890 and is fronted at https://kokoro.dudoxx.com (Apache2 reverse proxy).
  • Auth: DUDOXX_TTS_TOKEN Bearer for HTTP, ?token= for WebSocket. GET /health is unauthenticated.
  • Default voices: af_heart (English), ff_siwis / ff_sonia (French), de_thorsten via Piper or dm_frenz_q via Qwen3 (German).
  • Sanitisation discipline (CRITICAL): never feed raw LLM output. Two-layer strip in FlowAgent (step-helpers.ts + kokoro-tts.ts) removes <think>, working memory, markdown, repeated phrases, and the "section is complete" hallucination class — otherwise the agent reads internal reasoning aloud.
  • Pitfall — Qwen3 click suppression: v6.3.0 fixed a "tock" click on first frame via DC-offset removal + 1 ms silence + 30 ms fade-in (all 5 Qwen3 paths). If clicks reappear after a Qwen3 upgrade, regression is in these three steps.
  • Pitfall — Chatterbox MTL output_attentions error: transformers 4.57+ requires attn_implementation="eager" in chatterbox/models/t3/llama_configs.py. Documented in ddx-live-tts/CLAUDE.md troubleshooting table.
  • Concurrency: <=8 concurrent under 3 s; 9-16 needs a queue; 17+ requires horizontal scale (ddx-live-tts/CLAUDE.md performance table).
  • Demo + deploy: demo/index.html is published to /var/www/kokoro.dudoxx.com/public_html/index.html.

Related Topics

    Live TTS — Real-Time Text-to-Speech — Dudoxx Docs | Dudoxx