On-Prem STT/TTS/LLM — Self-Hosted Voice and Inference Stack
Audiences: developer, internal, partner, investor
On-Prem STT/TTS/LLM — Self-Hosted Voice and Inference Stack
Self-hosted STT + TTS + LLM-router inference services unified behind LiteLLM and the production CUDA embedder — every PHI-touching token of speech and text stays on Dudoxx infrastructure.
DRIFT NOTE (2026-07-02 re-verify): the original W4 text described the Kokoro TTS
stack (ddx-live-tts, kokoro-tts.plugin.ts, DEFAULT_TTS_PROVIDER=DUDOXX_KOKORO, af_heart
voice) and the Whisper/FunASR STT stack (ddx-live-stt). Both of those FastAPI services
are now DEPRECATED and moved to dudoxx-omni-intelligence (Kokoro fully retired 2026-06-12;
see live-stt.md / live-tts.md). The live ddx-api voice path is:
TTS → provider DUDOXX_TTS (CUDA Live TTS, Qwen3+XTTS), default voice ddx_heart,
plugin DudoxxTTSPlugin in livekit/plugins/dudoxx-tts.plugin.ts, env
VOICE_TTS_DUDOXX_TTS_URL=https://tts.home.dudoxx.com (NOT KOKORO_TTS_*).
STT → engine enums DUDOXX_DEEPGRAM (default) / DUDOXX_WHISPER / DUDOXX_STT (Forge
multi-engine gateway); the canonical on-prem voice-agent STT is CUDA Parakeet via DUDOXX_STT.
Historical Kokoro/Whisper/FunASR prose below is retained for lineage; where it conflicts with
this note, this note wins.
Business Purpose
The on-prem voice-and-inference stack is the single most expensive and the single most defensible piece of the Dudoxx platform. It exists because no SaaS combination of Deepgram + ElevenLabs + OpenAI is legally usable for a German clinic's dictation pipeline without a DPA chain that takes months to negotiate, and even then PHI leaves the country.
By owning the four hot-path inference services — STT (ddx-live-stt), TTS (ddx-live-tts), LLM (dudoxx-gemma via LiteLLM), embeddings (dudoxx-embed CUDA) — Dudoxx gains:
HIPAA / GDPR / BfArM data residency by construction — voice → transcript → LLM → embedding can all complete without leaving the cluster. Cloud providers are optional add-ons, not the default.
Zero per-minute STT/TTS cost — clinics pay for hardware once, then dictate unlimited hours. Deepgram at ~$0.0043/min × 200 dictation min/doctor/day × 30 doctors quickly outruns a single 4090 GPU.
Latency floor under 200 ms — STT runs in the same datacenter / on the same LAN; round-trip to AWS US-East from Hamburg adds ~120 ms before any inference happens.
Provider-portable contracts everywhere — TTS calls go to the Kokoro/Qwen3 endpoint via infra-litellm, LLM calls go through the same proxy, and embedding calls share the production CUDA endpoint at https://embedder.home.dudoxx.com/v1/embeddings. Swapping a model is a config change, not a code change.
Revenue lever: the "on-prem AI bundle" is the upsell that makes Dudoxx defensible against bundled GP suites. Every competitor that ships AI by re-selling OpenAI/Deepgram is exposed to data-residency objections; Dudoxx is not.
Audiences
Investor: ownership of STT + TTS + LLM + embedder is the unfair-advantage tier of the moat — and the answer to "how does your margin survive at scale?". A clinic with 30 dictating doctors pays cloud STT/TTS what it costs to amortise one self-hosted GPU per year.
Clinical buyer (doctor/nurse/receptionist): invisible — they hear "DDX scribe" / "voice command" / "AI assistant". The HIPAA/data-residency argument lives in procurement, not at the desk.
Developer/partner: each service is a FastAPI process with a public WebSocket and REST surface; all server-side consumers in ddx-api/src/voice-video/ import a thin plugin (KokoroTTSPlugin, voice processor) and route through the standard config resolution chain (frontend > user-db > org-db > env defaults).
Internal (ops/support): four endpoints to keep alive — localhost:6350 (Whisper STT), localhost:6360 (FunASR STT), localhost:8890 (Kokoro TTS via Docker — alias kokoro.dudoxx.com), localhost:4000 (LiteLLM gateway). A clinic where dictation suddenly fails almost always has one of these four health checks red.
STT split (Whisper vs FunASR) — Latin scripts (EN/DE/FR/ES/IT/PT) → Whisper (ddx-live-stt :6350, faster-whisper on CUDA or MLX on Apple Silicon). CJK + auto-detect short utterances → FunASR (ddx-live-stt :6360, SenseVoiceSmall). Both speak the same /v1/listen WebSocket protocol; the engine choice is org-level (whisperSttEndpoint vs funasrEndpoint columns on organizations).
TTS unification — Kokoro-82M (54 voices, 9 langs), Qwen3-TTS-0.6B (11 voices, EN/FR/DE), Piper TTS (DE), Chatterbox (emotion). All exposed by ddx-live-tts on :8890 behind POST /tts/stream; engine routing is automatic based on voice-id prefix (*_q → Qwen3, German voices → Piper, etc.).
LLM unification — every chat-completion request goes through LiteLLM gateway :4000 regardless of upstream (self-hosted dudoxx-gemma, OpenAI, Anthropic). The USE_DUDOXX_LITELLM=true flag makes the on-prem path the global default.
Embedder is shared — same https://embedder.home.dudoxx.com/v1 CUDA endpoint that powers the RAG sidecar (see rag-system-indexation). One model (dudoxx-embed, dim 2560) across every embedding consumer.
ddx-live-stt (FastAPI + faster-whisper / mlx-whisper / FunASR) — Python 3.11+, WebSocket-first, hot-reload, Prometheus metrics, API-key auth. Backends auto-selected: CUDA → faster-whisper (float16), Apple Silicon → mlx-whisper (int8), CPU → faster-whisper (int8). Two engine policies: LocalAgreement-2 (stable) or SimulStreaming/AlignAtt (ultra-low latency).
ddx-live-tts (FastAPI multi-engine) — Kokoro-82M ONNX + Qwen3-TTS 0.6B + Piper + Chatterbox Multilingual. Docker target kokoro-tts-cpu or kokoro-tts-gpu. Streams over WebSocket and chunked HTTP for LiveKit. 24 kHz native Qwen3 quality; click-suppression already applied at v6.3.0.
LiteLLM as the one model entrypoint — explicit choice over LangChain / vllm / custom proxy because it gives us: spend ledger, virtual keys, OpenAI-SDK compatibility, model aliasing. Detail in infra-litellm.
dudoxx-gemma as the default model — set globally by USE_DUDOXX_LITELLM=true + DUDOXX_BASE_URL=https://llm-router.dudoxx.com/v1. The default thinking-off invariant (enable_thinking:false) is enforced cluster-wide for cost predictability; verified in the RAG sidecar default LLM contract.
Production CUDA embedder external — embedder.home.dudoxx.com/v1/embeddings, dudoxx-embed, dim 2560, ~70 ms typical. Same endpoint everywhere — no duplicate embedding spaces.
Why NOT all-cloud (Deepgram / ElevenLabs / OpenAI): PHI residency, latency floor, per-minute cost, and vendor lock are each individually disqualifying — together they make the cloud path commercially impossible for a German clinic at scale.
Why NOT all-on-cluster GPU: small clinics without a GPU still need to deploy. The architecture supports a "thin clinic" tier that reaches out to a regional Dudoxx-managed GPU node over Traefik; the local clinic still runs the FastAPI proxies, just configures them to point at a shared upstream.
ddx-api/prisma-main/migrations/20260211000000_add_stt_engine_settings/migration.sql:15 — historical migration moved orgs off legacy port 4300 onto 6350.
Red on any of these → dictation / voice agent / TUCAN audio fail. Most "AI is broken" tickets resolve to one of these four.
STT engine choice is per-org, not per-user — switching a clinic from Whisper to FunASR is an organizations row update on whisperSttEndpoint / funasrEndpoint. The web UI exposes this in settings-stt-engine (see ddx-web specialist memory).
Kokoro voice ID convention: af_heart, af_bella, bm_george (Kokoro), *_q suffix → Qwen3 (e.g. af_heart_q, df_katharina), German thorst* → Piper. The frontend play_audio_feedback tool validates against this list (play-audio-feedback.tool.ts:39).
KOKORO_TTS_ENDPOINT is the legacy env name — prefer KOKORO_TTS_URL. The client SDK in ddx-live-tts/clients/ts/ warns once on legacy use. New deployments MUST use the new name.
HIPAA boundary inside LiteLLM — any prompt containing FHIR PHI MUST set model: 'dudoxx-gemma'. LiteLLM does NOT inspect content. Code review is the line of defense — see infra-litellm "HIPAA boundary".
Embedder is a hard dependency — both the RAG sidecar AND in-product semantic search hit embedder.home.dudoxx.com. An air-gapped clinic must run a local mirror; production CUDA endpoint is the default at ~70 ms.
enable_thinking:false is injected for dudoxx-gemma everywhere — verified in ~/.claude/CLAUDE.md (M-USER 2026-05-17). Verify by checking reasoning_tokens: 0 in the metrics block returned by rag_investigate.
GPU sharing: the Whisper service can run on CPU (int8) for receptionist call quality but NOT for dictation. Real dictation needs CUDA float16 or Apple-Silicon MLX int8. See backend matrix at ddx-live-stt/README.md:263.
Recording archive: dictation audio is persisted to MinIO bucket {clinic-slug}-recordings — see infra-minio and infra-jitsi (shares the recording webhook pattern).
Related Topics
infra-litellm — the unification point for every LLM call; sets the model alias, budget, virtual-key story
infra-livekit — voice-agent RTC; the STT/TTS pipeline plugs in via KokoroTTSPlugin + Whisper WS
infra-jitsi — sibling video stack; NOT a substitute for the voice pipeline
live-stt — deeper dive into the Whisper / FunASR engine details