/Knowledge Base
GitHub
07-voice-and-telemedwave: W4filled14 citations

Voice Commands — AI Assistant Voice Interface

Audiences: doctor, developer, internal

Voice Commands — AI Assistant Voice Interface

Cross-surface voice command layer: a transcript segment becomes (a) a corrected text, (b) one or more detected intents (SUBMIT/CANCEL/CLEAR/NEW_LINE/STOP/PAUSE/NAVIGATE/CONFIRM/REPEAT), and (c) an auto-submit hint based on VAD silence — letting any voice surface (dictation, FlowAgent, TUCAN Operator) react deterministically to "submit", "cancel", "go to anamnesis" in EN/DE/FR without each surface re-implementing intent parsing.

Business Purpose

Doctors dictating notes, patients filling FlowAgent forms, and nurses operating the TUCAN Operator panel all need the same primitives: "send it", "no wait", "scratch that", "go to medications". Building those primitives once at the API tier — with both fast pattern matching (~1 ms, deterministic, multilingual) AND LLM-corrected intent + ASR correction for the medical vocabulary — is the difference between a voice product clinicians actually trust and a demo. The fast path keeps UI snappy; the LLM path catches German compounds and homophones that pattern matching always misses (the "neue Zeile" / "nein, eine Zeile" class).

Audiences

  • Investor: Same intent layer powers dictation, FlowAgent, and the operator panel — three SKUs, one engine.
  • Doctor: Voice commands work in EN/DE/FR without retraining. "Senden", "abbrechen", "neue Zeile" do the right thing on Day 1.
  • Developer/partner: Two endpoints — POST /voice/process (LLM-corrected, parallel agents) and POST /voice/quick-intent (~1 ms patterns) — same DTOs. RBAC-exempt (session-scoped) but JWT-protected.
  • Internal (ops/support): Pattern table lives in code (COMMAND_PATTERNS) — adding a new locale is a PR. LLM config cascade (frontend > user > org > system) means tenants can pick their own model without redeploy.

Architecture

diagram
ddx-web  (dictation widget, TUCAN drawer, FlowAgent overlays)
    │
    │ POST /voice/process       (LLM correction + intent, parallel agents)
    │ POST /voice/quick-intent  (pattern only, real-time)
    ▼
ddx-api  VoiceProcessorController
    │
    ▼
VoiceProcessorService
    ├── ResolvedLLMConfig (priority: frontend > user-db > org-db > system)
    ├── COMMAND_PATTERNS (en/de/fr, 8 intents) — fallback when LLM unavailable
    │
    ├── Correction Agent   ──┐
    │   (OpenAI-compatible)  │  Promise.all() — parallel for latency
    └── Intent Agent       ──┘
        │
        ▼
ProcessTranscriptionResponseDto
   { correctedText, detectedIntents[], primaryAction,
     autoSubmit (VAD-based), processingMs, tokensUsed }

Consumers:
  - Dictation widget  → applies correctedText + reacts to SUBMIT/CANCEL
  - FlowAgent (out-of-form intents) → STOP/PAUSE/NAVIGATE → out-of-workflow
  - TUCAN Operator panel → voice-control.tool.ts → drives operator actions
  - tucan/tools/voice/commentary.tool.ts → narrates back to user

The pattern table covers EN, DE, FR (voice-processor.service.ts:57-127) — 8 intents each. VAD silence thresholds: 2 s → suggest submit, 5 s → suggest stop (voice-processor.service.ts:137-138).

Tech Stack & Choices

  • Layer: ddx-api NestJS module under voice-video/voice/. Controller VoiceProcessorController mounted at /voice.
  • Auth: @UseGuards(JwtAuthGuard) per endpoint + @RbacExempt() (session-scoped, JWT enforced by GatewayAuthGuard).
  • LLM client: openai package via DDX_LLM_CONFIG_SERVICE resolution — Frontend > User-DB > Org-DB > System (voice-processor.service.ts:31-36). Default base URL https://llm-router.dudoxx.com/v1 (:146), fallback model dudoxx (:151).
  • Parallel execution: correction + intent agents run via Promise.all() — latency = max(corr_ms, intent_ms), not sum.
  • Pattern table: COMMAND_PATTERNS (voice-processor.service.ts:57-127) — exact keyword list per locale. ~1 ms classification, no semantics, no medical correction.
  • Intents: VoiceIntent enum — SUBMIT, CANCEL, CLEAR, NEW_LINE, STOP, PAUSE, NAVIGATE, CONFIRM, REPEAT, NONE.
  • VAD hints: 2 s → submit, 5 s → stop. Surfaces decide what to do with the hint.
  • Why two endpoints? Quick-intent for real-time UI feedback during dictation; process for accurate batch decisions at end of utterance.
  • Why not put this inside FlowAgent? Then dictation + TUCAN Operator would each re-implement intent parsing. Centralising at ddx-api means the LLM config cascade applies uniformly, the pattern table is a single source of truth, and per-tenant model substitution works without per-surface code change.

Data Flow

  1. Client widget streams audio → STT yields TranscriptionFinal segments.
  2. While streaming (real-time): POST /voice/quick-intent per partial → pattern matches → UI shows "Submit?" hint.
  3. At silence boundary (VAD): POST /voice/process with all segments + llmSettings (optional frontend override).
  4. Service resolves LLM config (frontend → user prefs → org prefs → system) and runs correction + intent agents in parallel.
  5. Response carries correctedText, detectedIntents[] (with confidence), primaryAction, autoSubmit (VAD-based), processingMs, tokensUsed.
  6. Consumer reacts:
    • Dictation: applies correctedText to the field; if primaryAction === SUBMIT and autoSubmit, submit.
    • FlowAgent: out-of-form intents (STOP/PAUSE/NAVIGATE) bubble up to the workflow bridge → terminate or jump section.
    • TUCAN Operator (tucan-operator/agent/tools/voice-control.tool.ts): translates intents into operator-side tool calls (mute, switch agent, end session, etc.).
    • Commentary tool (ai/tucan/tools/voice/commentary.tool.ts): generates spoken acknowledgement back to user.

Implicated Code

MANDATORY: ≥3 file:line citations.

  • ddx-api/src/voice-video/voice/voice-processor.controller.ts:1-10 — controller preamble; two endpoints POST /voice/process, POST /voice/quick-intent.
  • ddx-api/src/voice-video/voice/voice-processor.controller.ts:54-60 — @ApiTags('Voice'), @RbacExempt(), @Controller('voice').
  • ddx-api/src/voice-video/voice/voice-processor.controller.ts:71-122 — processTranscription route handler + Swagger doc (LLM config cascade documented in @ApiOperation).
  • ddx-api/src/voice-video/voice/voice-processor.controller.ts:125-160+ — quickIntent real-time pattern-only endpoint.
  • ddx-api/src/voice-video/voice/voice-processor.service.ts:1-13 — service preamble: parallel Correction + Intent agents.
  • ddx-api/src/voice-video/voice/voice-processor.service.ts:31-36 — ResolvedLLMConfig priority enum (frontend > user-db > org-db > system).
  • ddx-api/src/voice-video/voice/voice-processor.service.ts:57-127 — COMMAND_PATTERNS (en/de/fr, 8 intents).
  • ddx-api/src/voice-video/voice/voice-processor.service.ts:137-138 — VAD thresholds (2 s submit / 5 s stop).
  • ddx-api/src/voice-video/voice/voice-processor.service.ts:140-157 — constructor: DUDOXX_BASE_URL, DUDOXX_API_KEY, AGENT_MODEL (default dudoxx).
  • ddx-api/src/ai/tucan-operator/agent/tools/voice-control.tool.ts — Operator-side consumer.
  • ddx-api/src/ai/tucan/tools/voice/commentary.tool.ts — narration-back tool.

Operational Notes

  • LLM default model: dudoxx alias (resolved by LiteLLM at https://llm-router.dudoxx.com/v1). Per project rule (root CLAUDE.md) this maps to dudoxx-gemma with enable_thinking: false.
  • @RbacExempt() rationale: voice processing is session-scoped (one user's transcript) — RBAC interceptor would over-block. Tenancy enforced by JwtAuthGuard + TenantIsolationInterceptor. Never remove @RbacExempt without re-architecting the cascade.
  • Adding a locale: add a new key to COMMAND_PATTERNS (voice-processor.service.ts:57); add the locale to ddx-web language picker; ensure STT/TTS support that locale (CUDA Live TTS dudoxx-tts + CUDA Parakeet / on-prem Deepgram STT cover EN/DE/FR).
  • Pitfall — pattern overlap: "stop" in EN matches STOP, but "stopp" in DE could also surface as a section-skip intent in FlowAgent. The pattern table is intentionally coarse — let the LLM intent agent disambiguate when ambiguity matters; pattern is for real-time hints only.
  • Pitfall — VAD silence too aggressive: at 2 s, mid-sentence pauses trigger SUBMIT hints. Surfaces should not auto-submit on hint alone; require confirmation (CONFIRM intent or explicit click) for irreversible actions.
  • Pitfall — Correction Agent burning tokens: medical-vocabulary correction is the expensive half. If a tenant has a clean ASR (on-prem Whisper large-v3), they can disable correction via llmSettings.disableCorrection (the DTO supports this).
  • Throttling: inherits global ThrottlerModule tiers from ddx-api/CLAUDE.md (10/1s burst, 100/60s, 1000/3600s).

Related Topics

    Voice Commands — AI Assistant Voice Interface — Dudoxx Docs | Dudoxx