Voice Commands — AI Assistant Voice Interface

Voice Commands — AI Assistant Voice Interface

Cross-surface voice command layer: a transcript segment becomes (a) a corrected text, (b) one or more detected intents (SUBMIT/CANCEL/CLEAR/NEW_LINE/STOP/PAUSE/NAVIGATE/CONFIRM/REPEAT), and (c) an auto-submit hint based on VAD silence — letting any voice surface (dictation, FlowAgent, TUCAN Operator) react deterministically to "submit", "cancel", "go to anamnesis" in EN/DE/FR without each surface re-implementing intent parsing.

Business Purpose

Doctors dictating notes, patients filling FlowAgent forms, and nurses operating the TUCAN Operator panel all need the same primitives: "send it", "no wait", "scratch that", "go to medications". Building those primitives once at the API tier — with both fast pattern matching (~1 ms, deterministic, multilingual) AND LLM-corrected intent + ASR correction for the medical vocabulary — is the difference between a voice product clinicians actually trust and a demo. The fast path keeps UI snappy; the LLM path catches German compounds and homophones that pattern matching always misses (the "neue Zeile" / "nein, eine Zeile" class).

Audiences

• Investor: Same intent layer powers dictation, FlowAgent, and the operator panel — three SKUs, one engine.
• Doctor: Voice commands work in EN/DE/FR without retraining. "Senden", "abbrechen", "neue Zeile" do the right thing on Day 1.
• Developer/partner: Two endpoints — POST /voice/process (LLM-corrected, parallel agents) and POST /voice/quick-intent (~1 ms patterns) — same DTOs. RBAC-exempt (session-scoped) but JWT-protected.
• Internal (ops/support): Pattern table lives in code (COMMAND_PATTERNS) — adding a new locale is a PR. LLM config cascade (frontend > user > org > system) means tenants can pick their own model without redeploy.

Architecture

VoiceProcessorService

ProcessTranscriptionResponseDto = { correctedText, detectedIntents[], primaryAction, autoSubmit (VAD-based), processingMs, tokensUsed }

Consumers

The pattern table covers EN, DE, FR (voice-processor.service.ts:57-127) — 8 intents each. VAD silence thresholds: 2 s → suggest submit, 5 s → suggest stop (voice-processor.service.ts:137-138).

Tech Stack & Choices

• Layer: ddx-api NestJS module under voice-video/voice/. Controller VoiceProcessorController mounted at /voice.
• Auth: @UseGuards(JwtAuthGuard) per endpoint + @RbacExempt() (session-scoped, JWT enforced by GatewayAuthGuard).
• LLM client: openai package via DDX_LLM_CONFIG_SERVICE resolution — Frontend > User-DB > Org-DB > System (voice-processor.service.ts:31-36). Default base URL https://llm-router.dudoxx.com/v1 (:146), fallback model dudoxx (:151).
• Parallel execution: correction + intent agents run via Promise.all() — latency = max(corr_ms, intent_ms), not sum.
• Pattern table: COMMAND_PATTERNS (voice-processor.service.ts:57-127) — exact keyword list per locale. ~1 ms classification, no semantics, no medical correction.
• Intents: VoiceIntent enum — SUBMIT, CANCEL, CLEAR, NEW_LINE, STOP, PAUSE, NAVIGATE, CONFIRM, REPEAT, NONE.
• VAD hints: 2 s → submit, 5 s → stop. Surfaces decide what to do with the hint.
• Why two endpoints? Quick-intent for real-time UI feedback during dictation; process for accurate batch decisions at end of utterance.
• Why not put this inside FlowAgent? Then dictation + TUCAN Operator would each re-implement intent parsing. Centralising at ddx-api means the LLM config cascade applies uniformly, the pattern table is a single source of truth, and per-tenant model substitution works without per-surface code change.

Data Flow

1. Client widget streams audio → STT yields TranscriptionFinal segments.
2. While streaming (real-time): POST /voice/quick-intent per partial → pattern matches → UI shows "Submit?" hint.
3. At silence boundary (VAD): POST /voice/process with all segments + llmSettings (optional frontend override).
4. Service resolves LLM config (frontend → user prefs → org prefs → system) and runs correction + intent agents in parallel.
5. Response carries correctedText, detectedIntents[] (with confidence), primaryAction, autoSubmit (VAD-based), processingMs, tokensUsed.
6. Consumer reacts:
   • Dictation: applies correctedText to the field; if primaryAction === SUBMIT and autoSubmit, submit.
   • FlowAgent: out-of-form intents (STOP/PAUSE/NAVIGATE) bubble up to the workflow bridge → terminate or jump section.
   • TUCAN Operator (tucan-operator/agent/tools/voice-control.tool.ts): translates intents into operator-side tool calls (mute, switch agent, end session, etc.).
   • Commentary tool (ai/tucan/tools/voice/commentary.tool.ts): generates spoken acknowledgement back to user.

Implicated Code

MANDATORY: ≥3 file:line citations.

• ddx-api/src/voice-video/voice/voice-processor.controller.ts:1-10 — controller preamble; two endpoints POST /voice/process, POST /voice/quick-intent.
• ddx-api/src/voice-video/voice/voice-processor.controller.ts:54-60 — @ApiTags('Voice'), @RbacExempt(), @Controller('voice').
• ddx-api/src/voice-video/voice/voice-processor.controller.ts:71-122 — processTranscription route handler + Swagger doc (LLM config cascade documented in @ApiOperation).
• ddx-api/src/voice-video/voice/voice-processor.controller.ts:125-160+ — quickIntent real-time pattern-only endpoint.
• ddx-api/src/voice-video/voice/voice-processor.service.ts:1-13 — service preamble: parallel Correction + Intent agents.
• ddx-api/src/voice-video/voice/voice-processor.service.ts:31-36 — ResolvedLLMConfig priority enum (frontend > user-db > org-db > system).
• ddx-api/src/voice-video/voice/voice-processor.service.ts:57-127 — COMMAND_PATTERNS (en/de/fr, 8 intents).
• ddx-api/src/voice-video/voice/voice-processor.service.ts:137-138 — VAD thresholds (2 s submit / 5 s stop).
• ddx-api/src/voice-video/voice/voice-processor.service.ts:140-157 — constructor: DUDOXX_BASE_URL, DUDOXX_API_KEY, AGENT_MODEL (default dudoxx).
• ddx-api/src/ai/tucan-operator/agent/tools/voice-control.tool.ts — Operator-side consumer.
• ddx-api/src/ai/tucan/tools/voice/commentary.tool.ts — narration-back tool.

Operational Notes

• LLM default model: dudoxx alias (resolved by LiteLLM at https://llm-router.dudoxx.com/v1). Per project rule (root CLAUDE.md) this maps to dudoxx-gemma with enable_thinking: false.
• @RbacExempt() rationale: voice processing is session-scoped (one user's transcript) — RBAC interceptor would over-block. Tenancy enforced by JwtAuthGuard + TenantIsolationInterceptor. Never remove @RbacExempt without re-architecting the cascade.
• Adding a locale: add a new key to COMMAND_PATTERNS (voice-processor.service.ts:57); add the locale to ddx-web language picker; ensure STT/TTS support that locale (CUDA Live TTS dudoxx-tts + CUDA Parakeet / on-prem Deepgram STT cover EN/DE/FR).
• Pitfall — pattern overlap: "stop" in EN matches STOP, but "stopp" in DE could also surface as a section-skip intent in FlowAgent. The pattern table is intentionally coarse — let the LLM intent agent disambiguate when ambiguity matters; pattern is for real-time hints only.
• Pitfall — VAD silence too aggressive: at 2 s, mid-sentence pauses trigger SUBMIT hints. Surfaces should not auto-submit on hint alone; require confirmation (CONFIRM intent or explicit click) for irreversible actions.
• Pitfall — Correction Agent burning tokens: medical-vocabulary correction is the expensive half. If a tenant has a clean ASR (on-prem Whisper large-v3), they can disable correction via llmSettings.disableCorrection (the DTO supports this).
• Throttling: inherits global ThrottlerModule tiers from ddx-api/CLAUDE.md (10/1s burst, 100/60s, 1000/3600s).

Related Topics

• Voxial LLM — model + cascade behind the agents
• FlowAgent Scenarios — primary consumer of STOP/PAUSE/NAVIGATE for out-of-form intents
• LiveKit FlowAgent TS — agent runtime that bubbles out-of-form intents up to this service
• TUCAN Dispatcher (W3) — eventually hit by NAVIGATE and operator intents
• AI Assistant Visit Wiring — voice commands inside a visit context