/Knowledge Base
GitHub
07-voice-and-telemedwave: W4filled9 citations

FlowAgent — Voice-Driven Clinical Workflow Scenarios

Audiences: doctor, developer, internal, investor

FlowAgent — Voice-Driven Clinical Workflow Scenarios

Author-once, run-anywhere clinical form-filling assistant: clinicians author a "scenario" (form sections, agent behaviour, voice tools) in ddx-web; the runtime compiles it into a Mastra createWorkflow() chain where each section is a suspend/resume voice step — letting a patient or nurse fill an entire questionnaire by voice with deterministic completion detection, while ddx-api persists every transcript and checkpoint.

Business Purpose

Two clinical realities collide:

  1. Forms have to be filled — intake, anamnesis, consent, triage, follow-up — yet typing on a tablet kills consultation flow.
  2. LLM-only voice agents drift — they hallucinate "section complete" before all fields are filled, lock onto one question, or wander.

FlowAgent solves both by separating the scenario (clinic-authored structured form + agent persona + tool overlay) from the orchestrator (Mastra workflow that owns completion state). LLM does conversation; workflow owns advancement. This is what we sell to chronic-disease clinics (Mesalvo demo 2026-04-20), GP practices, and triage centres — a deterministic voice intake bot that the clinic can re-author for any speciality without touching code.

Audiences

  • Investor: Replaces tablet kiosks. Authoring is a no-code-ish flow in ddx-web. Same engine drives Mesalvo demo + future Direct-Medical onboarding bots — one runtime, many SKUs.
  • Doctor: Lets you reuse your existing paper or PDF form as a voice flow without IT involvement; you change a section's agentBehavior.approach and the next session uses it (after re-seed).
  • Developer/partner: Mastra createWorkflow() chain. Each section = createStep() with inputSchema / outputSchema and suspend() / resume(). Tools are scoped per step (max 7 — voice-essential set).
  • Internal (ops/support): All scenario state is versioned (FlowAgentScenarioVersion); session state and transcript live in prisma-ai. Re-seed after every form/code change (see Pitfalls).

Architecture

diagram
ddx-web (scenario author)                       ddx-api (NestJS 11)
    │                                                  │
    │ POST /flowagent/scenarios                        │
    ▼                                                  ▼
┌──────────────────────────────────────────────┐   ┌──────────────────────────┐
│ ScenarioService (CRUD)                       │   │ SessionService (lifecycle) │
│ scenario-data.mapper.ts → snapshot           │   │ runtime.service → snapshot │
│ FlowAgentScenarioVersion (versioned)         │   │ checkpoint.service         │
└──────────────────┬───────────────────────────┘   │ audit.service              │
                   │                                │ flowagent-events.publisher │
                   │ runtime fetches snapshot       └──────────────┬───────────┘
                   ▼                                               │ SSE: flowagent:ui:command
        ┌──────────────────────────────┐                           │
        │ ddx-livekit-agents-ts/       │                           │
        │   agents/flowagent (Node.js) │                           ▼
        │                              │                ┌──────────────────────┐
        │ workflow-compiler.ts         │                │ ddx-web FlowAgent UI │
        │   → Mastra createWorkflow()  │◄───────────────│ useFlowAgentSSE      │
        │ workflow-steps.ts            │  data channel  │ useFlowAgentLiveKit  │
        │   greeting / section / done  │   (state_changed,│ TUCANLiveKitRoom    │
        │   suspend/resume per turn    │    transcript)  └──────────────────────┘
        │ workflow-bridge.ts           │
        │   STT → resume(transcript)   │
        │   tool calls → fill_field…   │
        │   TTS ← sanitizeForTTS(text) │
        └──────┬───────────────────────┘
               │
               ▼
        LiveKit room ↔ patient/clinician mic

Two processes: NestJS owns authoring + persistence + SSE bridge; a separate Node process (ddx-livekit-agents-ts/agents/flowagent) hosts LiveKit + Mastra workflow + STT/TTS plumbing. (Text/agentic consumers use the AG-UI SSE bridge instead of the DataChannel — see flowagent-agui-bridge.md.)

Tech Stack & Choices

  • Framework: Mastra v1.20.0 (ddx-livekit-agents-ts/agents/flowagent/) — a DIFFERENT Mastra line from ddx-api's embedded @mastra/core (1.37.1 — verify live from node_modules/@mastra/core/package.json; the ddx-api CLAUDE.md "1.5.0"/prior "1.13.0" line is stale). Uses @mastra/core/workflows createWorkflow() + createStep().
  • Persistence: MASTRA_STORAGE_URL=file:./mastra-flowagent.db (LibSQL) for in-process memory; clinical state in prisma-ai Postgres schema (FlowAgentScenario / Version / Session, via PrismaAiService).
  • Workflow shape: greeting step → section steps × N → completion step (workflow-steps.ts:1-13). Sections suspend/resume per turn; when all fields handled, the step returns output and the chain auto-advances (no suspend() call).
  • LLM: createDudoxxMastraModel({ thinkingBudgetTokens: 0 }) — voice latency budget (see voxial-llm.md).
  • Memory: lastMessages: 20 (config.ts), Mastra Memory backed by LibSQL.
  • Voice tool set: 7 essential per phase — fill_field, skip_field, batch_fill, validate_field, update_field, get_status, clarify (NOT 26 — MEMORY_LIVEKIT.md).
  • Sanitisation: two-layer — step-helpers.ts.sanitizeForTTS() + the TTS-plugin prefilter (src/plugins/tts-prefilter.ts, Dudoxx CUDA TTS — Kokoro is deprecated) — strips <think>, working memory, markdown, "section is complete" hallucinations.
  • Silence handler: first re-prompt at 6 s, then 2 s check, subsequent at 20 s — beats LiveKit's ~8 s idle exit.
  • Transcript dedup: 3 s window prevents duplicate STT resumes.
  • Why a separate Node process? LiveKit Agents native runtime is Node + @livekit/rtc-node; embedding it in NestJS adds churn and crashes (C++ mutex on disconnect — MEMORY_LIVEKIT.md).
  • Why suspend/resume instead of LLM-driven state? LLMs cannot reliably track form completion — see [[feedback_voice_state_machine_design]] memory shard: in both v3 FSM and v4 Mastra, the orchestrator MUST detect completion; LLM produces conversation, not transitions.

Data Flow

  1. Author: clinician creates a scenario via ddx-web → ScenarioService.create (ddx-api/src/ai/flowagent/services/scenario.service.ts) → FlowAgentScenario + FlowAgentScenarioVersion with a JSON snapshot built by buildSnapshot in scenario-data.mapper.ts.
  2. Start session: ddx-web posts POST /flowagent/sessions → SessionService.create (session.service.ts) → creates a linked ChatSession (TUCAN dispatch route) → publishes SSE on SSEChannels.session(...).
  3. Compile workflow: the FlowAgent Node process fetches the scenario snapshot + form definition → workflow-compiler.compile() returns a Mastra workflow with [greeting, section_1, …, section_N, completion].
  4. LiveKit room joined: ddx-livekit-agents-ts/agents/flowagent/src/agents/flowagent-voice.ts (livekit-flowagent-ts.md) wires STT (Dudoxx CUDA / Deepgram) + TTS (Dudoxx CUDA) + LLM provider.
  5. Per turn:
    • STT TranscriptionFinal → workflow bridge.resume({ transcript }).
    • Section step calls createSectionAgent(...).generate(...) (timeout 30 s — workflow-steps.ts:33).
    • Tool calls (fill_field/skip_field) update FlowAgentState (workflow-state.ts).
    • trackFieldsFromToolCalls() updates handled-field set.
    • If allFieldsHandled → return output (auto-advance). Else → sanitised assistant text → TTS → state_changed DataChannel event.
  6. Persistence: every fill_field / skip_field hits ddx-api /flowagent/sessions/{id} via runtime.service.ts; checkpoint.service.ts snapshots state for crash recovery; audit.service.ts logs transitions.
  7. Frontend state: useFlowAgentSSE (ddx-web/src/lib/hooks/flowagent/useFlowAgentSSE.ts) + useFlowAgentLiveKit (ddx-web/src/lib/flowagent-client/hooks/useFlowAgentLiveKit.ts) read state_changed (IDLE/GREETING/NAVIGATING/COLLECTING/SECTION_COMPLETE/COMPLETING/DONE) via flowagent-client/events/data-channel-handler.ts. HTTP POST = DB persist only; SSE is the source of truth for UI state. (The text/agentic surface folds AG-UI frames instead — useFlowAgentAguiStream + createFlowAgentRunStateFold, see flowagent-agui-bridge.md.)

Implicated Code

MANDATORY: ≥3 file:line citations.

  • ddx-livekit-agents-ts/agents/flowagent/src/mastra/workflow-compiler.ts:1-9 — workflow compiler entry: scenario snapshot + form definition → Mastra createWorkflow() chain.
  • ddx-livekit-agents-ts/agents/flowagent/src/mastra/workflow-compiler.ts:39-83 — FormFieldDefinition, SectionOverlay, FieldOverlay types (the scenario contract).
  • ddx-livekit-agents-ts/agents/flowagent/src/mastra/workflow-steps.ts:1-13 — three step types (greeting / section / completion) using suspend/resume.
  • ddx-livekit-agents-ts/agents/flowagent/src/mastra/workflow-steps.ts:23-26 — createStep, MastraModelConfig, Memory imports.
  • ddx-livekit-agents-ts/agents/flowagent/src/mastra/workflow-steps.ts:33 — 30 s agent.generate() timeout to prevent LLM hangs.
  • ddx-livekit-agents-ts/agents/flowagent/src/mastra/workflow-bridge.ts — STT → resume(transcript) bridge.
  • ddx-livekit-agents-ts/agents/flowagent/src/mastra/step-helpers.ts — sanitizeForTTS, trackFieldsFromToolCalls, extractToolCalls, buildSessionContext, buildFieldProgressContext.
  • ddx-api/src/ai/flowagent/services/scenario.service.ts:1-28 — scenario CRUD + snapshot building, tenant-scoped.
  • ddx-api/src/ai/flowagent/services/session.service.ts:1-38 — session lifecycle, ChatSession linking, SSE publish via flowagent-events.publisher.
  • ddx-api/src/ai/flowagent/services/runtime.service.ts — snapshot retrieval + checkpoint orchestration.
  • ddx-api/src/ai/flowagent/services/checkpoint.service.ts — state snapshot for crash recovery.

Operational Notes

  • Re-seed after form/code changes: feedback_flowagent_scenarios_reseed.md — if you change form fields or step code, sessions started before the change have a stale snapshot. Re-seed scenarios before testing.
  • Voice FSM rule (CRITICAL): Orchestrator auto-advances on allFieldsHandled. NEVER let the LLM say "section is complete" — that hallucination is stripped by sanitizeForTTS, but if you see it in TTS output, the strip regex regressed.
  • thinkingBudgetTokens: 0 — non-negotiable for voice. Raising it adds 10-21 s latency.
  • Cleanup race: ddx-web/src/lib/flowagent-client/hooks/useFlowAgentLiveKit.ts calls stopVoiceSession on mount cleanup (React Strict Mode double-render) → known bug, tracked in MEMORY_LIVEKIT.md.
  • Silence handler: first re-prompt at 6 s, then every 2 s check, subsequent at 20 s.
  • Transcript dedup: 3 s window — important when STT emits a final immediately followed by another with corrected casing.
  • State channel: UI reads state_changed from DataChannel only. HTTP POSTs to /flowagent/sessions/{id} are DB persistence (no SSE roundtrip).
  • Mastra version: 1.20.0 in this package; ddx-api's embedded @mastra/core is a different line (1.37.1 — verify live). Do NOT cross-import between the two.

Related Topics

    FlowAgent — Voice-Driven Clinical Workflow Scenarios — Dudoxx Docs | Dudoxx