Phoenix Agent — Knowledge Base Mastra Agent with Hidden-Plan Retrieval

Phoenix is the Mastra agent that powers Ask Phoenix. It grounds every answer in the dudoxx_kb_techdocs Qdrant corpus via a two-tool plan-then-act protocol: a hidden kb-plan decomposition turn followed by kb-search retrieval calls — one per sub-question. The plan is private to the agent; the user only ever sees a "Planning…" chip and the final grounded answer.

Business Purpose

Investors, doctors, and partners asking about the Dudoxx HMS need a trustworthy assistant that:

1. Never fabricates — every claim must trace back to a corpus chunk ([layer/slug] citation chip rendered as a clickable deep-link).
2. Handles multi-aspect questions — a question like "Compare ddx-api authentication vs FHIR partition isolation" requires retrieving from distinct sub-topics, not a single search.
3. Refuses out-of-corpus questions politely (pricing for external products, personal advice, generic coding help) without hallucinating.

Phoenix solves the first two with the hidden-plan turn; the third with an explicit refusal protocol baked into the system prompt. It is the second tier of the Phoenix KB system — Tier 1 (CR-038) hardened output discipline; Tier 2 (CR-039) restored multi-search recall after Tier 1 inadvertently regressed it.

Audiences

• Investor: Phoenix is the public-facing demonstration that the Dudoxx HMS knowledge base is real, queryable, and grounded — not just static documentation. The same Mastra runtime that powers internal TUCAN agents serves the public Ask Phoenix surface, on a single shared LLM gateway.
• Developer / partner: To extend Phoenix's corpus, ingest new chunks into dudoxx_kb_techdocs (Qdrant collection name in ddx-web/src/mastra/tools/kb-search.ts:94). To extend its behaviour, edit PHOENIX_SYSTEM_PROMPT in ddx-web/src/mastra/agents/phoenix.ts:17 — preserve the Output-language-strict and Off-corpus-refusal sections verbatim, those are Tier 1 invariants that carry forward.
• Internal (ops / support): Phoenix has falsifiable acceptance gates measured by the 20-question eval harness (scripts/kb-phoenix-eval.ts). Any prompt change must pass AC-1 multi-search rate ≥ 10/20, AC-10 length non-inflation ≤ +20 chars, AC-22 output-language strictness, before shipping. See phoenix-kb-eval-harness.

Architecture

ddx-web/src/app/api/phoenix/chat/route.ts:21 — validates messages, builds DudoxxEnv · calls streamPhoenix(opts) → Response.body (SSE stream).

streamPhoenix (phoenix.ts:84)

buildPhoenixTools (phoenix.ts:62)

const kbPlanTurnState = {
  prevCalls: []
};
return {
  'kb-plan': tool({
    execute: input =>
      runKbPlan(input,
        kbPlanTurnState)
  }),
  'kb-search': tool({
    execute: input =>
      runKbSearch(input, env)
  })
}

The plan-then-act protocol (system prompt)

1. On every in-scope question, MUST call kb-plan FIRST with 1-4 sub-questions.
2. After kb-plan ok:true with N sub-questions, MUST call kb-search exactly N
   times — one per sub_question — BEFORE writing any reply text.
   (If N=1 run one; N=2 run two; N=3 run three; N=4 run four.)
3. NEVER call kb-plan twice for the same question. After ok:true the next
   tool call MUST be kb-search. On ok:false, retry kb-plan EXACTLY ONCE
   using its suggestion, then succeed or fall back to a single kb-search.
4. kb-plan is private — NEVER mention plan / sub-questions / tool itself
   in the reply.

Refusal-class questions (out-of-scope, vendor pricing, generic coding requests) skip kb-plan entirely and apply the refusal protocol directly.

The tool-layer anti-loop guard (T-09, CR-039)

The "max 2 kb-plan calls per turn" rule is enforced deterministically at the tool layer, not just by the prompt:

• kb-plan accepts an optional turnState: { prevCalls: [] } (closure-scoped inside buildPhoenixTools).
• On the 2nd call: if prevCalls.length >= 2 OR Jaccard similarity ≥ 0.7 between current sub_questions and previous ones (lowercased + stoplisted token sets), kb-plan short-circuits with { ok: false, suggestion: 'Plan already produced — call kb-search using the existing sub_questions, do not retry kb-plan.' }.
• The model sees ok:false, follows the prompt's existing "retry on ok:false with suggestion" rule, and the suggestion redirects to kb-search.

This pattern fixes a class of failure where dudoxx-gemma at temperature 0.3 ignored the prompt-level "max 2 calls" rule on single-identifier lookups (Q6 "port 6100", Q8 "LiteLLM master key"), looping kb-plan with synonymed sub_questions until the step budget exhausted and the answer came back empty. See phoenix-kb-eval-harness for the eval trajectory.

Tech Stack & Choices

Why hidden plan over visible plan?

Tier 1 (CR-038) suppressed a visible 5-step PLAN/SEARCH/JUDGE/SYNTHESIZE/SELF-CHECK scaffold to clean up output discipline. Side effect: multi-search recall regressed from 7/20 → 3/20 because the visible PLAN line had been autoregressively conditioning the model into multi-search behaviour. Tier 2 restores the conditioning via a structured tool call (kb-plan with typed sub_questions[]) whose existence in the model's context window primes multi-search WITHOUT leaking the plan to the user.

Data Flow

Per-question telemetry (used by the eval harness, NOT in production):

• plan_sub_question_count = tool_calls[0].args.sub_questions.length
• num_searches = count of tool_calls where name === 'kb-search'
• top_score = max score across all kb-search hits
• assistant_text = concatenated text-delta chunks (UI ALSO sees these)

Implicated Code

Mandatory ≥3 file:line citations.

• ddx-web/src/mastra/agents/phoenix.ts:17 — PHOENIX_SYSTEM_PROMPT template literal: Output-language-strict (line 27), Plan-then-act protocol (line 30), Forbidden list (line 43), Off-corpus refusal (line 53).
• ddx-web/src/mastra/agents/phoenix.ts:62 — buildPhoenixTools(env) declares kbPlanTurnState closure-scoped at line 63, wires runKbPlan
  • runKbSearch via AI SDK tool({...}) factory, returns { 'kb-plan', 'kb-search' } (kebab tool ids = matched by the model's tool calls).
• ddx-web/src/mastra/agents/phoenix.ts:84 — streamPhoenix(opts) is the AI SDK streamText call; Tier 1 invariants at lines 87/91/94 (dudoxx-gemma, temperature: 0.3, enable_thinking: false).
• ddx-web/src/mastra/tools/kb-plan.ts:106 — runKbPlan(input, turnState?): short-circuits with ok:false when prevCalls.length >= MAX_PLAN_CALLS_PER_TURN or jaccardSimilarity(...) >= JACCARD_THRESHOLD(=0.7). Otherwise calls validateDecomposition and pushes to turnState.prevCalls.
• ddx-web/src/mastra/tools/kb-plan.ts:90 — jaccardSimilarity(a, b): intersection-over-union on lowercased + stoplist-filtered tokens. Returns 0 if union empty.
• ddx-web/src/mastra/lib/decompose.ts:30 — export const STOPLIST: ReadonlySet<string> used by both validateDecomposition and kb-plan.ts:jaccardSimilarity.
• ddx-web/src/mastra/lib/decompose.ts:76 — validateDecomposition(input): 4 reject branches → { ok: false, suggestion }. AC-9 pure heuristic (zero LLM imports).
• ddx-web/src/app/api/phoenix/chat/route.ts:21 — POST handler validates the request, builds env, returns the SSE stream from streamPhoenix.

Operational Notes

Hard invariants (carry-forward from Tier 1 + Tier 2)

Pitfalls

1. Spec text vs codebase pattern: kb-search.ts exports schemas + runKbSearch and the tool({}) wrap happens in phoenix.ts:buildPhoenixTools — there is NO createTool factory in either tool file. New tools must mirror this shape, not "createTool".
2. Anti-synonym discipline: kb-plan tool description MUST avoid "retrieve", "search", "find", "lookup" — those keywords belong to kb-search. Routing confusion at temp 0.3 is the failure mode.
3. Forbidden tokens in system prompt: words like PLAN / SEARCH / JUDGE / SYNTHESIZE / SELF-CHECK MUST NOT appear as line-leading markers in the system prompt — dudoxx-gemma will echo them back into the user-visible output. Keep them inline within sentences.
4. Per-turn state must be closure-scoped: a module-level let prevCalls = [] would corrupt across concurrent users. buildPhoenixTools(env) is invoked fresh per streamPhoenix(opts) call — declare state inside that function.
5. Sub_questions cap = 4: Tier 2 spec caps decomposition at 4. The eval stopWhen: steps.length >= 8 budgets exactly 1 kb-plan + up to 4 kb-search + answer + buffer. Raising the cap requires re-budgeting stopWhen.

Validation

cd ddx-web
pnpm tsc --noEmit
pnpm eslint src/mastra/lib/decompose.ts src/mastra/tools/kb-plan.ts src/mastra/agents/phoenix.ts
# Anti-LLM grep (AC-9):
rg -n '(openai|anthropic|dudoxx-provider|@mastra/core/llm|qdrant|prisma)' src/mastra/tools/kb-plan.ts src/mastra/lib/decompose.ts | (! grep -q .)
# Falsification eval (multi-minute):
pnpm tsx scripts/kb-phoenix-eval.ts

Common defects

• Empty answer + 6+ kb-plan calls → tool-layer anti-loop guard not wired (check kbPlanTurnState declared in buildPhoenixTools and passed to runKbPlan).
• Multi-search rate ≤ 5/20 in the eval → system prompt rule 2 is missing the imperative N-times wording. Soft instructions don't carry at temp 0.3.
• CJK tokens in output → enable_thinking flipped to true, or Output-language-strict section truncated in a prompt edit.
• Citation chips render as plain text → PhoenixAnswer markdown pipeline broken or [layer/slug] payload missing layer field (kb-search filter on layer value must use the raw slug, e.g. infrastructure, NOT 01-infrastructure).

Related Topics

• Ask Phoenix — Public KB Chat Assistant
• Phoenix KB Eval Harness — 20-Question Falsification Test
• TUCAN RAG Tools — Tool Discovery and Filtering
• RAG System — Document Indexation and Retrieval
• Mastra Codebase — Monorepo AI Framework
• Qdrant — Vector Search Infrastructure