Phoenix KB Eval Harness — 20-Question Falsification Test

The Phoenix KB eval is the falsification gate for every CR that touches Ask Phoenix. It runs a fixed 20-question corpus (14 in-scope + 6 refusal-class) against the live Mastra agent and emits a per-question record + a Markdown report with computable Acceptance Criteria. No CR ships without a passing eval run.

Business Purpose

Phoenix's value depends on three properties that are easy to break in subtle ways:

1. Multi-aspect questions get multi-search retrieval (recall, not just precision).
2. Refusal-class questions get refused (no hallucination of out-of-corpus answers).
3. Output stays clean (no CJK leaks, no protocol meta-narration, no length inflation, refusal protocol triggers before kb-plan on refusal questions).

The 20-Q eval is a fixed corpus that converts each property into a falsifiable AC. When AC-1 (multi-search recall) regressed from 7/20 → 3/20 in Tier 1 (CR-038), the eval caught it within one run. When CR-039's first Tier 2 attempt made it WORSE (1/20), the eval caught that too. The harness is the load-bearing artifact that lets the Phoenix system evolve without silent regression.

Audiences

• Developer: Run the eval after any change to phoenix.ts, kb-plan.ts, decompose.ts, or kb-search.ts. The eval is the fastest way to know whether a prompt edit actually helped or just changed the failure mode.
• Internal (ops / support): Compare eval reports across releases to spot regressions. Each report stamps a timestamp in findings/kb-phoenix-eval/YYYY-MM-DD-HH-MM/report.md so trends are trivially diff-able.
• Partner: Shipping a partner-side KB ingestion (adding new content to the corpus) MUST be followed by a fresh eval run to verify the new chunks don't shift the multi-search distribution unexpectedly.

Architecture

ddx-web/scripts/kb-phoenix-eval.ts pipeline:

The 20-question corpus

The corpus is fixed by design — changing it would invalidate trend comparisons across releases.

Acceptance Criteria (per CR-039 Tier 2)

Tech Stack & Choices

• Runner: pnpm tsx scripts/kb-phoenix-eval.ts (no compile step; tsx for speed of iteration).
• Record schema: Zod (EvalRecord in the script). Every field is REQUIRED — missing values default to safe sentinels (null for scores, 0 for counts) so the report never crashes on partial data.
• Helper: extractPlanSubQuestionCount(tool_calls) at scripts/kb-phoenix-eval.ts:70 — defaults to 0 when: empty tool_calls, first call is not kb-plan, args not an array, any unexpected shape. Belt-and-braces extraction.
• Report: Markdown with summary table + per-question detail. JSON twin (report.json) for programmatic comparison across runs.
• Output: findings/kb-phoenix-eval/<YYYY-MM-DD-HH-MM>/ — timestamped folders prevent overwrites. Git-ignored by default (findings are ephemeral); drop .keep-in-git + git add -f to promote a specific run as a durable baseline.

Data Flow

streamPhoenix(...) → async iterator of stream events:

After the stream ends, build the record:

{
  idx, question,
  finish_reason, num_searches, top_score,
  citations_count, citations_unique,
  meta_protocol_leak (regex on assistantText for PLAN:/SEARCH:/...),
  cjk_leak (regex on assistantText for CJK ranges),
  assistant_text, tool_calls,
  answer_len_tokens (whitespace word count proxy),
  plan_sub_question_count,
  latency_ms_ttft, latency_ms_total
}

Implicated Code

Mandatory ≥3 file:line citations.

• ddx-web/scripts/kb-phoenix-eval.ts:70 — extractPlanSubQuestionCount(toolCalls): required helper that wires AC-4 across every emission path (success, refusal, error, timeout). Returns 0 on any unexpected shape; never throws.
• ddx-web/scripts/kb-phoenix-eval.ts:307 — streamPhoenix(question, flags): per-question driver that invokes the real Mastra agent and aggregates stream events into an EvalOutcome.
• ddx-web/scripts/kb-phoenix-eval.ts:496 — Record emission point: plan_sub_question_count: extractPlanSubQuestionCount(outcome.tool_calls). Centralized in runQuestion so all paths land here.
• ddx-web/scripts/kb-phoenix-eval.ts:560 — multi_subq_rate aggregation surfaced in the report header (rows with plan_sub_question_count >= 2), for visibility.
• ddx-web/scripts/kb-phoenix-eval.ts:666 — main(): orchestrates the 20-question loop, writes the timestamped findings/kb-phoenix-eval/<stamp>/ folder.
• findings/kb-phoenix-eval/2026-05-19-09-17/report.md — pre-CR-039 regression baseline (3/20 multi-search). Reference for AC-10 length delta computation.
• findings/kb-phoenix-eval/2026-05-19-10-59/report.md — current passing baseline (12/20 multi-search, Q6+Q8 fixed by T-09 tool-layer anti-loop guard).

Operational Notes

When to run

• Before any PR that touches: phoenix.ts, kb-plan.ts, decompose.ts, kb-search.ts, kb-phoenix-eval.ts, or the Phoenix prompt-anywhere.
• After any Qdrant dudoxx_kb_techdocs re-ingest (new content shifts retrieval scores → multi-search distribution may change).
• After any LLM gateway model swap (different model = different tool-routing behaviour).
• Periodically (weekly cron candidate) to catch corpus drift — topics ingested elsewhere that affect retrieval.

How to run

cd ddx-web
pnpm tsx scripts/kb-phoenix-eval.ts
# Output: findings/kb-phoenix-eval/<YYYY-MM-DD-HH-MM>/report.md

Wall-clock: ~3 minutes for 20 questions on a warm Qdrant cache. Each question runs sequentially; concurrent eval is NOT supported (would corrupt latency measurements + the kb-plan turnState test).

Reading the report

The report header carries the load-bearing summary:

- Rows: 20
- finish_reason distribution: stop: 20
- Total latency: 41896 ms
- Presentation hardening (CR-038 T-09): meta-protocol leaks=0/20, cjk leaks=0/20
- Plan decomposition (CR-039 T-04): multi_subq_rate=65.0% (rows with plan_sub_question_count >= 2)

A healthy run shows:

• 20/20 finish_reason=stop (never tool-calls — that means truncated)
• 0/20 leaks
• multi_subq_rate ≥ 50% (matches AC-1 floor)
• 14/14 non-refusal questions have num_searches ≥ 1
• 6/6 refusal questions have num_searches = 0 and finish_reason=stop

Iteration trajectory (CR-038 + CR-039)

The eval has captured three landmark trajectories worth knowing:

Each landmark surfaced an architectural lesson:

• 3/20 → 1/20: Hidden plan turn alone isn't enough — soft prompt instructions don't carry weight at temp 0.3.
• 1/20 → 12/20: Imperative MUST clauses with concrete numerical expansion ("If N=1 run one. If N=2 run two...") work where prose doesn't.
• Q6+Q8 fix: Prompt-level "max N calls" rules are unreliable — move enforcement to deterministic tool-layer state.

Pitfalls

1. Latency comparison across runs: each run uses real LLM calls; latency varies with gateway load. Compare medians across at least 3 runs before flagging a regression. AC-5 (mean delta ≤ +600ms) is informational; do not gate a CR on a single-run latency.
2. Re-running an eval is expensive: each run = 20 real LLM calls × multiple tool steps. Treat as a precious resource. If AC-1 fails by >2 points, the failure is structural (prompt or tool problem) — file a defect, do NOT rerun expecting a different result.
3. Citation count comparison: citations_count and citations_unique are sensitive to which Qdrant chunks the search hits. A re-ingest can shift these without any code change. Don't gate on citation deltas across re-ingests.
4. Refusal-class question order: questions 16–20 + Q4 (Mastra dispatcher) are the refusal-class set in the current corpus. The refusal classification is implicit — derived from the eval's num_searches=0 + finish_reason=stop pattern. Renaming questions changes their index but not their class.
5. Eval is single-tenant: the harness uses a shared Qdrant collection. Multi-tenant Phoenix surfaces (if ever introduced) would need a per-tenant eval variant.

Validation

# Type + lint the eval script itself
cd ddx-web
pnpm tsc --noEmit
pnpm eslint scripts/kb-phoenix-eval.ts
# Spot-check the latest report
ls -t findings/kb-phoenix-eval | head -1 | xargs -I{} head -30 findings/kb-phoenix-eval/{}/report.md

Promoting a baseline run

To keep a specific eval run as a durable baseline (for CR comparison):

RUN=findings/kb-phoenix-eval/2026-05-19-10-59
touch $RUN/.keep-in-git
git add -f $RUN/

Related Topics

• Phoenix Agent — Architecture and Hidden-Plan Retrieval
• Ask Phoenix — Public KB Chat Assistant
• Monitoring and Observability — Health, Logging and Jobs
• Qdrant — Vector Search Infrastructure
• RAG System — Document Indexation and Retrieval