Condor HITL Approval — the Two-Axis Mutation Gate and the One Suspend Funnel

Condor HITL Approval — the Two-Axis Mutation Gate and the One Suspend Funnel

Human-in-the-loop approval in Condor is three layers, not one. A policy engine decides whether a tool call needs a human (tool-platform/tool-policy.ts); a workflow leaf parks the run by returning Mastra's suspend sentinel (leaf-builders/tool-call.ts); and a single pump funnel persists the suspension and emits the AG-UI frame the frontend renders as an Approve/Reject card (pump/hitl-suspend.ts → pump/enqueue-suspend-frame.ts). The historical bug class in each layer is the same shape: a layer that is correct in isolation but wired at only ONE of its two call sites.

Purpose

Condor's clinical tools write real patient data — prescriptions, medications, billing, signed documents. The approval mechanism exists so that a class of mutation always parks for an explicit human decision before it executes, while the routine mutations a doctor typed the request for are not gated into uselessness. Approval is therefore a property of the tool's effect class and risk tier, never of an individual tool author's diligence (tool-policy.ts:26-38).

Layer 1 — policy-class derivation

Every tool resolves to exactly one PolicyClass — read | mutate | exec | comms (tool-policy.ts:39-40). Resolution order is explicit-declared → static registry → metadata.mutating → read (resolveToolPolicyClass, tool-policy.ts:304-314).

Two details are load-bearing:

• The mutating derivation is a security fix, not a convenience. The 28 clinical mutating tools declare metadata.mutating: true but no explicit policyClass. Before that derivation existed, declared === undefined fell straight through to read → the engine returned allow → mutating clinical tools executed with no approval suspend at all (tool-policy.ts:284-303 records the incident: an empty allergy persisted to FHIR under autonomy: 'autonomous').
• The exhaustiveness default fails closed. An unrecognised PolicyClass returns require-approval with ruleId: 'unknown.fail-closed', never allow (tool-policy.ts:247-256).

The engine is pure, synchronous and Mastra-free by design so it can be unit-tested in isolation — ddx-api's jest does not exempt @mastra/* from transformIgnorePatterns, so a spec that transitively imports Mastra dies with an ESM SyntaxError (tool-policy.ts:10-14).

Layer 2 — the two-axis mutate gate (tool-policy.ts:182-236)

mutate is the only class with a conditional verdict. It is decided by two axes, and planApprovalPolicy is explicitly not one of them (it is void-ed at tool-policy.ts:196).

• HIGH always suspends — ruleId: mutate.high-tier-always-approval. HIGH is reserved for prescription / medication / signed-document / billing writes; 5 descriptor rows declare it today against 22 LOW.
• LOW is autonomous-allow — ruleId: mutate.low-tier-autonomous-allow. The doctor who typed the request into operator chat is the authorising human, so a LOW mutation under an autonomous dispatch runs without parking. A supervised (guarded/ambient) dispatch of the same tool still suspends (mutate.low-tier-supervised-approval).
• A destructive op promotes LOW → HIGH per call. DESTRUCTIVE_OPS (tool-policy.ts:85-92) is a 6-verb set — delete, void, remove, cancel, discontinue, archive — matched case-insensitively by isDestructiveOp. Discontinuing a medication is destructive even though the tool's declared tier is LOW.
• A missing tier is treated as HIGH. effectiveHigh folds hitlTier === undefined in alongside 'high' (tool-policy.ts:199), so an undeclared tier over-suspends rather than silently allowing.

The tier is not free-form: MutateRequiresHitlTier makes hitlTier a compile-time required field on any descriptor row whose policyClass is mutate (descriptors/tool-descriptor.ts:217-232). It reaches the engine via the execution-policy context (execution-policy.service.ts:103), and every verdict carries a stable ruleId so the firing matrix row is queryable in audit events (tool-policy.ts:121-127).

Layer 3 — the suspend → approve → resume loop

The suspend sentinel (the Mastra 1.50 trap)

execCtx.suspend(payload) returns an opaque InnerOutput control sentinel that the execution engine reads from the execute return value to transition the run to suspended. The leaf must therefore return await suspend(...) — returning any marker object of its own overwrites the sentinel, and the engine records the step FAILED in ~1 ms with no error (tool-call.ts:344-357 documents the widened local suspend type for exactly this reason). In Mastra 1.37 the return value was inconsequential; the 1.50 bump made it load-bearing.

The run-level FSM

run-spine.fsm.ts gates the transitions. suspended is an active state (:34, :47), and the suspend ⇄ executing cycle is explicit so a run may pass multiple approval gates: executing → suspended (:135), suspended → executing on resume/approve (:139), suspended → failed when the resume payload is rejected by the resume schema (:140), suspended → aborted on deny/stop/idle-sweep (:145). A resume or approve aimed at a run that is not suspended throws RunNotSuspendedError (:176-182).

The TWO emit paths — and why that is the bug class

persistHitlSuspend is documented in-source as the one suspend funnel (hitl-suspend.ts:276-288, marked with a ★ and an explicit "do NOT add a 4th path"). Every suspend routes through it, and there are exactly two call sites:

Both then hand persisted.frame to enqueueSuspendFrame (:796 and :225). A suspend frame emitted at only one of the two sites is the recurring defect class: the run parks correctly and the backend state is right, but the user never sees a card on the path that was missed, so the run hangs until the resume timeout. The two sites share one funnel precisely so the frame shape cannot drift between them.

Idempotency is keyed on suspend IDENTITY, not stepId

Emit site #2 guards on ${stepId}:${contractHash} — the identity — not on the bare stepId (suspend-resume-phase.ts:185-198). A step that re-suspends on the SAME stepId for a DIFFERENT reason (approval resolved, then the tool ran and found a required field missing → clarification) has a different contractHash because the leaf's resolvedInput is re-stamped per suspend. Under a bare-stepId guard that second suspend was swallowed, the drain loop exited with status still suspended, and the clarification form never rendered. Site #2 also adopts a first identity already framed by site #1 without re-framing it (dispatchAlreadyFramedFirst), so a single suspend never yields two cards.

readSuspendReasonCode (hitl-suspend.ts:263-267) is the single reader that classifies a payload as 'clarification' (payload carries kind: 'clarification') vs 'approval' (everything else). Both the persist path and the drain-loop guard derive the reason code through that one function so they cannot disagree — an approval suspend and a clarification suspend on the same step share a contractHash, so reasonCode is the only axis distinguishing them.

The AG-UI suspend frame contract

The pump models a suspension as a SuspendSyntheticFrame shaped { type: 'tool-output-available', toolName: '__suspend__', output }. That raw shape is not a valid AI-SDK v6 UIMessageChunk, for two independent reasons (enqueue-suspend-frame.ts:1-21):

1. the tool-output-available union member does not permit a toolName key — it fails validation with unrecognized_keys: ["toolName"];
2. an orphaned tool-output-available with no preceding part opening that toolCallId is rejected by the frontend DefaultChatTransport, which then closes the SSE stream and aborts the run before a human can approve.

So enqueueSuspendFrame emits two chunks, in order (enqueue-suspend-frame.ts:27-44):

Both emit sites import this one helper so the two-chunk sequence cannot drift. See Condor SSE Pump & AG-UI for the wider frame projection rules this rides on.

The resume schema is shape-only, and its property set is exact

buildApprovalResumeSchema() returns a JSON schema whose properties are exactly {approved, reason} (approval-resume-schema.ts:29-37). The frontend isApprovalSchema() detector allowlists keys.includes('approved') && keys.every(k => k === 'approved' || k === 'reason'), so approverId — declared optional on the backend approval schema — is deliberately excluded: including it would fail the .every() allowlist and leave the card broken while looking correct. approverId is server-derived at resume time, never user-supplied. The schema carries no default; the sole authority on whether the tool actually runs remains the resumeData.approved === true gate in leaf-builders/tool-call.ts.

The file is deliberately import-free (approval-resume-schema.ts:1-12) because its natural home, hitl-suspend.ts, pulls @mastra/core transitively via stream-readers → compiler. Keeping the builder in a leaf module makes the contract unit-testable. Do not add imports to it.

A clarification suspend takes the other branch: buildClarificationResumeSchema derives a form from the missing fields plus per-field hints — enum options, array/multi-choice flags, date formats and bounds, textarea and radio flags (hitl-suspend.ts:108-251, :326-335). Every hint reader keeps only well-shaped sub-fields, so a malformed hint degrades to free-text rather than rendering a broken picker.

Governance — every suspend is audited

A HITL suspend is a clinical-governance event and must reach the audit log. The pump has no Nest DI, so it cannot inject AuditService; instead CondorController registers its injected service at construction via registerHitlAuditSink (hitl-suspend.ts:38-79), mirroring the existing registerSuspendBridge module-singleton pattern. The funnel then emits a decision: 'suspended' entry carrying the tool, reasonCode, runId, sessionId, the op verb and the hitlTier (:351-379).

Three properties of that emission are deliberate:

• it fires after persistWorkflowSuspend, so durable suspend state exists first;
• it is best-effort — a null or failing sink never blocks the suspend, because the frontend card is the load-bearing path;
• approvalId is set to the stepId, matching the approval.required bus event id, so a suspend row joins its later approve/deny row.

The actor recorded on suspend is the run's own session/user context; the operator identity is captured at approve/deny time in the controller (condor.controller.ts:1349 POST :id/resume, :1397 POST :id/approve).

Failure modes to check first

Related Topics

• Condor HMS Tool Layer — where policyClass and hitlTier are declared on the tool descriptors this gate reads.
• Condor SSE Pump & AG-UI — the frame wire the suspend card rides on, and the two-chunk projection rules.
• Condor Session & Run Spine — the run FSM that gates executing ⇄ suspended and persists the paused session.