TUCAN Agent Pool — Instance Management

TUCAN Agent Pool — Instance Management

AgentPoolService is the process-wide cache of constructed Mastra Agent instances keyed by (tenantSlug, agentType) — built once per tenant + type, reused across turns until the tenant is evicted on org switch or LRU. Replaces per-request Agent construction.

Business Purpose

Constructing a Mastra Agent is not free. Each construction resolves the YAML template, filters the tool catalog by category, renders Jinja2 instructions, builds the model object, and wires memory. Doing this on every turn would:

• Burn CPU on identical constructions.
• Cause memory churn (Mastra agents hold callbacks + closures).
• Slow per-turn latency by 30-100ms (template parse + tool filter on a 350-tool catalog).

The pool gives Dudoxx a O(1) reuse with tenant-aware eviction. A clinic admin switching organizations evicts their pool entries; LRU prevents unbounded growth in long-running processes.

Audiences

• Developer / partner: To get a constructed agent for a turn, call agentPool.acquire(tenantSlug, agentType, factoryFn). The pool returns the cached instance or runs your factory once. Concurrent calls for the same key share a single construction Promise.
• Internal (ops / support): Set AGENT_POOL_MAX_ENTRIES=256 by default. Watch the stats() shape if you need to tune for large-fleet deployments. Stale agents after a tenant config change require an evictTenant(slug) call.
• Investor: One unsexy line: the pool is what lets a tenant scale to thousands of concurrent sessions on one ddx-api process without spawning thousands of duplicate agent constructions.

Architecture

AgentPoolService (singleton, process-wide)

Public API:

Callers — each calls acquire and passes a thunk () => buildAgent(...):

Identity, lifetime, concurrency

Factory thunk pattern

acquire accepts a factory function, not a constructed agent. The factory is invoked LAZILY only on miss. This means:

• Cache hit: no construction happens; the bound closure is never evaluated.
• Cache miss + concurrent calls: only the first call's factory runs; the others await the shared Promise.

This pattern is what lets the specialist and network factories preserve their "construct on first need" semantics while sharing the pool.

Tech Stack & Choices

Alternatives rejected:

• TTL-based eviction — would refresh templates that haven't changed; cost without benefit.
• Process-per-tenant pools — would multiply ddx-api memory; the tenant slug key in one pool is sufficient.
• Mastra-internal caching — would not be tenant-aware; we need the org slug in the key.

Data Flow

Business outcome: Dr. Huber (Hamburg clinic) and Dr. Schmidt (Munich clinic) both make a TUCAN turn at the same time. Each gets a constructed clinical specialist agent; the next turn for Dr. Huber reuses the Hamburg agent without re-running the template parse.

Technical mechanism:

1. Dispatcher calls AgentFactory.createAgent(context).
2. Factory builds the pool key from context.organization.slug + 'clinical'.
3. Factory calls agentPool.acquire('ddx-hamburg-clinic', 'clinical', () => specialistFactory.create({...})).
4. Pool miss → runs the factory thunk → caches the result → returns the agent.
5. Concurrent turn for Dr. Schmidt: agentPool.acquire('ddx-munich-clinic', 'clinical', () => specialistFactory.create({...})) → different key → also miss → also constructs.
6. Next turn for Dr. Huber (~1s later) → cache hit → returns immediately, bumps lastUsed.
7. Hamburg admin changes template config → API calls agentPool.evictTenant('ddx-hamburg-clinic') → next Hamburg turn reconstructs from new config; Munich is untouched.

Implicated Code

• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:46 — type PoolableAgent = Agent (Mastra Agent today; Network later).
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:47 — interface AgentPoolEntry<T> with lastUsed mutable on hit.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:54 — interface AgentPoolStats.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:61 — DEFAULT_MAX_ENTRIES = 256.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:66 — buildPoolKey(tenantSlug, agentType) exported for tests + diagnostics.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:72 — class AgentPoolService.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:74 — entries: Map<string, AgentPoolEntry>.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:75 — pendingConstruction: Map<string, Promise<PoolableAgent>>.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:81 — constructor reads AGENT_POOL_MAX_ENTRIES env via ConfigService.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:103 — acquire<T>(tenantSlug, agentType, factoryFn) public entry.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:108-115 — empty-slug defensive bypass.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:119-122 — cache-hit lastUsed = Date.now() update.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:125-128 — in-flight async construction dedupe.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:130-148 — construction Promise with finally to remove from pending map.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:154 — evictTenant(tenantSlug) prefix scan.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:173 — evictEntry(tenantSlug, agentType).
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:178 — evictAll() (admin / shutdown).
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:192 — tenantsInPool() distinct slugs.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:202 — stats() diagnostics snapshot.
• ddx-api/src/ai/tucan/pool/agent-pool.service.ts:216 — enforceMaxEntries() LRU safety net.

Operational Notes

• Tenant key MUST be org slug, NEVER UUID. agent-pool.service.ts:27-31 calls this out explicitly. Mirrors SSE contract channel rules (SSEChannels.org(slug) is slug-keyed) — UUID typechecks but produces wrong cache hits/misses across tenants.
• Explicit eviction is required on tenant config change. The template-loader doesn't know who is in the pool; the operator (or a tenant-template service hook) must call evictTenant(slug) after a template update.
• No TTL means stale agents until eviction. If you change the YAML template on disk and restart, the pool starts empty — fine. If you change tenant runtime overrides via the agent-templates REST and DON'T evict, the cached agent stays on the old config. Wire evictTenant into the override write path.
• AGENT_POOL_MAX_ENTRIES default is 256. For multi-tenant fleets with 100+ active tenants and 5 specialist types each, bump this — otherwise LRU will churn during peak hours.
• Empty tenant slug = pool bypass + log warning. Catch the warning in tests; in production it means a caller forgot to populate context.organization.slug.
• PoolableAgent = Agent today but the type is intentionally open. When the orchestrated-dispatch workflow replaces the specialist/network agents, the pool will store the new primitive without API churn.
• @Optional() configService — if no Nest config is present (unit tests, scripts), defaults to DEFAULT_MAX_ENTRIES = 256.
• Single Node thread — sync get-or-construct is safe. Multi-process scaling (cluster, PM2) requires per-process pools; this is fine because each process serves a different worker.
• Concurrent construction dedupe is per-key. Two concurrent acquires for ('hamburg','clinical') share one factory call; one for ('hamburg','clinical') and one for ('hamburg','scheduling') run their factories independently.

Related Topics

• tucan-assistant — orchestration entry that triggers acquire per turn.
• agent-templates — the YAML factory whose create* methods the pool wraps.
• tucan-agent-types — the AgentTypeKey enum that forms half of the pool key.
• tucan-context-builder — supplies the organization.slug that forms the tenant half of the pool key.
• tucan-dispatcher — consumer of pool-acquired agents in DispatcherStage.run.