Backend Pool — Deployment Topology, High Availability and Circuit Breaker
Audiences: developer, internal
Backend Pool — Deployment Topology, High Availability and Circuit Breaker
Honest current-state: today's deployment is a single-host, multi-process stack. The managed service registry (scripts/ddx-lib.sh:39) is ALL_SERVICES="fhir ocr api saas web deepsearch pdf-engine flowagent-ts vivoxx livekit-agents-py stripe" (11 processes — note saas, livekit-agents-py, stripe are current; the older stt / vivoxx-headtts entries were removed), plus the Docker infrastructure containers (postgres, redis, minio, qdrant, livekit, jitsi, litellm), all started by ./ddx-manage.sh start and supervised by the shell script's PID tracking (no pm2, no systemd, no Kubernetes). The BackendPool code in ddx-web is fully implemented for N-node clusters but ships with BACKEND_NODES unset → falls back to a single-node configuration pointing at http://127.0.0.1:6100. The circuit breaker, health checker, load balancer, and failover retry logic all run with that one node — so what they really protect against today is "ddx-api crashed and is being restarted" rather than "host #2 failed over to host #3".
Business Purpose
A clinic that loses access to its API for two minutes during a restart should not see "Backend unavailable" splash screens for ten of those two minutes because the browser cached a dead URL. The Backend Pool makes that gap invisible to the user: when ddx-api restarts after a deploy, the ddx-web BackendPool's fast health checker re-marks the node HEALTHY within ~2 seconds and the circuit breaker closes within another ~5 seconds, so the user's next click resumes — no refresh, no error toast that requires a clinic admin to explain. The same code, with BACKEND_NODES populated, scales to a multi-node SaaS deployment without a rewrite — but that's a future-state, not a current commitment. See ha-strategy for the data-engine side of the HA story (Postgres × 2, Qdrant × 1, MinIO × 1, all single-node today).
Audiences
Investor: HA at the application tier today = "fast recovery from a single-process crash", not "multi-host failover". The code is ready for both; the production posture is the first. Moving from one node to N nodes is a config-file change (BACKEND_NODES=[{...},{...}]) plus a load-balancer in front of the ddx-web instances — no application rewrite. The same trade-off shape as the data tier in ha-strategy — durable single-node, with a clear seam for HA when revenue justifies it.
Clinical buyer (doctor/nurse/receptionist): When the system needs to restart (after a software update, or after a temporary problem), the application does not show "Server unavailable" for the full restart window. The browser pauses for one or two seconds, then resumes — the page you were on, the work you were typing, are still there. If the system stays down longer than expected, the application shows a clear status banner instead of a generic error.
Developer/partner: BackendPool is a server-side singleton in ddx-web (src/lib/api/backend-pool/pool.ts) wrapped by BACKEND_URL / BACKEND_BASE_URL exports in backend-client.ts. Every Server Component / Server Action / API route handler that hits ddx-api goes through it. The pool runs a HealthChecker that probes /api/v1/health/live every 2s and /api/v1/health/ready every 15s (config.ts:37-45) against each configured node; circuit breaker opens after 5 consecutive failures and re-tests after 5 seconds (config.ts:47-51); failover retries up to 2 times on 502/503/504 (config.ts:53-57). For a single-clinic on-prem deploy, leave BACKEND_NODES unset — the fallback config at config.ts:111-123 produces one node pointing at API_URL (default http://127.0.0.1:6100/api/v1).
Internal (ops/support): The pool is server-side only (backend-client.ts:30 — import 'server-only'). Browsers do not run circuit-breaker code; they get a banner via BackendStatusProvider (ddx-web/src/components/system/BackendStatusProvider.tsx) that polls /api/v1/health with exponential backoff and shows SystemDownView if it stays red. There is no external load balancer in front of ddx-web today; if you want one (HAProxy, nginx, an AWS ALB), put it upstream of the Next.js process(es). The pool's metrics live in memory only — pool.getMetrics() returns a PoolMetrics snapshot with per-node NodeState, but nothing scrapes that today (no Prometheus endpoint — see monitoring-observability).
Architecture
diagram
┌──────── Today: single-host, multi-process (no pm2) ─────────────────────┐
│ │
│ ./ddx-manage.sh start │
│ ├─ start_service api → pnpm start:dev → ddx-api/dist/main.js │
│ │ PID stored in ./logs/.api.pid, stdout to ./logs/api.log │
│ │ Listens on :6100 (Trusted Gateway only — see api-architecture) │
│ │ │
│ ├─ start_service web → pnpm dev / pnpm start → next │
│ │ PID stored in ./logs/.web.pid, stdout to ./logs/web.log │
│ │ Listens on :6200 (the only browser-facing port) │
│ │ │
│ └─ start_service {deepsearch | stt | ocr | pdf-engine | …} │
│ (each has its own type — node-nextjs | node-nest | python | java) │
│ │
│ Docker infra (live 6xxx dev band): │
│ postgres :6432 redis :6379 minio :6000/:6001 qdrant :6333/:6334 │
│ livekit :15880 jitsi :15180/:15543 (telemed 15xxx) litellm :4000 │
│ (containers: ddx-001-postgres-1 / -redis-1 / -minio-1 / -qdrant-1) │
│ │
│ ./ddx-manage.sh status → HTTP probes get_url() for each app │
│ + docker ps for each infra container │
└──────────────────────────────────────────────────────────────────────────┘
┌──────── ddx-web → ddx-api request path (the BackendPool seam) ──────────┐
│ │
│ Browser (Next.js client) │
│ ↓ fetch('/api/v1/patients') ← same-origin, no CORS │
│ Next.js server process (:6200, server-only code) │
│ ├─ backend-client.ts → BACKEND_URL = getPoolBackendUrl() │
│ ├─ BackendPool.getInstance().executeWithFailover(fn) │
│ │ → loadBalancer.selectNode(healthy nodes) │
│ │ → fn(node.url + '/api/v1/patients') │
│ │ → on 502/503/504: retry next node × maxRetries │
│ │ → on success: recordSuccess(latencyMs); circuit CLOSED │
│ │ → on N failures: circuit OPEN; node not selected for 5s │
│ └─ HealthChecker (background) │
│ ├─ every 2s → GET node.url/api/v1/health/live │
│ └─ every 15s → GET node.url/api/v1/health/ready │
│ ↓ │
│ ddx-api (:6100) │
│ → GatewayAuthGuard validates X-API-Key + X-Gateway-* headers │
│ → handler │
│ ← response │
│ │
└──────────────────────────────────────────────────────────────────────────┘
┌──────── Roadmap: N-node pool (not deployed; code already supports it) ──┐
│ │
│ BACKEND_NODES='[ │
│ {"id":"primary","url":"http://api-1.internal:6100","weight":100}, │
│ {"id":"replica","url":"http://api-2.internal:6100","weight":100}]' │
│ BACKEND_POOL_STRATEGY=LATENCY_BASED (or WEIGHTED_ROUND_ROBIN / │
│ LEAST_CONNECTIONS) │
│ + an external LB in front of ddx-web (HAProxy / nginx / ALB) │
│ + shared Postgres + Redis (see ha-strategy.md) │
└──────────────────────────────────────────────────────────────────────────┘
Architecture anchor: see coding_context/ddx-hms-context.md for the HMS service topology and the Browser → Next.js → NestJS Trusted Gateway flow (the pool sits at the Next.js → NestJS seam, never on the browser side). ha-strategy covers the data-engine side; this page covers the application-tier side.
Tech Stack & Choices
Pool implementation (current): Hand-rolled TypeScript at ddx-web/src/lib/api/backend-pool/. Eight files, ~1,500 lines total — pool.ts (singleton + executeWithFailover), config.ts (env-driven config loader), circuit-breaker.ts (3-state machine), health-checker.ts (background timers), load-balancer.ts (3 strategies), node.ts (per-node state), types.ts (interfaces), index.ts (public exports). No external dependencies — no opossum, no cockatiel, no resilience4j-js. Reason: keeping the resilience logic in-repo means the maintainer who has to debug a flapping node at 02:00 can read straight through the state machine without learning a third-party API.
Process supervisor (current): ./ddx-manage.sh shell script — not pm2, not systemd, not Kubernetes. Service registry lives in scripts/ddx-lib.sh:39 (ALL_SERVICES="fhir ocr api saas web deepsearch pdf-engine flowagent-ts vivoxx livekit-agents-py stripe"), with per-service get_port(), get_url(), get_name(), get_pattern() (regex for matching ps output), get_type() (node-nextjs | node-nest | node-agent | python | java | node-server), get_dir(). Lifecycle commands (start_service, stop, restart) in scripts/ddx-services.sh:14-90 spawn each service in a backgrounded subshell, write the PID to ./logs/.{service}.pid, and pipe stdout to ./logs/{service}.log. Stop uses PID first, then falls back to the regex pattern from get_pattern() — this is what stops voice agents (flowagent-ts, vivoxx) that bind no port. Why not pm2: a clinic admin who needs to restart a service does it via ./ddx-manage.sh restart api; one script, one log directory, no NodeJS-version-pin issues with pm2's daemon.
Pool singleton: BackendPool.getInstance() (pool.ts:99-104). Initialized on module load by backend-client.ts:48 (const pool = getPool();) so the first server request doesn't pay the initialization cost. The start() method begins health checking; shutdown() is called explicitly only in tests (resetInstance() at pool.ts:109). Next.js's standard module caching keeps the singleton alive across requests; a process restart resets pool state (acceptable — health checks rebuild it within the fast-check interval of 2s).
Load balancer (current selected): Default WEIGHTED_ROUND_ROBIN (config.ts:179). Two alternatives ship and are switched via BACKEND_POOL_STRATEGY env var: LEAST_CONNECTIONS (picks the node with the fewest active connections — useful when nodes have asymmetric request handlers) and LATENCY_BASED (picks the node with the lowest moving-average latency — useful in geo-distributed setups). Implementation factory at load-balancer.ts (exported via createLoadBalancer() at index.ts:57). For the current single-node config, the strategy choice is irrelevant — there's one node to pick.
Health checker (current intervals): Fast /api/v1/health/live every 2,000 ms; full /api/v1/health/ready every 15,000 ms. Probe timeout 3,000 ms. consecutiveSuccessThreshold: 1 — one success after a failure flips the node back to HEALTHY; consecutiveFailureThreshold: 3 — three failures flip it to UNHEALTHY (config.ts:37-45). The threshold-of-1 success is a deliberate change documented inline at config.ts:43 ("1 success = recovered (was 2)") — the previous threshold-of-2 created a ~4s recovery gap on restart that operators noticed in the field. The checker hits the live + ready endpoints documented in monitoring-observability §Architecture.
Circuit breaker (current thresholds): 5 failures opens the circuit; the breaker stays OPEN for 5,000 ms before transitioning to HALF_OPEN; 3 max requests allowed in HALF_OPEN before the breaker decides CLOSED (success) or OPEN (failure) (config.ts:47-51). The 5,000 ms recovery window is also documented inline as a tuning change ("5 seconds (was 30s — caused ~30s dead window on restart)" at config.ts:49) — same reason as the success-threshold tweak: ddx-api restarts faster than 30 seconds, so the old window forced a healthy node to stay blocked.
Failover retry (current): maxRetries: 2, retryDelayMs: 100, retryOnStatuses: [502, 503, 504] (config.ts:53-57). 4xx errors are not retried — those are user errors and retrying would mask them. The retry delay backs off linearly (retries++ is multiplied into the delay at pool.ts:303 — retryDelayMs * retries), so the first retry waits 100ms and the second waits 200ms. Total worst-case latency added by failover: ~300 ms before bailing out with 503.
URL normalization: localhost is rewritten to 127.0.0.1 (config.ts:148-158) because Node's DNS resolution path on macOS prefers IPv6 (::1) for localhost, but the NestJS process binds IPv4 only — a mismatch that breaks the loopback fetch with ECONNREFUSED until 127.0.0.1 is used. This is a workaround, not an elegant design; it would not be necessary if NestJS dual-stack listened by default.
Why no external LB today: The single browser-facing port is ddx-web's :6200. A reverse proxy (nginx/Caddy/Apache) is the customer's choice for TLS termination + domain routing, not ours. We do not bundle a TLS terminator because every German clinic deploy uses the customer's existing certbot/wildcard cert. When a customer needs HA at the Next.js tier, they put their existing LB in front of multiple :6200 instances; the BackendPool inside each instance still loads the same BACKEND_NODES config and picks ddx-api nodes downstream.
Why not pm2/systemd: pm2 adds a daemon that locks the Node version, complicates pnpm dev hot-reload, and surfaces its own status table that competes with ./ddx-manage.sh status. systemd works on Linux but not on the macOS dev machines several team members use. The shell script's PID + log-file pattern is portable and surfaces every service's actual stdout in one directory — easy to grep across services.
Process supervisor (roadmap): When the SaaS hosted offering arrives, replace ./ddx-manage.sh with systemd unit files (Linux production) or, for a container-orchestrated deploy, Kubernetes Deployments + Services. The HTTP health endpoints (/health/live + /health/ready — see monitoring-observability) are already Kubernetes-compatible — they 503 cleanly on failure and don't require special probe formatting.
Data Flow
Per-request failover sequence (the most common happy + sad paths):
Server Component / Server Action calls fetch(BACKEND_URL/...). BACKEND_URL is computed at module load from pool.getNodeUrl() which calls loadBalancer.selectNode(this.nodes) — but backend-client.ts typically wraps each call in pool.executeWithFailover(fn) so the node is re-selected per request.
executeWithFailover (pool.ts:218-330) builds a triedNodes: Set<string> and enters a while (retries <= maxRetries) loop.
Filter available nodes: this.nodes.filter(n => n.isAvailable() && !triedNodes.has(n.id)). isAvailable() = circuit CLOSED or HALF_OPEN with allowance.
Load balancer selects one node (round-robin / least-connections / latency-based). node.incrementConnections() and node.circuitBreaker.beforeRequest() are called before the fetch.
Run user fetch. On success: node.recordSuccess(latencyMs), return {...result, nodeId, latencyMs, retries}.
On retryable status (502/503/504): node.recordFailure(), retries++, sleep retryDelayMs * retries, continue loop.
On non-retryable status (e.g. 400/404): treat as a normal response — don't penalize the node for client errors; return the result.
On thrown error: node.recordFailure(), exit through the catch branch with failover_triggered event.
If the loop exhausts maxRetries: emit failover_exhausted, return the last error (or a generic 503 {ok:false, status:503, error:'All backend nodes are unavailable'} at pool.ts:236-243).
Health-check cycle:
HealthChecker.start() (health-checker.ts:46-73) fires one immediate check on each interval then schedules setInterval timers for both fast (2s) and full (15s) cadences.
For each node, the fast probe hits ${node.url}/api/v1/health/live with a 3s AbortController timeout and an X-API-Key: ${apiKey} header so the GatewayAuthGuard accepts the request (the live endpoint is @Public() so the API key is not strictly required, but sending it keeps the request shape consistent with all other gateway traffic).
On success: node.recordHealthSuccess(). If consecutiveSuccesses >= 1 and the node was UNHEALTHY/DEGRADED, mark it HEALTHY.
On failure (timeout / non-2xx / network): node.recordHealthFailure(). If consecutiveFailures >= 3, mark it UNHEALTHY.
The full check hits ${node.url}/api/v1/health/ready — same logic, but the readiness endpoint also pings Postgres + FHIR; a HEALTHY-by-live + UNHEALTHY-by-ready state surfaces as DEGRADED.
The circuit breaker is separate from the health checker — it tracks recordSuccess/recordFailure from actual user-request fetches, not from health-probe fetches. So a node can be HEALTHY-by-probe but have its circuit OPEN because real requests are 500ing.
Circuit breaker state machine (circuit-breaker.ts:25-104):
CLOSED — normal operation; every request passes through.
OPEN — canExecute() returns false; the load balancer's filter at pool.ts:229-230 skips this node.
After recoveryTimeoutMs (5000) elapses, the next getState() call transitions OPEN → HALF_OPEN automatically (circuit-breaker.ts:42-51); resets halfOpenRequests = 0, successes = 0.
HALF_OPEN — up to halfOpenMaxRequests (3) are allowed through. Each success increments successes; if successes reaches the threshold, transition HALF_OPEN → CLOSED. Any failure during HALF_OPEN transitions HALF_OPEN → OPEN (lastFailureTime resets — another full 5s wait).
Client component mounts on every authenticated page; useEffect calls checkBackendHealth() (BackendStatusProvider.tsx:10-28) which does a 2.5s-timeout fetch against /api/v1/health (same-origin → Next.js → BackendPool).
On failure, schedule a retry with exponential backoff: 1500ms → 2250ms → 3375ms → … → 15000ms cap (BackendStatusProvider.tsx:49-57).
While status === 'down', render SystemDownView instead of children. On recovery, fall back to 1500ms baseline and render normally.
RTO/RPO targets:
Single-node restart (today's main scenario): RTO ≈ 2-7 seconds from the moment ddx-api accepts connections again. Fast health check (2s interval) re-marks HEALTHY → next user request succeeds. RPO = 0 (no data is "lost" by the pool itself — failed in-flight requests bubble up as 503 to the calling Server Component).
Multi-node failover (roadmap): RTO ≈ 100-300 ms (one failover retry). The pool already implements this — the gap to production is operational: BACKEND_NODES config + a second host + a way to make ddx-api stateless (currently it holds in-memory caches; see src/platform/cache/).
Implicated Code
≥3 file:line citations required. Count: 12 below.
ddx-web/src/lib/api/backend-pool/pool.ts:67-128 — BackendPool class + getInstance() singleton + start() lifecycle; the entry point for all server-side requests to ddx-api. Initialized at backend-client.ts:48.
ddx-web/src/lib/api/backend-pool/pool.ts:218-330 — executeWithFailover(); the failover loop — filters by isAvailable(), calls loadBalancer.selectNode(), retries on 502/503/504 up to maxRetries, emits failover_triggered / failover_exhausted events.
ddx-web/src/lib/api/backend-pool/config.ts:37-57 — DEFAULT_HEALTH_CHECK + DEFAULT_CIRCUIT_BREAKER + DEFAULT_FAILOVER constants; the four numbers that drive recovery time (fast=2s, full=15s, circuit recovery=5s, retry max=2).
ddx-web/src/lib/api/backend-pool/config.ts:85-123 — loadNodes() + createSingleNodeConfig(); the fallback that produces a one-node config from API_URL when BACKEND_NODES is unset. This is the path running in production today.
ddx-web/src/lib/api/backend-pool/circuit-breaker.ts:25-104 — the three-state CLOSED → OPEN → HALF_OPEN → CLOSED machine with automatic timeout-driven OPEN → HALF_OPEN transition at getState() (circuit-breaker.ts:42-51).
ddx-web/src/lib/api/backend-pool/health-checker.ts:31-73 — HealthChecker.start(); sets up the setInterval timers for fast (2s) and full (15s) probes, performs an immediate check on startup so the pool knows node state before the first user request.
ddx-web/src/lib/api/backend-pool/types.ts:39-48 — NodeHealth = 'HEALTHY' | 'DEGRADED' | 'UNHEALTHY' | 'UNKNOWN' and CircuitBreakerState = 'CLOSED' | 'OPEN' | 'HALF_OPEN'; the canonical state vocabulary used across pool + load-balancer + circuit-breaker.
ddx-web/src/lib/api/backend-client.ts:30-80 — import 'server-only' + pool initialization + BACKEND_URL / BACKEND_BASE_URL / GATEWAY_API_KEY exports; the file every server-side API caller imports.
ddx-web/src/components/system/BackendStatusProvider.tsx:10-60 — client-side health-watch with exponential backoff; renders SystemDownView when /api/v1/health stays red; the only browser-visible piece of the HA story.
ddx-api/src/platform/health/health.controller.ts:161 (getLiveness(), the fast-check target) and :191 (getReadiness(), the full-check target); the endpoints the BackendPool HealthChecker hits, also documented in monitoring-observability §Architecture.
scripts/ddx-lib.sh:39 — the ALL_SERVICES registry (+ get_port / get_url / get_pattern / get_type / get_dir helpers) that drives the shell-script lifecycle. Single source of truth for what "managed services" exist today.
scripts/ddx-services.sh:14-90 — start_service(); the data-driven service launcher that handles node-nextjs | node-nest | node-agent | python | java | node-server types, spawns each in a subshell, writes PID files, pipes stdout to ./logs/{service}.log.
Business → Technical match.Business outcome: a deploy that takes 30 seconds to swap binaries should not produce a 30-second "Backend Unavailable" splash for the user mid-edit. Technical mechanism: ddx-api restart kills the listener on :6100. The next user request through Next.js hits pool.executeWithFailover(); the fetch errors (ECONNREFUSED); node.recordFailure() increments, and after 5 failures the circuit OPENS for 5 seconds (circuit-breaker.ts:configured at config.ts:47-51). During those 5 seconds the request itself bubbles up as a 503; BackendStatusProvider shows the banner. ddx-api comes back up, the HealthChecker's 2s fast probe succeeds (health-checker.ts:31-73), the node returns to HEALTHY; the circuit auto-transitions OPEN → HALF_OPEN after 5s; the next 3 user requests succeed → circuit CLOSES. Total user-visible disruption: ~7-12 seconds, depending on where in the cycle the restart landed.
Operational Notes
The default config is the production config. BACKEND_NODES unset → one-node pool pointing at http://127.0.0.1:6100 via createSingleNodeConfig() (config.ts:111-123). Do not set BACKEND_NODES to a JSON array unless a second ddx-api host actually exists — an unreachable second node will inflate health-check noise without improving availability.
./ddx-manage.sh restart api is the canonical deploy step. It stops the api PID (or falls back to the get_pattern api regex if the PID is stale), waits for the port to free up, restarts via pnpm start:dev or pnpm start:prod. The BackendPool inside ddx-web handles the gap automatically — no need to also restart ddx-web.
Tune the four numbers in config.ts before adding new code. Fast interval, full interval, circuit recovery, failure threshold — the comments at config.ts:38-50 document why these moved from the original defaults; respect that history before tuning them again. The consecutiveSuccessThreshold=1 and recoveryTimeoutMs=5000 choices specifically reflect operator feedback that earlier values produced visible dead-windows on restart.
There is no metrics export today. pool.getMetrics() returns a rich PoolMetrics object (per-node state, latencies, request counts) but nothing scrapes it. If you need observability of pool behaviour, add a /api/internal/pool-metrics route in ddx-web that returns the snapshot; do not add a Prometheus client (prom-client) without a discussion — the project does not pull Prometheus today and adding it crosses an architectural boundary.
The pool only knows about ddx-api. Other services (deepsearch, stt, ocr, pdf-engine, flowagent-ts, vivoxx, fhir) are accessed via direct URLs from server code — no failover, no circuit breaker. If any of them needs the same HA treatment, the pattern is replicable: copy the pool to a new module, swap the health endpoint, give it its own singleton. None of them needs it today because none is on the user's critical request path the way ddx-api is.
Voice agents have no port to probe. flowagent-ts and vivoxx are outbound LiveKit workers — get_port() returns empty (scripts/ddx-lib.sh:50-51), get_url() returns empty (scripts/ddx-lib.sh:80-81). ./ddx-manage.sh status reports them based on PID file alone; stop falls back to the get_pattern regex. Don't try to add them to the BackendPool.
pm2 status will return nothing. The project does not use pm2. The status surface is ./ddx-manage.sh status (table) or ./ddx-manage.sh status --json (machine-readable) — see monitoring-observability §Architecture.
The IPv6/IPv4 localhost gotcha is real. If you change localhost back into a node URL, the loopback fetch will fail with ECONNREFUSED on macOS. The rewrite at config.ts:148-158 is load-bearing; do not undo it without testing on macOS.
Body parser limit is 150 MB. Set in ddx-api at src/main.ts:80-93 to accommodate base64-encoded FHIR DocumentReference uploads. If the pool's failover retry sends the same large payload twice and both attempts fail downstream, the user sees 503 after ~300 ms of retry budget — short enough not to feel like a hang.
Client-side fallback uses exponential backoff with a 15s cap. BackendStatusProvider.tsx:50-52 (Math.min(backoffMsRef.current * 1.5, 15000)). The cap matters because in a long outage the polling load on Next.js stays bounded; without the cap, an idle laptop would silently grow its polling interval into the minutes range and miss the recovery.
Roadmap candidates (no code today; record as design intent):
N-node deployment: populate BACKEND_NODES, add an external LB in front of ddx-web, make ddx-api fully stateless (today it holds in-memory caches; audit src/platform/cache/ and the SSE event-bus channels). Replace ./ddx-manage.sh with systemd unit files or Kubernetes Deployments.
Pool metrics scrape: expose pool.getMetrics() over an internal route (or migrate to pino + push to Loki/Datadog — see monitoring-observability §Tech Stack on why pino is not in yet).
Per-service breakers for downstream calls: ddx-api itself does not wrap its FHIR / Qdrant / MinIO / LiteLLM calls in circuit breakers. A LiteLLM outage today produces immediate 5xx responses that fail through to the user; a small breaker there would let the agent fail fast with a clear UI message rather than a per-tool timeout.
Validation commands:
./ddx-manage.sh status — table of all managed app services (ALL_SERVICES, 11 today) + infra containers.
./ddx-manage.sh restart api — exercise the recovery loop end-to-end.
curl -fsS http://localhost:6100/api/v1/health/live — the fast-check probe URL the pool hits every 2s.
curl -fsS http://localhost:6100/api/v1/health/ready — the full-check probe URL the pool hits every 15s.
DEBUG_BACKEND_POOL=true pnpm dev (in ddx-web) — enables isPoolDebugEnabled() (config.ts:274) and prints [BackendPool]-prefixed lines for every state transition.
tail -F logs/api.log to watch ddx-api restart timing; tail -F logs/web.log to watch BackendPool decisions.
Related Topics
Monitoring and Observability — Health, Logging and Jobs — the /health/live + /health/ready + /health/databases + /health/sse endpoints the pool's HealthChecker probes, plus the access-log table that captures every request the pool dispatches.
Canonical operational references (not duplicated here): ddx-web/CLAUDE.md §Key Files (BackendPool row), ddx-api/CLAUDE.md §Architecture (Browser → Next.js → NestJS Trusted Gateway diagram), ddx-presentation/CLAUDE.md §What this folder is (the deployment context for how these services are documented and demo'd).
Backend Pool — Deployment Topology, High Availability and Circuit Breaker — Dudoxx Docs | Dudoxx