/Knowledge Base
GitHub
01-infrastructurewave: W1filled21 citations

LiteLLM — LLM Router and Proxy

Audiences: developer, internal, investor

LiteLLM — LLM Router and Proxy

A single OpenAI-compatible endpoint in front of every LLM provider (self-hosted Gemma/Llama, OpenAI, Anthropic, Google) — cost-tracked, key-scoped, HIPAA-routable.

Business Purpose

LiteLLM is the only entry point any Dudoxx code is allowed to use when calling a language model. By routing 100% of AI traffic through one proxy, the platform gains:

  • Cost control & attribution — per-org, per-user, per-key spend tracked in PostgreSQL; budgets enforced; auto-shutoff on cap breach.
  • HIPAA routability — sensitive prompts can be pinned to the self-hosted dudoxx-gemma model (zero data leaves the cluster) while non-sensitive flows can opt-in to cloud (OpenAI/Anthropic) for quality.
  • Provider portability — if Anthropic raises prices 30 %, we swap claude-sonnet-4 to a Gemma alias in litellm-config.yaml; no application code changes.
  • Vendor lock-in elimination — the entire ddx-api codebase uses the OpenAI SDK; LiteLLM makes that SDK call any provider.

Revenue lever: per-organization spend caps + per-key rotation make it safe to sell "AI add-on" tiers without giving customers raw OpenAI keys.

Audiences

  • Investor: a single rate-limited, observable choke point is the only acceptable answer to "how do you control LLM cost as you scale to N hospitals?". Without it, one runaway prompt burns the margin.
  • Clinical buyer (doctor/nurse/receptionist): invisible — they see "AI scribe" / "intake summary" / "TUCAN agent"; the routing decision (local vs cloud) is policy, not user-facing.
  • Developer/partner: import the OpenAI SDK, point baseURL at LITELLM_BASE_URL, use the master key. No per-provider client code anywhere in the monorepo.
  • Internal (ops/support): dashboard at :4000/ui shows live spend, key-by-key. Budget exhaustion shows up as HTTP 429 with BudgetExceededError.

Architecture

diagram
ddx-api ───► LiteLLM Proxy :4000 (gateway) ─┬──► self-hosted dudoxx-gemma  (https://llm-router.dudoxx.com/v1)
                       │                    ├──► OpenAI (gpt-4-turbo-preview)
                       │                    ├──► Anthropic (claude-sonnet-4-20250514)
                       │                    └──► Google (gemini-pro)
                       │
                       ▼
                litellm-postgres :5433  (host, LITELLM_PG_PORT; spend, keys, budgets, users)

The stack is two containers:

  1. litellm (container ${TENANT_TRIGRAM:-hms}-litellm-main) — ghcr.io/berriai/litellm:latest, 1 worker process (--num_workers 1), mounts the read-only litellm-config.yaml defining the model_list. Health-checked at /health/liveliness.
  2. litellm-postgres (container ${TENANT_TRIGRAM:-hms}-litellm-postgres) — Postgres 15-alpine holding keys, virtual keys, budgets, spend rows on host port :5433 (LITELLM_PG_PORT, LiteLLM's own sub-stack band, distinct from the HMS 6xxx band).

Both join the same compose network (dudoxx-network). External callers (ddx-api in production) hit :4000 (LITELLM_PORT) directly; in self-hosted edge mode, Traefik fronts it at https://llm-router.dudoxx.com and adds wildcard TLS — see infra-traefik.

For deeper integration context see coding_context/ddx-hms-context.md (AI section).

Tech Stack & Choices

  • LiteLLM (BerriAI) — chosen because it is OpenAI-API-compatible (drop-in) AND has spend / key / budget management built in. Alternatives — langchain.llms, custom client, vllm proxy — all lacked the spend ledger.
  • Postgres 15 for the LiteLLM control plane — isolated from the main HMS Postgres so a LiteLLM schema migration cannot break the application database.
  • Master key + virtual keys — LITELLM_MASTER_KEY (sk-1234 in dev) generates downstream virtual keys per organization or per user; rotation is a single API call.
  • response_format + tool_choice whitelisted for dudoxx-gemma (litellm-config.yaml:22-25) — Gemma's OpenAI-shim only supports a subset of params; explicit allowlist prevents 422s.
  • num_workers: 1 (current dev/edge setting, docker-compose.yml:30) — the proxy is I/O-bound on upstream latency, so a single worker suffices for a single-clinic box; raise it (and scale horizontally) only under sustained concurrent load.
  • No Anthropic/OpenAI direct calls anywhere — enforced by code review; every embedded HTTP client uses LiteLLM's URL.

Data Flow

Outbound chat completion

  1. Application code (TUCAN, FlowAgent, scribe, summarizer) builds an OpenAI-SDK ChatCompletion.create({ model: 'dudoxx-gemma', ... }).
  2. SDK posts to ${LITELLM_BASE_URL}/v1/chat/completions with Authorization: Bearer ${LITELLM_MASTER_KEY} (or the virtual key minted for that org).
  3. LiteLLM looks up dudoxx-gemma in model_list, resolves litellm_params.model = openai/dudoxx-gemma, api_base = os.environ/DUDOXX_FORGE_API_BASE (both dudoxx and its alias dudoxx-gemma point at the forge vLLM upstream — litellm-config.yaml:2,20).
  4. Forwards the upstream call with provider-native auth; streams back SSE chunks to the caller.
  5. Records: request_id, model, input/output tokens, dollar cost, virtual key, user id → litellm-postgres.

Key management

POST /key/generate → returns sk-<rand> bound to:

  • models: ['dudoxx-gemma', 'claude-sonnet-4'] allow-list
  • max_budget, budget_duration
  • tpm_limit, rpm_limit

Application surfaces this through ddx-api/src/ai/ai-core/ddxllm.controller.ts, called from the web UI at ddx-web/src/app/api/v1/litellm/keys/{generate,delete}/route.ts.

Default-routing override (dudoxx-gemma global default)

When USE_DUDOXX_LITELLM=true (the production setting), DDXLLMConfigService falls back to dudoxx-gemma for every user that has not configured their own provider. Implementing this single env-flag + service swap is what made it possible to "flip the whole platform onto local Gemma" without touching application code.

Implicated Code

MANDATORY: ≥3 file:line citations.

  • ddx-docker-litellm/docker-compose.yml:4 — litellm service definition (container_name ${TENANT_TRIGRAM:-hms}-litellm-main), image ghcr.io/berriai/litellm:latest.
  • ddx-docker-litellm/docker-compose.yml:22 — "${LITELLM_PORT:-4000}:4000" host-port binding.
  • ddx-docker-litellm/docker-compose.yml:30 — command: ["--config", "/app/config.yaml", "--port", "4000", "--num_workers", "1"] — single worker process.
  • ddx-docker-litellm/docker-compose.yml:38 — health-check hits http://localhost:4000/health/liveliness.
  • ddx-docker-litellm/docker-compose.yml:51 — litellm-postgres Postgres 15-alpine container (host port ${LITELLM_PG_PORT:-5433}:5432).
  • ddx-docker-litellm/config/litellm-config.yaml:6 — model_list — root config that defines every callable alias.
  • ddx-docker-litellm/config/litellm-config.yaml:17 — dudoxx-gemma model definition with api_base: os.environ/DUDOXX_FORGE_API_BASE (alias of dudoxx, both → forge vLLM).
  • ddx-docker-litellm/config/litellm-config.yaml:22 — allowed_openai_params — Gemma-specific param allow-list.
  • ddx-docker-litellm/config/litellm-config.yaml:42 — general_settings.master_key: os.environ/LITELLM_MASTER_KEY.
  • ddx-api/src/ai/ai-core/ddxllm.service.ts:75 — export class DDXLLMService — typed client wrapper used by the AI domain.
  • ddx-api/src/ai/ai-core/ddxllm.service.ts:77 — private readonly litellmServer: string + litellmMasterKey: string fields read from config.
  • ddx-api/src/platform/common/config/ddxllm.config.ts:24 — interface LiteLLMConfig — shape of the per-user/per-org runtime config.
  • ddx-api/src/platform/common/config/ddxllm.config.ts:48 — USE_DUDOXX_LITELLM = process.env.USE_DUDOXX_LITELLM === 'true' — global flag that routes everyone through the self-hosted gateway.
  • ddx-api/src/platform/common/config/ddxllm.config.ts:49 — DUDOXX_BASE_URL ||= 'https://llm-router.dudoxx.com/v1' — production default.
  • ddx-api/src/platform/common/config/ddxllm.config.ts:61 — DEFAULT_MODELS = ['dudoxx', 'dudoxx-home', 'embedder'].
  • ddx-api/src/platform/common/config/ddxllm.config.ts:68 — export class DDXLLMConfigService — per-user / per-org config resolver.
  • ddx-api/src/platform/common/config/ddxllm.config.ts:241 — if (!USE_DUDOXX_LITELLM || !DUDOXX_BASE_URL || !DUDOXX_API_KEY) return DEFAULT_MODELS — fallback path.
  • ddx-api/src/organization/tenant-provisioner/templates/tenant.env.tpl:131 — LITELLM_URL={{LITELLM_URL}} injected at tenant provisioning; both LITELLM_URL and LITELLM_BASE_URL exported.

Operational Notes

  • HIPAA boundary: any prompt containing FHIR PHI MUST be routed to dudoxx-gemma (self-hosted). The application enforces this by passing model: 'dudoxx-gemma' for those code paths; the proxy does NOT inspect content. Code review is the line of defense — there is no automated content classifier in LiteLLM.
  • Budget exhaustion surfaces as HTTP 429 with body BudgetExceededError. Clients MUST handle this distinctly from rate-limit 429 (tpm_limit/rpm_limit).
  • num_workers: 1 is the current setting (docker-compose.yml:30); raise it only if the proxy becomes the concurrency bottleneck under sustained multi-tenant load.
  • Dashboard: http://localhost:4000/ui (UI_USERNAME / UI_PASSWORD env vars; default admin/admin — must be rotated in production).
  • Self-hosted upstream URL: DUDOXX_BASE_URL=https://llm-router.dudoxx.com/v1 (memory: M-USER 2026-05-17: default LLM is dudoxx-gemma with enable_thinking:false — RAG sidecar enforces this for all internal traffic; production deployments inherit the same default).
  • Postgres data volume: ./data/postgres is host-mounted. Back this up on the same cadence as primary clinic data — losing it loses key history (re-issuance possible, but spend ledger is gone).
  • Health validation:
    bash
    curl -f http://localhost:4000/health                         # gateway up
curl http://localhost:4000/v1/models -H "Authorization: Bearer $LITELLM_MASTER_KEY"  # list models

Related Topics

  • infra-postgres — pattern for spend/key/budget storage (LiteLLM uses its OWN Postgres instance, NOT the HMS main DB)
  • infra-traefik — TLS termination for llm-router.dudoxx.com in production
  • infra-redis — separate rate-limit primitive; not used by LiteLLM here (LiteLLM has its own in-memory limiter)
  • infra-full-hms-compose — production deployment references external.litellmUrl
  • CLAUDE_MASTRA_CODEBASE_USAGE — TUCAN agent uses these aliases