High-Availability Strategy — Database, Vector Store, Object Storage

High-Availability Strategy — Database, Vector Store, Object Storage

Honest current-state: Dudoxx HMS today ships as a single-node, snapshot-restore stack. In the dev/on-prem ddx-docker-* layout that is one consolidated Postgres (ddx-001-postgres-1, :6432, hosting all app DBs and ddx_fhir_core), Qdrant × 1, MinIO × 1, each restart: unless-stopped with a named volume. (The multi-tenant full-hms compose still splits postgres-main + postgres-fhir — see infra-full-hms-compose.) There is no replication, no clustering, and no automatic failover in any engine. HA is a roadmap topic; what exists today is durable single-node operation plus a pre-migration backup script that dumps all five Prisma databases, and ddx-api/scripts/safe-reset.sh which co-snapshots app DBs and ddx_fhir_core before any reset (guards the FHIR↔Prisma dual-store drift, memory [[project-fhir-prisma-dual-store-drift]]).

Business Purpose

Healthcare platforms must answer two operational questions independent of each other: what happens when a schema migration goes wrong? and what happens when an entire host dies? Today the Dudoxx HMS stack answers the first question with disciplined per-database pg_dump snapshots (the backup-before-migrate.sh script runs on every pnpm run prisma:migrate:deploy) and the second question with a single-host recovery procedure — start the containers, restore the volumes, restore the FHIR partition catalog. This is the appropriate posture for an on-prem single-clinic install where the host is also the customer's own hardware: clinics don't pay for a second machine that they don't need, and migrations — not hardware failures — are by far the more frequent recovery trigger. The HA roadmap below describes what an enterprise/multi-clinic SaaS install would add on top of this baseline.

Audiences

• Investor: HA in healthcare = "can the clinic still see patients if the server reboots?" Today the answer is yes, after a few minutes of container restart, with zero data loss from the most recent committed transaction (Postgres durable WAL, MinIO durable writes, Qdrant durable WAL). What is not yet on the table is sub-minute RTO via hot-standby — that lives in the roadmap and is the natural next investment when we move from single-clinic on-prem to multi-clinic SaaS hosting.
• Clinical buyer (doctor/nurse/receptionist): The platform survives a power cut, a kernel update, and a botched ORM migration — the last because a fresh pre-migration dump exists for every one of the five application databases at ddx-api/backups/<timestamp>/. What it does not yet survive (without a manual restart cycle) is a hardware failure on the host itself; that scenario is handled by your IT department's host-level backup process, same as any other on-prem application you run.
• Developer/partner: All three engines are intentionally single-node. There is no Postgres physical replication, no Qdrant raft cluster (cluster: enabled: false in production.yaml), and no MinIO erasure coding (the command is server /data, not server /data{1...4}). HA decisions live downstream of two facts: (1) Hibernate's FHIR schema cannot tolerate being rewritten by Prisma → ddx_fhir_core is a distinct DATABASE (in the dev stack, inside the same ddx-001-postgres-1 instance; in full-hms, its own postgres-fhir container) with its own backup path, and Prisma carries no connection string to it; (2) Qdrant snapshots and MinIO buckets are functions of business state already persisted in the application Postgres, so a full recovery is rebuildable from ddx_api_main + the FHIR partition catalog + the MinIO data directory + the most recent Qdrant snapshot.
• Internal (ops/support): The only automated HA-adjacent surface that ships today is pnpm run prisma:backup (see ddx-api/package.json:78), which is invoked automatically by the migration tooling — see pnpm run prisma:migrate:deploy in ddx-api/CLAUDE.md. Qdrant snapshots and MinIO bucket replication are manual operator actions; the directories and policies exist but no cron emits them. Adding cron-driven backups is a small task — see Operational Notes below.

Architecture

Current state (2026-05) — what actually ships today:

Backup chain (ddx-api/scripts/backup-before-migrate.sh):

Roadmap (NOT implemented) — none of the following exists today:

Architecture anchor: see coding_context/ddx-hms-context.md for the HMS service topology and the five-Prisma-schema layout that drives the backup strategy. The Prisma-vs-Hibernate split exists primarily for data isolation — in the dev stack it is a database-level split inside one instance; in full-hms a container-level split (postgres-main/postgres-fhir), see infra-postgres §Architecture — and only secondarily for "independent backup chains"; once HA is added each will get its own replica.

Tech Stack & Choices

• Postgres replication mode (current): None. Both Postgres containers run as standalone single-node instances pinned to postgres:16-alpine (ddx-docker-postgres/docker-compose.yml:30, :87). postgresql.conf is the production-tuned profile (shared_buffers=2GB, effective_cache_size=6GB, max_wal_size=4GB) but does not enable wal_level=replica, archive_mode, or any standby-related parameters — see ddx-docker-postgres/config/postgresql.conf:1-57.
• Postgres replication mode (roadmap): Native streaming replication (one warm standby per primary) is the obvious next step — it requires only a config change (wal_level=replica, max_wal_senders, archive_mode=on, plus primary_conninfo on the standby) and a second compose service per database. Logical replication is the alternative if we ever need to replicate only certain databases (e.g. ship ddx_api_log to a long-term archive without copying clinical writes). pgBackRest is the candidate for point-in-time recovery once we move past pg_dump.
• Qdrant single-node-vs-cluster (current): Explicit cluster: enabled: false at ddx-docker-qdrant/config/production.yaml:51-52. Qdrant cluster mode uses raft consensus and at least 3 nodes — overkill for a single-clinic on-prem deployment where the embeddings can be rebuilt from ddx_api_main source data + the CUDA embedder.
• Qdrant snapshot strategy (current): A snapshots directory is bind-mounted from ./snapshots into /qdrant/snapshots (ddx-docker-qdrant/docker-compose.yml:38), so an operator can call POST /collections/{name}/snapshots against the Qdrant HTTP API and the resulting file lands on the host filesystem. There is no cron triggering this — snapshots today are operator-initiated. Qdrant's WAL is configured (wal_capacity_mb: 256, wal_segments_ahead: 2 at production.yaml:24-26) so the engine itself is durable across restarts; the missing piece is automated off-box copy.
• Qdrant backup-to-MinIO cadence (roadmap): The cleanest pattern would be a NestJS-side cron (or a tiny sidecar) that calls POST /collections/{name}/snapshots, then mc cp the resulting file to a system MinIO bucket (clinic-temp or a new system-backups bucket) on a nightly cadence. None of this exists yet; nothing in ddx-api/src/platform/qdrant/ references snapshot APIs.
• MinIO erasure coding (current): Not configured. The command in ddx-docker-minio/docker-compose.yml:28 is server /data --console-address ":9001" — single-disk single-node mode. Erasure coding requires server /data{1...N} with N ≥ 4 drives (or 4 paths within the container) and a parity scheme; nothing in the current compose maps multiple data paths.
• MinIO replication mode (current): Not configured. The minio-init container (docker-compose.yml:68-169) sets up system buckets, encryption (SSE-S3), versioning, the ddx_api_user service account, and the all-clinics-rw policy — but does not call mc replicate or set up any site-replication. The encryption (mc encrypt set sse-s3 at docker-compose.yml:102-104) plus versioning (mc version enable at :96-98) give us protection against accidental overwrites and at-rest reads, but not against site loss.
• MinIO data-residency considerations: Region is hard-coded to eu-central-1 (docker-compose.yml:37) and the MinIO console runs on localhost only — both choices reinforce the on-prem German-clinic deployment target. When site-replication is added it will need to respect the same residency boundary (replicate to a second EU region, not to a US bucket).
• Cross-engine backup orchestration (current): Only Postgres has automated backup. The shell script at ddx-api/scripts/backup-before-migrate.sh:1-72 dumps all five Prisma databases per migration cycle, writes timestamped output to ddx-api/backups/<TIMESTAMP>/, and prunes anything older than 7 days (backup-before-migrate.sh:67-71). It is wired into pnpm run prisma:backup (ddx-api/package.json:78) and called automatically by prisma:migrate:deploy.

Data Flow

Current restore sequence (single-host recovery, manual):

1. Stop application: pm2 stop ddx-api ddx-web (or whichever process manager). Application stop is a precondition — restoring under live writes corrupts referential integrity between the engines.
2. Restore Postgres app: gunzip < backups/ddx_api_main.dump | docker exec -i ddx-001-postgres-1 pg_restore -U dudoxx_user -d ddx_api_main --clean (and repeat for log/ai/com/acc). The pattern is documented at ddx-docker-postgres/README.md:185-206.
3. Restore Postgres FHIR: same shape, against the ddx_fhir_core DB in ddx-001-postgres-1 in the ddx_fhir_core DB (same :6432 instance). CRITICAL: never restore the FHIR dump using Prisma — only pg_restore against ddx_fhir_core. See infra-postgres Operational Notes.
4. Restore MinIO: rsync/copy the named volume ddx-001-minio-data-1 from the off-host backup. Object versioning means the latest copy of each binary is durable as long as the volume image is intact.
5. Restore Qdrant: copy the most recent snapshot file into ./snapshots/, then POST /collections/{name}/snapshots/upload to re-hydrate. If no snapshot exists, the embeddings can be rebuilt from ddx_api_main text content + the CUDA embedder (embedder.home.dudoxx.com) — this is the planned fallback because embeddings are derived data, not source-of-truth.
6. Restart application — Trusted Gateway header flow (see infra-postgres) resumes without ceremony once the four engines are healthy.

Order matters because (a) Postgres app must be healthy before MinIO is fronted (the Prisma attachments rows are the source-of-truth for what the buckets should contain), and (b) Qdrant should be the last to come up because it's the only engine whose data can be reconstructed from the others — bringing it up last lets operations confirm a clean state before re-embedding.

Roadmap failover sequence (when streaming replication exists): primary unreachable → standby promoted (manual pg_ctl promote or repmgr/Patroni-driven), connection string flipped via env-var swap or HAProxy in front, application restart, Qdrant snapshot freshness checked, MinIO replica activated. Sub-minute RTO is realistic with Patroni + etcd; without orchestration the realistic RTO is "as long as your on-call takes to ssh in".

RTO/RPO targets:

• Current single-node: RPO ≈ "since last prisma:backup run" (typically pre-migration cadence — could be hours to days for the apps that don't ship migrations frequently). RTO ≈ minutes (container restart + volume mount) for soft failures, hours for hardware failures because the volume must be re-attached or restored from off-host backup.
• Roadmap with streaming replication + scheduled snapshots: RPO ≤ 1 minute (WAL streaming lag), RTO ≤ 5 minutes (standby promotion + DNS/env-var flip).

Implicated Code

≥3 file:line citations required. Counts: 9 below.

• ddx-docker-postgres/docker-compose.yml:25 — the single consolidated postgres service (ddx-001-postgres-1, :6432); restart: unless-stopped is the only HA-adjacent directive today. This one instance hosts all app DBs + ddx_fhir_core (the dev stack no longer runs a separate postgres-fhir container; full-hms still does).
• ddx-docker-postgres/scripts/init-fhir-database.sh — creates/initialises ddx_fhir_core inside the same instance; its dump is a distinct backup artifact from the Prisma DBs and MUST be restored via pg_restore, never Prisma.
• ddx-docker-postgres/config/postgresql.conf:15-19 — WAL configuration that is durable but not replication-enabled (wal_buffers=16MB, max_wal_size=4GB, no wal_level=replica).
• ddx-docker-postgres/README.md:183-206 — documented manual backup + restore commands (per-DB pg_dump | gzip and gunzip | psql).
• ddx-docker-qdrant/config/production.yaml:51-52 — explicit cluster: enabled: false; the file also documents the WAL (production.yaml:24-26) and snapshots path (production.yaml:20) that make single-node durability work.
• ddx-docker-qdrant/docker-compose.yml:38 — ./snapshots:/qdrant/snapshots bind-mount; without this the operator-triggered snapshot API has nowhere to land output.
• ddx-docker-minio/docker-compose.yml:28 — command: server /data --console-address ":9001" — single-disk MinIO; no mc replicate or server /data{1...4} erasure-coding pattern.
• ddx-docker-minio/docker-compose.yml:95-104 — versioning (mc version enable) + SSE-S3 encryption (mc encrypt set sse-s3) on system buckets — the HA-adjacent posture that we do have today (protect against accidental overwrite + at-rest read).
• ddx-api/scripts/backup-before-migrate.sh:34-71 — the only automated cross-engine backup: per-database pg_dump of all five Prisma DBs into a timestamped directory + 7-day rolling cleanup; wired in via pnpm run prisma:backup at ddx-api/package.json:78.

Business → Technical match. Business outcome: a developer ships a migration that turns out to have a bad DROP COLUMN; the clinic can rewind to the state the system held five minutes before the migration without losing any patient data. Technical mechanism: pnpm run prisma:migrate:deploy invokes scripts/backup-before-migrate.sh first; the script writes ddx_api_main.dump, ddx_api_log.dump, ddx_api_ai.dump, ddx_api_com.dump, ddx_api_acc.dump into ddx-api/backups/<UTC-timestamp>/ (backup-before-migrate.sh:20-22 and :56-60); if the migration breaks, pg_restore -d ddx_api_main backups/<timestamp>/ddx_api_main.dump --clean returns the database to its pre-migration state. The five-DB split (see infra-postgres) means a bad migration on ddx_api_log does not require restoring ddx_api_main — schemas are restored independently.

Operational Notes

• What ships today is durable, not highly-available. Single-node with named volumes plus disciplined backups is sufficient for a clinic where the host is owned by the customer. HA is a follow-on investment driven by either (a) a hosted multi-clinic SaaS offering, or (b) a customer contract that requires sub-minute RTO. Don't promise HA in pre-sales unless one of those triggers is on the table.
• The pre-migration backup is the only safety net for ORM mistakes. It runs automatically on prisma:migrate:deploy (see Forbidden Patterns in ddx-api/CLAUDE.md — raw npx prisma migrate bypasses the backup hook). Anyone who shells out to npx prisma migrate directly has just disarmed the safety net; refuse that pattern in code review.
• Qdrant snapshots are operator-initiated. Until a scheduled snapshot job ships, treat Qdrant data as rebuildable-from-source: embeddings are a function of ddx_api_main text content + the CUDA embedder, and extractTopScores() post-processing in QdrantService is deterministic (see infra-qdrant).
• MinIO single-disk durability depends on the host filesystem. Use ZFS, btrfs, or LVM with checksums underneath the named volume; MinIO's own erasure coding is not active (server /data mode). Bucket versioning + SSE-S3 (minio-init step in ddx-docker-minio/docker-compose.yml:95-104) cover the "fat-finger delete" and "at-rest read" threats, not the "drive failure" threat.
• Restore sequence is documented but untested. The procedure in §Data Flow above is derived from the docker-compose layout and the README backup commands; we have not run a full restore drill against a production-shaped dataset. REMINDER for the team: schedule a quarterly recovery drill before promising any RTO number to a clinic in writing.
• Never restore a Prisma pg_dump into ddx_fhir_core. The FHIR Hibernate schema and the Prisma schemas are non-overlapping; cross-restoring will either fail (missing tables) or, worse, succeed and silently corrupt the FHIR partition catalog. See infra-postgres Operational Notes.
• The FHIR partition catalog is separate state. HAPI FHIR maintains tenant partitions in its own metadata tables inside ddx_fhir_core. A backup of ddx_api_main alone is not a complete tenant backup — the partition catalog must come from the FHIR Postgres dump. Cross-engine restores must keep these two in sync (same timestamp window).
• Roadmap candidates (no code today; record as design intent):
  • Postgres: streaming replication + Patroni + pgBackRest (PITR).
  • Qdrant: nightly snapshot cron → MinIO + retention policy; consider 3-node raft only when patient-embedding write volume justifies it.
  • MinIO: distributed mode (4+ drives) + mc replicate to a second site/region for off-site DR.
  • Cross-engine: a small NestJS scheduled job (@nestjs/schedule, same pattern as the existing jobs module in ddx-api/src/platform/jobs/) that orchestrates pg_dump + Qdrant snapshot + MinIO sync atomically and writes the manifest into a recovery index.
• Validation commands (current state):
  • pnpm run prisma:backup from inside ddx-api/ — confirms backup pipeline functional.
  • ls -lh ddx-api/backups/$(ls -t ddx-api/backups/ | head -1)/ — last backup directory contents.
  • docker exec ddx-001-postgres-1 pg_isready -U dudoxx_user and the ddx_fhir_core DB check on the same :6432 instance.
  • curl -H "api-key: $QDRANT_API_KEY" http://localhost:6333/collections/{name}/snapshots — list existing Qdrant snapshots (empty list is the expected state until automation lands).
  • mc admin info ddx — MinIO node + drive state.

Related Topics

• PostgreSQL — Multi-Schema Database Infrastructure — the two-container split is the foundation that any HA design has to respect; Prisma and Hibernate get independent backup/replication chains.
• Qdrant — Vector Search Infrastructure — explains why Qdrant data is rebuildable-from-source and what a single-node durable WAL covers.
• MinIO — Object Storage Infrastructure — bucket-per-clinic isolation means HA decisions can be scoped per tenant (e.g. replicate only premium tenants) once distributed mode is added.
• Canonical operational reference (not duplicated here): ddx-docker-postgres/README.md §Backup & Restore and ddx-api/CLAUDE.md §Prisma Commands.