OCR Server — Document Text Extraction

A self-hosted FastAPI service (port 6800, v0.1.0) that extracts text + structured data from uploaded documents (PDF, image, audio, DICOM, HL7, office files) using a strategy-dispatched extractor chain, then optionally runs a declarative processor pipeline (classify → entity_extract → chunk → embed → summarize).

Business Purpose

Every clinical workflow starts with documents that nobody on the staff wants to retype: scanned referral letters, lab PDFs (text-layer or pure raster), DICOM headers, HL7 v2 messages, dictated voice memos, faxed prescriptions, photographed insurance cards. The OCR Server is the single gateway that converts all of those into structured text + medical entities that the rest of the HMS can index, search, and present.

Why in-house instead of Google Document AI / AWS Textract:

• PHI never leaves the cluster — the only network hop is to the in-cluster Dudoxx LLM router (port 4000), not a third-party cloud.
• Cost is fixed: a single GPU box can handle a clinic's daily document volume; AWS Textract is ~$0.0015/page and adds up fast across thousands of patients.
• The strategy chain is deterministic and inspectable — when an extraction fails, the operator can see which strategy was tried and why it lost.
• The processor pipeline is declarative (pipeline_steps: ["classify", "entity_extract", "chunk", "embed"]) so business logic can rewire the chain without code changes.

Audiences

• Investor: in-house OCR + medical-NER + audio transcription on a single endpoint is unusual — most competitors stitch together 3-4 cloud SaaS APIs. This is a meaningful CapEx-vs-OpEx differentiator for hospital procurement.
• Clinical buyer: the receptionist uploads a scanned PDF; within seconds it appears as searchable text on the patient record with diagnoses, medications, and lab values already extracted as structured fields.
• Developer/partner: a small REST surface — POST /api/v1/ocr/{image|pdf|audio|document} for sync extraction, POST /api/v1/ocr/hausarzt-erstgespraech for structured template extraction (German GP first-consultation), POST /api/v1/jobs for async + pipeline + SSE progress, POST /api/v1/search for semantic queries against the indexed chunks.
• Internal (ops/support): process-managed via ddx-manage.sh start ocr-server; structured logs with request_id propagation via RequestIdMiddleware; pause/resume support on long jobs.

Architecture

Caller (ddx-api documents domain, ddx-doxing-server)
        │
        ▼  HTTP (X-API-Key)
┌────────────────────────────────────────────────┐
│ FastAPI app (ddx-ocr-server:6800)              │
│  /api/v1/ocr/{image|pdf|audio|document}  (sync)│
│  /api/v1/jobs                            (async│
│           + SSE)                              )│
│  /api/v1/search                                │
└──────────────────┬─────────────────────────────┘
                   │
           ┌───────┴────────┐
           ▼                ▼
    ┌──────────────┐  ┌──────────────────┐
    │ Strategy     │  │ Pipeline runner  │
    │ Dispatcher   │  │ (declarative)    │
    └──────┬───────┘  └────────┬─────────┘
           │                   │
           ▼                   ▼
  ┌────────────────┐   ┌──────────────────────┐
  │ Extractors     │   │ Processors           │
  │  pdf_extract   │   │  classify            │
  │  tesseract     │   │  entity_extract      │
  │  vision_llm    │   │  chunk               │
  │  docling       │   │  embed → Qdrant      │
  │  dicom / hl7   │   │  summarize           │
  │  audio (whisper│   │  describe / enrich   │
  │   + deepgram)  │   │  speaker_role / soap │
  └────────────────┘   └──────────────────────┘
        │                       │
        ▼                       ▼
   Dudoxx LLM router       PostgreSQL + Qdrant
   (vision, whisper)       (jobs, chunks, embeds)

Two orthogonal axes:

1. Extractors turn bytes → raw text + structured fields (one-shot per file).
2. Processors enrich raw text → entities, chunks, embeddings, SOAP notes, summaries (chain of steps on a JobContext).

Both axes use a registry pattern — extractors via the dispatcher's MIME-table, processors via a @register("name") decorator (see pipeline/registry.py).

Tech Stack & Choices

Supported file types (via FileDetector magic-byte detection)

• Raster images: PNG, JPEG, TIFF, WEBP, BMP
• PDF: native-text + scanned (auto-detected by character count threshold)
• Audio: WAV, MP3, M4A, FLAC, OGG
• Clinical: DICOM, HL7 v2, FHIR JSON/XML
• Office / text: DOCX, XLSX, PPTX, HTML, plain text (via Docling)

Strategy fallback chains

The StrategyDispatcher (services/dispatcher.py) selects an ordered list of extractors per MIME class and tries them in order; the first to produce a confident ExtractionResult wins. Per the module docstring:

PDF with text layer → PdfExtractor
PDF scan           → TesseractExtractor → VisionLlmExtractor
Image              → VisionLlmExtractor → TesseractExtractor
Audio              → WhisperLocalExtractor → DeepgramExtractor
Clinical           → DicomExtractor | Hl7Extractor
Office / text      → DoclingExtractor

Each step is logged; if every strategy in the chain fails, OcrServiceError(503) propagates back as a 503.

Processor pipeline (declarative)

Jobs submit a pipeline_steps: [...] list. Available steps (registered via @register at import time, see pipeline/registry.py:53-67):

• classify — document type classification
• entity_extract — medical NER (conditions, medications, procedures, lab values)
• chunk — text segmentation for embedding
• embed — vector embedding into Qdrant
• summarize — narrative summary
• enrich — additional metadata enrichment
• describe — image / page description
• contextual_prefix — prepend context to chunks for retrieval
• speaker_role — speaker diarization label per segment (audio)
• soap — SOAP-note structuring (Subjective/Objective/Assessment/Plan)

Step names are validated against the registry at Pipeline.__init__ — unknown names fail-fast before any I/O.

Data Flow

Sync extraction (POST /api/v1/ocr/pdf, etc.)

1. Caller uploads file (multipart) + auth header X-API-Key.
2. validate_upload checks size + MIME against Settings.max_file_size_mb.
3. read_upload_bytes streams bytes with the size limit enforced.
4. FileDetector().detect_mime(file_bytes, fname) runs magic-byte detection — the filename extension is a hint, the byte signature is authoritative.
5. StrategyDispatcher.dispatch() picks the chain by MIME class, iterates extractors, logs each attempt, returns the first ExtractionResult that succeeds.
6. _to_result(...) maps to the API-level OcrResult schema (pages, segments, raw text, confidence, elapsed_ms).
7. Response returned synchronously — no job created, no SSE.

Async + pipeline (POST /api/v1/jobs + GET /api/v1/jobs/{id}/events)

1. Caller POSTs files + pipeline_steps (form-encoded JSON).
2. JobService persists a job row in Postgres and returns 202 Accepted with the job id.
3. Background runner drives:
   • extraction (same dispatcher path)
   • then Pipeline(steps).run(ctx) — each processor emits typed SSE events (ProcessorStartEvent, ProcessorEndEvent, ProcessorRetryEvent, ProcessorPausedEvent, ProcessorErrorEvent)
4. Caller subscribes to /jobs/{id}/events (SSE) to track live progress; if the job is already done, the route replays from the DB-persisted event log.
5. chunk + embed processors write to Qdrant under collection ddx-ocr-chunks (configurable via qdrant_collection).
6. Pause / resume via POST /jobs/{id}/control with {"action": "pause" | "resume"}.
7. Cancel via DELETE /jobs/{id}.

Structured template extraction (POST /api/v1/ocr/hausarzt-erstgespraech)

A domain-specific route (mounted from api/routes/hausarzt_erstgespraech.py, router tag ocr-templates) that takes a patient PDF (Vorbefunde) and returns a fully structured German first-consultation document (HausarztErstgespraechDoc) with 9 sections: meta, sicherheit, kurzprofil, diagnosen, medikation, befunde, eingriffe, fachaerzte, offene_punkte.

Key design points:

• Schema keys are German — Pydantic models use Field(alias=...) + ConfigDict(populate_by_name=True) and the response is serialised by_alias=True so the JSON contract preserves German field names (see core/schemas/hausarzt_erstgespraech.py).
• Safety-first ordering: sicherheit (Allergien + kritische Hinweise) is always populated first so downstream UIs can surface allergies/critical warnings immediately.
• Runs the dedicated HausarztErstgespraechExtractor (uses dudoxx_text_model via the shared LlmClient). On failure it raises OcrServiceError with codes SCHEMA_INVALID / EXTRACTOR_FAILED / EMPTY_TEXT, mapped to structured JSON by the global handler.
• Auth: X-API-Key required. This is a template extractor — a first example of the template-driven structured-extraction pattern layered on top of the generic OCR chain.

Semantic search (POST /api/v1/search, POST /api/v1/search/sources, GET /api/v1/search/chunks/{task_id})

After a job runs chunk + embed, the resulting vectors are queryable via the search routes (api/routes/search.py, all X-API-Key-gated):

• POST /api/v1/search — RAG semantic search: embeds the query via the Dudoxx router and searches Qdrant with optional clinical metadata filters; returns ranked SourceChunk results with parent-job metadata.
• POST /api/v1/search/sources — fetch full chunk content + parent metadata by Qdrant point UUIDs (for citation rendering / source highlighting in the UI).
• GET /api/v1/search/chunks/{task_id} — list all chunks persisted in PostgreSQL for a given job, including Qdrant UUIDs for cross-referencing.

Request-Id propagation

Every request gets an X-Request-Id header (generated if missing) bound to structlog.contextvars for the duration of the request and echoed back on the response — see main.py:31-44.

Implicated Code

MANDATORY: ≥3 file:line citations. No page accepted without this section populated.

• ddx-ocr-server/engine/src/ddx_ocr_server/main.py:87 — create_app() wires 6 routers (jobs, ocr, hausarzt_erstgespraech, search, health incl. /providers), middleware, exception handler under prefix /api/v1.
• ddx-ocr-server/engine/src/ddx_ocr_server/main.py:48 — Lifespan: asyncpg pool init + migrations, LlmClient singleton, Qdrant singleton, graceful shutdown drain.
• ddx-ocr-server/engine/src/ddx_ocr_server/api/routes/hausarzt_erstgespraech.py:65 — POST /ocr/hausarzt-erstgespraech — structured German template extraction (HausarztErstgespraechDoc, 9 sections, safety-first).
• ddx-ocr-server/engine/src/ddx_ocr_server/api/routes/search.py:157 — RAG search routes (POST /search, POST /search/sources, GET /search/chunks/{task_id}).
• ddx-ocr-server/engine/src/ddx_ocr_server/services/dispatcher.py:61 — StrategyDispatcher class — MIME-class routing + per-class fallback chains.
• ddx-ocr-server/engine/src/ddx_ocr_server/api/routes/ocr.py:1 — Four sync routes: /ocr/image, /ocr/pdf, /ocr/audio, /ocr/document.
• ddx-ocr-server/engine/src/ddx_ocr_server/api/routes/ocr.py:89 — _run() shared pipeline: validate → read → MIME-detect → dispatch → map.
• ddx-ocr-server/engine/src/ddx_ocr_server/api/routes/jobs.py:54 — POST /jobs (async submission, multipart + pipeline_steps).
• ddx-ocr-server/engine/src/ddx_ocr_server/pipeline/runner.py:30 — Pipeline class with cancel/pause/retry/SSE-emit semantics.
• ddx-ocr-server/engine/src/ddx_ocr_server/pipeline/registry.py:53 — ensure_processors_imported() — force-imports all 10 processor modules so @register decorators populate the registry.
• ddx-ocr-server/engine/src/ddx_ocr_server/extractors/ — extractor modules (pdf_extract, tesseract, vision_llm, docling_extract, dicom, hl7, audio/ = whisper_local + whisper_api + deepgram, hausarzt_erstgespraech).
• ddx-ocr-server/engine/src/ddx_ocr_server/processors/ — 10 processor modules (classify, entity_extract, chunk, embed, summarize, enrich, describe, contextual_prefix, speaker_role, soap).
• ddx-ocr-server/engine/src/ddx_ocr_server/config.py:23 — Settings(BaseSettings): port 6800, file-size cap, Tesseract languages, Qdrant + LLM endpoints, feature flags.
• ddx-ocr-server/DDX-OCR-CONFIG.md — env-var reference and configuration recipes.

Operational Notes

Environment variables (Pydantic Settings)

Key knobs (see config.py):

• PORT (default 6800)
• DDX_OCR_API_KEY — single shared key for service-to-service auth (X-API-Key header)
• DUDOXX_BASE_URL / DUDOXX_API_KEY — Dudoxx LLM router endpoint (vision + NER + Whisper)
• DUDOXX_TEXT_MODEL / DUDOXX_VISION_MODEL / DUDOXX_WHISPER_MODEL — model selection
• QDRANT_URL / QDRANT_API_KEY / QDRANT_COLLECTION (default ddx-ocr-chunks)
• MAX_FILE_SIZE_MB (default 50, capped at 500)
• PDF_DPI (default 300)
• TESSERACT_LANG (default eng+deu+fra)
• OCR_HYBRID_THRESHOLD (default 60.0 — Tesseract confidence below which Vision LLM fallback fires)
• OCR_MIN_NATIVE_CHARS (default 50 — PDFs with fewer native-text chars are treated as scanned)
• ENABLE_VISION_LLM, ENABLE_TESSERACT, ENABLE_DIARIZATION, ENABLE_DEEPGRAM — feature flags
• DEEPGRAM_API_KEY + DEEPGRAM_MODEL (default nova-3-medical) — required when ENABLE_DEEPGRAM=true (enforced by model_validator)
• DEEPGRAM_ONPREMISE_BASE_URL / DEEPGRAM_ONPREMISE_API_KEY — optional self-hosted Deepgram endpoint; when the on-premise base URL is set it overrides the public DEEPGRAM_BASE_URL (https://api.deepgram.com/v1) for full sovereignty

Process management

• ./ddx-manage.sh start ocr-server from the monorepo root.
• Health probe: GET /api/v1/health (no auth) — checks db / qdrant / llm_router (each 5 s timeout); status is 'ok' only when all three pass, else 'degraded'.
• Readiness probe: GET /api/v1/readyz (no auth) — 200 when the DB pool can execute a query, 503 otherwise (k8s readinessProbe).
• Providers listing: GET /api/v1/providers (X-API-Key required) — reports which extraction strategies are enabled in the current build with their supported MIME types.
• Swagger UI: http://localhost:6800/docs.

Validation

• cd ddx-ocr-server/engine && uv run pytest
• uv run ruff check src/ tests/
• uv run mypy src/

Known pitfalls

1. Tesseract language packs must be installed at the OS layer — tesseract_lang = "eng+deu+fra" fails silently if the system Tesseract install only has eng. Check tesseract --list-langs on the host.
2. Magic-byte detection is authoritative, not the filename — a .pdf upload with WAV bytes will route to the audio chain. This is correct behavior; if callers want filename-based routing, they must validate upstream.
3. Vision LLM fallback adds latency — each vision_llm call hits the LLM router and can take 5-30 seconds per page. If the corpus is mostly clean PDFs, lower OCR_HYBRID_THRESHOLD to avoid unnecessary fallbacks.
4. Pipeline registry depends on import-time side effects — adding a new processor requires both @register("name") on the class AND adding the module to ensure_processors_imported() in pipeline/registry.py:53-67. The validator fails fast on unknown names — good — but missing the registry-import line means the name is never registered.
5. Qdrant client is a singleton on app.state — do NOT instantiate per-request. Closing it on request scope will break subsequent jobs.
6. CORS_ORIGINS default is http://localhost:6100 (ddx-api) — production deployments must override.

Auth posture

• Single shared X-API-Key enforced via dependencies=[Depends(verify_api_key)] at each route (see e.g. api/routes/jobs.py:58).
• Only /health and /readyz are public for probes; /providers does require X-API-Key (it exposes the enabled-strategy inventory).
• No per-user auth — calling services (ddx-api, ddx-doxing-server) gate human identity upstream and pass through trusted gateway headers if needed.

Related Topics

• Document Management — upstream consumer; persists OCR results to the patient record alongside the source binary.
• Document ACL — access-control rules for the extracted text + entities.
• Doxing — AI Document Intelligence — sibling service that calls into OCR Server for raw text, then runs higher-level RAG / summarization.
• PDF Engine — inverse path (text → PDF); together they form the document round-trip.
• FHIR ResourceFacade — extracted text is stored alongside DocumentReference.content[].attachment for downstream FHIR consumers.