Doxing — Document Intelligence and RAG Pipeline

"Doxing" is Dudoxx's internal name for the document-intelligence stack: RAG-based document ingestion, semantic search over clinical attachments, OCR text extraction, and on-demand AI summarization — originally implemented as two separate services (ddx-doxing-server, ddx-doxing-web) that were deactivated 2026-04-18 and replaced by in-process modules inside ddx-api calling a shared backend server.

Architecture History and Current Status

IMPORTANT — honest disclosure: The architecture has gone through two generations:

Generation 1 (deactivated 2026-04-18, parked 2026-06-14):

• ddx-doxing-server/ — standalone FastAPI (Python 3.13) + Docling + Qdrant + PostgreSQL backend. Ran as a separate process. Deactivated from the running stack 2026-04-18; parked off root to _claude-archive/parked-2026-06-14/dead-code-twins/ddx-doxing-server/ in the 2026-06-14 housekeeping (twin of ddx-ocr-server).
• ddx-doxing-web/ — standalone Next.js UI for document search. Also deactivated + parked to the same dead-code-twins/ folder.

Generation 2 (current):

• ddx-api/src/documents/doxing/ — NestJS module inside ddx-api that calls the DDX Doxing Server over HTTP. Provides REST endpoints for document collection management, ingestion, semantic querying, and extraction. This is the active code path.
• ddx-api/src/documents/ocr/ — NestJS OCR pipeline that can use multiple providers: the Doxing server's VLM OCR, ddx-ocr-server (separate), or Tesseract.
• ddx-api/src/documents/attachment-summarization/ — on-demand AI summarization per attachment, using TUCAN AI agents via the SummarizationAgentAdapter.

The DDX Doxing Server (ddx-doxing-server/) may still be the backend that DoxingIngestionService calls over HTTP — the NestJS module configures a DOXING_SERVER_URL env var and sends documents to it via HTTP. Whether the server is running in production is a deployment concern, not a code concern.

Business Purpose

Clinical documents — lab reports, discharge summaries, consent forms, diagnostic images — are rarely useful as raw files. Clinicians need to ask questions across a patient's document history ("What was the creatinine trend over the last 6 months?"), and AI assistants need to answer questions grounded in specific attachments. The doxing stack delivers this by:

1. Document ingestion to a vector store — documents are downloaded from MinIO, processed through Docling (structure extraction), chunked, and embedded into Qdrant collections organised by clinic and patient.
2. Semantic search — natural language queries are embedded and matched against the Qdrant collection, optionally enhanced with LLM reranking.
3. OCR extraction — scanned documents and images are run through VLM or Tesseract OCR, with a human-correction workflow for low-confidence results.
4. AI summarization — on-demand per-attachment summaries driven by configurable templates, streamed via SSE using the TUCAN AI agent infrastructure.
5. TUCAN session RAG — during AI assistant sessions, the TUCAN system uses the doxing/indexation pipeline to ground responses in the patient's actual document history.

Audiences

• Investor: Document intelligence is a strong differentiation for Dudoxx in the "AI-native HMS" category. The ability to ask questions against a patient's full document history, inside the EHR, without a separate BI tool, is a premium capability. The architecture (Docling → Qdrant → TUCAN) is extensible to multi-modal inputs (lab PDFs, DICOM reports, ECG PDFs).
• Clinical buyer (doctor/nurse/receptionist): Doctors benefit from AI-summarised discharge letters and lab reports directly in the patient view. The OCR pipeline makes scanned paper records machine-readable for search.
• Developer/partner: The active NestJS entry points are DoxingModule (src/documents/doxing/), OcrModule (src/documents/ocr/), and AttachmentSummarizationModule (src/documents/attachment-summarization/). All three are in documents-group.module.ts. The standalone ddx-doxing-server/ submodule is an independent FastAPI server — read its CLAUDE.md and ARCHITECTURE.md for the Python-side API contract.
• Internal (ops/support): The DOXING_SERVER_URL env var controls where the NestJS module sends ingestion requests. If the server is not reachable, ingestion silently fails with HTTP errors logged by DoxingIngestionService. OCR jobs use OcrJobService which tracks job status in ddx_api_main.

Architecture

Documents Group (ddx-api/src/documents/)
│
├── DoxingModule (documents/doxing/)
│     DoxingService (Orchestrator, since v2.7.1)
│       └── DoxingCollectionService  — collection naming + stats ({clinic}-patient-{id}, {clinic}-documents)
│       └── DoxingDocumentService    — per-document ingestion records in Prisma
│       └── DoxingQueryService       — semantic search via HTTP → Doxing Server
│       └── DoxingTaskService        — async ingestion task lifecycle (SSE progress)
│       └── DoxingIngestionService   — MinIO download → multipart POST → Doxing Server
│       └── DoxingExtractionService  — structured extraction (tables, entities)
│       └── DoxingStorageService     — document storage helpers
│       └── DoxingChatService        — RAG-enhanced chat completion
│
├── OcrModule (documents/ocr/)
│     OcrService / OcrFactoryService
│       └── DdxOcrServerProvider     — HTTP calls to ddx-ocr-server (separate microservice)
│       └── DoxingOcrProvider        — uses Doxing server's VLM OCR endpoint
│       └── HausarztErstgespraechProvider — specialty-specific OCR form extraction
│     OcrJobService                  — job lifecycle in ddx_api_main
│     OcrSearchService               — search over OCR-extracted text
│
└── AttachmentSummarizationModule (documents/attachment-summarization/)
      AttachmentSummarizationService
        → SummarizationAgentAdapter (implements SummarizationAgentServiceInterface)
        → forwardRef(() => TucanModule)  ← uses Mastra LLM + MemoryService
      SummarizationTemplateService   — per-attachment template CRUD
      AttachmentSummaryExportService — export summaries to PDF
      SSEProgressEmitterService      — streams summarization progress via SSE

External: DDX Doxing Server (FastAPI Python)
  → POST /ingest     ← DoxingIngestionService sends MinIO-downloaded docs
  → GET  /search     ← DoxingQueryService sends embedded queries
  → POST /extract    ← DoxingExtractionService requests structured extraction
  Backend: Docling (IBM) + Qdrant 1.12+ + PostgreSQL 16 + CUDA VLM

Tech Stack & Choices

Data Flow

Document Ingestion (MinIO → Doxing Server → Qdrant)

POST /doxing/ingest (or triggered from document upload flow)
  → DoxingIngestionService.ingest(attachmentId, clinicSlug, patientId)
      1. PrismaService: look up attachment → get MinIO bucket + key
      2. downloadFromMinIO(bucket, key)   ← S3Client + GetObjectCommand
      3. FormData multipart POST → DOXING_SERVER_URL/ingest
             {file: <buffer>, collection: '{clinic}-patient-{patientId}', ...}
      4. Doxing Server (Python FastAPI):
             Docling parses document → structured chunks
             Chunks embedded (CUDA VLM or sentence-transformers)
             Upsert chunks to Qdrant collection
      5. DoxingDocumentService: update ingestion status in ddx_api_main
      6. DocumentEventPublisher: emit 'document.indexed' event (optional, @Optional() inject)

Semantic Search

POST /doxing/query
  → DoxingQueryService.query(dto, clinicSlug, patientId)
      1. Build query params: {question, collection: '{clinic}-patient-{patientId}', limit}
      2. HTTP GET → DOXING_SERVER_URL/search?question=...&collection=...
      3. Doxing Server: embed query → Qdrant similarity search → optional LLM rerank
      4. Return DoxingQueryResult {chunks, sources, answer}

AI Summarization (on-demand, SSE-streamed)

POST /documents/attachments/:id/summarize
  → AttachmentSummarizationController
    → AttachmentSummarizationService.summarize(attachmentId, templateId, userId)
        1. SummarizationTemplateService.getTemplate(templateId)
        2. SummarizationAgentAdapter.summarize(attachment, template)
               → TucanModule.LLMConfigFactory (LiteLLM 4000)
               → Mastra agent with attachment content + template prompt
        3. SSEProgressEmitterService.emit(sessionId, {type: 'progress', ...})
        4. On completion: emit {type: 'complete', summary: ...}

Implicated Code

DoxingModule (active NestJS wrapper)

• ddx-api/src/documents/doxing/doxing.module.ts:1 — DoxingModule; 8 domain services + orchestrator; imports HttpModule, StorageModule, OcrModule, DocumentsModule; comment at :27 documents generation history and collection naming convention.
• ddx-api/src/documents/doxing/doxing.service.ts:36 — DoxingService; orchestrator pattern since v2.7.1 (refactored from 2,612-line monolith); delegates to 7 domain services.
• ddx-api/src/documents/doxing/services/doxing-ingestion.service.ts:36 — DoxingIngestionService; constructs S3Client from MINIO_ENDPOINT/MINIO_ACCESS_KEY; initDoxingServerConfig(configService) reads DOXING_SERVER_URL; sends multipart POST to Doxing server.
• ddx-api/src/documents/doxing/services/doxing-ingestion.service.ts:47 — @Optional() @Inject(forwardRef(() => DocumentEventPublisher)) — event publisher is optional to avoid circular dep in standalone ingestion context.
• ddx-api/src/documents/doxing/services/doxing-query.service.ts:1 — DoxingQueryService; HTTP GET to Doxing server /search endpoint; also queries OcrSearchService for OCR-extracted text (dual engine: ddx_ocr_server | doxing).
• ddx-api/src/documents/doxing/services/doxing-collection.service.ts:1 — DoxingCollectionService; collection naming ({clinic}-patient-{patientId}, {clinic}-documents); stats aggregation from Qdrant (374 lines — exceeds 300-line refactoring limit; stats extraction is a known TODO).

OCR module

• ddx-api/src/documents/ocr/ocr.module.ts:1 — OcrModule; three OCR providers: DdxOcrServerProvider (ddx-ocr-server HTTP), DoxingOcrProvider (Doxing server VLM), HausarztErstgespraechProvider (specialty form OCR).
• ddx-api/src/documents/ocr/ocr-factory.service.ts:1 — OcrFactoryService; selects OCR provider by document type and config.
• ddx-api/src/documents/ocr/services/ocr-job.service.ts:1 — OcrJobService; job lifecycle tracking in ddx_api_main.
• ddx-api/src/documents/ocr/services/ocr-search.service.ts:1 — OcrSearchService; search over OCR-extracted text.

Attachment summarization

• ddx-api/src/documents/attachment-summarization/attachment-summarization.module.ts:1 — AttachmentSummarizationModule; forwardRef(() => TucanModule) for LLM config + memory; SUMMARIZATION_AGENT_TOKEN injection token pattern.
• ddx-api/src/documents/attachment-summarization/services/attachment-summarization.service.ts:1 — AttachmentSummarizationService; template-driven per-attachment summarization; SSE progress events.
• ddx-api/src/documents/attachment-summarization/services/summarization-agent.adapter.ts:1 — SummarizationAgentAdapter; adapts TUCAN Mastra agent API to SummarizationAgentServiceInterface.

Legacy standalone servers (on disk, deactivated)

• _claude-archive/parked-2026-06-14/dead-code-twins/ddx-doxing-server/src/ — FastAPI Python server; Docling + Qdrant + PostgreSQL; pyproject.toml lists Docling 2.55+, FastAPI 0.115+, Qdrant-client 1.12+. NOT running in production stack (deactivated 2026-04-18, parked 2026-06-14).
• .../ddx-doxing-server/ARCHITECTURE.md — authoritative description of the Python server's API contract (ingestion endpoints, collection schema, search response shape).
• .../ddx-doxing-server/CLAUDE.md — Python server agent instructions; read before modifying the server-side API.

Operational Notes

• DOXING_SERVER_URL env var — controls where DoxingIngestionService sends ingestion requests. If unset or the server is unreachable, ingestion fails with HTTP errors. Monitor ddx-api logs for [DoxingIngestionService] Failed to ingest.
• Deactivated standalone servers — ddx-doxing-server and ddx-doxing-web were removed from the PM2 / Docker Compose stack on 2026-04-18 and parked to _claude-archive/parked-2026-06-14/dead-code-twins/ on 2026-06-14 (reversible via git mv). Do not restart them without checking the current ddx-manage.sh configuration — the NestJS module may conflict with a running standalone server on the same Qdrant collections.
• Collection naming isolation — patient collections use {clinic-slug}-patient-{patientId} to prevent cross-patient vector searches. If a patient is transferred between clinics, their Qdrant collection is NOT automatically migrated; vectors would need manual re-ingestion under the new clinic slug.
• DoxingCollectionService over-limit — doxing-collection.service.ts is 374 lines (exceeds the 300-line service limit per project rules). The stats aggregation methods are a known extraction candidate. Do not add further methods without refactoring.
• Summarization uses TUCAN LLM routing — AttachmentSummarizationService routes through LiteLLM (port 4000). Summarization jobs inherit the TUCAN enable_thinking:false default. If the LiteLLM router is down, summarization fails with 503.
• OCR job status — OCR extraction results and job statuses are stored in ddx_api_main via OcrJobService. Query SELECT * FROM ocr_job WHERE status = 'FAILED' for failed extractions; the errorMessage field contains the provider error.

Related Topics

• Document Management — document upload, ACL, MinIO storage; doxing ingestion is triggered after successful document upload.
• RAG System Indexation — DDX RAG sidecar (~/.claude/rag-sidecar/) for the coding meta-agents; shares Qdrant infrastructure with the doxing clinical collections but uses separate collection namespaces.
• Prisma 7 Schemas — OCR job records and extraction metadata live in ddx_api_main (PrismaService); doxing module writes ingestion status to the same DB.