THE CHALLENGE

Harmonizing Legacy Lab Data for Research

The University Hospital Cologne (UKK) Medical Data Integration Centre (MeDIC) required a high-performance solution to migrate legacy laboratory results into the HL7 FHIR® standard. The objective was to transform massive volumes of proprietary CSV exports into high-fidelity FHIR Observation resources compliant with the German Medical Informatics Initiative (MII) Core Data Set and securely persist them into a production Firely server. While the project validated the workflow with an initial dataset, the architecture was designed to scale to the millions of diagnostic records required for a live production environment.

THE SOLUTION

Configuring a High-Performance Ignifyr Pipeline

Ignifyr replaced ad-hoc conversion scripts with a managed ETL architecture. By deploying Ignifyr within the hospital’s secure infrastructure, the team established a “plug-and-play” environment where data transformation is decoupled from the underlying software code.

Intelligent Integration Workflow

By configuring specialized mapping jobs within the Ignifyr environment, we addressed the hospital’s specific infrastructure and security requirements:

• Automated Folder Watchers: We configured Ignifyr’s native file-system watchers to trigger transformations automatically the moment new CSV exports were detected in the hospital’s data directories.
• Streaming ETL via Apache Spark: With Ignifyr’s underlying Spark-powered engine, we enabled the processing of massive clinical datasets via streaming, ensuring consistent high throughput and system stability even as data volumes scaled into the millions.
• Secured Connectivity & Identity Mapping: We developed mapping jobs within Ignifyr that utilized OAuth 2.0 for secure communication. Ignifyr’s “Identity Service” helped automatic links from new Observations to existing Patient and Encounter resources, ensuring clinical continuity.

Mastering Semantic Translation

We utilized Ignifyr’s advanced terminology and transformation features to handle the complex, non-standardized reality of hospital data:

• LOINC & UCUM Mapping: We configured Ignifyr’s integrated terminology service to map approximately 800 local laboratory codes to international LOINC standards. Simultaneously, hospital-specific units (such as RE/ml) were normalized to UCUM using Ignifyr’s built-in unit conversion functions to ensure profile compatibility.
• Standardizing Qualitative Results: For non-numeric results (e.g., “+”, “positive”), we implemented specialized concept maps that parse text and output standardized SNOMED CT codes.
• Data Quality Guardrails: Leveraging Ignifyr’s flexible logic, the pipeline was configured to identify and handle invalid rows or missing units. This ensures that individual data errors flag notifications for the IT team rather than halting the entire ingestion process.

Operational “Quality of Life” Configurations

The system had to stay maintainable by UKK’s own IT team, without us in the loop. Three configurations make that possible:

• Intelligent Backlog Management: By configuring Ignifyr settings such as maxFileAge and latestFirst triggers, the hospital can prioritize the most recent clinical data while simultaneously processing massive historical backlogs in the background.
• Automated Data Lifecycle: We enabled Ignifyr’s archiveMode to automatically move or delete processed files, preventing data duplication and maintaining optimal storage performance.
• Containerized Deployment: The entire configuration—including mapping definitions and environment settings—was packaged as a containerized microservice for seamless deployment within the hospital’s in-house servers.

THE impact

A Foundation for Scalable Clinical Research

The UKK MeDIC deployment established a robust, automated FHIR pipeline that successfully transitioned from a pilot phase to a production-scale system. By externalizing the mapping logic, the hospital can now update the mapping scripts, medical terminologies or MII profiles independently, without modifying the core software.

• Engineering Rigor: Replaced ad-hoc transformation scripts with a versioned, monitored, and fully automated pipeline.
• Clinical Fidelity: Nothing was dropped. Where no international match existed, the local code was preserved alongside the mapped value.
• Production Scalability: Established a technical blueprint that has successfully scaled to process millions of laboratory records, providing a high-availability data source for the national research network.

SEE OUR PACKAGES

Do you want to transform your health data into FHIR?

Discover our implementation packages designed to support healthcare organizations in building scalable FHIR pipelines. Or contact us to discuss your data integration challenges and project requirements.