THE CHALLENGE

Scaling Data Access for Research

The University Hospital Cologne (UKK) Medical Data Integration Centre (MeDIC) required a robust mechanism to provide researchers with high-quality, analytics-ready clinical data. While the hospital had established a FHIR-native repository, the “last mile” of secondary use remained a bottleneck. Researchers needed a way to extract specific patient cohorts, validate that data against complex national profiles, and export it into high-performance tabular formats like Parquet and CSV for large-scale analysis.

THE SOLUTION

A Microservices Suite Powered by Ignifyr

We built a suite of RESTful microservices on Ignifyr, each handling one stage of the pipeline. UKK’s team calls them individually from their existing workflows, so extraction, transformation and validation can be composed in whatever order a study needs.

High-Performance Extraction and Transformation

Within the Ignifyr environment, we implemented specialized modules to handle massive clinical datasets with enterprise-grade efficiency:

• Spark-Powered Extraction: With Ignifyr’s underlying Apache Spark integration to implement a batch extraction service. By configuring patient batch sizes and repartitioning strategies, we enabled the high-speed retrieval of FHIR resources from the central repository without impacting clinical operations.
• Complex Data Transformation: We executed sophisticated mapping jobs that can transform data between FHIR versions or into specialized research models. These transformations are managed through RESTful endpoints, allowing for fully automated, hands-off data preparation.

Automated Conformance and Validation

To ensure that exported data meets the rigorous standards of the Medical Informatics Initiative (MII), we enabled Ignifyr’s validation capabilities:

• Profile Validation: Resources are checked against UKK’s local profiles alongside the MII profiles, value sets and code systems.
• Granular Severity Control: The system can be configured to filter results by severity, allowing researchers to prioritize “Fatal” or “Error” issues while maintaining a record of “Warnings” for data quality improvement.

Analytics-Ready Export Services

The final stage of the pipeline focused on making data usable for modern data science tools:

• Multi-Format Tabular Export: We implemented an export service that translates hierarchical FHIR resources into flat formats such as CSV, TSV, and Parquet. This allows researchers to load clinical data directly into tools like R, Python, or specialized RWE platforms.
• FHIR-to-FHIR Synchronization: For multi-site studies, the same service exports directly to another FHIR server, so data crosses the research network in its native format.

THE impact

A Modern Research Data Factory

The deployment of the Data Export Microservices at UKK MeDIC provided a production-grade bridge between the clinical repository and the research community.

• High-Throughput Analytics: By utilizing Apache Spark within the Ignifyr architecture, UKK can now process millions of resources in parallel, drastically reducing the time required to prepare large-scale datasets.
• Guaranteed Data Quality: The automated validation layer ensures that any data leaving the MeDIC environment is natively compliant with international and national standards.
• Seamless Integration: By wrapping every service in a documented RESTful API (Swagger/OpenAPI), we enabled UKK to orchestrate complex data flows through their central business process engines with minimal manual intervention.

SEE OUR PACKAGES

Are you looking to unlock clinical data for research?

See our packages to discover how we help institutions transform FHIR repositories into high-performance research data pipelines. Or contact us to discuss your specific data extraction and analytics needs.