Articles

Healthcare is increasingly recognized as a data-intensive industry. Multi-hospital networks, among other organizations, face mounting operational and governance challenges because of rigid data-integration pipelines that support all data sources and destinations in the network. These pipelines have become difficult to modify, causing them to lag behind the changing needs of the clinical operation. Scalable data-pipeline architectures better support clinical decision making, optimize hospital operations, ease data quality and compliance concerns, and contribute to improved patient outcomes. Meeting scalability goals requires breaking up monolithic data-integration pipelines into smaller decoupled components and aligning service-level agreements of pipeline components and source systems. Parallelization and adoption of distributed data-warehouse technology mitigate the burden of ingesting data into a multi-hospital network. However, latency requirements still warrant the construction of separate pipelines for data ingress from clinical devices, electronic health records, and external laboratory-information systems. Healthcare associations recommend near real-time data availability for a growing list of clinical and operational applications. Mishandling the real-time ingestion of data from clinical devices, in particular, compromises availability and performance. Scalable architectural patterns for real-time streaming Ingestion from heterogeneous data sources, transport processes, and back-end processing structures are detailed.