Articles

Machine Learning (ML) and Artificial Intelligence (AI) are having an increasingly transformative impact on all industries and are already used in many mission-critical use cases in production, bringing considerable value. Data engineering, which combines ETL pipelines with other workflows managing data and machine learning operations, is also significantly impacted. The Intelligent Data Engineering and Automation framework offers the groundwork for intelligent automation processes. However, ML/AI are not the only disruptive forces; new Big Data technologies inspired by Web2.0 companies are also reshaping the Internet. Companies having the largest Big Data footprints not only provide applications with a Big Data operational model but also source their competitive advantage from data in the form of AI services and, consequently, impact the cost/performance equilibrium of ETL pipelines. All these technologies and reasons help explain why the traditional ETL pipeline design should adapt to current and emerging technologies and may be enhanced through artificial intelligence.

The medallion architecture applied to large-scale ETL pipelines and Azure Data Lake is examined. The Azure-specific aspects of medallion architecture are deciphered for improved design, operation, and governance of ETL pipelines. Bronze-layering covers data ingestion and retention, storage organization, naming conventions, schema-on-read and schema-on-write trade-offs, and accessibility by non-technical users. The design of the silver layer focuses on cleansing, standardization, deduplication, conformance, and data enrichment. Finally, gold-layering deals with the creation of data products—curated datasets that are cataloged, ready to be consumed by visual-analysis tools, and offered through published APIs. The work centers on using medallion architecture to tackle the challenges of ETL pipelines running in Azure Data Lake. Such pipelines face increasing volumes and complexities, including new data sources, customer mandates for improved product quality, stricter regulatory requirements, and the need for customer-oriented APIs. The analysis is further filtered by success criteria expressed in data quality, observability, and monitoring för the whole medallion architecture, and data-organizational strategy for the silver and gold layers with specific focus on the definition and creation of data products.

Data quality and data lineage are critical concerns for organizations mandated to comply with stringent regulatory regimes. This paper analyses the latest developments in the governance of data quality and data lineage within a regulated financial services organisation. It sets out the underlying regulatory context, describes the concepts employed in the business environment, summarizes how data quality is captured and monitored, examines the artefacts that record data lineage, reviews the roles and responsibilities of staff who implement the necessary processes, and maps areas where improvements are possible. The internal organization and processes of regulated data platforms are shaped not only by the capabilities prescribed by their technical architecture but also by the regulatory regimes under which they operate. These mandates, in particular, require rigorous examination of four aspects of data quality — accuracy, completeness, consistency, and timeliness — and detailed documentation of how data arrives in its final form in the repository. Although data monitoring, alerting, assessment, and remediation are well established, provenance capture remains an area ripe for further investment.

Predictive modeling, supported by machine learning technology, aims to analyze data in order to guide decision-making towards actions generating desired values in the future. It encompasses the set of techniques used to build models that estimate the value of a certain variable predicting a forthcoming event from the past or current values of relevant attributes. In predictive healthcare modeling, the built models represent the relationship among the data concerning customer, provider, production, and other aspects of the healthcare environment in order to assist the decision processes in the prevention of diseases and in the planning of preventive actions by detection of high-risk patients. Contrary to trend analysis, whose goal is to describe past events, predictive models aim to provide useful indications regarding future events and changes. Predictive healthcare modeling supports actions that try to prevent the manifestation of diseases in healthy individuals or try to diagnose as early as possible the incidence of a disease in patients at risk. A sound predictive analysis encompasses not only the model-training task, but also the aspects of data quality, preprocessing, and fusion during its entire implementation lifecycle to ensure appropriate input data preparation. The robustness of the predictive model and its results depends highly on data quality. Due to the variety of data sources in healthcare environments, it becomes essential to use preprocessing in order to remove noise and inconsistencies. The increasing number of endorsable data exchange standards makes each data exchange achievable, but it demands the implementation of a data-governance program. In addition, the influence of the hospital-database architect on the architecture of an early-diagnosis model is important to guarantee appropriate input-formatting modularity.