Systems, Architecture and Systems Architecture

Metadata-driven pipelines are a data management solution that eliminates the need for considerable coding by identifying patterns in data connectivity and processing and therefore replaces the manual coding necessary for each data source. This method not only saves time but also decreases the possibility of mistakes. Furthermore, metadata-driven pipelines are highly scalable and versatile, easily responding to different data sources and destinations without extensive reconfiguration. This adaptability is especially beneficial in data-driven contexts where data requirements change typically. Metadata-driven pipelines, in essence, provide a streamlined, efficient, and scalable solution to data processing and management.

The following are critical components of a generic metadata-driven pipeline design within Azure Data Factory (ADF):

Metadata Database

A SQL database that maintains metadata about the data sources and processing logic. It serves as a governance layer, supplying information required for the pipeline to function.

Parameterized Linked Services

These are ADF setups that specify how to link to various data sources. They’re parameterized to be dynamically updated, providing for greater flexibility when linking to data defined by metadata.

Parameterized Data Sets

These data sets, like linked services, are parameterized to dynamically adapt to the data structure provided by the metadata, guaranteeing that the data is handled correctly throughout processing.

Metadata Driven Pipeline

ADF’s central processing unit, which contains activities such as the ‘Lookup Activity’. It determines the actions to do on the data using metadata from the Metadata Database.

Lookup Activity

A pipeline activity that retrieves information from the metadata database in order to determine how to process the data.

Pipeline Activities

These are the many jobs that the pipeline will carry out, such as data movement, transformation, and any other essential data processing.

This architecture, in essence, allows a dynamic and scalable approach to data integration and transformation, removing the need for manual workflow. When data sources or structures change, updates must be made.

References

Dearandyxu, D. (2023, April 12). Build large-scale data copy pipelines with metadata-driven approach in copy data tool – Azure Data Factory. Microsoft Learn. Retrieved November 16, 2023, from https://learn.microsoft.com/en-us/azure/data-factory/copy-data-tool-metadata-driven

Layered Architecture for Data Platforms: the place that turns data into insights. (n.d.). Deloitte Netherlands. https://www2.deloitte.com/nl/nl/pages/data-analytics/articles/layered-architecture-for-data-platforms-the-place-that-turns-data-into-insights.html

Mahajan, G. (2021, October 25). Creating metadata-driven data pipelines using Azure Data Factory. SQL Shack – Articles About Database Auditing, Server Performance, Data Recovery, and More. https://www.sqlshack.com/creating-metadata-driven-data-pipelines-using-azure-data-factory/

Metadata Driven Pipelines for Microsoft Fabric. (n.d.). TECHCOMMUNITY.MICROSOFT.COM. https://techcommunity.microsoft.com/t5/fasttrack-for-azure/metadata-driven-pipelines-for-microsoft-fabric/ba-p/3891651

Microsoft Learn. (2023, July 17). Parameterize linked services – Azure Data Factory & Azure Synapse. Retrieved November 16, 2023, from https://learn.microsoft.com/en-us/azure/data-factory/parameterize-linked-services?tabs=data-factory