ADF.procfwk – Video Setup How To Guide

Code Project Overview

This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages of dependencies to be executed in sequence. Then secondly, all pipelines within a stage to be executed in parallel offering scaled out control flows where no inter-dependencies exist.

The framework is designed to integrate with any existing Data Factory solution by making the lowest level executor a stand alone Worker pipeline that is wrapped in a higher level of controlled (sequential) dependencies. This level of abstraction means operationally nothing about the monitoring of orchestration processes is hidden in multiple levels of dynamic activity calls. Instead, everything from the processing pipeline doing the work (the Worker) can be inspected using out-of-the-box ADF features.

This framework can also be used in any Azure Tenant and allow the creation of complex control flows across multiple Data Factory resources by connecting Service Principal details through metadata to targeted Subscriptions > Resource Groups > Data Factory’s and Pipelines, this offers very granular administration over data processing components in a given environment.

ADF.procfwk Resources

Blogs mrpaulandrew.com/ADF.procfwk
GitHub github.com/mrpaulandrew/ADF.procfwk
Twitter #ADFprocfwk
Vlogs youtube.com/mrpaulandrew

Setup How To Guide

To support the community in the setup and deployment of my metadata driven processing framework I’ve put together a short video series to guide you through the steps needed. I’ve done this without rehearsal and using a completely clean ‘Free Trial’ Azure subscription.

The 10 videos below are aligned to the Deployment Steps.md file available within the source code repository… https://github.com/mrpaulandrew/ADF.procfwk

The complete YouTube playlist is available here.

Part 1 – Getting a copy of the code from GitHub

Part 2 – Creating our new Azure Resources

Part 3 – Adding access and service keys to Key Vault

Part 4 – Publishing the metadata SQLDB schema

Part 5 – Publishing the Azure Functions

Part 6 – Create an SPN and deploy Data Factory

Part 7 – Adding Worker pipelines

Part 8 – Updating the database metadata

Part 9 – Running it!

Part 10 – Email Alerting (Optional)

 


Many thanks

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.