Category: SQLDB

ADF.procfwk v1.7 – Pipeline Email Alerting

Posted on by mrpaulandrew

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.7 – Pipeline Email Alerting →

ADF.procfwk v1.6 – Error Details for Failed Activities Captured

Posted on by mrpaulandrew

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.6 – Error Details for Failed Activities Captured →

ADF.procfwk v1.5 – Power BI Dashboard for Framework Executions

Posted on by mrpaulandrew

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.5 – Power BI Dashboard for Framework Executions →

ADF.procfwk v1.4 – Enhancements for Long Running Pipelines

Posted on by mrpaulandrew

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.4 – Enhancements for Long Running Pipelines →

ADF.procfwk v1.3 – Metadata Integrity Checks

Posted on by mrpaulandrew

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.3 – Metadata Integrity Checks →

ADF.procfwk v1.2 – Execution Restartability

Posted on by mrpaulandrew

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.2 – Execution Restartability →

ADF.procfwk v1.1 – Service Principal Handling via Metadata

Posted on by mrpaulandrew

Hi data friends! Version 1.1 of my ADF.procfwk is ready! Following the great response I got to the version 1.0 blog series I decided to push ahead and get version 1.1 out there as soon as possible. The main thing I've addressed with this release is the big problem of having hard coded service principal … Continue reading ADF.procfwk v1.1 – Service Principal Handling via Metadata →

Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 4 of 4

Posted on by mrpaulandrew

In this four part blog series I want to share my approach to delivering a metadata driven processing framework in Azure Data Factory. This is very much version 1 of what is possible and where can we build on this to deliver a hardened solution for our platform orchestrator. For the agenda of this blog … Continue reading Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 4 of 4 →

Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 3 of 4

Posted on by mrpaulandrew

In this four part blog series I want to share my approach to delivering a metadata driven processing framework in Azure Data Factory. This is very much version 1 of what is possible and where can we build on this to deliver a hardened solution for our platform orchestrator. For the agenda of this blog … Continue reading Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 3 of 4 →

Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 2 of 4

Posted on by mrpaulandrew

In this four part blog series I want to share my approach to delivering a metadata driven processing framework in Azure Data Factory. This is very much version 1 of what is possible and where can we build on this to deliver a hardened solution for our platform orchestrator. For the agenda of this blog … Continue reading Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 2 of 4 →