Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API

Hi friends, just a very quick how to guide style post on something I had to build in Azure Data Factory. Scenario: I want to trigger a Data Factory pipeline, but when I do I want the pipeline to know if it's already running. If it is already running, stop the new run. Sounds simple … Continue reading Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API

An Idea for Self Service Using Azure Synapse Analytics

I've been playing around with Azure Synapse Analytics for a while now exploring the preview features and trying to find a meaningful use case for the 'single pane of glass' capabilities. In this post I'm exploring one possible option/idea for creating a very simple self service approach to dataset ingestion and consumption. Full disclosure, the … Continue reading An Idea for Self Service Using Azure Synapse Analytics

Capturing YouTube Video Views with Azure Data Factory

As a result of the pandemic like many other community speakers I've taken to YouTube as the current method for sharing knowledge and content. Recording video's isn't as much fun as speaking at a physical event, but it partially ticks the box. Plus, my daughter thinks its "cool" that I have a YouTube channel, so … Continue reading Capturing YouTube Video Views with Azure Data Factory

ADF.procfwk v1.8.5 – Execution Precursor

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.5 – Execution Precursor

ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring

ADF.procfwk v1.8.2 – Optionally Store SPN Details in Azure Key Vault

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.2 – Optionally Store SPN Details in Azure Key Vault

How To Use ‘Specify dynamic contents in JSON format’ in Azure Data Factory Linked Services

Make any Azure Data Factory Linked Service dynamic! In a few different community circles I've been asked 'how to handle dynamic Linked Service connections in Azure Data Factory if the UI doesn't naturally support the addition of parameters'. Depending on the Linked Service the support for this varies. Well, the answer, or should I say, … Continue reading How To Use ‘Specify dynamic contents in JSON format’ in Azure Data Factory Linked Services

ADF.procfwk v1.8 – Complete Pipeline Dependency Chains For Failure Handling

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8 – Complete Pipeline Dependency Chains For Failure Handling

ADF.procfwk v1.7.2 – NULL Pipeline Parameters Handled

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.7.2 – NULL Pipeline Parameters Handled

ADF.procfwk – The 300! Scaled Out Worker Pipeline Test

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk – The 300! Scaled Out Worker Pipeline Test