Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions

A quick blog friends... I've done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I've: Executed Any Azure Data Factory Pipeline with an Azure Function Get Any Azure Data Factory Pipeline Run Status with Azure Functions Building on … Continue reading Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions

ADF.procfwk v1.3 – Metadata Integrity Checks

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.3 – Metadata Integrity Checks

Using the Azure Data Factory Switch Activity

Switching Between Different Azure Databricks Clusters Depending on the Environment (Dev/Test/Prod) As far as I can gather at some point last year, probably around the time of Microsoft Ignite Azure Data Factory (ADF) got another new Activity called Switch. This is excellent and exactly what ADF needed. Nested If activities can get very messy so … Continue reading Using the Azure Data Factory Switch Activity

Summarise An Azure Data Factory ARM Template Using T-SQL

While documenting a customers data platform solution I decided it would be far easier if we could summarise the contents of a fairly complex Data Factory using its ARM Template. So, this is what I've done using T-SQL to parse the ARM Template JSON and output of series of tables containing details about the factory … Continue reading Summarise An Azure Data Factory ARM Template Using T-SQL

Best Practices for Implementing Azure Data Factory

Blog post updated 15th July 2020 My colleagues and friends from the community keep asking me the same thing... What are the best practices from using Azure Data Factory (ADF)? With any emerging, rapidly changing technology I'm always hesitant about the answer. However, after 5 years of working with ADF I think its time to … Continue reading Best Practices for Implementing Azure Data Factory

Get Any Azure Data Factory Pipeline Run Status with Azure Functions

Microsoft.Azure.Management.DataFactory.Models.RunFilterParameters So I recently found myself with code withdrawal… Yes, its a real thing!  Anyone in an architect type role will understand, I hope… You can’t spend all your time drawing pictures of how things “should” work without occasionally getting hands-on with some code. Anyway, my recent episode of code withdrawal led me to creating … Continue reading Get Any Azure Data Factory Pipeline Run Status with Azure Functions

Using Data Factory Parameterised Linked Services

Microsoft recently announced that we can now make our Azure Data Factory (ADF) v2 pipelines even more dynamic with the introduction of parameterised Linked Services. This now completes the set for our core Data Factory components meaning we can now inject parameters into every part of our Data Factory control flow orchestration processes. Pipelines ✔ Activities … Continue reading Using Data Factory Parameterised Linked Services

Creating an Azure Data Factory v2 Custom Activity

Many moons ago and in a previous job role I wrote a post for creating an Azure Data Factory v1 Custom Activity here. According to Google Analytics this proved to be one of my most popular blog posts on that site. I therefore feel I need to do an update post with the same information … Continue reading Creating an Azure Data Factory v2 Custom Activity

Using the ADFv2 Until Activity

Hello friends, with this post I wanted to give you a quick 'how to' guide when working with the Azure Data Factory (ADF) v2 Until activity... Having implemented the activity several times in production I've repeatedly found the current Microsoft documentation here falling a little short in its explanations and examples. So, I've created my … Continue reading Using the ADFv2 Until Activity

ADFv2 Variable String Interpolation – REST API Body

As we know Azure Data Factory (ADF) Version 2 now has the ability to use expressions, parameters and system variables in various components throughout the service. This is brilliant and the best thing about v2 allowing us to build some very dynamic parameter driven pipelines. However, the syntax isn't always that obvious when you use … Continue reading ADFv2 Variable String Interpolation – REST API Body