Summarise An Azure Data Factory ARM Template Using T-SQL

While documenting a customers data platform solution I decided it would be far easier if we could summarise the contents of a fairly complex Data Factory using its ARM Template. So, this is what I've done using T-SQL to parse the ARM Template JSON and output of series of tables containing details about the factory … Continue reading Summarise An Azure Data Factory ARM Template Using T-SQL

Best Practices for Implementing Azure Data Factory

My colleagues and friends from the community keep asking me the same thing... What are the best practices from using Azure Data Factory? With any emerging, rapidly changing technology I'm always hesitant about the answer. However, after 4 years of working with ADF I think its time to start suggesting what I'd expect to see … Continue reading Best Practices for Implementing Azure Data Factory

Get Any Azure Data Factory Pipeline Run Status with Azure Functions

Microsoft.Azure.Management.DataFactory.Models.RunFilterParameters So I recently found myself with code withdrawal… Yes, its a real thing!  Anyone in an architect type role will understand, I hope… You can’t spend all your time drawing pictures of how things “should” work without occasionally getting hands-on with some code. Anyway, my recent episode of code withdrawal led me to creating … Continue reading Get Any Azure Data Factory Pipeline Run Status with Azure Functions

Using Data Factory Parameterised Linked Services

Microsoft recently announced that we can now make our Azure Data Factory (ADF) v2 pipelines even more dynamic with the introduction of parameterised Linked Services. This now completes the set for our core Data Factory components meaning we can now inject parameters into every part of our Data Factory control flow orchestration processes. Pipelines ✔ Activities … Continue reading Using Data Factory Parameterised Linked Services

Creating an Azure Data Factory v2 Custom Activity

Many moons ago and in a previous job role I wrote a post for creating an Azure Data Factory v1 Custom Activity here. According to Google Analytics this proved to be one of my most popular blog posts on that site. I therefore feel I need to do an update post with the same information … Continue reading Creating an Azure Data Factory v2 Custom Activity

Using the ADFv2 Until Activity

Hello friends, with this post I wanted to give you a quick 'how to' guide when working with the Azure Data Factory (ADF) v2 Until activity... Having implemented the activity several times in production I've repeatedly found the current Microsoft documentation here falling a little short in its explanations and examples. So, I've created my … Continue reading Using the ADFv2 Until Activity

ADFv2 Variable String Interpolation – REST API Body

As we know Azure Data Factory (ADF) Version 2 now has the ability to use expressions, parameters and system variables in various components throughout the service. This is brilliant and the best thing about v2 allowing us to build some very dynamic parameter driven pipelines. However, the syntax isn't always that obvious when you use … Continue reading ADFv2 Variable String Interpolation – REST API Body