Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 1 of 4

In this four part blog series I want to share my approach to delivering a metadata driven processing framework in Azure Data Factory. This is very much version 1 of what is possible and where can we build on this to deliver a hardened solution for our platform orchestrator. For the agenda of this blog … Continue reading Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 1 of 4

Execute Any Azure Data Factory Pipeline with an Azure Function

Following on from a previous blog post that I wrote a few months ago where I got an Azure Data Factory Pipeline run status with an Azure Function (link below). I recently found the need to create something very similar to execute any pipeline from an Azure Function. https://mrpaulandrew.com/2019/11/21/get-any-azure-data-factory-pipeline-run-status-with-azure-functions/ Happily, this pipeline execution is basically … Continue reading Execute Any Azure Data Factory Pipeline with an Azure Function

Preparing for SQLBits 2020 – My Azure Data Factory Sessions

Another year and another excellent SQLBits conference is in the events calendar. This time back to London and the massive ExCel exhibition centre. For me, once again I'm very humbled to have been selected to speak, not once, but twice. Thank you SQLBits. This will be my: Fifth time attending the event. Fourth time helping … Continue reading Preparing for SQLBits 2020 – My Azure Data Factory Sessions

Azure Data Factory Resource Limitations

Hello friends, I'm creating this post hopefully to raise awareness for my followers of the service limitations for Azure Data Factory. Like most resources in the Microsoft Cloud Platform at various levels (Resource/Resource Group/Subscription/Tenant) there are limitations, these are enforced by Microsoft and most of the time we don't hit them, especially when developing. That … Continue reading Azure Data Factory Resource Limitations

Using the Azure Data Factory Switch Activity

Switching Between Different Azure Databricks Clusters Depending on the Environment (Dev/Test/Prod) As far as I can gather at some point last year, probably around the time of Microsoft Ignite Azure Data Factory (ADF) got another new Activity called Switch. This is excellent and exactly what ADF needed. Nested If activities can get very messy so … Continue reading Using the Azure Data Factory Switch Activity

Summarise An Azure Data Factory ARM Template Using T-SQL

While documenting a customers data platform solution I decided it would be far easier if we could summarise the contents of a fairly complex Data Factory using its ARM Template. So, this is what I've done using T-SQL to parse the ARM Template JSON and output of series of tables containing details about the factory … Continue reading Summarise An Azure Data Factory ARM Template Using T-SQL

Best Practices for Implementing Azure Data Factory

My colleagues and friends from the community keep asking me the same thing... What are the best practices from using Azure Data Factory? With any emerging, rapidly changing technology I'm always hesitant about the answer. However, after 4 years of working with ADF I think its time to start suggesting what I'd expect to see … Continue reading Best Practices for Implementing Azure Data Factory