Following on from a previous blog post that I wrote a few months ago where I got an Azure Data Factory Pipeline run status with an Azure Function (link below). I recently found the need to create something very similar to execute any pipeline from an Azure Function. https://mrpaulandrew.com/2019/11/21/get-any-azure-data-factory-pipeline-run-status-with-azure-functions/ Happily, this pipeline execution is basically … Continue reading Execute Any Azure Data Factory Pipeline with an Azure Function
Another year and another excellent SQLBits conference is in the events calendar. We were supposed to be back in London and at the massive ExCel exhibition centre. However, the event (like most others) is going virtual. For me, once again I'm very humbled to have been selected to speak, not once, but three times. Thank … Continue reading Preparing for SQLBits 2020 – My Azure Data Factory Sessions
Hello friends, I'm creating this post hopefully to raise awareness for my followers of the service limitations for Azure Data Factory. Like most resources in the Microsoft Cloud Platform at various levels (Resource/Resource Group/Subscription/Tenant) there are limitations, these are enforced by Microsoft and most of the time we don't hit them, especially when developing. That … Continue reading Azure Data Factory Resource Limitations
With this post I wanted to write something a little different, something that isn't directly focused on a particular technology, but is just as important when delivering an enterprise grade solution. Or any solution for that matter! Typically we all work in a team delivering Data Platform solutions for our internal/external customers. In order for … Continue reading My Script for Peer Reviewing Code
Switching Between Different Azure Databricks Clusters Depending on the Environment (Dev/Test/Prod) As far as I can gather at some point last year, probably around the time of Microsoft Ignite Azure Data Factory (ADF) got another new Activity called Switch. This is excellent and exactly what ADF needed. Nested If activities can get very messy so … Continue reading Using the Azure Data Factory Switch Activity
While documenting a customers data platform solution I decided it would be far easier if we could summarise the contents of a fairly complex Data Factory using its ARM Template. So, this is what I've done using T-SQL to parse the ARM Template JSON and output of series of tables containing details about the factory … Continue reading Summarise An Azure Data Factory ARM Template Using T-SQL
Blog post updated 14th August 2021 🙂 My colleagues and friends from the community keep asking me the same thing... What are the best practices from using Azure Data Factory (ADF)? With any emerging, rapidly changing technology I'm always hesitant about the answer. However, after 6 years of working with ADF I think its time … Continue reading Best Practices for Implementing Azure Data Factory
Renewing my post from last year (here) I've gathered my thoughts and put together the following content that I'd like to deliver to our wonderful data platform community in 2020. 2019 has been a great year for me having delivered 30 talks, in 17 cities and 8 different countries. This brings my grand total to … Continue reading My Community Talks 2020
For my first appearance on the Altius Data Lounge show I talk about and demonstrate the 'out-of-the-box' capabilities within Azure Data Factory for Git integration and how to publish developed pipelines. In my demo you'll see what each of the developer UI toolbar functions means in the context of the code repository and what happens … Continue reading Deploying Azure Data Factory in a Single Service Environment (Option 1)
Just a short post following a recent question I got from my delivery team... Are there any best practices for structuring our Databricks Notebooks in terms of code comments and markdown? Having done a little Googling I simply decided to whip up a quick example that could be adopted as a technical standard for the … Continue reading Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments
Welcome to the Blog of Paul Andrew 