Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk

From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Kamil Nowinski talks about how you can use his PowerShell Module 'azure.datafactory.tools' to deploy Azure Data Factory with ease and in a highly flexible way. Kamil kindly helped me simplify the deployment steps needed for publishing my ADF.procfwk Data Factory pipelines via PowerShell and using his Azure DevOps … Continue reading Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk

Using NUnit to Automate the Testing of Azure Data Factory Pipelines

From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Richard Swinbank talks about how you can use an NUnit project in Visual Studio to automate the testing of Data Factory pipelines. Richard kindly helped me apply this and test the pipelines in the ADF.procfwk to ensure post release everything still works in the open source code project. … Continue reading Using NUnit to Automate the Testing of Azure Data Factory Pipelines

When Should I Use Multiple Azure Data Factory’s?

Hi all, I get asked this question a lot. So I decided to do a full post on the subject based on my current thoughts and experience. When should I use multiple Azure Data Factory's? The consultants answer, it depends πŸ˜‰ Ok, let's go deeper and explore why it depends! Also, to be clear, we … Continue reading When Should I Use Multiple Azure Data Factory’s?

Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions

A quick blog friends... I've done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I've: Executed Any Azure Data Factory Pipeline with an Azure Function Get Any Azure Data Factory Pipeline Run Status with Azure Functions Building on … Continue reading Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions

ADF.procfwk v1.3 – Metadata Integrity Checks

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.3 – Metadata Integrity Checks

First Time Playing with Spark.Net on Azure Synapse Analytics

Exciting times friends, today (25th March) the lovely people at Microsoft granted me access to the private preview MVP workspace for Azure Synapse Analytics πŸ˜€ In this quick blog I wanted to share my experience so far, which I'm basically writing as I'm playing around... https://azure.microsoft.com/en-gb/services/synapse-analytics/ The main reason I wanted access to Synapse is … Continue reading First Time Playing with Spark.Net on Azure Synapse Analytics

Using the Azure Data Factory Switch Activity

Switching Between Different Azure Databricks Clusters Depending on the Environment (Dev/Test/Prod) As far as I can gather at some point last year, probably around the time of Microsoft Ignite Azure Data Factory (ADF) got another new Activity called Switch. This is excellent and exactly what ADF needed. Nested If activities can get very messy so … Continue reading Using the Azure Data Factory Switch Activity

Summarise An Azure Data Factory ARM Template Using T-SQL

While documenting a customers data platform solution I decided it would be far easier if we could summarise the contents of a fairly complex Data Factory using its ARM Template. So, this is what I've done using T-SQL to parse the ARM Template JSON and output of series of tables containing details about the factory … Continue reading Summarise An Azure Data Factory ARM Template Using T-SQL

Best Practices for Implementing Azure Data Factory

Blog post updated 15th July 2020 My colleagues and friends from the community keep asking me the same thing... What are the best practices from using Azure Data Factory (ADF)? With any emerging, rapidly changing technology I'm always hesitant about the answer. However, after 5 years of working with ADF I think its time to … Continue reading Best Practices for Implementing Azure Data Factory

Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments

Just a short post following a recent question I got from my delivery team... Are there any best practices for structuring our Databricks Notebooks in terms of code comments and markdown? Having done a little Googling I simply decided to whip up a quick example that could be adopted as a technical standard for the … Continue reading Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments