Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.7.1 – Alerting Bug Fix And Pipeline Parameter Size Limit Removed
Author: mrpaulandrew
Principal consultant and architect specialising in big data solutions on the Microsoft Azure cloud platform.
Data engineering competencies include Azure Data Factory, Data Lake, Databricks, Stream Analytics, Event Hub, IoT Hub, Functions, Automation, Logic Apps and of course the complete SQL Server business intelligence stack.
Many years’ experience working within healthcare, retail and gaming verticals delivering analytics using industry leading methods and technical design patterns.
STEM ambassador and very active member of the data platform community delivering training and technical sessions at conferences both nationally and internationally.
Father, husband, swimmer, cyclist, runner, blood donor, geek, Lego and Star Wars fan!
ADF.procfwk – Video Setup How To Guide
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk – Video Setup How To Guide
ADF.procfwk v1.7 – Pipeline Email Alerting
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.7 – Pipeline Email Alerting
ADF.procfwk and Azure Synapse Orchestrate – Preview Limitations
With Azure Synapse Analytics now in public preview is was time to find out how compatible my Azure Data Factory metadata driven processing framework (ADF.procfwk) is with the Synapse Orchestrate features. Firstly, as Synapse doesn't yet have any source control or DevOps support I had to manually rebuild the framework pipelines in the browser, copying … Continue reading ADF.procfwk and Azure Synapse Orchestrate – Preview Limitations
When Should I Use Multiple Azure Data Factory’s?
Hi all, I get asked this question a lot. So I decided to do a full post on the subject based on my current thoughts and experience. When should I use multiple Azure Data Factory's? The consultants answer, it depends 😉 Ok, let's go deeper and explore why it depends! Also, to be clear, we … Continue reading When Should I Use Multiple Azure Data Factory’s?
Microsoft Book of News – Build 2020
Hi friends, its conference time again and Microsoft have kindly provided another Book of News. This time its for Build 2020 and it contains details of all the announcements made. Based on the 55 pages of content it would appear Microsoft is once again focused on all things Azure Data/AI in this current release period, … Continue reading Microsoft Book of News – Build 2020
ADF.procfwk v1.6 – Error Details for Failed Activities Captured
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.6 – Error Details for Failed Activities Captured
A Complete Introduction to Azure Data Factory
Following a recent talk I did for the Europe - Data Platform Discovery Day, here is the YouTube recording for those that didn't get to see it live. https://www.youtube.com/watch?v=CtDTJPbXVpQ Slides available in GitHub here: https://github.com/mrpaulandrew/CommunityEvents/blob/master/DataPlatformDiscoveryDay-2020/Slides.pdf Thanks again to the event organisers. Please reach out if you'd like me to speak at your community event. A … Continue reading A Complete Introduction to Azure Data Factory
ADF.procfwk v1.5 – Power BI Dashboard for Framework Executions
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.5 – Power BI Dashboard for Framework Executions
Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions
A quick blog friends... I've done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I've: Executed Any Azure Data Factory Pipeline with an Azure Function Get Any Azure Data Factory Pipeline Run Status with Azure Functions Building on … Continue reading Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions
