Following a recent talk I did for the Europe - Data Platform Discovery Day, here is the YouTube recording for those that didn't get to see it live. https://www.youtube.com/watch?v=CtDTJPbXVpQ Slides available in GitHub here: https://github.com/mrpaulandrew/CommunityEvents/blob/master/DataPlatformDiscoveryDay-2020/Slides.pdf Thanks again to the event organisers. Please reach out if you'd like me to speak at your community event. A … Continue reading A Complete Introduction to Azure Data Factory
Author: mrpaulandrew
Principal consultant and architect specialising in big data solutions on the Microsoft Azure cloud platform.
Data engineering competencies include Azure Data Factory, Data Lake, Databricks, Stream Analytics, Event Hub, IoT Hub, Functions, Automation, Logic Apps and of course the complete SQL Server business intelligence stack.
Many years’ experience working within healthcare, retail and gaming verticals delivering analytics using industry leading methods and technical design patterns.
STEM ambassador and very active member of the data platform community delivering training and technical sessions at conferences both nationally and internationally.
Father, husband, swimmer, cyclist, runner, blood donor, geek, Lego and Star Wars fan!
ADF.procfwk v1.5 – Power BI Dashboard for Framework Executions
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.5 – Power BI Dashboard for Framework Executions
Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions
A quick blog friends... I've done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I've: Executed Any Azure Data Factory Pipeline with an Azure Function Got Any Azure Data Factory Pipeline Run Status with Azure Functions Building on … Continue reading Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions
ADF.procfwk v1.4 – Enhancements for Long Running Pipelines
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.4 – Enhancements for Long Running Pipelines
ADF.procfwk v1.3 – Metadata Integrity Checks
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.3 – Metadata Integrity Checks
First Time Playing with Spark.Net on Azure Synapse Analytics
Exciting times friends, today (25th March) the lovely people at Microsoft granted me access to the private preview MVP workspace for Azure Synapse Analytics 😀 In this quick blog I wanted to share my experience so far, which I'm basically writing as I'm playing around... https://azure.microsoft.com/en-gb/services/synapse-analytics/ The main reason I wanted access to Synapse is … Continue reading First Time Playing with Spark.Net on Azure Synapse Analytics
ADF.procfwk v1.2 – Execution Restartability
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is delivered by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows stages … Continue reading ADF.procfwk v1.2 – Execution Restartability
ADF.procfwk v1.1 – Service Principal Handling via Metadata
Hi data friends! Version 1.1 of my ADF.procfwk is ready! Following the great response I got to the version 1.0 blog series I decided to push ahead and get version 1.1 out there as soon as possible. The main thing I've addressed with this release is the big problem of having hard coded service principal … Continue reading ADF.procfwk v1.1 – Service Principal Handling via Metadata
Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 4 of 4
In this four part blog series I want to share my approach to delivering a metadata driven processing framework in Azure Data Factory. This is very much version 1 of what is possible and where can we build on this to deliver a hardened solution for our platform orchestrator. For the agenda of this blog … Continue reading Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 4 of 4
Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 3 of 4
In this four part blog series I want to share my approach to delivering a metadata driven processing framework in Azure Data Factory. This is very much version 1 of what is possible and where can we build on this to deliver a hardened solution for our platform orchestrator. For the agenda of this blog … Continue reading Creating a Simple Staged Metadata Driven Processing Framework for Azure Data Factory Pipelines – Part 3 of 4
