An Idea for Self Service Using Azure Synapse Analytics

I've been playing around with Azure Synapse Analytics for a while now exploring the preview features and trying to find a meaningful use case for the 'single pane of glass' capabilities. In this post I'm exploring one possible option/idea for creating a very simple self service approach to dataset ingestion and consumption. Full disclosure, the … Continue reading An Idea for Self Service Using Azure Synapse Analytics

Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity

Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity

Capturing YouTube Video Views with Azure Data Factory

As a result of the pandemic like many other community speakers I've taken to YouTube as the current method for sharing knowledge and content. Recording video's isn't as much fun as speaking at a physical event, but it partially ticks the box. Plus, my daughter thinks its "cool" that I have a YouTube channel, so … Continue reading Capturing YouTube Video Views with Azure Data Factory

SQLBits 2020 – My Review & Thoughts

Like so many events SQLBits 2020 was delivered from our homes... Training sessions and content delivered via webcam and recordings. Catching up with friends done via chat rooms and breakout sessions. Hanging out in the coffee line done my hanging out in your kitchen with a headset. Banter and consume sharing done via social media. … Continue reading SQLBits 2020 – My Review & Thoughts

ADF.procfwk v1.8.5 – Execution Precursor

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.5 – Execution Precursor

Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk

From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Kamil Nowinski talks about how you can use his PowerShell Module 'azure.datafactory.tools' to deploy Azure Data Factory with ease and in a highly flexible way. Kamil kindly helped me simplify the deployment steps needed for publishing my ADF.procfwk Data Factory pipelines via PowerShell and using his Azure DevOps … Continue reading Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk

ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring

ADF.procfwk v1.8.2 – Optionally Store SPN Details in Azure Key Vault

Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.2 – Optionally Store SPN Details in Azure Key Vault

Using NUnit to Automate the Testing of Azure Data Factory Pipelines

From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Richard Swinbank talks about how you can use an NUnit project in Visual Studio to automate the testing of Data Factory pipelines. Richard kindly helped me apply this and test the pipelines in the ADF.procfwk to ensure post release everything still works in the open source code project. … Continue reading Using NUnit to Automate the Testing of Azure Data Factory Pipelines