I've been playing around with Azure Synapse Analytics for a while now exploring the preview features and trying to find a meaningful use case for the 'single pane of glass' capabilities. In this post I'm exploring one possible option/idea for creating a very simple self service approach to dataset ingestion and consumption. Full disclosure, the … Continue reading An Idea for Self Service Using Azure Synapse Analytics
Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity
Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity
Capturing YouTube Video Views with Azure Data Factory
As a result of the pandemic like many other community speakers I've taken to YouTube as the current method for sharing knowledge and content. Recording video's isn't as much fun as speaking at a physical event, but it partially ticks the box. Plus, my daughter thinks its "cool" that I have a YouTube channel, so … Continue reading Capturing YouTube Video Views with Azure Data Factory
SQLBits 2020 – My Review & Thoughts
Like so many events SQLBits 2020 was delivered from our homes... Training sessions and content delivered via webcam and recordings. Catching up with friends done via chat rooms and breakout sessions. Hanging out in the coffee line done my hanging out in your kitchen with a headset. Banter and consume sharing done via social media. … Continue reading SQLBits 2020 – My Review & Thoughts
A Day Full of Azure Data Factory
Join me for a full day of training on Azure Data Factory. If we want to achieve any data processing in Azure you need an umbrella service to manage, monitor and schedule your solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS … Continue reading A Day Full of Azure Data Factory
ADF.procfwk v1.8.5 – Execution Precursor
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.5 – Execution Precursor
Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk
From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Kamil Nowinski talks about how you can use his PowerShell Module 'azure.datafactory.tools' to deploy Azure Data Factory with ease and in a highly flexible way. Kamil kindly helped me simplify the deployment steps needed for publishing my ADF.procfwk Data Factory pipelines via PowerShell and using his Azure DevOps … Continue reading Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk
ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring
ADF.procfwk v1.8.2 – Optionally Store SPN Details in Azure Key Vault
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.2 – Optionally Store SPN Details in Azure Key Vault
Using NUnit to Automate the Testing of Azure Data Factory Pipelines
From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Richard Swinbank talks about how you can use an NUnit project in Visual Studio to automate the testing of Data Factory pipelines. Richard kindly helped me apply this and test the pipelines in the ADF.procfwk to ensure post release everything still works in the open source code project. … Continue reading Using NUnit to Automate the Testing of Azure Data Factory Pipelines
