Azure Data Factory & Azure Synapse Analytics Integrate Pipelines In this post I want us to explore and understand the difference between an internal and external activity when using our favourite orchestration pipelines. I'll focus predominately on Azure Data Factory (ADF), but the same applies to Azure Synapse Analytics. *Warning: this is a fairly dry, … Continue reading Pipelines – Understanding Internal vs External Activities
Category: Data Factory
Data Factory Activity Concurrency Limits – What Happens Next?
As a follow up to my blog about Data Factory resource limitations here. I decided to dig deeper and expose some of these limitations with a view to understanding what happens to your pipeline when/if you hit them. In this post I'm focusing on the Activity Concurrency limits, as a reminder: Resource Default limit Maximum … Continue reading Data Factory Activity Concurrency Limits – What Happens Next?
Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API
Hi friends, just a very quick how to guide style post on something I had to build in Azure Data Factory. Scenario: I want to trigger a Data Factory pipeline, but when I do I want the pipeline to know if it's already running. If it is already running, stop the new run. Sounds simple … Continue reading Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API
Best Practices for Implementing Azure Data Factory – Auto Checker Script v0.1
Building on the work done and detailed in my previous blog post (Best Practices for Implementing Azure Data Factory) I was tasked by my delightful boss to turn this content into a simple check list of what/why that others could use.... I slightly reluctantly did so. However, I wanted to do something better than simply … Continue reading Best Practices for Implementing Azure Data Factory – Auto Checker Script v0.1
Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity
Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity
Capturing YouTube Video Views with Azure Data Factory
As a result of the pandemic like many other community speakers I've taken to YouTube as the current method for sharing knowledge and content. Recording video's isn't as much fun as speaking at a physical event, but it partially ticks the box. Plus, my daughter thinks its "cool" that I have a YouTube channel, so … Continue reading Capturing YouTube Video Views with Azure Data Factory
A Day Full of Azure Data Factory
Join me for a full day of training on Azure Data Factory. If we want to achieve any data processing in Azure you need an umbrella service to manage, monitor and schedule your solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS … Continue reading A Day Full of Azure Data Factory
ADF.procfwk v1.8.5 – Execution Precursor
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.5 – Execution Precursor
Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk
From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Kamil Nowinski talks about how you can use his PowerShell Module 'azure.datafactory.tools' to deploy Azure Data Factory with ease and in a highly flexible way. Kamil kindly helped me simplify the deployment steps needed for publishing my ADF.procfwk Data Factory pipelines via PowerShell and using his Azure DevOps … Continue reading Using the PowerShell Module ‘azure.datafactory.tools’ to Deploy ADF.procfwk
ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring
Code Project Overview This open source code project delivers a simple metadata driven processing framework for Azure Data Factory (ADF). The framework is made possible by coupling ADF with an Azure SQL Database that houses execution stage and pipeline information that is later called using an Azure Functions App. The parent/child metadata structure firstly allows … Continue reading ADF.procfwk v1.8.4 – Database Schema Reorganise and Restructuring
