Tag: Azure Data Factory

Data Integration, Architecture and Real-Time Analytics Full Training Days with Me 2024

Posted on by mrpaulandrew

SQL Bits, UK, go to >> SQL Day, Poland, go to >> Data Platform Next Steps, Denmark, go to >> Big Data Europe, Lithuania, go to >> SQLBits | March | Online & In Person Microsoft Data Integration Pipelines – The Fundamentals to Level 300 Event Link: https://sqlbits.com/attend/the-agenda/full-day-training/Microsoft_Data_Integration_Pipelines__The_Fundamentals_to_Level_300 Overview In this full day of training, … Continue reading Data Integration, Architecture and Real-Time Analytics Full Training Days with Me 2024 →

Training Events in October 2022

Posted on by mrpaulandrew

Azure Data Factory & Azure Synapse Analytics Pipeline Training Get your tickets for more in person training with me this October. Limited places available, don't miss out. Azure Data Integration Pipelines: Fundamentals to Level 300 When: Monday, 10th October Where: Lamot Conference & Heritage Centre - Mechelen, Belgium Host: dataMinds Connect Tickets: https://tickets.datamindsconnect.be/ Azure Data … Continue reading Training Events in October 2022 →

Training Events in 2022 – Q2/Q3

Posted on by mrpaulandrew

Azure Data Factory & Azure Synapse Analytics Pipelines Training More in person training confirmed with me in June and September 2022. Limited places available, don't miss out. Azure Data Integration Pipelines: Advanced Design & Delivery Monday 27th June - Birmingham Hosted by Cloud Formations Event tickets here: https://www.cloudformations.org/event-details/azure-integration-pipelines-advanced-design-delivery-with-paul-andrew Azure Data Integration Pipelines: Fundamentals to Level … Continue reading Training Events in 2022 – Q2/Q3 →

Azure Data Integration Pipelines – A Deep Dive

Posted on by mrpaulandrew

Having worked with Azure Data Factory and now Azure Synapse Analytics data integration pipelines ever since the resources first appeared in the Microsoft cloud platform, I've definitley earned (more than) my fair share of experience using the products. Both good and bad. As a data engineer and data archtiect. To help soften those bad experiences, … Continue reading Azure Data Integration Pipelines – A Deep Dive →

Using Mermaid to Create a ProcFwk Pipeline Lineage Diagram

Posted on by mrpaulandrew

The other day Kamil (@NowinskiK) messaged me on Teams asking if I liked Mermaid's 🙂 I prepared myself for a conversation about which was better; the fish half being on top vs on the bottom. Ha! Of course not!! Kamil was actually referring to the... Javascript based diagramming and charting tool that renders Markdown-inspired text … Continue reading Using Mermaid to Create a ProcFwk Pipeline Lineage Diagram →

Pipelines – Understanding Internal vs External Activities

Posted on by mrpaulandrew

Azure Data Factory & Azure Synapse Analytics Integrate Pipelines In this post I want us to explore and understand the difference between an internal and external activity when using our favourite orchestration pipelines. I'll focus predominately on Azure Data Factory (ADF), but the same applies to Azure Synapse Analytics. *Warning: this is a fairly dry, … Continue reading Pipelines – Understanding Internal vs External Activities →

Data Factory Activity Concurrency Limits – What Happens Next?

Posted on by mrpaulandrew

As a follow up to my blog about Data Factory resource limitations here. I decided to dig deeper and expose some of these limitations with a view to understanding what happens to your pipeline when/if you hit them. In this post I'm focusing on the Activity Concurrency limits, as a reminder: Resource Default limit Maximum … Continue reading Data Factory Activity Concurrency Limits – What Happens Next? →

Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API

Posted on by mrpaulandrew

Hi friends, just a very quick how to guide style post on something I had to build in Azure Data Factory. Scenario: I want to trigger a Data Factory pipeline, but when I do I want the pipeline to know if it's already running. If it is already running, stop the new run. Sounds simple … Continue reading Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API →

Best Practices for Implementing Azure Data Factory – Auto Checker Script v0.1

Posted on by mrpaulandrew

Building on the work done and detailed in my previous blog post (Best Practices for Implementing Azure Data Factory) I was tasked by my delightful boss to turn this content into a simple check list of what/why that others could use.... I slightly reluctantly did so. However, I wanted to do something better than simply … Continue reading Best Practices for Implementing Azure Data Factory – Auto Checker Script v0.1 →

Capturing YouTube Video Views with Azure Data Factory

Posted on by mrpaulandrew

As a result of the pandemic like many other community speakers I've taken to YouTube as the current method for sharing knowledge and content. Recording video's isn't as much fun as speaking at a physical event, but it partially ticks the box. Plus, my daughter thinks its "cool" that I have a YouTube channel, so … Continue reading Capturing YouTube Video Views with Azure Data Factory →