Having worked with Azure Data Factory and now Azure Synapse Analytics data integration pipelines ever since the resources first appeared in the Microsoft cloud platform, I've definitley earned (more than) my fair share of experience using the products. Both good and bad. As a data engineer and data archtiect. To help soften those bad experiences, … Continue reading Azure Data Integration Pipelines – A Deep Dive
Inspired by an earlier blog where we looked at 'How Interchangeable Delta Tables Are Between Databricks and Synapse' I decided to do a similar exercise, but this time with the integration pipeline components taking centre stage. As I said in my previous blog post, the question in the heading of this blog should be incredibly … Continue reading How Interchangeable Are Integration pipelines Between Azure Data factory and Azure Synapse Analytics?
Azure Data Factory & Azure Synapse Analytics Following a recent talk I did for the MeetUp 'Let's Talk - Data Engineering' , here is the recording on YouTube for those that didn't catch the live broadcast. I squeezed a lot of content into an hour for those that are new to Azure Data Factory and … Continue reading An Introduction to Azure Data Integration Pipelines
Scaling Azure Data Factory & Azure Synapse Analytics Pipelines Context Back in May 2020 I wrote a blog post about 'When You Should Use Multiple Azure Data Factory's'. Following on from this post with a full year+ now passed and having implemented many more data platform solutions for some crazy massive (technical term) enterprise customers … Continue reading Scaling Azure Data Integration Pipelines – Decoupling Data Extract And Transform
The other day Kamil (@NowinskiK) messaged me on Teams asking if I liked Mermaid's 🙂 I prepared myself for a conversation about which was better; the fish half being on top vs on the bottom. Ha! Of course not!! Kamil was actually referring to the... Javascript based diagramming and charting tool that renders Markdown-inspired text … Continue reading Using Mermaid to Create a ProcFwk Pipeline Lineage Diagram
On 2nd June 2020 Apress reached out to me about the role of technical reviewer on a book about Azure Data Factory. The exact conversation may amuse you, so I've included it exactly as it happened 🙂 THE BEGINNING Paul,I know writing is out of the question, but do you have bandwidth to tech review … Continue reading Azure Data Factory by Example – A Review of My Technical Review
Azure Data Analytics End to End Let's start with a story, not a 'once upon a time story', a story for your backlog 🙂 As a solution architect I need to design and build an Azure data analytics platform end to end to deliver data insights for my customer. In February 2021 I delivered a … Continue reading An Architects Guide to Delivering Data Insights Using the Microsoft Azure Data Platform
Azure Data Factory & Azure Synapse Analytics Integrate Pipelines In this post I want us to explore and understand the difference between an internal and external activity when using our favourite orchestration pipelines. I'll focus predominately on Azure Data Factory (ADF), but the same applies to Azure Synapse Analytics. *Warning: this is a fairly dry, … Continue reading Pipelines – Understanding Internal vs External Activities
As a follow up to my blog about Data Factory resource limitations here. I decided to dig deeper and expose some of these limitations with a view to understanding what happens to your pipeline when/if you hit them. In this post I'm focusing on the Activity Concurrency limits, as a reminder: Resource Default limit Maximum … Continue reading Data Factory Activity Concurrency Limits – What Happens Next?
Hi friends, just a very quick how to guide style post on something I had to build in Azure Data Factory. Scenario: I want to trigger a Data Factory pipeline, but when I do I want the pipeline to know if it's already running. If it is already running, stop the new run. Sounds simple … Continue reading Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API
Welcome to the Community Blog of Paul Andrew 