How Interchangeable Are Integration pipelines Between Azure Data factory and Azure Synapse Analytics?

Inspired by an earlier blog where we looked at 'How Interchangeable Delta Tables Are Between Databricks and Synapse' I decided to do a similar exercise, but this time with the integration pipeline components taking centre stage. As I said in my previous blog post, the question in the heading of this blog should be incredibly … Continue reading How Interchangeable Are Integration pipelines Between Azure Data factory and Azure Synapse Analytics?

An Introduction to Azure Data Integration Pipelines

Azure Data Factory & Azure Synapse Analytics Following a recent talk I did for the MeetUp 'Let's Talk - Data Engineering' , here is the recording on YouTube for those that didn't catch the live broadcast. I squeezed a lot of content into an hour for those that are new to Azure Data Factory and … Continue reading An Introduction to Azure Data Integration Pipelines

Scaling Azure Data Integration Pipelines – Decoupling Data Extract And Transform

Scaling Azure Data Factory & Azure Synapse Analytics Pipelines Context Back in May 2020 I wrote a blog post about 'When You Should Use Multiple Azure Data Factory's'. Following on from this post with a full year+ now passed and having implemented many more data platform solutions for some crazy massive (technical term) enterprise customers … Continue reading Scaling Azure Data Integration Pipelines – Decoupling Data Extract And Transform

Using Mermaid to Create a ProcFwk Pipeline Lineage Diagram

The other day Kamil (@NowinskiK) messaged me on Teams asking if I liked Mermaid's πŸ™‚ I prepared myself for a conversation about which was better; the fish half being on top vs on the bottom. Ha! Of course not!! Kamil was actually referring to the... Javascript based diagramming and charting tool that renders Markdown-inspired text … Continue reading Using Mermaid to Create a ProcFwk Pipeline Lineage Diagram

Azure Data Factory by Example – A Review of My Technical Review

On 2nd June 2020 Apress reached out to me about the role of technical reviewer on a book about Azure Data Factory. The exact conversation may amuse you, so I've included it exactly as it happened πŸ™‚ THE BEGINNING Paul,I know writing is out of the question, but do you have bandwidth to tech review … Continue reading Azure Data Factory by Example – A Review of My Technical Review

Thinking About an Azure Synapse Analytics Physical Architecture v1

Background Building on a previous blog post where I explored what a possible Azure Synapse Analytics logical architecture might look like in terms of end-to-end data curation/enrichment, here: Thinking about an Azure Synapse Analytics Logical Architecture v1 Now, with the recently added Synapse 'Managed Virtual Network' I'm exploring the same architecture from a physical network connectivity … Continue reading Thinking About an Azure Synapse Analytics Physical Architecture v1

Thinking about an Azure Synapse Analytics Logical Architecture v1

Hi folks, as some of you may be aware it was the Microsoft MVP Summit in March, for MVP's this is an exciting time where Microsoft product teams share plans for the coming year and talk about features on the roadmap. That said, I have the privilege of knowing what the future looks like for … Continue reading Thinking about an Azure Synapse Analytics Logical Architecture v1

An Architects Guide to Delivering Data Insights Using the Microsoft Azure Data Platform

Azure Data Analytics End to End Let's start with a story, not a 'once upon a time story', a story for your backlog πŸ™‚ As a solution architect I need to design and build an Azure data analytics platform end to end to deliver data insights for my customer. In February 2021 I delivered a … Continue reading An Architects Guide to Delivering Data Insights Using the Microsoft Azure Data Platform

Azure Synapse Analytics – Feature Summary vs Status

Last Updated - 28th may 2021 Following my previous (more detailed) blog post on 'Is Azure Synapse Analytics Ready for Production?' I was asked to summarize this understanding/opinion of Synapse into a simpler view. Updated: I've tweaked a couple of the traffic lights and narrative based on the annoucements this week from the Microsoft Build … Continue reading Azure Synapse Analytics – Feature Summary vs Status