An Introduction to Azure Data Integration Pipelines

Azure Data Factory & Azure Synapse Analytics Following a recent talk I did for the MeetUp 'Let's Talk - Data Engineering' , here is the recording on YouTube for those that didn't catch the live broadcast. I squeezed a lot of content into an hour for those that are new to Azure Data Factory and … Continue reading An Introduction to Azure Data Integration Pipelines

Scaling Azure Data Integration Pipelines – Decoupling Data Extract And Transform

Scaling Azure Data Factory & Azure Synapse Analytics Pipelines Context Back in May 2020 I wrote a blog post about 'When You Should Use Multiple Azure Data Factory's'. Following on from this post with a full year+ now passed and having implemented many more data platform solutions for some crazy massive (technical term) enterprise customers … Continue reading Scaling Azure Data Integration Pipelines – Decoupling Data Extract And Transform

Thinking About an Azure Synapse Analytics Physical Architecture v1

Background Building on a previous blog post where I explored what a possible Azure Synapse Analytics logical architecture might look like in terms of end-to-end data curation/enrichment, here: Thinking about an Azure Synapse Analytics Logical Architecture v1 Now, with the recently added Synapse 'Managed Virtual Network' I'm exploring the same architecture from a physical network connectivity … Continue reading Thinking About an Azure Synapse Analytics Physical Architecture v1

Thinking about an Azure Synapse Analytics Logical Architecture v1

Hi folks, as some of you may be aware it was the Microsoft MVP Summit in March, for MVP's this is an exciting time where Microsoft product teams share plans for the coming year and talk about features on the roadmap. That said, I have the privilege of knowing what the future looks like for … Continue reading Thinking about an Azure Synapse Analytics Logical Architecture v1

Azure Synapse Analytics – Feature Summary vs Status

Last Updated - 28th may 2021 Following my previous (more detailed) blog post on 'Is Azure Synapse Analytics Ready for Production?' I was asked to summarize this understanding/opinion of Synapse into a simpler view. Updated: I've tweaked a couple of the traffic lights and narrative based on the annoucements this week from the Microsoft Build … Continue reading Azure Synapse Analytics – Feature Summary vs Status

Is Azure Synapse Analytics Ready for Production?

It's now been well over a year since Microsoft announced Azure Synapse Analytics as an offering at the big U.S.A based conferences back in November 2019. While I completely share and actually like Microsoft's vision of an analytics resource... "that brings together data integration, enterprise data warehousing and big data analytics" https://azure.microsoft.com/en-gb/services/synapse-analytics/ ... the marketing, … Continue reading Is Azure Synapse Analytics Ready for Production?

How Interchangeable Are Delta Tables Between Azure Databricks and Azure Synapse Analytics?

Databricks vs Synapse Analytics As an architect I often get challenged by customers on different approach's to a data transformation solutions, mainly because they are concerned about locking themselves into a particular technology, resource or vendor. One example of this is using a Delta Lake to deliver an Azure based warehousing/analytics platform. Given this context, … Continue reading How Interchangeable Are Delta Tables Between Azure Databricks and Azure Synapse Analytics?

Pipelines – Understanding Internal vs External Activities

Azure Data Factory & Azure Synapse Analytics Integrate Pipelines In this post I want us to explore and understand the difference between an internal and external activity when using our favourite orchestration pipelines. I'll focus predominately on Azure Data Factory (ADF), but the same applies to Azure Synapse Analytics. *Warning: this is a fairly dry, … Continue reading Pipelines – Understanding Internal vs External Activities

Trying to Deploy Azure Synapse Analytics Using ARM Templates

Just last week we heard the announcement from Microsoft that Azure Synapse Analytics is now generally available (GA)... A full year on, plus a few weeks, since first seeing Synapse at the big USA conferences in November 2019. Today I've been attempting to use the resource with a view to implementing it for several customer … Continue reading Trying to Deploy Azure Synapse Analytics Using ARM Templates