Recording from the Hybrid Virtual Group MeetUp Abstract Once upon a time, there was a data warehouse and it lived happily as a set of tables within our relational database management system (RDMS) called Microsoft SQL Server. The data warehouse had three children known as extract, transform, and load. One day a blue/azure coloured cloud … Continue reading An Introduction to Delta Lakes and Delta Lake-Houses
Having worked with Azure Data Factory and now Azure Synapse Analytics data integration pipelines ever since the resources first appeared in the Microsoft cloud platform, I've definitley earned (more than) my fair share of experience using the products. Both good and bad. As a data engineer and data archtiect. To help soften those bad experiences, … Continue reading Azure Data Integration Pipelines – A Deep Dive
Inspired by an earlier blog where we looked at 'How Interchangeable Delta Tables Are Between Databricks and Synapse' I decided to do a similar exercise, but this time with the integration pipeline components taking centre stage. As I said in my previous blog post, the question in the heading of this blog should be incredibly … Continue reading How Interchangeable Are Integration pipelines Between Azure Data factory and Azure Synapse Analytics?
Azure Data Factory & Azure Synapse Analytics Following a recent talk I did for the MeetUp 'Let's Talk - Data Engineering' , here is the recording on YouTube for those that didn't catch the live broadcast. I squeezed a lot of content into an hour for those that are new to Azure Data Factory and … Continue reading An Introduction to Azure Data Integration Pipelines
A Great Outcome Following the Acquisition of Altius Once upon a time, a grumpy consultant was made even more grumpy (if that was possible) by the news that Altius Data Consulting Ltd was being/had been acquired by the much-much larger Avanade Inc orange machine, complete with Accenture and Microsoft parents, Mum & Dad. I must … Continue reading A Spring Board into the Avanade Centre of Excellence (CoE)
Scaling Azure Data Factory & Azure Synapse Analytics Pipelines Context Back in May 2020 I wrote a blog post about 'When You Should Use Multiple Azure Data Factory's'. Following on from this post with a full year+ now passed and having implemented many more data platform solutions for some crazy massive (technical term) enterprise customers … Continue reading Scaling Azure Data Integration Pipelines – Decoupling Data Extract And Transform
The other day Kamil (@NowinskiK) messaged me on Teams asking if I liked Mermaid's 🙂 I prepared myself for a conversation about which was better; the fish half being on top vs on the bottom. Ha! Of course not!! Kamil was actually referring to the... Javascript based diagramming and charting tool that renders Markdown-inspired text … Continue reading Using Mermaid to Create a ProcFwk Pipeline Lineage Diagram
When logging into my MVP profile to update community activities etc I still have to do a double take when seeing the bookmark image (on the left). Specifically, the 'Number of MVP Awards' sub heading, reading 5. Five consecutive years I've been given this great award from Microsoft as a Most Valuable Professional. Smug face … Continue reading The 5 Year Microsoft MVP Milestone
What are the most common phrases that everyone tends to use during Teams meetings, things people only say when you join a call? Just for fun, here is my top 10 list. I'll just quickly share my screen. You're on mute.Can you hear me?Can you see my screen?You're breaking up.You've got your hand raised.Sorry, Teams … Continue reading Teams Meeting Phrase Bingo
On 2nd June 2020 Apress reached out to me about the role of technical reviewer on a book about Azure Data Factory. The exact conversation may amuse you, so I've included it exactly as it happened 🙂 THE BEGINNING Paul,I know writing is out of the question, but do you have bandwidth to tech review … Continue reading Azure Data Factory by Example – A Review of My Technical Review
Welcome to the Community Blog of Paul Andrew 