Recording from the Hybrid Virtual Group MeetUp Abstract Once upon a time, there was a data warehouse and it lived happily as a set of tables within our relational database management system (RDMS) called Microsoft SQL Server. The data warehouse had three children known as extract, transform, and load. One day a blue/azure coloured cloud … Continue reading An Introduction to Delta Lakes and Delta Lake-Houses
Inspired by an earlier blog where we looked at 'How Interchangeable Delta Tables Are Between Databricks and Synapse' I decided to do a similar exercise, but this time with the integration pipeline components taking centre stage. As I said in my previous blog post, the question in the heading of this blog should be incredibly … Continue reading How Interchangeable Are Integration pipelines Between Azure Data factory and Azure Synapse Analytics?
Azure Data Factory & Azure Synapse Analytics Following a recent talk I did for the MeetUp 'Let's Talk - Data Engineering' , here is the recording on YouTube for those that didn't catch the live broadcast. I squeezed a lot of content into an hour for those that are new to Azure Data Factory and … Continue reading An Introduction to Azure Data Integration Pipelines
Scaling Azure Data Factory & Azure Synapse Analytics Pipelines Context Back in May 2020 I wrote a blog post about 'When You Should Use Multiple Azure Data Factory's'. Following on from this post with a full year+ now passed and having implemented many more data platform solutions for some crazy massive (technical term) enterprise customers … Continue reading Scaling Azure Data Integration Pipelines – Decoupling Data Extract And Transform
On 2nd June 2020 Apress reached out to me about the role of technical reviewer on a book about Azure Data Factory. The exact conversation may amuse you, so I've included it exactly as it happened 🙂 THE BEGINNING Paul,I know writing is out of the question, but do you have bandwidth to tech review … Continue reading Azure Data Factory by Example – A Review of My Technical Review
Azure Data Analytics End to End Let's start with a story, not a 'once upon a time story', a story for your backlog 🙂 As a solution architect I need to design and build an Azure data analytics platform end to end to deliver data insights for my customer. In February 2021 I delivered a … Continue reading An Architects Guide to Delivering Data Insights Using the Microsoft Azure Data Platform
How nice is this!... All the Microsoft Ignite announcements in one place. Very cool. Data Platform and AI things in chapter 3, starting on page 39. Sadly not that many new things for us data people this year, but still, its all great news. Azure Synapse Analytics seems to be the main data platform news, … Continue reading Microsoft Ignite 2019 Announcements – The Ignite Book of News 2019
Building on the excellent PowerShell Databricks module created by Gerhard Brueckl here, I've added another layer of code to recursively export all items in a given Databricks workspace using PowerShell. I accept this does need to be hardened as a PowerShell cmdlet on its own and added to a module. However, I wanted to share the … Continue reading PowerShell Export Databricks Workspace Items – Recurse
Continuous delivery of data services has always been a battle for us data analytics peeps. Things are getting better when we think about business intelligence and dev ops, however there are still gaps. This blog post is an account of one such issue I recently faced and how I worked around the problem. Problem While … Continue reading Inject into Azure Logic Apps ARM Templates with PowerShell
Its finally here!! After being teased for nearly 4 months with all the great new functionality available in Azure Data Factory version 2 we finally have the tools to start developing it. Microsoft, it appears, have leap frogged my long standing request and feedback votes to support ADF templates in Visual Studio 2017 and gone … Continue reading Azure Data Factory v2 Developer Interface Launched
Welcome to the Community Blog of Paul Andrew 