An Introduction to Delta Lakes and Delta Lake-Houses

Recording from the Hybrid Virtual Group MeetUp Abstract Once upon a time, there was a data warehouse and it lived happily as a set of tables within our relational database management system (RDMS) called Microsoft SQL Server. The data warehouse had three children known as extract, transform, and load. One day a blue/azure coloured cloud … Continue reading An Introduction to Delta Lakes and Delta Lake-Houses

An Architects Guide to Delivering Data Insights Using the Microsoft Azure Data Platform

Azure Data Analytics End to End Let's start with a story, not a 'once upon a time story', a story for your backlog 🙂 As a solution architect I need to design and build an Azure data analytics platform end to end to deliver data insights for my customer. In February 2021 I delivered a … Continue reading An Architects Guide to Delivering Data Insights Using the Microsoft Azure Data Platform

How Interchangeable Are Delta Tables Between Azure Databricks and Azure Synapse Analytics?

Databricks vs Synapse Analytics As an architect I often get challenged by customers on different approach's to a data transformation solutions, mainly because they are concerned about locking themselves into a particular technology, resource or vendor. One example of this is using a Delta Lake to deliver an Azure based warehousing/analytics platform. Given this context, … Continue reading How Interchangeable Are Delta Tables Between Azure Databricks and Azure Synapse Analytics?

Using the Azure Data Factory Switch Activity

Switching Between Different Azure Databricks Clusters Depending on the Environment (Dev/Test/Prod) As far as I can gather at some point last year, probably around the time of Microsoft Ignite Azure Data Factory (ADF) got another new Activity called Switch. This is excellent and exactly what ADF needed. Nested If activities can get very messy so … Continue reading Using the Azure Data Factory Switch Activity

Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments

Just a short post following a recent question I got from my delivery team... Are there any best practices for structuring our Databricks Notebooks in terms of code comments and markdown? Having done a little Googling I simply decided to whip up a quick example that could be adopted as a technical standard for the … Continue reading Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments

PowerShell Export Databricks Workspace Items – Recurse

Building on the excellent PowerShell Databricks module created by Gerhard Brueckl here, I've added another layer of code to recursively export all items in a given Databricks workspace using PowerShell. I accept this does need to be hardened as a PowerShell cmdlet on its own and added to a module. However, I wanted to share the … Continue reading PowerShell Export Databricks Workspace Items – Recurse

What is Azure Data Factory Data Flow?

There were many great announcements to come out of the Microsoft Ignite 2018 conference, but the main one for me was the introduction of Azure Data Factory Data Flow. As a business intelligence person with many years experience working with SSIS packages this new feature is very exciting! Since I first saw the mock up … Continue reading What is Azure Data Factory Data Flow?

Preparing My Community Sessions for 2019

Hey friends & SQL family, its been a great year of tech and events so far, but now with several big conferences planning next years agendas its time to think ahead. Below I've prepared titles and abstracts for talks I'd like to deliver to our awesome data platform community in 2019. 5x regular sessions. 2x … Continue reading Preparing My Community Sessions for 2019