Welcome to the Blog & Website of Paul Andrew
Technical Leadership Centred Around the Microsoft Data Platform
Recording from the Hybrid Virtual Group MeetUp Abstract Once upon a time, there was a data warehouse and it lived happily as a set of tables within our relational database management system (RDMS) called Microsoft SQL Server. The data warehouse had three children known as extract, transform, and load. One day a blue/azure coloured cloud … Continue reading An Introduction to Delta Lakes and Delta Lake-Houses
Databricks vs Synapse Analytics As an architect I often get challenged by customers on different approach's to a data transformation solutions, mainly because they are concerned about locking themselves into a particular technology, resource or vendor. One example of this is using a Delta Lake to deliver an Azure based warehousing/analytics platform. Given this context, … Continue reading How Interchangeable Are Delta Tables Between Azure Databricks and Azure Synapse Analytics?
Just a short post following a recent question I got from my delivery team... Are there any best practices for structuring our Databricks Notebooks in terms of code comments and markdown? Having done a little Googling I simply decided to whip up a quick example that could be adopted as a technical standard for the … Continue reading Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments
Building on the excellent PowerShell Databricks module created by Gerhard Brueckl here, I've added another layer of code to recursively export all items in a given Databricks workspace using PowerShell. I accept this does need to be hardened as a PowerShell cmdlet on its own and added to a module. However, I wanted to share the … Continue reading PowerShell Export Databricks Workspace Items – Recurse
There were many great announcements to come out of the Microsoft Ignite 2018 conference, but the main one for me was the introduction of Azure Data Factory Data Flow. As a business intelligence person with many years experience working with SSIS packages this new feature is very exciting! Since I first saw the mock up … Continue reading What is Azure Data Factory Data Flow?
Welcome to the Blog & Website of Paul Andrew 