As Azure Data Factory continues to evolve as a powerful cloud orchestration service we need to update our knowledge and understanding of everything the service has to offer. Mainly, so we can make the right design decisions when developing complex, dynamic solution pipelines. In this post I want to explore and share the reasons for … Continue reading Azure Data Factory – Web Hook vs Web Activity
Author: mrpaulandrew
Principal consultant and architect specialising in big data solutions on the Microsoft Azure cloud platform.
Data engineering competencies include Azure Data Factory, Data Lake, Databricks, Stream Analytics, Event Hub, IoT Hub, Functions, Automation, Logic Apps and of course the complete SQL Server business intelligence stack.
Many years’ experience working within healthcare, retail and gaming verticals delivering analytics using industry leading methods and technical design patterns.
STEM ambassador and very active member of the data platform community delivering training and technical sessions at conferences both nationally and internationally.
Father, husband, swimmer, cyclist, runner, blood donor, geek, Lego and Star Wars fan!
PowerShell Export Databricks Workspace Items – Recurse
Building on the excellent PowerShell Databricks module created by Gerhard Brueckl here, I've added another layer of code to recursively export all items in a given Databricks workspace using PowerShell. I accept this does need to be hardened as a PowerShell cmdlet on its own and added to a module. However, I wanted to share the … Continue reading PowerShell Export Databricks Workspace Items – Recurse
Interacting with SQL Server 2019 Big Data Clusters
Just a very quick post with lots of pictures 🙂 SQL Server 2019 Big Data Clusters are still in private preview and I'm currently running with version CTP 2.1. However, I wanted to share some of the exciting new features we can expect from this latest big data tech in terms of tools, management and … Continue reading Interacting with SQL Server 2019 Big Data Clusters
Using Data Factory Parameterised Linked Services
Microsoft recently announced that we can now make our Azure Data Factory (ADF) v2 pipelines even more dynamic with the introduction of parameterised Linked Services. This now completes the set for our core Data Factory components meaning we can now inject parameters into every part of our Data Factory control flow orchestration processes. Pipelines ✔ Activities … Continue reading Using Data Factory Parameterised Linked Services
Creating an Azure Data Factory v2 Custom Activity
Many moons ago and in a previous job role I wrote a post for creating an Azure Data Factory v1 Custom Activity here. According to Google Analytics this proved to be one of my most popular blog posts on that site. I therefore feel I need to do an update post with the same information … Continue reading Creating an Azure Data Factory v2 Custom Activity
Future Decoded 2018 – My Quick Review
This week saw the return of Microsoft's Future Decoded event at London's ExCel. It was my third time at this two-day exhibitor backed event and it certainly won't be my last. As a Data Platform person it's always great attending conferences for all things under the Microsoft umbrella, not just data, it's a real eye … Continue reading Future Decoded 2018 – My Quick Review
What is Azure Data Factory Data Flow?
There were many great announcements to come out of the Microsoft Ignite 2018 conference, but the main one for me was the introduction of Azure Data Factory Data Flow. As a business intelligence person with many years experience working with SSIS packages this new feature is very exciting! Since I first saw the mock up … Continue reading What is Azure Data Factory Data Flow?
