Just a quick post friends following a talk I did on Azure Data Factory (ADF) v2 at SQL Saturday Manchester. In the talk I demonstrated using ADF more dynamically now version 2 supports expressions and parameters... Amongst other cool stuff I specifically showed how you could run your SSIS packages, scaled out across all the … Continue reading Execute All SSIS Packages in Parallel with ADFv2 Lookup and ForEach Activities
Author: mrpaulandrew
Principal consultant and architect specialising in big data solutions on the Microsoft Azure cloud platform.
Data engineering competencies include Azure Data Factory, Data Lake, Databricks, Stream Analytics, Event Hub, IoT Hub, Functions, Automation, Logic Apps and of course the complete SQL Server business intelligence stack.
Many years’ experience working within healthcare, retail and gaming verticals delivering analytics using industry leading methods and technical design patterns.
STEM ambassador and very active member of the data platform community delivering training and technical sessions at conferences both nationally and internationally.
Father, husband, swimmer, cyclist, runner, blood donor, geek, Lego and Star Wars fan!
How To Control the SSIS IR From Your ADFv2 Pipelines – Start/Stop
Hello friends, in this post I want to share with you a couple of methods I've developed for controlling the start up and stop of the Azure Data Factory (ADF) v2 SSIS Integration Runtime (IR) from your pipelines. This requirement came about mainly because of cost... The SSIS IR underneath is simply a cluster of … Continue reading How To Control the SSIS IR From Your ADFv2 Pipelines – Start/Stop
Using the ADFv2 Until Activity
Hello friends, with this post I wanted to give you a quick 'how to' guide when working with the Azure Data Factory (ADF) v2 Until activity... Having implemented the activity several times in production I've repeatedly found the current Microsoft documentation here falling a little short in its explanations and examples. So, I've created my … Continue reading Using the ADFv2 Until Activity
T-SQL to U-SQL Data Type Conversion
This is just a quick post, mainly for my own reference. When working with code generated solutions we often need to convert datasets from SQL Server (T-SQL) data types to Azure Data Lake Analytics (U-SQL) data types. As you probably know U-SQL has a hybrid syntax of T-SQL and C# which uses .Net data types. … Continue reading T-SQL to U-SQL Data Type Conversion
ADFv2 Variable String Interpolation – REST API Body
As we know Azure Data Factory (ADF) Version 2 now has the ability to use expressions, parameters and system variables in various components throughout the service. This is brilliant and the best thing about v2 allowing us to build some very dynamic parameter driven pipelines. However, the syntax isn't always that obvious when you use … Continue reading ADFv2 Variable String Interpolation – REST API Body
Azure Data Lake Store – Files Dashboard
A tool I've used in the past to query my local disk drives is WinDirStat. It's great for finding out where large files are hiding in your directories using a heat map visual. In this post I talk you through how I've created a similar tool for Azure Data Lake Store using PowerShell, SQL Server … Continue reading Azure Data Lake Store – Files Dashboard
Inject into Azure Logic Apps ARM Templates with PowerShell
Continuous delivery of data services has always been a battle for us data analytics peeps. Things are getting better when we think about business intelligence and dev ops, however there are still gaps. This blog post is an account of one such issue I recently faced and how I worked around the problem. Problem While … Continue reading Inject into Azure Logic Apps ARM Templates with PowerShell
