While documenting a customers data platform solution I decided it would be far easier if we could summarise the contents of a fairly complex Data Factory using its ARM Template. So, this is what I've done using T-SQL to parse the ARM Template JSON and output of series of tables containing details about the factory … Continue reading Summarise An Azure Data Factory ARM Template Using T-SQL
Best Practices for Implementing Azure Data Factory
My colleagues and friends from the community keep asking me the same thing... What are the best practices from using Azure Data Factory? With any emerging, rapidly changing technology I'm always hesitant about the answer. However, after 4 years of working with ADF I think its time to start suggesting what I'd expect to see … Continue reading Best Practices for Implementing Azure Data Factory
My Community Talks 2020
Renewing my post from last year (here) I've gathered my thoughts and put together the following content that I'd like to deliver to our wonderful data platform community in 2020. 2019 has been a great year for me having delivered 30 talks, in 17 cities and 8 different countries. This brings my grand total to … Continue reading My Community Talks 2020
Deploying Azure Data Factory in a Single Service Environment (Option 1)
For my first appearance on the Altius Data Lounge show I talk about and demonstrate the 'out-of-the-box' capabilities within Azure Data Factory for Git integration and how to publish developed pipelines. In my demo you'll see what each of the developer UI toolbar functions means in the context of the code repository and what happens … Continue reading Deploying Azure Data Factory in a Single Service Environment (Option 1)
Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments
Just a short post following a recent question I got from my delivery team... Are there any best practices for structuring our Databricks Notebooks in terms of code comments and markdown? Having done a little Googling I simply decided to whip up a quick example that could be adopted as a technical standard for the … Continue reading Structuring Your Databricks Notebooks with Markdown, Titles, Widgets and Comments
Get Any Azure Data Factory Pipeline Run Status with Azure Functions
Microsoft.Azure.Management.DataFactory.Models.RunFilterParameters So I recently found myself with code withdrawal… Yes, its a real thing! Anyone in an architect type role will understand, I hope… You can’t spend all your time drawing pictures of how things “should” work without occasionally getting hands-on with some code. Anyway, my recent episode of code withdrawal led me to creating … Continue reading Get Any Azure Data Factory Pipeline Run Status with Azure Functions
Microsoft Ignite 2019 Announcements – The Ignite Book of News 2019
How nice is this!... All the Microsoft Ignite announcements in one place. Very cool. Data Platform and AI things in chapter 3, starting on page 39. Sadly not that many new things for us data people this year, but still, its all great news. Azure Synapse Analytics seems to be the main data platform news, … Continue reading Microsoft Ignite 2019 Announcements – The Ignite Book of News 2019
