It's now been well over a year since Microsoft announced Azure Synapse Analytics as an offering at the big U.S.A based conferences back in November 2019. While I completely share and actually like Microsoft's vision of an analytics resource... "that brings together data integration, enterprise data warehousing and big data analytics" https://azure.microsoft.com/en-gb/services/synapse-analytics/ ... the marketing, … Continue reading Is Azure Synapse Analytics Ready for Production?
Databricks vs Synapse Analytics As an architect I often get challenged by customers on different approach's to a data transformation solutions, mainly because they are concerned about locking themselves into a particular technology, resource or vendor. One example of this is using a Delta Lake to deliver an Azure based warehousing/analytics platform. Given this context, … Continue reading How Interchangeable Are Delta Tables Between Azure Databricks and Azure Synapse Analytics?
New & Updated Talks for the Community With 2020 in the rear view mirror its time to reflect on another year of experience gained as a consultant designing and delivering data platform solutions for my customers in this every changing world of technology. I've previously confessed that my job is my hobby, so its great … Continue reading Preparing My Community Talks for 2021
Azure Data Factory & Azure Synapse Analytics Integrate Pipelines In this post I want us to explore and understand the difference between an internal and external activity when using our favourite orchestration pipelines. I'll focus predominately on Azure Data Factory (ADF), but the same applies to Azure Synapse Analytics. *Warning: this is a fairly dry, … Continue reading Pipelines – Understanding Internal vs External Activities
Just last week we heard the announcement from Microsoft that Azure Synapse Analytics is now generally available (GA)... A full year on, plus a few weeks, since first seeing Synapse at the big USA conferences in November 2019. Today I've been attempting to use the resource with a view to implementing it for several customer … Continue reading Trying to Deploy Azure Synapse Analytics Using ARM Templates
As a follow up to my blog about Data Factory resource limitations here. I decided to dig deeper and expose some of these limitations with a view to understanding what happens to your pipeline when/if you hit them. In this post I'm focusing on the Activity Concurrency limits, as a reminder: Resource Default limit Maximum … Continue reading Data Factory Activity Concurrency Limits – What Happens Next?
Hi friends, just a very quick how to guide style post on something I had to build in Azure Data Factory. Scenario: I want to trigger a Data Factory pipeline, but when I do I want the pipeline to know if it's already running. If it is already running, stop the new run. Sounds simple … Continue reading Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API
Building on the work done and detailed in my previous blog post (Best Practices for Implementing Azure Data Factory) I was tasked by my delightful boss to turn this content into a simple check list of what/why that others could use.... I slightly reluctantly did so. However, I wanted to do something better than simply … Continue reading Best Practices for Implementing Azure Data Factory – Auto Checker Script v0.1
I've been playing around with Azure Synapse Analytics for a while now exploring the preview features and trying to find a meaningful use case for the 'single pane of glass' capabilities. In this post I'm exploring one possible option/idea for creating a very simple self service approach to dataset ingestion and consumption. Full disclosure, the … Continue reading An Idea for Self Service Using Azure Synapse Analytics
Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity