The other day Kamil (@NowinskiK) messaged me on Teams asking if I liked Mermaid's 🙂 I prepared myself for a conversation about which was better; the fish half being on top vs on the bottom. Ha! Of course not!! Kamil was actually referring to the... Javascript based diagramming and charting tool that renders Markdown-inspired text … Continue reading Using Mermaid to Create a ProcFwk Pipeline Lineage Diagram
On 2nd June 2020 Apress reached out to me about the role of technical reviewer on a book about Azure Data Factory. The exact conversation may amuse you, so I've included it exactly as it happened 🙂 THE BEGINNING Paul,I know writing is out of the question, but do you have bandwidth to tech review … Continue reading Azure Data Factory by Example – A Review of My Technical Review
Azure Data Analytics End to End Let's start with a story, not a 'once upon a time story', a story for your backlog 🙂 As a solution architect I need to design and build an Azure data analytics platform end to end to deliver data insights for my customer. In February 2021 I delivered a … Continue reading An Architects Guide to Delivering Data Insights Using the Microsoft Azure Data Platform
Azure Data Factory & Azure Synapse Analytics Integrate Pipelines In this post I want us to explore and understand the difference between an internal and external activity when using our favourite orchestration pipelines. I'll focus predominately on Azure Data Factory (ADF), but the same applies to Azure Synapse Analytics. *Warning: this is a fairly dry, … Continue reading Pipelines – Understanding Internal vs External Activities
As a follow up to my blog about Data Factory resource limitations here. I decided to dig deeper and expose some of these limitations with a view to understanding what happens to your pipeline when/if you hit them. In this post I'm focusing on the Activity Concurrency limits, as a reminder: Resource Default limit Maximum … Continue reading Data Factory Activity Concurrency Limits – What Happens Next?
Hi friends, just a very quick how to guide style post on something I had to build in Azure Data Factory. Scenario: I want to trigger a Data Factory pipeline, but when I do I want the pipeline to know if it's already running. If it is already running, stop the new run. Sounds simple … Continue reading Get Data Factory to Check Itself for a Running Pipeline via the Azure Management API
Building on the work done and detailed in my previous blog post (Best Practices for Implementing Azure Data Factory) I was tasked by my delightful boss to turn this content into a simple check list of what/why that others could use.... I slightly reluctantly did so. However, I wanted to do something better than simply … Continue reading Best Practices for Implementing Azure Data Factory – Auto Checker Script v0.1
Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity
As a result of the pandemic like many other community speakers I've taken to YouTube as the current method for sharing knowledge and content. Recording video's isn't as much fun as speaking at a physical event, but it partially ticks the box. Plus, my daughter thinks its "cool" that I have a YouTube channel, so … Continue reading Capturing YouTube Video Views with Azure Data Factory
Join me for a full day of training on Azure Data Factory. If we want to achieve any data processing in Azure you need an umbrella service to manage, monitor and schedule your solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS … Continue reading A Day Full of Azure Data Factory
Welcome to the Blog of Paul Andrew 