I've been playing around with Azure Synapse Analytics for a while now exploring the preview features and trying to find a meaningful use case for the 'single pane of glass' capabilities. In this post I'm exploring one possible option/idea for creating a very simple self service approach to dataset ingestion and consumption. Full disclosure, the … Continue reading An Idea for Self Service Using Azure Synapse Analytics
Category: C#
Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity
Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity
Using NUnit to Automate the Testing of Azure Data Factory Pipelines
From my YouTube channel: https://www.youtube.com/c/mrpaulandrew Special guest Richard Swinbank talks about how you can use an NUnit project in Visual Studio to automate the testing of Data Factory pipelines. Richard kindly helped me apply this and test the pipelines in the ADF.procfwk to ensure post release everything still works in the open source code project. … Continue reading Using NUnit to Automate the Testing of Azure Data Factory Pipelines
Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions
A quick blog friends... I've done a few different thing now with Azure Functions and Azure Data Factory (ADF). They are definitely two of my favourite Azure Resources. In previous post I've: Executed Any Azure Data Factory Pipeline with an Azure Function Get Any Azure Data Factory Pipeline Run Status with Azure Functions Building on … Continue reading Get Any Azure Data Factory Pipeline Activity Error Details with Azure Functions
First Time Playing with Spark.Net on Azure Synapse Analytics
Exciting times friends, today (25th March) the lovely people at Microsoft granted me access to the private preview MVP workspace for Azure Synapse Analytics đ In this quick blog I wanted to share my experience so far, which I'm basically writing as I'm playing around... https://azure.microsoft.com/en-gb/services/synapse-analytics/ The main reason I wanted access to Synapse is … Continue reading First Time Playing with Spark.Net on Azure Synapse Analytics
Using the Azure Data Factory Switch Activity
Switching Between Different Azure Databricks Clusters Depending on the Environment (Dev/Test/Prod) As far as I can gather at some point last year, probably around the time of Microsoft Ignite Azure Data Factory (ADF) got another new Activity called Switch. This is excellent and exactly what ADF needed. Nested If activities can get very messy so … Continue reading Using the Azure Data Factory Switch Activity
Get Any Azure Data Factory Pipeline Run Status with Azure Functions
Microsoft.Azure.Management.DataFactory.Models.RunFilterParameters So I recently found myself with code withdrawal⌠Yes, its a real thing! Anyone in an architect type role will understand, I hope⌠You canât spend all your time drawing pictures of how things âshouldâ work without occasionally getting hands-on with some code. Anyway, my recent episode of code withdrawal led me to creating … Continue reading Get Any Azure Data Factory Pipeline Run Status with Azure Functions
Welcome to the Blog & Website of Paul Andrew 