Spark.Net – Welcome to the Blog of Paul Andrew

An Idea for Self Service Using Azure Synapse Analytics

Posted on October 20, 2020October 20, 2020 by mrpaulandrew

I've been playing around with Azure Synapse Analytics for a while now exploring the preview features and trying to find a meaningful use case for the 'single pane of glass' capabilities. In this post I'm exploring one possible option/idea for creating a very simple self service approach to dataset ingestion and consumption. Full disclosure, the … Continue reading An Idea for Self Service Using Azure Synapse Analytics →

Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity

Posted on October 14, 2020October 14, 2020 by mrpaulandrew

Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity →

First Time Playing with Spark.Net on Azure Synapse Analytics

Posted on March 25, 2020June 3, 2020 by mrpaulandrew

Exciting times friends, today (25th March) the lovely people at Microsoft granted me access to the private preview MVP workspace for Azure Synapse Analytics 😀 In this quick blog I wanted to share my experience so far, which I'm basically writing as I'm playing around... https://azure.microsoft.com/en-gb/services/synapse-analytics/ The main reason I wanted access to Synapse is … Continue reading First Time Playing with Spark.Net on Azure Synapse Analytics →

mrpaulandrew

Paul (AKA @mrpaulandrew) is the Founder & CTO of Cloud Formations, a specialist data consultancy based in the UK. With nearly 20 years’ experience designing and delivering Microsoft data architectures, Paul leads a passionate team of engineers, supporting businesses small and large with scalable cloud platforms. Business value delivered through data insights. Over the years, Paul has covered the breadth and depth of design patterns and industry leading concepts, including Lambda, Kappa, Delta Lake, Data Mesh and Data Fabric. Paul is also a Microsoft Data Platform MVP, director for the Data Relay community conference, East Midlands user group leader, book author and mentor. In addition to the day job(s), Paul is a father of three, husband, foodie, runner, blood donor, geek, Lego, and Star Wars fan! Lastly, Paul confesses to enjoying a Ramstein playlist when given half a chance to do some coding for a customer project.

View Full Profile →