An Idea for Self Service Using Azure Synapse Analytics

I've been playing around with Azure Synapse Analytics for a while now exploring the preview features and trying to find a meaningful use case for the 'single pane of glass' capabilities. In this post I'm exploring one possible option/idea for creating a very simple self service approach to dataset ingestion and consumption. Full disclosure, the … Continue reading An Idea for Self Service Using Azure Synapse Analytics

Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity

Here's a quick bit of information I thought was worth sharing... For file types that don't contain there own metadata (CSV, Text etc) we typically have to go and figure out there structure including; attributes and data types before doing any actual transformation work. Often I've used the Data Factory Metadata Activity to do this … Continue reading Spark Data Frame Infer Schema vs Data Factory Get Metadata Activity

ADF.procfwk and Azure Synapse Orchestrate – Preview Limitations

With Azure Synapse Analytics now in public preview is was time to find out how compatible my Azure Data Factory metadata driven processing framework (ADF.procfwk) is with the Synapse Orchestrate features. Firstly, as Synapse doesn't yet have any source control or DevOps support I had to manually rebuild the framework pipelines in the browser, copying … Continue reading ADF.procfwk and Azure Synapse Orchestrate – Preview Limitations

First Time Playing with Spark.Net on Azure Synapse Analytics

Exciting times friends, today (25th March) the lovely people at Microsoft granted me access to the private preview MVP workspace for Azure Synapse Analytics πŸ˜€ In this quick blog I wanted to share my experience so far, which I'm basically writing as I'm playing around... https://azure.microsoft.com/en-gb/services/synapse-analytics/ The main reason I wanted access to Synapse is … Continue reading First Time Playing with Spark.Net on Azure Synapse Analytics