Preparing My Community Sessions for 2019

Hey friends & SQL family, its been a great year of tech and events so far, but now with several big conferences planning next years agendas its time to think ahead. Below I’ve prepared titles and abstracts for talks I’d like to deliver to our awesome data platform community in 2019.

5x regular sessions.
2x full pre-con training days.
2x lightning talks.
1x Azure icon game 🙂

My focus in 2019 as you’ll see in the session content will be data engineering in Azure and delivering modern data warehouse solutions using Azure services.

Any thoughts and comments you have on the below sessions would be very welcome. Also, please reach out if you’d like me to present any of this content at your event.

Thanks 🙂

Regular Sessions:

An Introduction to SQL Server 2019 Big Data Clusters (Level 100)

The world of data is moving quickly and traditional relational database technology can be a limiting factor in responding to change. Teams want to move quicker, work with a wider array of data, handle massive datasets and augment their code with open-source libraries and projects. Data delivery and demand for immediate insights mean we no longer have the luxury to extract, transform and load datasets before we need to realise the value locked within our data. SQL Server 2019 has made radical architecture changes to meet these challenges, introducing in-built data lakes, spark clusters, massive data ingestion engines and the ability to harness massively parallel processing architectures. These engines are all implemented behind a single, scalable interface that streamlines data acquisition, transparently and without costly movement operations.

In this talk we will outline the problems that can be tackled with the new SQL Server Big Data Clusters, provide an overview of how they have been implemented and discuss how SQL Server can now handle your Big Data problems. We will be drawing parallels to the Azure Data Platform and highlight where we can adopt similar patterns in our on-premises data platforms.

Data Factory – An Introduction to Azure Control Flows & Data Flows (Level 100)

SQL Server Integration Services has been a good friend since its first appearance in SQL Server 2005. But now, after a slightly bumpy start, Azure Data Factory is here and ready to replace all our DTSX package capabilities. This cloud native orchestration tool is a powerful equivalent for SSIS and the SQL Agent as a primary component within the Modern Data Warehouse. In this session we will start with the basics of Azure Data Factory. What do we need to build cloud ETL pipelines? What’s the integration runtime? Do we have an SSIS equivalent cloud data flow engine? Can we easily lift and shift existing SSIS packages into the cloud? The answers to all these questions and more in this session.

ETL in Azure made easy with Data Factory Data Flow (Level 200)

What happens when you combine a cloud orchestration service with a Spark cluster?! The answer is a feature rich, graphical, scalable data flow environment to rival any ETL tech we’ve previously had available in Azure. In this session we’ll look at Azure Data Factory and how it integrates with Azure Databricks to produce a powerful abstraction over the Apache Spark analytics ecosystem. Now we can transform data in Azure using Databricks, but without the need to write a single line of Scala or Python! If you haven’t used either service yet, don’t worry, you’ll get a quick introduction to both before we go deeper into the Data Factory Data Flow feature.

Complex Azure Orchestration with Dynamic Data Factory Pipelines (Level 400)

If you have already mastered the basics of Azure Data Factory (ADF) and are now looking to advance your knowledge of the tool this is the session for you. Yes, Data Factory can handle the orchestration of our ETL pipelines in Azure. But what about our wider Azure environment? In this session we’ll go beyond the basics looking at how we build custom activities, metadata driven dynamic design patterns for Data Factory. Plus, considerations for optimising compute costs by controlling other service scaling as part of normal data processing. Once we can hit a REST API with an ADF web activity anything is possible, extending our Data Factory and orchestrating everything. Furthermore, we’ll explore how we can truly benefit from the scale out capabilities of the cloud when we fully parallelise our pipeline activities.

Beyond IoT Real-time Data Ingestion with Azure Stream Analytics (Level 300)

The desire and expectation to use real-time data is constantly growing, businesses need to react to market trends instantly. In this new data driven age a daily ETL load/processing window isn’t enough. We need a constant stream of information and analytics achieved in real-time. In this session will look at how that can be achieved using Azure Stream Analytics. Building streaming jobs that can blend and aggregate data as it arrives to drive live Power BI dashboards. Plus, we’ll explore how a complete lambda architecture can be created when combining stream and batch data together.

Pre-conference Full Training Days:

Azure Data Factory – Zero to Hero

If we want to achieve any data processing in the cloud we need a service to bootstrap, manage, schedule and handle our solution. For a long time when on premises, the SQL Agent has been our go to tool, combined with T-SQL and SSIS packages. It’s now time to upgrade and start using cloud native services to achieve the same thing in Azure. Within a Modern Data Warehouse, the primary component for delivering this orchestration is Azure Data Factory, combined with Azure Data Bricks.

In this full day of training we’ll start with the basics and then get hands on with the tools. We’ll build our own cloud ETL pipelines using all Data Factory has to offer. Plus, we’ll take a look at hybrid architectures and where cloud scale out capabilities can support the lift and shift of our beloved SSIS packages into Azure.

A breakdown of the day:

An introduction to Azure Data Factory.
How to extend our orchestration with Custom Activities and Azure Functions.
Using SSIS packages in Azure.
Data Factory Data Flow with Azure Data Bricks
Dynamic metadata driven pipelines.
Factory alerting and monitoring.
Data Factory DevOps.

Have your laptops to hand and come armed with your Azure subscription, including credit and rights to deploy resources.

As part of the training all content and labs will be made available to take away.

Building an Azure Business Intelligence Solution End to End

There are so many ways to ingest, transform, aggregate and visualise our data in Azure. Architecting an end to end Modern Data Warehouse now requires a huge breadth of knowledge on a vast amount of data services, or does it? Where do you start? What tool is right for your solution?

In this hands on full day of training we’ll answer those questions. But we won’t stop there. We’ll build it! Using a simplified generic architecture as a guide.

Using the Microsoft Cloud platform, we’ll learn about the data focused subset of these Azure services and how they can be harnessed to build a working, modern, cloud based data warehouse solution.

A breakdown of the day:

Ingestion of raw data into storage or using real-time data sources with event hubs.
Preparation and data cleansing with batch service jobs and custom operations.
Streamed data flowing through aggregated query windows for real-time analytics.
Transformation using Data Bricks or Data Lake Analytics.
The role of the SQL Data Warehouse.
Handling semantic layer models with SSAS.
Rich visualisations and reporting anywhere with Power BI.
Orchestration of the entire platform with server less compute and management from Data Factory.

Have your laptops to hand and come armed with an Azure subscription, including credit and rights to deploy resources.

As part of the training all content and labs will be made available to take away.

Lightning Talks:

Azure Data Factory just got an SSIS Style Data Flow Engine

Our scale out cloud orchestration tool has just been given a game changing upgrade in the form of an SSIS style data flow engine. In this short talk we’ll take a sneak peek at this up and coming feature and the powerful transformation service it uses behind the scenes.

Cognitive Services Embedded in Azure Data Lake & U-SQL

Microsoft’s Cognitive Services are basically the best thing since sliced bread, especially for anybody working with data. Artificial intelligence just got packaged and made available for the masses to download. In this short talk, I’ll take you on a whirl wind tour of how to use these massively powerful libraries directly in Azure Data Lake with that offspring of T-SQL and C# … U-SQL. How do you get hold of the DLL’s and how can you wire them up for yourself?