Renewing my post from last year (here) I’ve gathered my thoughts and put together the following content that I’d like to deliver to our wonderful data platform community in 2020.
2019 has been a great year for me having delivered 30 talks, in 17 cities and 8 different countries. This brings my grand total to just short of 100 talks delivered to the community so far. You can view the full dashboard in my GitHub repository if you want to have a poke around 🙂
Anyway, for 2020, here are my proposed session titles with complete abstracts against each below:
An Introduction to Azure Synapse – What is it? Why use it? And how?
An Architects Guide to Delivering Data Insights on the Microsoft Azure Data Platform
Beyond IoT – How to Handle Streaming Data in Azure
Azure Orchestration – Applying Data Factory in Production
Creating a Metadata Driven Processing Framework for Azure Data Factory
ETL in Azure Made Easy with Data Factory Data Flows (Mapping & Wrangling)
A Complete Introduction to Azure Data Factory
Using Azure DevOps for Azure Data Factory
Azure Data Factory Data Flows
Azure Data Factory – Zero to Hero
If you’d like to see any of this content at your event, conference, meetup, etc please don’t hesitate to get in touch. Or even if you’d like a session on something not listed, I’m open to new ideas.
Regular Sessions
An Introduction to Azure Synapse – What is it? Why use it? And how?
Level: 100
The Microsoft abstraction machine is at it again with this latest veneer over what we had come to understand as the ‘modern data warehouse’. Or is it?! When creating an Azure PaaS data platform/analytics solution we would typically use a set of core Azure services; Data Factory, Data Lake, Databricks and SQL Data Warehouse. Now with the latest round of enhancements from the MPP team it seems in the third generation of the Azure SQLDW offering we can access all our core services as a bundle. Ok, so what? Well, this is a reasonable starting point in our understanding of Azure Synapse, but is also far from the whole story. In this session we’ll go deeper into the evolution of our SQLDW to complete our knowledge on why Synapse Analytics is a game changer for various data warehouse architectures. We’ll discover what Synapse has to offer with its Data Virtualisation layer, flexible storage and multi model compute engines. A simple veneer of things, this new resource is not. In this introduction to Synapse we’ll cover the what, that why and importantly the how for this emerging bundle of exciting services.
An Architects Guide to Delivering Data Insights on the Microsoft Azure Data Platform
Level: 200
The resources on offer in Azure are constantly changing, which means as data professionals we need to constantly change too. Updating knowledge and learning new skills. No longer can we rely on products matured over a decade to deliver all our solution requirements. Today, data platform architectures designed in Azure with best intentions and known good practices can go out of date within months. That said, is there now a set of core components we can utilise in the Microsoft cloud to ingest and deliver insights from our data? When does ETL become ELT? When is IaaS better than PaaS? Do we need to consider scaling up or scaling out? And should we start making cost the primary factor for choosing certain technologies? In this session we’ll explore the answers to all these questions and more from an architects viewpoint. Based on real world experience lets think about just how far the breadth of our knowledge now needs to reach when starting from nothing and building a complete Microsoft Azure Data Platform solution.
Beyond IoT – How to Handle Streaming Data in Azure
Level: 200
The desire and expectation to use real-time data is constantly growing, businesses need to react to market trends instantly. In the new data driven age a daily ETL load/processing window is not enough. We need a constant stream of information and analytics achieved in real-time. In this session will look at how that can be achieved using Azure Stream Analytics. Building streaming jobs that can blend and aggregate data as it arrives to drive live Power BI dashboards. Plus, we will consider options for using Stream Analytics in production and what other tools could be used in the ever-growing list of Azure services. Lastly, we will explore the theory behind a complete Lambda architecture where we combine streaming and batch data together and how this different from the Databricks Delta architecture.
Azure Orchestration – Applying Data Factory in Production
Level: 400
If you have already mastered the basics of Azure Data Factory (ADF) and you are now looking to advance your knowledge of the tool, this is the session for you. Yes, Data Factory can handle the orchestration of our ETL pipelines. But what about our wider Azure environment? In this session we will take a deeper dive into the service, considering how to build custom activities, create metadata driven dynamic pipelines and think about hierarchical design patterns. Plus, explore ways for optimising our Azure compute costs by controlling other resource scaling as part of our normal data processing pipelines. How? Well, once we can hit a REST API from an ADF web activity anything is possible, extending our Data Factory and orchestrating everything in any data platform solution. All this and more in a series of short lessons (based on real world experience) I will take you through how to use Azure Data Factory in production. Finally, we will look at how Data Factory can be deployed using Azure DevOps.
Creating a Metadata Driven Processing Framework for Azure Data Factory
Level: 300
Azure Data Factory is the undisputed PaaS resource within the Microsoft Cloud Platform for orchestrating our data workloads. With a growing set of 100+ Linked Service connections, combined with an array of control flow and data flow Activities there isn’t much Data Factory cannot do in terms of solution management. That said, the service may still require the support of other Azure resources for the purposes of logging, compute and storage. In this session we will focus on exactly that point by extending our Data Factory with an Azure SQL Database and Azure Functions. The result is the ability to create a dynamic, flexible, metadata driven processing framework that complements our existing solution orchestrator. Furthermore, we will explore how to bootstrap multiple Data Factory’s, design for cost with nearly free Consumption Plans and deliver an operational abstraction over our Data Factory worker pipelines without hiding any error details in layers of dynamic JSON.
ETL in Azure Made Easy with Data Factory Data Flows (Mapping & Wrangling)
Level: 200
What happens when you combine a cloud orchestration service with a Spark cluster?! The answer is a feature rich, graphical, scalable data flow environment to rival any ETL tech we’ve previously had available in Azure. In this session we will look at Azure Data Factory and how it integrates with Azure Databricks to produce a powerful abstraction over the Apache Spark analytics ecosystem in the form of Mapping and Wrangling Data Flows. If you have ever transformed datasets using SQL Server Integration Services (SSIS) packages or via Power BI’s Power Query tool this is the session for you. Now we can transform data in Azure using our favourite interfaces but with the support of Azure Databricks doing the heavy lifting. In this session you will get a quick introduction to Azure Data Factory before we go deeper into the services new Mapping and Wrangling Data Flows features. Start using cloud native technology and scale out compute within a convenient, easy to use Data Factory rich graphical interface.
A Complete Introduction to Azure Data Factory
Level: 100
SQL Server Integration Services has been a good friend since its first appearance in SQL Server 2005. But now, after a slightly bumpy start, Azure Data Factory is here and ready to replace all our DTSX package capabilities. This cloud native orchestration tool is a powerful equivalent for SSIS and the SQL Agent as a primary component within the Modern Data Warehouse. In this complete introduction session, we will cover the basics of Azure Data Factory. What do we need to build cloud ETL/ELT pipelines? What’s the integration runtime? Do we have an SSIS equivalent cloud data flow engine? Can we easily lift and shift existing SSIS packages into the cloud? The answers to all these questions and more in this session. Come to this session knowing nothing about Azure Data Factory and leave with enough knowledge to start building pipelines tomorrow.
Lightning Talks
Using Azure DevOps for Azure Data Factory
Level: 200
DevOps as a process is here to stay and is typically a must have requirement for any data platform solution. But how does the concept translate to the technology when implemented? Sadly, the answer isn’t always straight forward. In this short session I’ll introduce how we can continuously integrate and deliver our cloud orchestration resource Azure Data Factory (ADF). We’ll discuss three options for getting our service JSON deployed to production using the popular Azure DevOps environment, previously known as VSTS, and think about the suitability of the Microsoft provided ARM templates for our highly dynamic orchestrator. Does the ADF portal UI really support the DevOps methodology? Can we confidently publish our pipelines? How should we handle the branching of our source code for ADF developers? The answers to all these questions and more in this lightning session. All based on real world experience with the products.
Azure Data Factory – Data Flows
Level: 100
Our favourite scale out cloud orchestration tool has now got a game changing upgrade in the form of an SSIS style data flow engine. In this short talk we’ll take a quick peek at this feature and the powerful transformation service it uses behind the scenes, Azure Databricks. If you’re an SSIS developer looking to move to Azure this session should not be missed. Now in Azure Data Factory we have a Control Flow plane and a choice of two Data Flow tools to deliver all our ETL/ELT needs.
Full Day of Training
Azure Data Factory – Zero to Hero
If we want to achieve any data processing in Azure we need an umbrella service to manage, monitor and schedule our solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS packages. It’s now time to upgrade our skills and start using cloud native services to achieve the same thing on the Microsoft Cloud Platform. Within a PaaS only Modern Data Warehouse, the primary component for delivering that orchestration is Azure Data Factory, combined with Azure Databricks.
In this full day of training we’ll start with the basics and then get hands on with the tools. We’ll build our own Azure ETL/ELT pipelines using all Data Factory has to offer. Plus, consider hybrid architectures, think about lifting and shifting legacy packages, and explore complex bootstrapping to orchestrate everything in Azure with Data Factory.
Specifically, we’ll be covering the following topics:
- An introduction to Azure Data Factory. What is it and why use it?
- How to extend our orchestration processes with Custom Activities, Azure Functions and Web Hooks.
- Using SSIS packages in Azure.
- Data Factory Data Flows (Mapping & Wrangling) with support from Azure Databricks
- Dynamic metadata driven pipelines.
- Data Factory alerting and monitoring.
- Data Factory DevOps.
- Lessons learnt from using Azure Data Factory in production.
Azure Data Factory has been around since 2015 and has matured massively to form a complete feature rich cloud offering. Add this tech to your development toolbox today! Join me for this complete lesson in everything you need to deliver Azure Data Factory within your data platform solution.
Have your laptops to hand and come armed with your Azure subscription, including credit and rights to deploy resources.
Please check with bill payer.
Infrastructure not included.
Welcome to the Blog & Website of Paul Andrew 
One thought on “My Community Talks 2020”