A Day Full of Azure Data Factory

Posted on by mrpaulandrew

Join me for a full day of training on Azure Data Factory.

If we want to achieve any data processing in Azure you need an umbrella service to manage, monitor and schedule your solution. For a long time when working on premises, the SQL Agent has been our go-to tool, combined with T-SQL and SSIS packages. It’s now time to upgrade our skills and start using cloud native services to achieve the same thing on the Microsoft Cloud Platform. Within a PaaS only Modern Data Platform, the primary component for delivering that orchestration is Azure Data Factory, combined with various other compute resources.

In this full day of training we’ll start with the basics and learn how to orchestrate your Azure Data Platform end to end. You will learn how to build our Azure ETL/ELT pipelines using all Data Factory has to offer. Plus, consider hybrid architectures, dynamic design patterns, think about lifting and shifting legacy packages, and explore complex bootstrapping to orchestrate everything within your solution.

Complete Session Agenda

  • Module 1: Data Factory Fundamentals
    • What is it and why use it?
    • Resource Components
    • Common Activities
    • Execution Dependencies
  • Module 2: Uploading Data to Azure
    • Integration Runtimes
    • Hosted IR Patterns
    • Private Endpoints
  • Module 3: Using SSIS Packages in Azure
    • SSIS Integration Runtime
    • Packages Running on PaaS
    • Scaling Out Package Execution
  • Module 4: Data Flows
    • Mapping Data Flows
    • Wrangling Data Flows
    • Configuration
    • Use Cases
  • Module 5: Metadata Driven Pipelines
    • Expressions
    • Dynamic Pipelines
    • Orchestration Framework – ADFprocfwk.com
  • Module 6: Monitoring Alerting Security
    • Logging – Kusto Queries
    • Alerting
    • Roll Based Access
    • Using Azure Key Vault
  • Module 7: Pricing & Limitations
    • Data Integration Units
    • Data Flow Compute
    • Wider Platform Orchestration & Cost Control
    • Resource Limitations
  • Module 8: CI/CD with Azure DevOps
    • Source Control vs Developer UI
    • ARM Template Deployments
    • PowerShell Deployments
  • Module 9: Data Factory in Production
    • Testing
    • Bootstrapping
    • Best Practices
  • Module 10: Wrap Up
    • Conclusions
    • Questions
    • Homework

If that’s not enough content for one day, you will also get access to a set of hands-on labs that you can work through at your own pace. Whether you are new to Azure Data Factory or have some experience, you will leave this workshop with new skills and ideas for your projects.

20 thoughts on “A Day Full of Azure Data Factory”

    1. Sorry, probably not. There is already a 1 hour intro session on ADF available via my YouTube channel though. Maybe start with that if you think it’s a resource students would use.

      Like

      Reply

    1. Hi, yes, I’m delivering a session next week for the Belgium based Data Minds conference, although virtual this year. Cheers

      Like

      Reply

  6. Hi Paul,
    I am working on a POC in which i need to load the feed files(.txt) from client onedrive(sharepoint) to azure blob storage. i am using data factory in which using sharepointonlinelist linked service to connect to the sharepoint URL to load the files. while configuring the linked service i am getting error as
    The access token generated failed, status code: BadRequest, error message: {“error”:”unauthorized_client”,”error_description”:”AADSTS700016: Application with identifier ‘aad-feedfiletest’ was not found in the directory @[tenantid]

    Like

    Reply

  7. Hi
    I am rajesh kumar, i have recently got a job after losing last job almost for 1 year. I have to design data factory job with respect to pattern matching.
    I wanted to go through the course A DAY FULL OF AZURE DATA FACTORY, How to get access, please reply

    Like

    Reply

Leave a reply to mrpaulandrew Cancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.