Azure Data Factory Online Course
This course teaches you to design and implement a Metadata-Driven Ingestion Framework using Azure Data Factory—a reusable, plug-and-play system that saves time and costs by enabling adoption across multiple business units without extra development. You’ll start by building your first Azure Data Factory pipeline, then progress to creating a dynamic, fully functional ingestion framework with auditing capabilities for both batch orchestrations and individual runs. By the end, you’ll be equipped to deliver production-ready data ingestion solutions in Azure.
Who should take this Course?
The Azure Data Factory Online Course is ideal for data engineers, ETL developers, and cloud professionals who want to learn how to build, manage, and orchestrate data pipelines in Microsoft Azure. It’s also suitable for those seeking to integrate, transform, and load data from various sources for analytics and business intelligence purposes. Basic knowledge of data concepts, SQL, and Azure services is recommended for effective learning.
What you will learn
- Learn about Azure Data Factory and Azure Blob Storage
- Understand data engineering, data lake, and metadata-driven frameworks concepts
- Look at the industry-based example of how to build ingestion frameworks
- Learn dynamic Azure Data Factory pipelines and email notifications with logic apps
- Study tracking of pipelines and batch runs
- Look at version management with Azure DevOps
Course Outline
Introduction – Build Your First Azure Data Pipeline
- Introduction to the Course
- Introduction to ADF (Azure Data Factory)
- Requirements Discussion and Technical Architecture
- Register a Free Azure Account
- Create a Data Factory Resource
- Create a Storage Account and Upload Data
- Create Data Lake Gen 2 Storage Account
- Download Storage Explorer
- Create Your First Azure Pipeline
- Closing Remarks
Metadata-Driven Ingestion
- Introduction to Metadata-Driven Ingestion
- High-Level Plan
- Create Active Directory User
- Assign the Contributor Role to the User
- Disable Security Defaults
- Creating the Metadata Database
- Install Azure Data Studio
- Create Metadata Tables and Stored Procedures
- Reconfigure Existing Data Factory Artifacts
- Set Up Logic App to Handle Email Notifications
- Modify the Data Factory Pipeline to Send an Email Notification
- Create Linked Service for Metadata Database and Email Dataset
- Create Utility Pipeline to Send Email Notifications
- Explaining the Email Recipients Table
- Explaining the Get Email Addresses Stored Procedure
- Modify Ingestion Pipeline to Use the Email Utility Pipeline
- Tracking the Triggered Pipeline
- Making the Email Notifications Dynamic
- Making Logging of Pipeline Information Dynamic
- Add a New Way to Log the Main Ingestion Pipeline
- Change the Logging of Pipelines to Send Fail Message Only
- Creating Dynamic Datasets
- Reading from Source to Target - Part 1
- Reading from Source to Target - Part 2
- Explaining the Source to Target Stored Procedure
- Add Orchestration Pipeline - Part 1
- Add Orchestration Pipeline - Part 2
- Fixing the Duplicating Batch Ingestions
- Understanding the Pipeline Log and Related Tables
- Understanding the GetBatch Stored Procedure
- Understanding the Set Batch Status and GetRunID
- Setting Up an Azure DevOps Git Repository
- Publishing the Data Factory to Azure DevOps
- Closing Remarks
Event-Driven Ingestion
- Introduction
- Read from Azure Storage Plan
- Create Finance Container and Upload Files
- Create Source Dataset
- Write to Data Lake - Raw Plan
- Create Finance Container and Directories
- Create Sink Dataset
- Data Factory Pipeline Plan
- Create Data Factory and Read Metadata
- Add Filter by CSV
- Add Dataset to Read Files
- Add the For Each CSV File Activity and Test Ingestion
- Adding the Event-Based Trigger Plan
- Enable the Event Grid Provider
- Delete File and Add Event-Based Trigger
- Create Event-Based Trigger
- Publish Code to Main Branch and Start Trigger
- Trigger Event-Based Ingestion
- Closing Remarks