The Complete Course Of Apache Airflow 2024

The Complete Course Of Apache Airflow 2024
Published 7/2024
MP4 | Video: h264, 1920×1080 | Audio: AAC, 44.1 KHz

Learn Apache Airflow in a Professional way. Become an expert in Data Pipelines and Workflows, from ZERO to HERO!

What you’ll learn

At the end of the course you will fully master Apache Airflow, to be able to programmatically author, schedule, and monitor complex workflows from scratch

You will be able to conduct Apache Airflow projects step by step, understanding all the logic and ending with advanced practical examples and complete projects

You will establish a foundational understanding of Apache Airflow and its primary components

You will develop skills to write, organize, and manage DAGs effectively using best practices

You will understand the different types of operators and executors to manage and execute tasks efficiently

You will master techniques for monitoring and logging to manage DAG runs and task statuses effectively

You will learn to scale Apache Airflow and configure high availability setups for production environments

You will explore advanced features of Apache Airflow and learn to integrate it with external systems

You will develop skills in testing, debugging, and deploying DAGs and tasks using best practices

You will design and implement a complex, real-world workflow using Apache Airflow, integrating all learned concepts

You will be able to practice the content learned in a practical way by following all the steps in the complete exercises and the hands-on projects

Requirements

Preparing and installing the needed environment to follow the practical sessions (if you don’t know how, don’t worry, it’s very easy, and I’ll explain it to you in the course!)

A decent computer and of course, desire to learn!

Description

Overview

Section 1: Introduction to Apache Airflow

Lecture 1 Welcome to the course

Lecture 2 Introduction to Apache Airflow

Lecture 3 Understanding DAGs (Directed Acyclic Graphs) and operators

Section 2: Apache Airflow Architecture

Lecture 4 Components of Apache Airflow: Scheduler, Executor, Metadata Database, Web Server

Lecture 5 Understanding the role of each component in workflow orchestration

Section 3: Installation and Setup

Lecture 6 Installing Apache Airflow using different methods (e.g., pip, Docker)-1

Lecture 7 Installing Apache Airflow using different methods (e.g., pip, Docker)-2

Lecture 8 Exploring Airflow’s web interface

Section 4: Writing and Managing DAGs

Lecture 9 Understanding DAGs in detail

Lecture 10 How to Write DAGs

Lecture 11 Best practices for organizing and managing DAG code

Lecture 12 How to Write DAGs Assignment

Section 5: Operators and Executors

Lecture 13 Understanding Executors

Lecture 14 Understanding different types of operators (BashOperator, PythonOperator, etc.)

Lecture 15 Operators and Executors Assignment

Section 6: Monitoring and Logging

Lecture 16 Monitoring DAG runs and task statuses

Lecture 17 Monitoring and Logging Assignment

Section 7: Scaling and High Availability

Lecture 18 Scaling Airflow horizontally and vertically

Lecture 19 Configuring High Availability (HA) setups for production deployments-1

Lecture 20 Configuring High Availability (HA) setups for production deployments-2

Section 8: Advanced Features and Integrations

Lecture 21 Working with sensors for external triggers and dependencies

Lecture 22 Integrating Airflow with external systems

Section 9: Testing and Debugging

Lecture 23 Writing unit tests for DAGs and tasks

Lecture 24 Deployment and Best Practices

Lecture 25 Practical Assignment

Section 10: Designing and implementing a real-world workflow using Apache Airflow

Lecture 26 Final Project-1

Lecture 27 Final Project-2

Lecture 28 Final Project-3

Lecture 29 Final Project-4

Lecture 30 Final Project-5

Lecture 31 Course Closure

Beginners who have never used Apache Airflow before,Data Engineers, Data Scientists, DevOps Engineers, Software Engineers, IT Professionals, Students. who want to learn a new way to create Directed Acyclic Graphs (DAGs),Intermediate or advanced Airflow users who want to improve their skills even more!