Kubernetes For Data Engineering: Hands On End To End Guide

Kubernetes For Data Engineering: Hands On End To End Guide
Published 4/2024
MP4 | Video: h264, 1920×1080 | Audio: AAC, 44.1 KHz

Boost the efficiency of your Data Engineering Solutions by Deploying them on Kubernetes Cluster.

What you’ll learn

Understand the core concepts of Kubernetes, including pods, services, deployments, and more. Learn how to set up and manage a Kubernetes cluster

Gain practical experience in deploying and managing the Kubernetes Dashboard, a powerful tool for managing Kubernetes clusters through a user-friendly interface

Learn how to deploy Apache Airflow in a Kubernetes environment. Understand how to schedule and monitor data pipelines efficiently using Airflow.

Dive into the world of Directed Acyclic Graphs (DAGs) and learn how to create, schedule, and monitor them in an Airflow environment running on Kubernetes.

Understand how to secure your Kubernetes cluster and monitor its performance. Learn about Kubernetes namespaces, RBAC, secrets, and network policies.

Learn how to scale your data applications and ensure high availability within your Kubernetes cluster.

Develop skills to troubleshoot common issues in Kubernetes and optimize the performance of your data pipelines.

Requirements

Basic Programming Experience

Description

Overview

Section 1: Introduction

Lecture 1 Introduction

Lecture 2 What this course covers

Lecture 3 Kubernetes Architecture Explained

Lecture 4 KubeProxy and Container Runtime Deep Dive

Lecture 5 Kubernetes Additional Services

Lecture 6 Kubernetes Networking Fundamentals

Lecture 7 Kubernetes Core Concepts

Lecture 8 Kubernetes Behind The Scenes

Lecture 10 How Kubernetes can help you as an engineer

Section 2: Infrastructure Setup

Lecture 11 Installing Docker Desktop

Lecture 12 Setting up, enabling and verifying Kubernetes

Lecture 13 Installing Cluster Managers on all Operating Systems

Lecture 14 Cluster Manager Commands

Section 3: Kubernetes Dashboard

Lecture 17 Generating Tokens for Kubernetes Dashboard

Lecture 18 Working with Kubernetes Dashboard – End to End

Section 4: Deploying Apache Airflow to Kubernetes Cluster

Lecture 19 Deploying Apache Airflow on Kubernetes

Lecture 21 Creating and Deploying Airflow DAGS to Kubernetes Cluster

Lecture 22 Deploying and Working with Multiple DAGS on Kubernetes Cluster

Lecture 23 Optimising your Airflow DAG pipeline on Kubernetes

Section 5: Apache Spark Deployment on Kubernetes Cluster

Lecture 24 Preparing Spark Jobs to run on Kubernetes

Lecture 25 Packaging your Spark Jobs on Kubernetes

Lecture 26 Deploying Spark Jobs to Kubernetes Cluster

Lecture 27 Fixing Potential Bugs during Spark Deployment on Kubernetes Cluster

Section 6: Next Steps (Resources and Documentations)

Lecture 28 Course Resources (Source Code)

Lecture 29 Commands, Text and Documentation

Everybody interested in building scalable and efficient infrastructures