Streaming Data Pipeline From Confluent Kafka To Aws Mysql Db

Streaming Data Pipeline From Confluent Kafka To Aws Mysql Db
Published 5/2025
MP4 | Video: h264, 1920×1080 | Audio: AAC, 44.1 KHz

Hands on course to build Real Time Data Streaming Pipeline from Confluent Kafka to Lambda to AWS MySQL RDS from scratch

What you’ll learn

Hands On course to learn to use Confluent Kafka & AWS Cloud Platforms

Learn basics of navigating & using AWS Cloud Platform services like Lambda, RDS-MYSQL, IAM, Cloudwatch, Secrets Manager & more

Learn basics of Confluent Platform and overview of services like: Topics, Connectors, API Keys, Clusters & more

Watch & Learn to build on your AWS + mConfluent Accounts- Real Time Data Streaming pipeline from Confluent to AWS RDS using Serverless Lambda Compute

Requirements

Be able to create AWS Cloud Platform Account (Free Tier)

Be able to create Confluent Kafka Platform Account

Basic understanding of Python

Description

Overview

Section 1: Introduction

Lecture 1 Introduction & High Level Architecture of end product that we will build

Section 2: Confluent related Activities

Lecture 2 Create Free Confluent.io account to use for data streaming using Kafka

Lecture 3 Confluent: Create Topic, API Keys & Explore Confluent Kafka Topic capabilities

Lecture 4 Confluent: Setup Datagen Source Connector to produce data Kafka Topic

Section 3: AWS Account creation & Exploration

Lecture 5 AWS Cloud Platform: Create Account & Explore Services

Section 4: AWS + Confluent Integration

Lecture 6 AWS: Create & Explore Lambda Serverless Compute

Lecture 7 Configure Lambda to consume from Confluent Kafka Topic & verify in CloudWatch

Lecture 8 Create & Explore AWS RDS MySQL Database

Lecture 9 Connect to AWS MySql DB via MySQL Workbench & create Schema/Table

Section 5: Parse the event & extract data fields

Lecture 10 Parse event and extract fields and write to Cloudwatch

Lecture 11 Update Lambda to connect to RDS + Create Layer and add pymysql module to it

Lecture 12 Update Lambda Python code to insert extracted fields into MySQL RDS

Section 6: End to End Data Pipeline execution & Validation

Lecture 13 Execute End to End flow and observe at all Intermediate steps

Lecture 14 Summary, Wrap Up and Next Steps!

Beginner Cloud Developers looking to build Real Time Data pipeline development skills and add a real life project to resume,MId Level to Sr. Cloud Developers to quickly gain understanding of building Real Time Streaming Data Pipelines using AWS + Confluent Platforms,Product Owners looking to build understanding of capabilities, risks & issues in building data pipeline using AWS + Confluent Kafka platform,Product Managers looking to build understanding of capabilities, risks & issues in building data pipeline using AWS + Confluent Kafka platform,Technology Leaders looking to deepen understanding of capabilities, efforts, risks & issues in building data pipeline using AWS + Confluent Kafka platform,Solution Architects looking for building hands-on Data Pipeline using AWS MySQL DB + Confluent Kafka Platforms