Creating Data Pipelines Using Aws & Confluent Kafka Platform

Creating Data Pipelines Using Aws & Confluent Kafka Platform
Published 5/2025
MP4 | Video: h264, 1920×1080 | Audio: AAC, 44.1 KHz

Hands on course to build Real Time Data streaming pipeline from Confluent, to Lambda to DynamoDB from scratch

What you’ll learn

Get an high level overview of building Data Pipelines using confluent Kafka and AWS services – Lambda, DynamoDB

Be able to build End to End Real time streaming Data Pipeline using AWS + Confluent Platforms

Build understanding of popular AWS services like Lambda, DynamoDB, IAM, Cloudwatch

Hands on practice with streaming data from Confluent Kafka all the way to Database on AWS

Requirements

No programming experience needed, Be able to create free tier AWS and Confluent accounts for hands on practice

Description

Overview

Section 1: Introduction

Lecture 1 Introduction to Real Time Streaming Cloud Native Data Pipeline Architecture

Section 2: Confluent Cloud Account Setup

Lecture 2 Setup Confluent.io cloud Account

Section 3: Review Cluster on Confluent & Create Topic

Lecture 3 Confluent – Create Topic & API Key

Section 4: Create Data Gen Source Connector and load the Confluent Topic

Lecture 4 Create Data Gen Source Connector, load Topic & validate

Section 5: Create & Setup AWS Account

Lecture 5 Create & Setup AWS Account

Section 6: Create Lambda Function & Connect to Confluent Topic

Lecture 6 Initial Lambda Setup, Test & Cloudwatch Exploration

Lecture 7 Setup Lambda-Confluent connection (ESM), use Secrets Mgr & IAM role

Lecture 8 Test Data flow from Confluent source connector to Topic to AWS Lambda Function

Section 7: Parse event using Python Lambda code & write to CloudWatch Logs

Lecture 9 Parse event & write Data elements to Cloudwatch

Section 8: Create DynamoDB Table & Update Lambda function code to insert data into it

Lecture 10 Create Dynamo DB table & Explore NoSQL Capabilities

Lecture 11 Update Lambda function to insert incoming data to DynamoDB table

Section 9: Run the Data Pipeline End to End & Verify

Lecture 12 Run the Data Pipeline End to End and Validate all steps!

Section 10: Summary & Wrap Up!

Lecture 13 Summary & Wrap up!

Beginner Cloud Developers looking to build AWS + Confluent Kafka Skills,Entry Level Cloud & Data Engineers looking for a Real Life Project to add to their Resumes,Mid Level and Sr. Data Engineers who are looking to enhance their skills in Confluent + AWS Platform areas,Product Owners and Product Managers looking to understand capabilities of the Confluent + AWS ‘s Real Time Streaming Platforms,Technology Leaders looking to understand Capabilities, Shortcomings, efforts it takes to build Real Time Data Streaming Pipeline