Build An Aws Machine Learning Pipeline For Object Detection

Build An Aws Machine Learning Pipeline For Object Detection
Published 3/2023
MP4 | Video: h264, 1280×720 | Audio: AAC, 44.1 KHz

Use AWS Step Functions + Sagemaker to Build a Scalable Production Ready Machine Learning Pipeline for Plastic Detection

What you’ll learn

Learn how you can use Google’s Open Images Dataset V7 to use any custom dataset you want

Create Sagemaker Domains

Upload and Stream data into you Sagemaker Environment

Learn how to set up secure IAM roles on AWS

Build a Production Ready Object detection Algorithm

Use Pandas, Numpy for Feature and Data Engineering

Understanding Object detection annotations

Visualising Images and Bounding Boxes with Matplotlib

Learn how Sagemaker’s Elastic File System(EFS) works

Use AWS’ built in Object detection detection algorithm with Transfer Learning

How to set up Transfer Learning with both VGG-16 and ResNet-50 in AWS

Learn how to save images to RecordIO format

Learn what RecordIO format is

Learn what .lst files are and why we need them with Object Detection in AWS

Learn how to do Data Augmentation for Object detection

Gain insights into how we can manipulate our input data with data augmentation

Learn AWS Pricing for SageMaker, Step Functions, Batch Transformation Jobs, Sagemaker EFS, and many more

Learn how to choose the ideal compute(Memory, vCPUs, GPUS and kernels) for your Sagemaker tasks

Learn how to install dependencies to a Sagemaker Notebook

Setup Hyperparameter Tuning Jobs in AWS

Set up Training Jobs in AWS

Learn how to Evaluate Object detection models with mAP(mean average precision) score

Set up Hyperparameter tuning jobs with Bayesian Search

Learn how you can configure Batch Size, Epochs, optimisers(Adam, RMSProp), Momentum, Early stopping, Weight decay, overfitting prevention and many more in AWS

Monitor a Training Job in Real time with Metrics

How to Test your model in a Sagemaker notebook

Learn what Batch Transformation is

Set up Batch Transformation Jobs

How to use Lambda functions

Saving outputs to S3 bucket

Prepare Training and Test Datasets

Data Engineering

How to build Complex Production Ready Machine Learning Pipelines with AWS Step Functions

Use any custom dataset to build an Object detection model

Use AWS Cloudformation with AWS Step Functions to set up a Pipeline

Learn how to use Prebuilt Pipelines to Configure to your own needs

Learn how you can Create any Custom Pipelines with Step Functions(with GUI as well)

Learn how to Integrate Lambda Functions with AWS Step Functions

Learn how to Create and Handle Asynchronous Machine Learning Pipelines

How to use Lambda to read and write from S3

AWS best practices

Using AWS EventBridge to setup CRON jobs to tell you Pipeline when to Run

Learn how to Create End-to-End Machine Learning Pipelines

Learn how to Use Sagemaker Notebooks in Production and Schedule Jobs with them

Learn Machine Learning Pipeline Design

Create a MERN stack web app to interact with our Machine Learning Pipeline

How to set up a production ready Mongodb database for our Web App

Learn how to use React, Nextjs, Mongodb, ExpressJs to build a web application

Create and Interact with JSON files

Put Convolutional Neural Networks into Production

Deep Learning Techniques

How to clean up an AWS account after you are done

Train Machine Learning models on AWS

How to use AWS’ GPUs to speed up Machine Learning Training jobs

Learn what AWS Elastic Container Registry(ECS) is and how you can download Machine Learning Algorithms from it

AWS Security Best practices

Requirements

Laptop with Internet Access

AWS account

Knowledge of Python and basic Machine Learning

Spend 20-50 dollars on AWS if you want to follow along with me. Note that you can still follow along without having to pay any money

Description

Overview

Section 1: What we are Building

Lecture 1 Let’s look at our End Project

Lecture 2 Source Code for the Course

Lecture 3 Setting up IAM User

Lecture 4 Clarification about AWS S3

Lecture 5 Getting Data for our Project

Section 3: Setting up AWS SageMaker

Lecture 10 Create SageMaker Domain

Lecture 11 Create SageMaker Studio Notebook

Lecture 14 Upload and Extract Data in SageMaker

Lecture 15 Deleting Unused Files

Section 4: Exploratory Data Analysis

Lecture 16 Loading and Understanding our Data

Lecture 17 Counting total Images and getting Image ids

Lecture 18 Getting Classname Identifier

Lecture 19 Looking at Random Samples from our Dataframe

Lecture 20 Understanding Annotations

Lecture 23 Matplotlib difference between plt.show() and plt.imshow()

Lecture 24 Visualising Multiples Images at Once

Lecture 25 Correcting our Function

Lecture 29 Wrong Print Statement

Lecture 31 Read this Lesson if you have issues with Data Visualization

Section 5: Cleaning and Splitting our Data

Lecture 32 Clean our Train and Validation Dataframes

Lecture 33 Split Dataframe into Test and Train

Lecture 34 Get Images IDs

Lecture 35 Splitting IDs Theory Lesson

Lecture 36 Explanation Regarding Next video

Lecture 37 Moving Images to Appropriate Folders

Lecture 38 Count how many Train and Test Images we have

Section 6: Date Engineering

Lecture 41 Using Mxnet

Lecture 42 Additional Info regarding RecordIO format

Lecture 43 Using Mxnet RecordIO

Lecture 44 Correction Regarding Label width

Lecture 47 Moving Images To Correct Directory

Lecture 48 Explanation Regarding the Previous Video

Lecture 49 Verifying that all Images have been Moved Properly

Lecture 50 Read Before Proceeding to the next Lecture

Lecture 51 Creating Production .lst files (Optional)

Section 7: Data Augmentation

Lecture 52 Data Augmentation Theory

Lecture 53 Augmenting a Random Image

Lecture 54 Moving Images to new Folder structure

Lecture 57 Read this Lesson if you have issues visualising your images

Lecture 60 Checking Image Counts Before running the Function

Lecture 61 Correctional Video regarding our Function

Lecture 62 Augmenting Test Dataset and Creating test .lst Files

Lecture 65 Verifying that Data Augmentation has Worked

Section 8: Setting up and Creating our Training Job

Lecture 66 Increasing Service Quotas

Lecture 67 Installing dependencies and Packages

Lecture 68 Creating our RecordIO Files

Lecture 69 Uploading our RecordIO data to our S3 bucket

Lecture 70 Downloading Object Detection Algorithm from AWS ECR

Lecture 71 Setting up our Estimator Object

Lecture 72 Setting up Hyperparameters

Lecture 73 Additional Information for Hyperparameter Tuning in AWS

Lecture 74 Setting up Hyperparameter Ranges

Lecture 75 Setting up Hyperparameter Tuner

Lecture 76 Additional Information about mAP( mean average precision)

Lecture 79 More on mAP Scores

Lecture 80 Monitoring the Training Job

Lecture 81 Looking at our Finished Hyperparameter Tuning Job

Section 9: Analysing Training Job Results

Lecture 82 Deploying our Model in a Notebook

Lecture 83 Creating Visualization Function for Inferences

Lecture 86 Testing our Endpoint from Random Images from the Internet

Section 10: Setting up Batch Transformation

Lecture 87 Setting up Batch Transformation Job locally first

Lecture 89 Analysing our Batch Transformation Job

Lecture 90 Visualising Batch Transformation Results

Lecture 91 Look at this lesson if you have trouble with the Visualisations

Section 11: Setting Up Our Machine Learning Pipeline

Lecture 92 Read this Before Watching the Next Lesson

Lecture 93 Setting up AWS Step Function

Lecture 94 Verify that CloudFormation has worked

Lecture 97 Create Check Batch Transform Job Lambda

Lecture 98 Fixing typos and Syntax Erros

Lecture 99 JSON output Format

Lecture 105 Upload Test Data to S3

Lecture 106 Testing our Step Function

Lecture 107 Fixing Errors

Lecture 108 Testing our Step Function with the Corrections

Lecture 109 Verifying that our Step Function Ran Successfully

Lecture 110 Donwloading our JSON file from S3

Lecture 111 Using Event Bridge to set up Cron Job for our Machine Learning Pipeline

Lecture 112 Verify that the Cron Job works

Lecture 113 Verifying that our Pipeline Ran Successfully

Lecture 114 Setting up Production Notebook

Lecture 115 Extending Our Machine Learning Pipeline

Lecture 120 Verifying that the Images have been Saved Properly

Lecture 123 Verify that the Entire Machine Learning Pipeline works

Lecture 124 Deleted Unused items from Sagemaker EFS

Section 12: Creating our Web Application

Lecture 125 Clone the Web Application from Github

Lecture 126 Setup MongoDB

Lecture 127 Connect to MongoDB and get AWS Credentials

Lecture 128 Configuring Env file

Lecture 129 Install Node modules

Lecture 133 Output Images Explanation

Section 13: Outro

Lecture 136 Clean Up Resources

Lecture 137 Congratulations

For developers who want to take their machine learning skills to the next lever by being able to not only build machine learning models, but also incorporate them in a complex, secure production ready machine learning pipeline