Complete Python Web Scraping: Real Projects & Modern Tools

Complete Python Web Scraping: Real Projects & Modern Tools
Published 8/2024
MP4 | Video: h264, 1920×1080 | Audio: AAC, 44.1 KHz

Learn BeautifulSoup, Selenium, Scrapy and Scrapy-Playwright. 4 Project-like Exercises + 4 Real Life Projects

What you’ll learn

Scraping websites using Python

Python’s most popular and effective web scraping libraries

Scrapy-Playwright

Web Automation with Selenium

Scrapy

Requests and Beautiful Soup

Using the right method according to the structure of the website

Reading and Analyzing HTML code

Saving scraped data

Downloading bulk images

Requirements

Just a PC with internet connection.

No programming experience is needed. I will teach you everything you need to know.

Fundamental Python knowledge is nice to have but not a must.

Description

Overview

Section 1: INTRODUCTION

Lecture 1 Course Introduction

Lecture 2 Section Introduction

Lecture 3 Course Material

Lecture 4 Checking If the Website is Static or Dynamic

Lecture 5 Deciding on the Method to be Used

Lecture 6 Reading and Analyzing HTML Code

Lecture 7 Installing Python and Pycharm (WINDOWS)
Lecture 8 Installing Python and Pycharm (LINUX)

Lecture 9 Installing Python and Pycharm (MACOS)

Lecture 10 Installing Jupyter Notebook on Windows, Linux and MacOS (OPTIONAL)
Lecture 11 About Real Life Examples.

Lecture 12 Course Syllabus

Section 2: BEAUTIFUL SOUP and REQUESTS 1 – BASICS

Lecture 13 Section Introduction

Lecture 14 Installing Libraries and Making the First Request

Lecture 15 Selectors of Beuatiful Soup

Lecture 16 Scraping 1 Book’s Data

Lecture 17 Constructing the Outer Loop and Dealing with Pagination

Lecture 18 Constructing the Inner Loop and Scraping Every Book

Lecture 19 Saving the Data and Summary of the Project

Lecture 20 Downloading Images with Requests

Lecture 21 Using Proxies with Requests

Lecture 22 About Proxy Authentication

Section 3: BEAUTIFUL SOUP and REQUESTS 2 – EXERCISE 1 (QUOTES)

Lecture 23 Section Introduction

Lecture 24 Exercise 1

Lecture 25 Exercise 1 – Solution

Section 4: BEAUTIFUL SOUP and REQUESTS 3 – REAL LIFE EXAMPLE 1 (EBAY)

Lecture 26 Section Introduction

Lecture 27 Scraping 1 Laptop’s Data, Usage of Full Headers and Cookies

Lecture 28 Pagination

Lecture 29 Saving the Data

Section 5: SELENIUM 1 – BASICS

Lecture 30 Section Introduction

Lecture 31 Installing Selenium and Chromedriver

Lecture 32 Creating Driver and Opening Browser

Lecture 33 CSS Selectors

Lecture 34 XPATH

Lecture 35 Login with Selenium

Lecture 36 Scraping the First Page

Lecture 37 Browsing Through Pages

Lecture 38 Infinite Scroll

Lecture 39 Waits

Lecture 40 Actions
Section 6: SELENIUM 2 – EXERCISE 2 (AUTHORS)

Lecture 41 Section Introduction

Lecture 42 Exercise 2

Lecture 43 Exercise 2 – Solutions

Section 7: SELENIUM 3 – REAL LIFE EXAMPLE 2 (IMDB)

Lecture 44 Section Introduction

Lecture 45 Creating Driver and Opening Browser

Lecture 46 Important Note

Lecture 47 Adjustment to the Next Video (Automation)

Lecture 48 Reaching the Page with Automation

Lecture 49 Scrolling to Load All Items

Lecture 50 Scraping the Data

Section 8: SELENIUM 4 – EXERCISE 3 (IMDB DIRECTORS)

Lecture 51 Section Introduction

Lecture 52 Important Note

Lecture 53 Exercise 3

Lecture 54 Important Note

Lecture 55 Exercise 3 – Solutions

Section 9: SCRAPY 1 – BASICS

Lecture 56 Section Introduction

Lecture 57 Installing Scrapy, Creating Scrapy Project and Spider

Lecture 58 Scrapy Shell and Scrapy’s Selectors

Lecture 59 Constructing Parse Method

Lecture 60 Items

Lecture 61 Constructing Parse Books Method

Lecture 62 Pipelines

Lecture 63 Dropping Items with Pipelines

Lecture 64 Saving Data to Excel with Pipelines

Lecture 65 Note about SQLite

Lecture 66 Saving Data to SQLite Database with Pipelines

Lecture 67 Middlewares

Lecture 68 Crawler

Section 10: SCRAPY 2 – EXERCISE 4 (QUOTES)

Lecture 69 Section Introduction

Lecture 70 Exercise 4

Lecture 71 Exercise 4 – Soluitons

Section 11: SCRAPY 3 – REAL LIFE EXAMPLE 3 (FLYING TIGER)

Lecture 72 Section Introduction

Lecture 73 Creating Project and Spider

Lecture 74 Scraping the Data

Lecture 75 Adjustment on Previous Video – Log Files and Default Image URL

Lecture 76 Json Writer Pipeline

Lecture 77 Downloading Images

Section 12: SCRAPY 4 – REAL LIFE EXAMPLE 4 (YELP)

Lecture 78 Section Introduction

Lecture 79 Addition to the Next Video

Lecture 80 Real Life Crawler

Section 13: SCRAPY – PLAYWRIGHT

Lecture 81 Section Introduction

Lecture 82 Installing Virtual Box and Ubuntu for Windows
Lecture 83 Installing Scrapy – Playwright

Lecture 84 Adding Playwright Features to Scrapy Project

Lecture 85 Scraping Dynamic Data

Lecture 86 Playwright Page

Lecture 87 Clicking with Playwright

Lecture 88 Wait for Selector and Wait for Timeout

Section 14: EXTRA – QUICK PYTHON FUNDAMENTALS COURSE

Lecture 89 Section Introduction

Lecture 91 Data Structures

Lecture 92 Control Flow Statements (if, for, while)

Lecture 93 Functions

Lecture 94 Errors, Modules-Libraries and File Operations

Lecture 95 Strings

Lecture 96 Mini Project – ToDo List

Lecture 97 OOP – Introduction to Object Oriented Programming

Section 15: CONGRATULATIONS

Lecture 98 End of the Course