Spark, Ray, and Python for Scalable Data Science
MP4 | Video: AVC 1280 x 720 | Audio: AAC 48 Khz 2ch | Duration: 07:01:37 | 8.57 GB
Conceptual overviews and code-along sessions get you scaling up your data science projects using Spark, Ray, and Python.
Overview
About the Instructor
Jonathan Dinu is the founder of Zipfian Academy-an advanced immersive training program for data scientists and data engineers in San Francisco-and served as its CAO/CTO before it was acquired by Galvanize, where he now is the VP of Academic Excellence. He first discovered his love of all things data while studying Computer Science and Physics at UC Berkeley, and in a former life he worked for Alpine Data Labs developing distributed machine learning algorithms for predictive analytics on Hadoop.
Jonathan is a dedicated educator, author, and speaker with a passion for sharing the things he has learned in the most creative ways he can. He has run data science workshops at Strata and PyData (among others), built a Data Visualization course with Udacity, and served on the UC Berkeley Extension Data Science Advisory Board. Currently he is writing a book on practical Data Science applications using Python. When he is not working with students, you can find him blogging about data, visualization, and education at
Skill Level
Beginner to Intermediate
Learn How To
Integrate Python and distributed computing
Scale data processing with Spark
Conduct exploratory data analysis with PySpark
Utilize parallel computing with Ray
Who Should Take This Course
Course Requirements
Familiarity with the vocabulary of data processing at scale, machine learning (dataset, training set, test set, model), and AI.
Lesson Descriptions
Lesson 1: Introduction to Distributed Computing in Python
Lesson 2: Scaling Data Processing with Spark
Lesson 2 goes into detail on the Spark framework beginning with a “Hello World” example of programming with Spark. Then Jonathan turns to the Spark APIs. You get some experience with one of Spark’s primary data structures, the resilient distributed dataset (RDD). Next is key-value pairs and how Spark does operations on them similar to MapReduce. The lesson finishes up with a bit of Spark internals and the overall Spark application lifecycle.
Lesson 3: Exploratory Data Analysis with PySpark
Lesson 4: Parallel Computing with Ray
Lesson 4 introduces the Ray programming API, with Jonathan comparing the similarities and differences between the Ray and Spark APIs. You learn how you can distribute functions with Ray, as well as how you can perform operations with distributed classes or objects with Ray actors. Finally, Jonathan finishes up with a large scale simulation to highlight the strengths of the Ray framework.
Lesson 5: Scaling AI Applications with Ray
