This workshop will cover the following topics:

• Understand Spark architecture and execution model
• Learn structured data processing with Spark SQL, DataFrames and Datasets
• Apply powerful Spark SQL functions and user defined function (UDF)
• Perform streaming processing with Spark Structured Streaming
• Learn main concepts in Spark MLlib component & have fun with classification task
• Apply Spark MLlib to build Recommender Systems

Prerequisite:

• The exercises will be done via Databricks notebooks. Please signup for a free Databricks Community Edition account.
• Highly recommended resources to checkout to learn about creating a cluster, importing notebooks and data on Databricks
  • Create clusters
  • Import notebooks
  • Importing data

Movie Recommendation Application

• Github

Workshop Materials

Lecture Notes

• Part 1
• Part 2

Import Databricks Notebooks

• Login into Databricks - https://community.cloud.databricks.com/login.html
• Click on "Workspace" icon on left hand vertical bar
• Next to "Workspace" label at the top of the column, click on arrow
• Select "Import" option
• Make "Import from" option is "File"
• Click on the gray box with label "Drop file here to upload or click to select." to bring up file browser
• In File browser, navigate where "qconsf.dbc" file was download to and select it
• Click on "Import" button to complete the import process
• Should see a folder called "qconsf" in Workspace