Disclaimer: I provide this list without any representation as to the suitability or sufficiency of any of the items referenced for any particular purpose. IANAE. Please do your own assessment or evaluation, particularly if the provider is charging a fee. :).

Also, some of the approaches listed may no longer work. Change is constant.

Core Concepts

Getting that Web Data

• The three ways to get web data

• A cool example of using the hidden AirBnB API (not sure if it still works, but you'll get the idea)

• Using Excel to get web table data (super handy!)

• A more in depth look at Excel to scrape table data from the web

HTML and CSS

• Codecademy.com HTML tutorial (very good and free!)

• ILoveCoding.Org's excellent free course on HTML and CSS

• W3C HTML tag reference

• CSS demo

• Understanding the components of a URL

Control Flow

• Control flow statements explained (Python)

General Computer Science Concepts and "Lingo"

• 40 Key Computer Science Concepts Explained In Layman’s Terms

Regular Expressions

• Good intro from Syd Johnson

• A very detailed tutorial

• Coursera course chapter on regular expressions

• MIT cheat sheet

• Stanford cheat sheet

• Regex 101: A great online regex tester

Programming Skills

Learn to Code for Free (or Very Little)

• Meta resource list of coding training for non-programmers

• Learnpython.org

• A great free introductory Python course by Scrimba

• 14 free courses to learn Python online

• 30 Days of Python (YouTube)

Python Resources

• A practical guide to web scraping with Python

• Python package index

• Python built-in functions

• Awesome list of Python resources

• Find answers to Python errors on Stack Overflow (make an account!)

Web Scraping YouTube Channels

• John Watson Rooney

• Build with Python

• Python Simplified

DIY Tools

Anaconda and Jupyter Notebook

• The Anaconda distribution (Jupyter Notebook) - Community Edition

• Anaconda User Guide

• Jupyter Notebook Tips, Tricks and Shortcuts

• Jupyter Lab - building on Jupyter Notebook

• Jupyter Notebook Tutorial (very good!)

• Another excellent Jupyter Notebook Tutorial

• Jupyter Github repository

• Google Colab online notebooks

• Intro to Colab

• Sample Notebook: The basics (you may get prompted to request access - just click the button)

• Sample Notebook: Web Scraping Example 1 (same as above)

Developer Tools and the Web Scraper Extension

• The Chrome Web Scraper Extension

• General Chrome Developer Tools Tutorial

• Web scraping specific Chrome Developer Tools Tutorial

• Promptcloud Web Scraper Extension Guide

• Web Scraper YouTube channel

• Another good written tutorial on the Web Scraper by Luxi Wang on Medium

Coding a Web Scraper

• Webscraping - a complete introduction

• Basic example

Beautiful Soup

• Beautiful Soup homepage

• Beautiful Soup documentation

Scrapy

• Scrapy home page

• Scrapy documentation

• Scrapy tutorial

• Another excellent Scrapy tutorial

Headless Browsing

• Selenium homepage

• Jupyter Notebook demo of Instagram scrape using Selenium

• A coded example

• Chrome Driver: Controlling Browser Action

• Phantom JS (deprecated)

• HtmlUnit (Java)

• MechanicalSoup (cannot do JavaScript)

• YouTube demo of MechanicalSoup

Node.js

• Intro to Node.js

• Webscraping with Node.js

• Puppeteer (headless browsing library for Node.js)

Proxy Servers and Services

• List of free proxies

• Scraping proxy server IP addresses

• A simple sample implementation

• Webshare

Captcha Mitigation

• Good Octoparse article explaining Captcha

• https://anti-captcha.com/ (not free, but relatively cheap)

• https://www.deathbycaptcha.com/ (also not free)

Other Tools

• Web scraping in a browser (with proxy)

• Online JSON to CSV converter

• Online HTML viewer and editor

• Python library to OCR images

API's

• List of free API's

• List of publicly available API's

• Another list of API's

• Facebook SDK for Python

• Request access to the Twitter API

• Twitter API documentation

• Google Search API

• Instagram (Basic) API

• Guide to getting a TikTok API key

Online Scraping Tools and Services

Fee Based

• https://www.browseai.com/

• https://www.diffbot.com/

• https://hexomatic.com/

• https://parsehub.com/

• https://prowebscraper.com/

• https://www.rtila.net/

• https://scrapex.ai/ (in Beta)

• https://www.getscrapio.com/

• Serpapi - Google Search API scraping

• https://webautomation.io/

Free, Limited Use, or Relatively Inexpensive

• https://scrape.do/

• https://www.scrapestorm.com/

• https://scrapfly.io/

• https://simplescraper.io/

• https://tryspider.com/

Browser Extension Based

• https://www.agenty.com/

• https://www.anypicker.com/

• https://axiom.ai/

• https://datagrab.io/

• https://dataminer.io/

• https://www.listly.io/

• https://parsers.me/

Scraping as a Service

• Apify

• Automatio

• Bright Data

• Import.io

• Mozenda

• PhantomBuster

Social Media Scrapers

• Scraping social media (detailed paper)

• An online tool to download a user's tweets (with nice export feature)

• Another tool that grabs a user's tweets

• Octoparse

• Dexi

• Outwit

• Scrapehero

• Socinator (more a marketing monitoring tool, but worth a mention here)

• Zyte (used to be Scrapinghub)

Specific Example Implementations

• Using the serpApi to grab tweets

• Octoparse Twitter example

• Octoparse Instagram example

• Scraping Tweets with Tweepy and SNScrape

• Scraping Telegram with the Telethon Python package

• The Python insta-scrape package for scraping Instagram

• Scraping Instagram with Instagramy

• Scrape Instagram by hashtag

• Grab a specific user's photos and videos on Instagram with instagram-scraper

• Scrape Facebook profiles with PhantomBuster

• Scrape Facebook with Python

• TikTok scraper using Node.js

• Praw - a Reddit scraping API implementation

• Download all reddit user posts to a Google Sheet using Google Scripts

• A universal Reddit scraper

• Build your own Reddit scraper

• Scrape Facebook without an API

Analyzing Data and Visualization

• How to analyze a data set

• Beginner steps in analyzing a data set

Free or free options

• You Suck at Excel (54 minutes well spent, just go watch it)

• Excel data analytics YouTube course

• The complete guide to Power Query

• Microsoft Power BI (download links here)

• Google Data Studio

• Google Data Studio Tutorial

• Radiant Shiny App (R)

• Apache Superset

• Metabase

• Overview of Python visualization libraries

Not Free (but Trial Versions Available)

• Tableau

• Bipp

Legal and Ethical Issues

• Century 21 Canada Limited Partnership v. Rogers Communications Inc., 2011 BCSC 1196 (CanLII)

• 2021 Clearview decision (Privacy Commissioner of Canada)

• Wired article on Clearview

• Recent overview article from a Canadian law firm

• What You Can Scrape and What Is Right to Scrape: A Proposal for a Tool to Collect Public Facebook Data

• A hosting company's perspective

• Article on European approach to crawling for LE purposes

• Good overview of the US status of things

• Article on "algorithmic policing" in Canada

Related Stuff

Intro to Data Science

• Data School YouTube Channel (very good and free (except for the ads :))

Protecting Your Own Data

• Discover what data the big social media companies have about you

• Learning about what data your social networks keep about you

• I Downloaded the Information That Facebook Has on Me. Yikes. (Watch the embedded video)

Miscellaneous

• W3C website markup standards

• Reddit subreddit on webscraping (great for staying up to date or asking questions)

• Search GitHub

Finding Anything Else You Need

• Use this obscure and largely unknown site!

Contact

• Email me

Some of my other resource pages are here.

This page last updated in February 2022