module 7.3: Web Scraping

Web scraping is the process of getting data from websites. We can do this by fetching web pages, parsing the HTML or other markup language, and then extracting information.
This is useful for data analysis, research, or building applications.
Creating a program that calls a website with updating data (think of a website that displays the weather) and extracts just the information desired.
Beautiful Soup → a library used for web scraping purposes to pull the data out of HTML and XML files. Helps with parsing HTML and XML documents to extract data. Think about the previous libraries we have imported (ie numpy)

E-Commerce: Monitor prices, analyze competitors, and gather product data (e.g., Amazon, eBay)
Market Research: Analyze trends, competitor strategies, and customer sentiment from reviews or social media.
Real Estate: Collect property listings, prices, and market trends (e.g., Zillow).
Finance: Track stock prices, crypto data, and financial news for analysis.
Job Market: Scrape job listings, skill demands, and salary data.
Academic Research: Extract papers, abstracts, or citation data.
News Aggregation: Gather articles and monitor media coverage.
Travel: Compare flight and hotel prices and analyze reviews.
Social Media: Track hashtags, user engagement, and content trends.
Education: Collect course data or research resources.
Sports: Track live scores, stats, and ticket prices.
Local Business: Extract directory data (e.g., Google Maps, Yelp) and competitor info.

Complete the Trinket below by filling in the blanks. Use the EdPuzzle to walk you through it!

Watch the Ed Puzzle & add the missing code to run the html parser from Beautiful Soup.

Now, find your favorite webpage and change the url to a new page. Run the code again and verify if the web page title comes back appropriately.

Before you start scraping data, examining a website and seeing where the data is located is helpful.
Open a basketball or swimming statistics website (ex. ESPN, SwimRankings, or Basketball-Reference)
- Identify key data groups you might want to scrap
- Think about how the website has the data (ex. tables, lists)

Explore the code below to familiarize yourself with web scraping of a weather forecast!
Replace the #TODO lines with comments explaining what the code below is doing. If you get stuck research the command.

Coding Assistance: Click this link if you need help solving this problem.

Why is web scraping important?
- Data Collection: scraping is quick and efficient
- Market Research: monitors competition and common trends
- Academic Research: gathering information from different sources
- Automation: automates repetitive taste which allows for a more time efficient process
What are some common uses of scraping in everyday life?
- Price Comparison: companies like Honey help users to compare prices to ensure they are getting the best deal
- Job Hunting: companies like Indeed and LinkedIn use scraping to keep they applications updated with the newest jobs
- Investment and Finance: financial services use web scraping to gather stock prices, financial news, and market trends, which helps investors make informed decisions.
We would use web scraping/API’s opposed to JSON/CSV files when we what the data we are extracting to be live and/or updating. For example, data about a rocket launch or wind turbines.
Keep in mind web scraping/API’s can be used as the data source for your final project!

SELF ASSESSMENT

Page updated

Report abuse