1 Introduction to Stata
Interface, do-files, loading and saving datasets
2 Data Cleaning and Variable Management
generate, replace, rename, drop, keep, order, label
3 Sorting and Group-wise Operations
sort, gsort, bysort, egen, duplicates
4 Combining and Reshaping Datasets
merge, append, reshape, data validation
5. Macros and Loops I
local, global, foreach, forvalues
6. Macros and Loops II
Conditional processing, nested loops, automated tasks
7. Advanced Data Management
Dates, strings, missing values, assert, isid
8. Data Aggregation and Panel Data Management
collapse, contract, panel structure, xtset
9. Reproducible Data Management
tempfile, log-files, automated cleaning, complete data pipelines
1 Introduction to Web Scraping
Understand the basics of web scraping and learn how to retrieve and parse web pages using requests and BeautifulSoup.
2 HTML Parsing and Data Extraction
Extract specific information from HTML documents, organize the collected data using pandas, and save datasets as CSV files.
3 Introduction to Web Crawling
Learn how to automatically navigate multiple web pages, follow hyperlinks, and collect data across pages.
4 APIs and Web Data Collection
Understand how APIs work, retrieve data from web services, and process JSON data using Python.
5 Building Research Datasets
Apply web scraping, crawling, and API techniques to collect, clean, validate, and organize data for empirical research.
Python
10
Introduction to Python and Google Colab
Variables, lists, dictionaries, conditions, loops
11
Data Management Using pandas
DataFrames, CSV/Excel, filtering, merging, exporting data
12
Introduction to Web Scraping
HTML, requests, BeautifulSoup
13
Practical Web Scraping
Extracting tables, text, links, multiple pages
14
APIs and Automated Web Data Collection
JSON, APIs, pagination, error handling, responsible scraping
15
Final Project: Building a Research Dataset
Web data collection, cleaning with pandas, exporting to Stata
Web Scraping and Crawling Using Python — 6 Lectures × 90 Minutes
1 Introduction to Python and Google Colab
Variables, lists, dictionaries, loops, functions
2 Introduction to Web Scraping
HTML, HTTP, requests, BeautifulSoup, extracting text and links
3 Practical Web Scraping and Data Management
Extracting HTML tables, structured data, pandas, CSV/Excel
4 Introduction to Web Crawling
Following links, multiple pages, pagination, URL management, avoiding duplicate visits
5 Automated Data Collection and APIs
JSON, REST APIs, error handling, rate limits, responsible scraping
6 Final Project: Building a Research Dataset