Machine Learning for Economists

I also tweet about new relevant Machine Learning papers. You can follow me on Twitter @SansoneEcon


Where to start

  • Mullainathan and Spiess (Journal of Economic Perspectives, 2017) is a good introduction to ML for economist. The Online Appendix has a lot of important technical details to implement ML algorithms in practice

  • The Impact of Machine Learning on Economics by Athey (NBER, 2018)

  • Machine Learning Methods Economists Should Know About by Athey and Imbens (ArXiv, 2019)


Online courses and slides

  • Ng (Coursera) is a great MOOC. Ng is clear and does not go too fast. Applications in Matlab

  • Markham (R-bloggers) is the MOOC accompanying the textbook An Introduction to Statistical Learning. Entry-level course. Applications in R

    • Bellemare wrote a few posts (here and here) on what he learned from this textbook.

  • Caffo, Leek and Peng (Coursera) explain how to implement ML algorithms in R. Knowledge of R is a pre-requisite

  • IMF course on ML for economists by Michal Andrle

  • Advances in Causality and Foundations of Machine Learning by Maximilian Kasy (additional slides here)

  • ML in Development Economics and Applied Micro by Michael Koelle


If you want to learn more

  • Key resources

  • Textbooks

    • Friedman, Hastie and Tibshirani (Springer, 2008) provides a comprehensive (detailed and advanced) overview of the most commons ML techniques

    • Kuhn and Johnson (Springer, 2013) is a good complement to the textbook by Friedman et al. They cover different topics, or explain the same techniques from different angles.

  • Main articles

    • Storm et al. (ERAE, 2020) review ML tools from an applied economist’s perspective

    • Molina and Garip (ARS, 2019) give a broader overview of ML in the social sciences

    • Athey (Science, 2017) on using big data for policy problems

    • Athey and G. Imbens (Journal of Economic Perspective, 2017) discuss recent advancements in ML methods for casual inference

    • Subrahmanian and Kumar (Science, 2017) discuss future directions and challenges for ML

    • Belloni et al. (Journal of Economic Perspective, 2014) on using ML for model selection

      • Angrist and Frandsen (NBER, 2019) investigate more the idea of using ML for selecting control variables in OLS, and in the first-stage of IV. They find that ML works better at selecting controls than instruments.

    • Varian (Journal of Economic Perspective, 2014) discusses the benefits of ML for economists


Tools and best practices

  • Predict treatment heterogeneity

    • Causal Forest developed by Wager and Athey (JASA, 2018) and Generalized Random Forests (Athey et al., AoS, 2019)

      • Application to summer jobs by Davis and Heller (AER P&P, 2017)

        • See Bertrand et al. (WP, 2017), O'Neill and Weeks (WP, 2018), Naguib (WP, 2019), Lechner (arXiv, 2019), Cockx et al. (IZA, 2020), Guo et al. (JMR, 2020), Knittel and Stolper (AEA P&P, 2021), Goller et al. (IZA, 2021), Athey et al. (RePec, 2021) for applications and extensions

        • See Knaus et al. (JHR, 2020) for another application using a LASSO-type estimator.

      • Accessible description by Athey and Imbens (PNAS, 2016)

      • Application to energy use by Knittel and Stolper (NBER, 2019)

      • Huntington-Klein (JCI, 2020) use casual forest to improve finite-sample performance of IV by modeling first-stage heterogeneity i

    • Generic Machine Learning Inference by Chernozhukov et al. (ArXiv, 2018)

      • Extension to quasi-experimental settings by Deryugina et al. (NBER, 2019)

    • Heterogeneity with instruments by Syrgkanis et al. (ArXiv, 2019). See also Biewen and Kugler (IZA, 2020)

  • Intersection of ML and Econometrics

    • Using ML to improve pre-analysis plans by Ludwig et al. (AEA P&P, 2019)

    • Choosing among regularized estimators in empirical economics by Abadie and Kasy (ReStat, 2019)

    • Ovaisi et al. (PWC, 2020) combine ML with Heckman's two-stage method in learning-to-rank systems

    • Wang et al. (PNAS, 2020) developed a method for correcting inference in a second-stage when using outcomes predicted by ML in a first-stage. With an open-source R package

    • Mostly Harmless Machine Learning by Chen et al. (ArXiv, 2020) is a guide to use ML in instrumental variable designs

    • Chang (EconometricsJournal, 2020): ML with difference-in-difference

    • Kasy and Sautmann (Econometrica, 2021) proposed an algorithm for adaptive treatment assignment in experiments for policy choice. See also their article on VoxDev

      • Hadad et al. (PNAS, 2021): discuss how to construct confidence intervals in adaptive experiments

    • Athey and Wager (Econometrica, 2021) developed an algorithm for choosing whom to treat using observational data

    • Narita and Yaka (ArXiv, 2021) develop an estimator exploting natural experiments following algorithmic decision rules

    • Athey et al. (JASA, 2021) link matric completion methods with the unconfoundedness (matching) and synthetic control approaches. See also blog post by Cunningham

    • RCT vs ML (by Prest et al, WP 2021): ML replicates the true treatment effects, but DiD replicate the experimental benchmark as well, suggesting little benefit from ML approaches over standard program evaluation methods

  • Applying economic thinking to ML

    • Baiardi and Naghi (RePec, 2021): revisit influential empirical studies to show the advantages of causal machine learning methods

    • Raghu et al. (ArXiv, 2019) start thinking on how to combine ML and human experts, rather than just comparing performances, while Hofman et al. (Nature, 2021) provide a big-picture interdisciplinary view of ML, and urgue that it should be seen as a complement to causal inference.

      • Gennatas et al. (PNAS, 2019): example of expert-augmented machine learning, an automated way to extract problem-specific human expert knowledge and integrate it with ML

      • Ribers and Ullrich (ArXiv, 2020) combine ML predictions with physician diagnostic skill to improve the efficiency in antibiotic prescribing

      • Tubadji et al. (Economic Inquiry, 2021) looks at the propensity of consumers to adopt AI in banking services

    • Sansone (OBES, 2018) uses ML to predict high school dropout. He uses economic theory to guide the calibration of the ML algorithms. Stata conference presentation

      • Coyle et al. (Science, 2020) discuss the importance of clarifying the objective function when applying ML to policy

    • Björkegren et al. (ArXiv, 2020) discuss a manipulation-proof ML method, with an application to a field experiment in Kenya

  • Fairness, ethics, and privacy

    • Kleinberg et al. (AER P&P, 2018) advocate for not excluding variables such as race from the set of ML inputs in the name of fairness

      • Similar argument used for gender by Sean Higgins when predicting creditworthiness (MIT, 2019)

      • See also Rambachan et al. (NBER, 2020) on regulating algorithms, as well as Rambachan et al. (AEA P&P, 2020)

      • Rodolfa et al. (ArXiv, 2020): Trade-offs between accuracy and fairness assumed to be inherent in ML may be small in practice, making reducing disparities more practical

    • A black box can and should be used when it produces the best results. (Holm on Science, 2019)

      • Babic et al. (Science, 2021) discuss the drawbacks of requiring black-box algorithms to be explainable

    • ML could actually be used to detect discrimination (Kleinberg et al., PNAS 2020)

    • Ozler on the ethics of ML (Development Impact, 2019)

    • Kleinberg et al. (QJE, 2018) on ML being less biased than human judges when making bail decisions. Good summary by J. Doleac (Medium). Related case study in Latin America (IDB, 2019). See also Ludwig and Mullainathan (NBER, 2021)

      • When providing judges with risk scores, there was an increase in racial disparities due to judges overriding the recommended action for moderate-risk black defendants (Albright, WP 2019). See also Arnold et al. (NBER, 2020, AEA P&P, 2021)

      • Stevenson and Doleac (APPAM, 2019): even if the ML predictions are perfectly fair, the way humans take them into account might be biased. Example from judges in Virginia

    • Li et al. (NBER, 2020) look at hiring as a contextual bandit problem and build a ML algorithm that improves both quality of candidates and demographic diversity, thus increasing equity and efficiency.

    • Bjerre-Nielsen et al. (PNAS, 2020): models in education using only administrative data perform considerably better and, importantly, do not improve when adding high-resolution, privacy-invasive behavioral data

  • Data

    • Text as data by Gentzkow et al. (JEL, 2019)

      • Thorsrud (JBES, 2020) constructs a daily business cycle index based on quarterly GDP growth and textual info from a business newspaper

    • Jayachandran et al. (NBER, 2021) use ML to identify the best survey closed-ended questions to predict an agency score measured through qualitative interviews

    • Abowd et al. (NBER Summer Institute, 2017) on data linkage, e.g. across different Census waves. See also Price et al. (NBER, 2019) and James Feigenbaum's research

      • Combes et al. (IZA, 2021) discuss how to use ML to extract data from historical documents

    • Rolf et al. (NBER, 2020) on how to combine ML with satellite imagery

    • Schierholz and Schonlau (JSSaM, 2020) compare different ML algorithms for automated occupational coding

  • Other tools

    • Peterson et al. (Science, 2021) use ML to test decision-making theories. See summary by Bhatia and He (Science, 2021)

    • Borup et al. (SSRN, 2020) on targeting predictors in random forest

    • Goller et al. (LabourEconomics, 2020) try to use LASSO and Random Forest to improve the first stage in propensity score matching

    • Steurer and Hill (RePec, 2019): evaluate different performance metrics used in the housing market

    • Lechner et al. (WP, 2019) have developed a Random Forest estimator of the ordered choice model

    • Anastasopoulos (RePec, 2019) uses an adaptive lasso algorithms to select covariates in regression discontinuity designs

    • How to do cross-validation with time series data? Schnaubelt (RePec, 2019)

    • Be aware of bad controls when selecting variables using ML! (Hünermund et al., ArXiv, 2021)

  • Bonus material


Lectures and podcasts

  • Athey (ASSA, 2019) on the impact of ML on economics and econometrics

    • Watch also her lecture at the European Central Bank (ECB, 2019)

  • Duflo (NBER Summer Instititute, 2018) summaries current ML techniques that can be used by economists, with special focus on RCT. Slides and codes on GitHub

    • See also Quistorff and Johnson (ArXiv, 2020) for ML applications to restrict randomization in the design of experiments

  • Athey and Imbens (AEA Continuing Education, 2018) on machine learning and econometrics

  • Shiferaw (APPAM, 2017) discusses ML for Policy Analysis with Susan Athey

  • Athey and Imbens (NBER Summer Institute, 2015) is a mini-course on supervised ML, unsupervised ML, and ML for causal inference

  • The 2021 Summer Institute in Machine Learning in Economics hosted by the Center for Applied AI at Chicago Booth (YouTube)

  • Additional webinars (mainly on ML in macro) recorded by AMLEDS


Software


Cool applications

  • Labor, Gender, and Education

    • Koffi (RePec, 2021) use ML to show that female-authored papers are more likely to be omitted from references of related papers than male-authored papers

    • Cengiz et al. (NBER, 2021) use ML to identify minimum-wage workers

    • "What do economic education scholars study?" is an example of text analysis and unsupervised ML by Fernandez et al. (JEE, 2021)

    • Burn et al. (NBER, 2021): develop a ML algorithm to detect whether a job ad is ageist and to find employers more likely to be engaging in age discrimination

    • Kizilcec et al. (PNAS, 2020) use ML to personalize behavioral science interventions in online education (with limited improvements)

    • Stachl et al. (PNAS, 2020) use ML to predict individuals’ Big Five personality dimensions

    • Salganik et al. (PNAS, 2020): hundreds of researchers attempted to predict six life outcomes using ML with a rich dataset. No one made very accurate predictions.

    • Brieland and Töpfer (RePec, 2020) use ML to compute the adjusted gender pay gap. See also Strittmatter and Wunsch (IZA, 2021)

    • Borup and Schütte (JBES, 2021) use ML and Google Trends to predict US employment growth

      • Borup et al. (SSRN, 2021) generate a sequence of now- and backcasts of weekly unemployment insurance initial claims based on Google Trends search-volume data for terms related to unemployment

    • Sajjadiani et al. (Applied Psychology, 2019) use ML to predict teacher performance and retention in Minneapolis using pre-hire work history

    • Beattie et al. (EER, 2018): no improvements from using ML to predict college success and failure. See also Bird et al. (EdWP, 2021)

      • Similar results in Orlov et al. (AEA P&P, 2021) when using LASSO to identify econ undergrad students at risk of underperforming

    • Chalfin et al. (AER P&P, 2016) apply ML for police hiring and teacher tenure decision

  • Poverty and Inequality

    • Bloise et al. (ECINEQ, 2020) use ML to predict parental income in two-sample estimations of intergenerational income mobility

    • Lentz (WorldDevelopment, 2019) predict food security status in Malawi by incorporating granular market data, remotely-sensed rainfall and geographic data, and demographic characteristics

    • Dong et al. (PNAS, 2019) predict neighborhoods’ socioeconomic attributes using restaurant data

    • Brunori et al. (WB, 2018) estimate inequality of opportunity from regression trees

    • Jean et al. (Science, 2016) combine satellite data with ML to predict poverty. See also Yeh et al. (NatureCommunications, 2020), Burke et al. (NBER, 2020; Science, 2021), Aiken et al. (NBER, 2021), Huang et al. (NBER, 2021), Ratledge et al. (NBER, 2021)

      • Mueller et al. (PNAS, 2021) use ML to measure was destruction from satellite images

  • Politics and Policy

    • Gentzkov et al. (Econometrica, 2019) use ML to measure trends in the partisanship of U.S. congressional speeches

    • Yeomans et al. (BDM, 2019) compare computer recommender systems to human recommenders in predicting which jokes people will find funny, and whether people are willing to rely on computer recommender systems

    • Kleinberg et al. (NBER, 2019) discuss the potential advantages of ML in increasing equity

    • Bertrand and Kamenica (NBER, 2018) use ML to measure culture distance between groups in the US

    • Bonica (AJPS, 2018) infers roll‐call scores from campaign contributions using ML

    • Hauser (Wharton, 2018) on combining ML with behavioral economics to reduce cheating

    • Celiku and Kraay (World Bank, 2017) and Musumba et al. (Sustainability, 2021) use ML to predict conflict

    • See also MIT Professor Kim's "Machine Learning and Data Science in Politics" syllabus

  • Health

    • Carrieri et al. (HealthEconomics, 2021) use ML to predict communities at a high risk of vaccine hesitancy

    • Weis and Jacobson (Nature Biotechnology, 2021) build algorithm to identify future high-impact biotech publications. See also summary on MIT News

    • Benetos et al. (IZA, 2021) apply ML to different audio features embedded within chart-topping songs to create an index correlated with survey-based life satisfaction

    • Grogger et al. (NBER, 2020) use ML to predict domestic abuse cases

    • Johnson et al (SSRN, 2020) use ML to target inspections to increase occupational safety and decrease injuries in the workplace

    • Mullainathan and Obermeyer (NBER, 2020) use ML to reduce over- and under-testing for heart attacks

    • Deryugina et al. (AER, 2019) use ML to estimate the life-years lost due to pollution exposure, plus treatment effect heterogeneity

    • Obermeyer et al. (Science, 2019) find evidence of racial bias in one popular health algorithm due to biased historical heath training data. Summary and general discussion by Benjamin (Science, 2019). Also nice NYT article by Mullainathan.

    • Hastings et al. (NBER, 2019; PNAS, 2020) use ML to predict the risk of future opioid dependence, abuse, or poisoning

    • Ribers and Ullrich (ArXiv, 2019) show to what extent ML predictions may improve antibiotic prescribing by predict diagnostic test outcomes for urinary tract infections, while also highlighting that physician expert knowledge is still necessary

    • Kleinberg et al. (AER P&P, 2015) apply ML to predict mortality risk in surgery

    • What to learn from past mistakes in Google Flu (Lazer et al., Science 2014)

  • Business and Finance

    • Naudé et al. (IZA, 2021) discuss a ML competition in Africa

    • Borgschulte et al. (NBER, 2021) use neural-network to assess signs of aging in pictures of CEOs in order to then estimate how exposure to a distress shock during the Great Recession affected CEOs’ apparent age

    • Lommers et al. (ArXiv, 2021) discuss ML applications in finance

      • Na and Kim (EconomicsLetters, 2021) predict stock prices with ML using informed traders' activities

    • Fuster et al. (JFinance, 2020): when applied to the US mortgage market, ML slightly increase credit provision overall, but also increase rate disparities

    • Farrell et al. (AEA P&P, 2020) apply ML in administrative banking data to estimate gross family income

    • Dumitrescu et al. (RePec, 2020): new, simple and interpretable credit scoring method which uses information from decision trees to improve the performance of logistic regression

    • Björkegren and Grissen (WBER, 2019) use mobile phone data to predict credit repayment

    • McKenzie and Sansone (JDE, 2019) use ML to predict successful entrepreneurs in a business plan competition. See also Coad and Srhoj (SBE, 2020), Bryan et al. (NBER, 2021)

    • Andini et al. (VoxEU, 2018) on ML-based targeting in policies aiming at increasing household consumption and access to credit by firms

    • "Algorithms Need Managers, Too" by Luca et al. (HBR, 2016) on the advantages and limitations of using algorithms in business. The gains from ML come from "identifying patterns too subtle to be detected by human observation, and using those patterns to generate accurate insights and inform better decision making"

  • Other

    • Fudenberg et al. (AER, 2019) use machine learning to uncover regularities in the initial play of matrix games


Please email me if you think I am missing some interesting (published) papers.