Treatment Evaluation

Treatment Evaluation

Last update: 2020/01/14 —— Starting with summer term 2020 Nicole Schneeweis will offer these courses.

The course schedule can be found in the syllabus.

Objective

This course is intended for graduate students with an interest in econometric methods for the analysis and evaluation of (public) policies. The main target audience are students who pursue a career in public sector, international organizations, research institutions or academia; but also for those seek to find employment as a research associate with consulting firms. The course will teach students the theoretical and quantitative concepts needed to evaluate a wide range of policies. The emphasis of the Course (CS) is on methods, and the Intensifying Course (IC) focuses on applications.

Topics

  1. Introduction: The evaluation-problem

  2. Rubin causal model (RCM)

  3. Randomized control trials (RCT)

    • Encouragement design vs. LATE

    • External validity

    • Ethical concerns

    • Social-experiments

    • Field-experiments

    • Power calculations

  4. Natural-experiments (standard tools)

    • Exact matching (ExM)

    • Propensity score matching (PSM)

    • Fixed-effects estimation (FE)

    • Difference-in-differences estimator (DiD)

    • Instrumental variable (IV) approach

    • Regression discontinuity design (RDD)

Prerequisites

This course builds on the courses Empirical Economics and Intermediate Econometrics. In particular, students should be familiar with basic probability theory, mathematical statistics, and multivariate regression analysis (OLS). These topics are covered, e.g., in the Appendix A-C and chapters 2-8 in Wooldridge (4th edition), Introductory Econometrics (2003).

Grading and Attendance

Your grade in the CS is determined by a homework (15%), a classroom presentation (20%), and Part A of an article report (65%). The homework is one page on a media report of your choice. More details will be porvided in class. The classroom presentations will take place in the last four meetings of the CS and will be scheduled ahead. The article report is due by date tba and has to be submitted to ursula.mittendorfer-juen@jku.at. Please find more details on the article report below. Your grade in the IC is determined by several smaller homework exercises (25%), and Part B of the article report (75%).

Students are expected to read all assigned reading. You find a reading list below, and the time-table informs you on which date which material is covered. Be prepared to answer questions. Further I expect you to read at least the introduction of all articles which will be presented by your fellow students.

Participants are asked to be present at every meeting. Regular attendance of the Faculty's Empirical Economics and Econometrics Seminar (link) is highly recommended.

Article Report and Classroom Presentation

Each student has to prepare a classroom presentation of her/his article and write an article report. The classroom presentation (and the article report) should contain:

  • a concise statement of the evaluation (outcome, treatment, theory of change, hypothesis)

  • a brief discussion of related literature and existing evidence

  • a short discussion of the institutional background

  • a summary of how the author goes about achieving those goals (data, evaluation method)

  • a critical discussion of the identifying assumptions (critiques, praise, open questions)

  • a short summary of the results a

    • focus on the main results of the article; do not go over every robustness check & every specification

  • a discussion of potential policy implications

  • a summary of your replication (Part B)

The presentation should be 25 minutes. Note, I will enforce a strict time limit. Following each presentation, we will have a short discussion (where the audience is asked to focus on the identification and the policy implications). Here you can download the schedule: link.

The article report consist of two parts. Part A one should be 5 pages (i.e. about 2,200 to 2,400 words). Make sure the document you hand-in, has your contact details, the word count, and page numbers. Part B reports on your replication exercise. Please submit also your log-file.

Required Readings

  • Gertler, P.J., Martinez, S. Premand, P. Rawlings, L.B. and Vermeersch, C. M. J. (2011). Impact Evaluation in Practice. Washington, DC: The World Bank. (link)

  • Angrist, J.D. and Pischke J.-S. (2010). The Credibility Revolution in Empirical Economics: How Better Research Design is Taking the Con out of Econometrics. Journal of Economic Perspectives 24(2): 3-30. (link)

  • Heckman, J.J. and Smith, J.S. (1995). Assessing the Case for Social Experiments. Journal of Economic Perspectives 9(2): 85-110. (link)

  • Angrist, J.D. and Pischke J.-S. (2008). Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press, Princton, NJ.

  • Card, D.E. and Krueger, A.B. (1994). Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania. American Economic Review 84(4), 772-793. (link)

  • Angrist, J.D. (2005). Instrumental Variables Methods in Experimental Criminological Research: What, Why and How. Journal of Experimental Criminology 2, 1–22. (link)

Further Suggested Readings

  • On RCTs

    • Galeman, A. and Weakliem, D. (2009) Of Beauty, Sex and Power. American Scientist 97(4), 310-316. (link)

  • On IVs

    • Angrist, J.D. and Krueger, A.B. (2001). Instrumental Variables and the Search for Identification: From Supply and Demand to Natural Experiments. Journal of Economic Perspectives 15(4), 69-85. (link)

    • Deaton, A. (2010). Instruments, Randomization, and Learning about Development. Journal of Economic Literature 48(2), 424-455. (link)

    • Imbens, G. W. (2010). Better LATE than nothing: Some comments on Deaton (2009) and Heckman and Urzua (2009). Journal of Economic Literature 48(2), 399-423. (link)

  • On RDD

    • Becker, S.O., Egger, P. and von Ehrlich, M. (2010). Going NUTS: The Effect of EU Structural Funds on Regional Performance. Journal of Public Economics 94(9-10), 578-590.

    • Cook, Thomas D. (2008). "Waiting for Life to Arrive'": A History of the Regression-Discontinuity Design in Psychology, Statistics and Economics. Journal of Econometrics 142(2), 636-654.

  • Other

    • Manski, C. (2012). Public Policy in an Uncertain World: Analysis and Decision. Harvard University Press.

Additional Material

Problem Sets for IC

How to Find an Article?

The article report covers an article of your choice. This article should be published recently in a peer-reviewed economic journal (see a list of suggest journals below), focus on an empirical evaluation of a policy/program from and the data used has to be available for replications.

The main evaluation method should be either a social experiment, a field experiment, or a natural experiment. The latter ist defined by the usage of a difference-in-differences estimator, a instrumental variables estimation, a regression discontinuity design, or a combination of these.

Please send me an email not later than date tba with a suggestion. I will check whether your suggested article is suitable, and ensure that we have a sufficient mix of methods across presentations. List of accepted submissions: link

List of Journals

Try to find an article which is published rather recently (or forthcoming) in one of the following journals:

  • General interest (first tier): American Economic Review , Quarterly Journal of Economics, Journal of Political Economy, Econometrica, Review of Economic Studies

  • General interest (second tier): American Economic Journal: Applied Economics, American Economic Journal: Economic Policy, Economic Journal, European Economic Review, Journal of the European Economic Association, Review of Economics and Statistics

  • Leading field journals:

    • Labor: Journal of Labor Economics

    • Health: Journal of Health Economics

    • Public finance: Journal of Public Economics

    • Development: Journal of Development Economics

    • Economic History: Journal of Economic History