October 12: Methods & Measurement

Abstract: Psychological constructs like goal-setting and self-efficacy are increasingly recognized as important for explaining human behavior. Yet reliable measures either don't exist or have only been validated in high-income countries. Our paper introduces 4 new scales, which we developed and validated across multiple parts of sub-Saharan Africa. The four scales are a new Goal-Setting Capacity scale, a Generalized Efficacy Livelihoods scale aimed at measuring efficacy applicable to general economic activities, an Agricultural Self-Efficacy scale aimed at capturing relevant constraints for farmers, especially women, and a new Locus of Control scale, which unlike prior scales, aims to distinguish who might be the powerful others within the external locus subscale.

Abstract: Evidence suggests that soft skills, such as interpersonal skills, personality traits, self-awareness, and personal initiative may impact future economic outcomes. Yet a lack of consensus on how to define skills and a dearth of longitudinal data presents questions around which skills predict later outcomes and how to effectively measure them. In this systematic review, we attempt to address these questions by compiling and analyzing all the known evidence, including ongoing research and seven commissioned studies, on the predictive validity of soft skills measures on labor market outcomes. We summarize the existing evidence, identify gaps in the literature, provide advice for the field, and identify which skills and instruments predict labor market outcomes and should be prioritized for future use. After eliminating hundreds of candidate studies via title and abstract analysis, this review includes 41 studies published between 1990 - 2022, encompassing a number of soft skills and measurement types. While the goal is to apply lessons learned to low- and middle-income countries, this review is unconstrained geographically, in order to cast the widest net. In addition, this review includes lessons learned from seven parallel studies that were commissioned by Innovations for Poverty Action to better understand the predictive validity of various soft skill measures on longer-term economic empowerment outcomes. The studies were conducted in the United States by International Youth Foundation, Algeria by World Learning, Peru by the University of Toronto, Tanzania by the World Bank Gender Innovation Lab, Uganda by the University of California-Berkeley, and South Africa by Oxford University and the World Bank. Each study used different survey instruments and methods of data collection, but the research questions and statistical approaches were harmonized.

Abstract: Despite the importance of socio-emotional skills (SES) for men and women’s economic empowerment, the lack of measures validated in LMICs has inhibited policy research on which skills matter most. Most studies of SES training to date use a list of commonly used, non-representative measures. Here, the World Bank Africa Gender Innovation Lab and Innovations for Poverty Action have partnered to develop, adapt, and test measures of 14 skills that span the range of SES. After translation and cognitive interviews, measures were adapted and tested in baseline studies from 5 countries: Nigeria, Tanzania, Rwanda, Congo, and Cote d’Ivoire. Results demonstrate that the latest iteration of these measures meet key psychometric thresholds for reliability, exploratory factor analysis, and confirmatory factor analysis. We also examine the relationship of each skill measure with employment, income, and gender. We will also share the results of a second paper examining the gender gap in skills found for a randomized control trial in Tanzania. Here, both self-reported scales and behavioral measures were used for each of the same set of 14 skills. We examine the role played by education, cognitive ability and social desirability in explaining the observed male advantage in self-reported measures across skills. In addition, we show that this gender gap is not observed when behavioral measures are used for the same skills. Both papers appear to confirm the gender similarities hypothesis on levels of SES. The team will also share forthcoming concurrent and predictive validity results on gender differences in returns to SES, as well as key practical takeaways for others pursuing research on SES.

Abstract: Social disruption occurs when a policy creates or destroys many network connections between agents. It is a costly side effect of many interventions and so a growing empirical literature recommends measuring and accounting for social disruption when evaluating the welfare impact of a policy. However, there is currently little work characterizing what can actually be learned about social disruption from an experiment in practice. In this paper, we consider the problem of identifying social disruption in a RCT design that is popular in the literature. We provide two sets of identification results. First, we show that social disruption is not generally point identified, but informative bounds can be constructed using the eigenvalues of the network adjacency matrices observed by the researcher. Second, we show that point identification follows from a theoretically motivated monotonicity condition, and we derive a closed form representation. We apply our methods in two empirical illustrations and find large policy effects that otherwise might be missed by alternatives in the literature.

Abstract: Despite advances in our understanding of quasi-experimental methods, there will likely remain demand to evaluate programs using observational methods like regression and matching. To evaluate the observational bias in these methods we collected data from a large number of RCTs with imperfect compliance (ICRCTs) conducted over the last 20 years. We create comparable observational and experimental estimates of treatment effects, and use these to estimate bias in each study. We then use meta-analysis to quantify the average direction of bias and uncertainty about its size. We find little evidence of average bias but large uncertainty. We suggest adjusting standard confidence intervals to take this uncertainty into account. Our preferred estimates imply that a hypothetical infinite N observational study has an effective standard error of over 0.16 standard deviations and hence a minimal detectable effect of more than 0.3 standard deviations. We conclude that -- given current evidence -- observational studies cannot be used to provide information about the impact of many programs that in truth have important policy relevant effects, but that collecting data from more ICRCTs may help to reduce uncertainty and increase the effective power of observational program evaluation.

Abstract: We apply bunching methods and exploit a minimum wage scheme to estimate the wage elasticity of labor supply for agricultural workers in Malawi. Our data comprises daily output and fortnightly payroll data spanning nine years for 10,000 piece-rate workers, at a company that enforced a minimum wage. Workers face a kinked budget constraint in this setting: their earnings are flat as a function of daily effort until they hit a minimum output level. We use this kinked budget constraint and draw on insights from the public finance literature on bunching in response to tax rates to estimate the elasticity of worker effort with respect to wages. Consistent with the theoretical model behind this approach, our preliminary results (show below) reveal that there is substantial missing mass in the output distribution, centered around the minimum wage cutoff. The results from our study will shed light on the impact of minimum wage policies on agricultural labor supply in developing countries, and our labor supply elasticity estimates will help inform macroeconomic models of developing countries.

The mission of the Agricultural Technology Adoption Initiative (ATAI), co-managed by CEGA and J-PAL, is to rigorously test programs that increase farmer welfare through the broader use of productive technologies in South Asia and sub-Saharan Africa. We aim to generate a rigorous evidence base that helps carefully identify whether particular approaches are successful in spurring agricultural transformation. The ATAI Data Portal (atai-data.org) maximizes the value of this evidence base by providing access to harmonized datasets from the initiative portfolio. Because ATAI funds different principal investigators at different points in time, they employ a number of formats, coding schemes, and documentation for their work.  ATAI has undertaken the complicated work of harmonizing or making uniform this dataset portfolio to facilitate analysis across all ATAI-funded research projects. Additionally, ATAI links all datasets with a standard set of geographic variables to enable innovative geographical analysis. Read more about the ATAI Data Portal here.

Abstract: This study compares the magnitude of the global well-being losses that the COVID-19 pandemic inflicted for the 122 on the most-populous countries over three dimensions: loss of life, loss of income, and loss of learning. The well-being consequences of excess mortality are expressed in years of life lost while those of income losses and school closures are expressed in additional years spent in poverty. Our estimates of the well-being losses from each source are substantial: an average person lost 15 days of life, spent an additional 19 days in poverty due to income losses, and may spend an additional 29 days in poverty due to school closures – if this lost learning is not remediated. While the 2020-2021 period witnessed the largest one-year increase in global poverty in many decades, widespread school closures may lead to even greater years life spent below the poverty line. Most high-income countries suffered more years of life lost than additional years in poverty, while the opposite situation held for many low- and middle-income countries. Overall, high-income countries suffered lower total well-being losses than lower-income countries, unless one year of life lost is valued at least as much as 7 years in poverty.

Abstract: Production of administrative data represents a core task of many bureaucracies. Globally, much of the data collected and used by national (central) governments to target resources and make policies is collected and reported by local (decentralized) government bureaucrats. Such data collection is subject to two important agency problems: one between political principals and bureaucrats who collect data within local governments and one between national and local governments. This paper first documents the incentives of local bureaucrats who compile and submit data for three distinct decentralized data collection processes in Colombia: a means-testing system for social programs, a program for distributing natural resource royalties, and the national contracts databases. An original survey of three bureaucrats per local government (one per data collection process) documents substantial within-local government variation in bureaucratic incentives as well as between-program and between-locality variation in national-local government relations. Analysis of the administrative data produced by these bureaucrats provides a mapping between these incentives and bureaucrats' reporting behavior. These analyses show how bureaucratic incentives shape the accuracy of administrative data and offer new diagnostic tools to national governments and researchers who use on these data.

Abstract: There is a growing literature focusing on measurement error in self-reported data. We study the case of women’s agency. Women might misreport their agency due to fear of retaliation or social desirability biases. We conducted a survey experiment in rural Guatemala where all groups received the same questions about women’s agency as part of a phone survey, but we vary the privacy of their responses. Women in the control group answered verbally. Those in the first treatment group answered using code words, allowing for enhanced privacy as the respondent does not risk disclosing potentially sensitive information to individuals in her surroundings. The second treatment group responded using the phone keypad. These women do not risk disclosing potentially sensitive information to individuals in their surroundings or to the enumerator because the enumerator only hears the tone made by the keypad. We find that women answering under higher levels of privacy reveal much lower rates of agency. We provide evidence that is consistent with social desirability bias.

Abstract: Researcher pre-specified experimental product quantity is the status-quo when conducting willingness-to-pay (WTP) estimates, resting on the assumption of economic rationality. Yet, recent consumer hypothetical studies have found experimental quantity can impact marginal bidding behavior. This study investigates if, and to what degree, varying bid quantity in WTP elicitation impacts marginal WTP via a non-hypothetical field experiment using 527 bean farmers in rural Zimbabwe. Farmers were randomly assigned to either a fixed quantity group (FQG) where farmers bid on a 2kg seed pack or a variable quantity group (VQG) where farmers’ experimental quantity was matched to their intended purchase quantity. Preliminary results find that, on average, marginal WTP is 55% higher in the FQG than the VQG and increases with highly differentiated bean seeds. We find evidence that this difference in WTP across treatment groups is due to a behavioral bias that can arise from mental budgeting, if intended purchase quantity is above the experimental bid quantity used. These results point to the need for researchers to think critically about the experimental quantity used when designing input-based producer WTP studies, setting the experimental quantity to intended purchase quantity, when possible, to avoid potential overestimation of WTP bids. Differing marginal WTP results from the experimental bid quantity can have major implications to the pricing strategies of agro-dealers, costs to international organizations and NGOs trying to launch a new potentially socially desirable agricultural input, or the decision for potential government price-level interventions for smallholder farmers.

Abstract: Measuring work is crucial for policy making, especially in low-income countries where informal work represents a high share of total employment. Despite the relevance of the informal work in the labor market, many surveys rely on questions designed to capture traditional types of employment, and do not use appropriate screening questions to encompass all types of activities, possibly leading to the undermeasurement of the activities performed by individuals engaging in atypical work. Moreover, data on youth and women may disproportionally suffer from ‘proxy response’ bias because the respondent may accurately report their own activities, but under‐ or overreport activities of other household members. This paper provides experimental evidence to overcome these limitations and to improve data collection on work using data from 1,008 households and 2,480 individuals aged 15 to 64 years living in rural and peri-urban areas in El Salvador. We designed a methodological experiment to evaluate how a screening list of activities and the acceptance of proxy responses affect the measurement of work. We randomly assigned the 1,008 enrolled households into two treatment arms and a control group. One-third received the standard labor module for which proxy responses were accepted, preceded by a set of screening questions including our list of work activities; another third participated in a traditional survey where self-responses were required; and the last group (control group) was surveyed using the standard labor module for which proxy responses were accepted. Our results indicate that female participants surveyed with the list of activities were 12 percent more likely to report formal work relative to male respondents surveyed with the same method. Moreover, young men who self-responded were between 15 and 19 percent more likely to report being employed or working in the formal sector, compared with older men who were interviewed using the same method.

Abstract: This research studies the extent to which teachers are able to accurately perceive the success of their students on standardized testing. As part of a larger RCT, we ask Ugandan secondary school teachers to estimate the share of the students at their school who received each mark from 1 (best) to 9 (worst) on two high-stakes national examinations, UCE (O-Level) and UACE (A-Level). Crucially, we collected these data after the examinations were sat by students, but before the results were released. We also collect the administrative data from the Uganda National Examinations Board (UNEB). We are thus able to link the estimated distribution of test scores with the actual distribution of student test scores. We observe first that teachers tend to vastly overestimate the performance of students in their school relative to how the students actually perform, with the average “estimated” score being 2-3 scores better than the empirical average for that school. We also find that teachers tend to overestimate the performance of girls on the history exam more than boys on the same exam. Conversely, the teachers overestimate the performance of boys on physics exams more than girls on the same exam. Although we are not powered to detect changes for just history and physics teachers, we do not observe evidence which suggests that knowledge of students in those specific subjects mitigates the gendered perception of student performance, as physics teachers further overestimate boys’ performance in physics exams, while history teachers appear to do the same. Finally, we compare this measure of gender bias with a self-reported measure where we ask teachers whether they believe boys are better in science subjects than girls and the reasons they believe so.

Abstract: Using data collected as part of a large randomized program evaluation in Namibia, we investigate how cattle weight measurements vary across two measurement methods (farmer estimated and scale weights). We find that self-reported cattle weights are biased downwards and are less precise than objective scale measures. We then show that the program intervention itself led to an increase in the bias of self-reported measures, and that relying only on these measures would have resulted in an incorrectly estimated treatment effect. We describe certain conditions and respondent characteristics for which self-reported weights may be less biased, and suggest solutions for alternative measurement methods that are both more accurate than self-reports and more feasible than acquiring scale measures in the field.