Vocabulary

2- Program Evaluation Terms

• Analysis of Covariance (ANCOVA): is a general linear model that blends ANOVA and regression. ANCOVA evaluates whether the means of a dependent variable are equal across levels of one or more categorical independent variables and across one or more continuous variables (Harrell, 2015, p. 16). 

• Analysis of Variance (ANOVA): it is a family of statistical methods used to compare the means of two or more groups by analyzing variance. ANOVA compares the amount of variation between the group means to the amount of variation within each group. It's also called k-sample-type model (Harrell, 2015, p. 16).

• Attrition: it is the formal name for research participants' drop out.

• Availability Bias: it arises when effect size estimates obtained from studies which are readily available to the reviewer (in context of meta-analysis) differ from those estimates reported in studies which are less accessible (Ellis, 2010, p. 117).

• Balance Test: it is a test to show that the treatment and control groups are comparable over selected baseline covariates (e.g. age, income, education, etc.) before the treatment (Kerwin, Rostom and Sterck, 2024). 

• Bonferroni Correction: it is a correction of significance threshold level when testing multiple hypotheses. Goal: compensate the fact that in multiple hypothesis testing, the probability of observing a rare event increases (Ellis, 2010, p. 79). 

• Cohen's d index of effect size:  it is the difference between two means divided by a standard deviation for the data (Ellis, 2010, p. 10). 

• Confidence Interval: it is a range of plausible values for the parameter being estimated. Confidence intervals are relevant when making an inference from a sample to a wider population. Adopting a "B" level of confidence interval means that in the long run, (1-B)% of intervals estimated will exclude the parameter of interest. Confidence interval is point estimate of a parameter plus and minus a margin of error (Ellis, 2010, pp. 17-18). 

• Credibility Interval: it is the distribution of effect sizes centered on the mean population effect size. It's width equals standard deviation of the population effect size times z score of the corresponding significance level (Ellis, 2010, pp. 106). 

• Cronbach's Alpha: It is a measure of the internal consistency of a test or scale and it is expressed as a number between 0 and 1. Internal consistency describes the extent to which all the items in a test measure the same concept or construct and hence it is connected to the inter-relatedness of the items within the test (Tavakol and Dennick, 2011). 

• Effect Size: it is the magnitude of the relationship between two variables. Meta analysis combines effect size from different studies to report the overall impact of programs  (Weiss, 1998, p. 241).

• Ethnography: it is the prototypical method of data collection in qualitative studies (Weiss, 1998, p. 256). It involves examining the behavior of the participants in a given social situation and understanding the group members' own interpretation of such behavior.

• Events per variable (EPV): it is the number of observations divided by the number of variables. A rule of thumb for number of observations in logistic and Cox models is to have a minimum EPV of 10. 

• Factorial Design:  it is an experimental design that studies the effects of multiple independent variables (factors) on a dependent variable, examining both the individual effects of each factor (main effects) and how they interact with each other.  

• Fail-Safe N: it is the minimum number of additional studies with conflicting evidence (= null result) that would been needed to overturn the conclusion reached in the review. It described the tolerance level of the results (Ellis, 2010, p 122). 

• Family-wise Error Rate: it is the probability of making one or more false discoveries, or type I errors when performing multiple hypotheses tests (Ellis, 2010, p 78).  

• File Drawer Problem: it is when negative results, those that do not support the initial hypotheses of researchers are often "filed away" and are not published by the researchers (Weiss, 1998, p. 238). The researchers' decision to file away the non-significant results creates a file drawer problem (Ellis, 2010, pp. 117). 

• Fisher's Combined Probability Test: it is used in meta analysis for doing a combined significance test.

• Focus Group: started off as a technique in market research, it is now a data collection method in qualitative studies (Weiss, 1998, p. 260).

• Funnel Plot: it is a scatter plot of the effect size estimates combined in the meta-analysis and is used in identifying the publication bias (Ellis, 2010, p 120). 

• Generalized variance: it is the determinant of the covariance matrix  and generalizes variance for multivariate random variables.  

• Glass's delta for effect size:  (Ellis, 2010, p. 10). 

• HARKing:  An acronym for "Hypothesizing after the results are known". It is what happens when the researcher plays with numbers, finds a statistically significant result, then positions the paper as if that particular result was the original object of the study (Ellis, 2010, p 78).  

• Hawthorne Effect: it is a positive response of individuals due merely to the attention that they receive (Weiss, 1998, p. 219). 

• Hedges' g for effect size:  (Ellis, 2010, p. 10). 

• Hoeffding’s D (Harrell, 2015, p 81): it is a general and robust similarity measure  for continous variables when variables. It is a better option compared with Pearson and Spearman correlation when variables are not monotonically related. 

• Ignorable Missing Data: The missing data mechanism is said be ignorable if it is missing at random and the probability of a missingness does not depend on the missing information itself.  

• Interrater Agreement: It is the degree of agreement among independent observers who rate, code, or assess the same phenomenon. It is calculated as the proportion of number of agreements to total of number of agreements and disagreements (Ellis, 2010, p 101).  

• Margin of Error (ME): The margin of error describes the precision of the estimate and depends on the sampling error in the estimate and the natural variability in the population (Ellis, 2010, pp. 18). The width of ME = SE* (t statistic for N-1 degrees of freedom at chosen level of confidence); where SE = standard error.

• Meta Analysis: it is a method of synthesis of quantitative data from multiple independent studies addressing a common research question with the goal of calculating an overall impact size captured by all the previous studies (Weiss, 1998, p. 236). This method completely ignores the conclusions that the other authors have drawn and looks instead at the effects that have been observed (Ellis, 2010, p 90). 

• Minimum Detectable Effect Size: The minimum detectable effect size is the effect size below which we cannot precisely distinguish the effect from zero, even if it exists. If a researcher sets MDES to 10%, for example, they may not be able to distinguish a 7% increase in income from a null effect. To be clear, MDES is not the effect size we expect or want. However, to select MDES, it is important to consider the expected effect size. 

• Missing at Random: is one of three classifications of missing values offered by Rubin (1976)- when the probability of missing data on a variable Y is related to some other measured variables in the analysis model, but not to the values of Y itself (Enders, 2010, p. 6). It is also called an ignorable non-response and most of the available methods for dealing with missing data assume the data are of this type (Harrell, 2015, page 46). 

• Missing Completely at Random: is one of three classifications of missing values offered by Rubin (1976)- when the probability of missing data on a variable Y is unrelated to other measured variables and is unrelated to the values of Y itself (Enders, 2010, p. 7). It is called an ignorable non-response (Harrell, 2015, page 46).

• Missing Not at Random: is one of three classifications of missing values offered by Rubin (1976)- when the probability of missing data on a variable Y is related to the values of Y itself, even after controlling for other variables (Enders, 2010, p. 8). It is also called an ignorable non-response and informative missing (Harrell, 2015, page 46).

• Multiple Hypothesis Testing:

• Multiple Hypothesis Testing Correction:

• Multiplicity Problem: It arises when studies report the results of multiple statistical tests raising the probability that at least some of the results will be found to be statistically significant even if there is no underlying effect (Ellis, 2010, p 78).  

• Narrative Review: it is using qualitative methods for deriving conclusions about a particular research theme. It summarizes and synthesizes the conclusions of other papers into a narrative about the effect of interest (Ellis, 2010, p 90).  

• Part Correlation (Semi-Partial Correlation): it is the correlation between two variables (independent and dependent) after controlling for one or more other variables. It only accounts for the influence of the control variables on the independent variable, and not on the dependent variable. 

• Partial Correlation: it is the correlation between an independent variable and a dependent variable after controlling for the influence of other variables on both the independent and dependent variable. 

• Participant-Observation:  it is a method of data collection in qualitative studies (Weiss, 1998, p. 257) and it involves observation of a scene by a researcher and who takes part in activities and events.

• Post-hoc Analysis: it is the statistical analyses after observing the resulting data from research. 

• Power: it is the probability of detecting a given effect using a given test in a given context. It is True Positive, divided by Total Positive.

• Pre-Analysis Plan: it is a document produced at the design stage of an impact evaluation that sets out in advance how the researcher will analyze data. 

• Publication Bias: it refers to the selective publication of research studies based on their results; such that studies with positive findings are more likely to be published than studies with negative findings. Positive findings are also likely to be published quicker than negative ones. As a consequence, bias is introduced: results from published studies differ systematically from results of unpublished studies (Weiss, 1998, p. 238;  Ellis, 2010, p. 119). Methods of identification: Funnel Plot, Trim-and-Fill and Egger’s Test. 

• Qualtrics: it is an online survey tool where users can build and distribute surveys, collect responses, and even analyze response data. 

• ROC Curve: it is a graphical plot that illustrates the performance of a binary classifier model by plotting the true positive rate against the false positive rate at each threshold setting. According to Frank Harrell (2015, p 6), ROC analysis is misleading except for the special case of mass one-time group decision making with unknown utilities.

• Sampling Distribution: it is the distribution of a statistic, when derived from a random sample of size n; it is the distribution of the statistic for all possible samples from the same population of a given sample size. 

• Sampling Error: it describes the discrepancy between the values in the population and the values observed in the sample. This error is inversely proportional to the square root of size of the sample. 

• Simulation Analysis:

• Single Imputation Methods: A series of methods for filling the missing values by generating a single replacement value for each missing data point. They include Arithmetic mean imputation, regression imputation, stochastic regression imputation, hot-deck imputation, similar response pattern imputation, averaging the available items, last observation carried forward (Enders, 2010, pp. 42-52). 

• Spline Regression: it is one method for testing non-linearity in the predictor variables and for modeling non-linear functions. It maximises power and only assumes a smooth relationship between the predictor and the outcome variables (Harrell, 2015, p. 21).

• Standard Deviation: it is a measure of the amount of variation of values of a variables around its mean.

• Standard Error: of a statistic (like mean) is the standard deviation of its sampling distribution.  It indicates the uncertainty of a sample-based statistic. It equals the standard deviation of the sample divided by the square root of the sample size.

• Standardized minimum detectable effect (MDE): 

• Statistical Power:

• Standard Score or  Z Score: it is the magnitude of an effect in terms of standard deviation units (Ellis, 2010, p 105).  

• SurveyCTO: it is a scalable mobile data collection platform for researchers and professionals working in offline settings. It is used in the RCTs and program evaluation projects.

• Tower of Babel Bias: its the exclusion of papers from meta analysis or literature review based on their language (Ellis, 2010, p 120). 

• Triangulation: it is the cross-check of data collected through different modes of inquiry (Weiss, 1998, p. 263).

• Type I Error:

• Type II Error:

• Type M Error:

• Type S Error:

• Variance Inflation Factor (VIF): is the ratio (quotient) of the variance of a parameter estimate when fitting a full model that includes other parameters to the variance of the parameter estimate if the model is fit with only the parameter on its own. The VIF provides an index that measures how much the variance of an estimated regression coefficient is increased because of collinearity. 

• Q Statistic:  (Ellis, 2010, p 107).  

• Zero-order Correlation: it is the correlation between two variables (the independent and dependent variable) without controlling for the influence of any other variables. 

3- Economics Methodology Terms

• Arellano and Bond estimator: Linear dynamic panel-data models include p lags of the dependent variable as covariates and contain unobserved panel-level effects, fixed or random. By construction, the unobserved panel-level effects are correlated with the lagged dependent variables, making standard estimators inconsistent. Arellano and Bond (1991) derived a consistent generalized method of moments (GMM) estimator for the parameters of this model. 

• Bonferroni band: it is a type of  simultaneous confidence bands (Montiel Olea  and Plagborg-Møller, 2018).

• Choleski identification assumption: it is an assumption in structural vector autoregressive (VAR) studies of the monetary transmission mechanism that the monetary policy shock does not affect macroeconomic variables contemporaneously.  This assumption amounts to including the contemporanous values of the system variables causally ordered first in exogenous variables (Jorda, 2023, p.618).

• Driscoll-kraay standard error: it is an standard error consistent when the time dimension is large and there exists spatial and temporal dependence across the observations (Driscoll & Kraay, 1998). 

• Inverse probability weighting estimator: it is used in causal inference and econometrics for program evaluation, treatment effects, and panel data models to handle selection bias, missing data, or endogenous treatment assignment. It reweights the sample so that the treated and untreated groups become comparable, as if treatment were randomly assigned. Each observation is weighted by the inverse of the probability of receiving the treatment actually observed. That probability is called the propensity score (Abadie and Cattaneo, 2018).

• Kitagawa-Oaxaca-Blinder (KOB) decomposition: is a statistical method that explains the difference in the means of a dependent variable between two groups by decomposing the gap into within-group and between-group differences in the effect of the explanatory variable. Fortin et al. (2011) provides an extensive review of the KOB decomposition.

• Local projection method: Local projections were introduced by Jordà (2005) as an alternative to traditional impulse response function (IRF) estimation based on vector autoregressive (VAR) models. Local-projection estimation is not constrained by a model and thus provides more flexible impulse–response coefficients. 

• Mahalanobis distance:  is a measure of the distance between a point and a probability distribution. The joint Null hypothesis in a vector of impulse response estimates could be tested with the traditional Wald statistic based on the Mahalanobis distance (Jorda, 2023, p.616).

• Newey-west heteroskedasticity and autocorrelation (HAC) consistent estimator: A Newey–West estimator is used in statistics and econometrics to provide an estimate of the covariance matrix of the parameters of a regression-type model where the standard assumptions of regression analysis do not apply. For cases where the residual serial correlation affects the construction of standard errors, Jorda (2005) offered a semiparametric solution using a HAC estimator (Jorda, 2023). 

• Pointwise error band: it quantifies the potential deviation of a single data point or measurement from its true or expected value. See Simultaneous error band.

• Projection band: it is a type of  simultaneous confidence bands (Montiel Olea  and Plagborg-Møller, 2018).

• Propensity score: 

• Scheffé's S method: it is a method for adjusting significance levels in a linear regression analysis to account for multiple comparisons. It is particularly useful in analysis of variance (a special case of regression analysis), and in constructing simultaneous confidence bands for regressions involving basis functions. Jorda (2009) proposed constructing the individual critical values for the confidence intervals of each coefficient in the local projection method using the Scheffé's S method  (Jorda, 2023, p. 616). 

• Simultaneous confidence bands (Simultaneous error bands): they are used in applied work to visualize estimation uncertainty for vector-valued parameters, such as impulse response functions (Montiel Olea  and Plagborg-Møller, 2018).  They are needed, since the impulse response coefficients are correlated with one another.

• Sup-t confidence band: it is a type of  simultaneous confidence bands, and is narrower than other common bands, like projection and Bonferroni bands (Montiel Olea  and Plagborg-Møller, 2018).

• Vector autoregressive (VAR) models:

• Wold decomposition (Wold representation theorem): it says that every covariance-stationary time series can be written as the sum of two time series, one deterministic and one stochastic. 

4- Remote Sensing/Geographic Information Systems Terms

• Buffering: it is one of the most commonly used proximity functions. A buffer is a region that is less than or equal to a specified distance from one or more features. There are raster buffers, and vector buffers for the data sources of raster and vector types. Vector buffers are of three types: simple, compound, nested and variable distance (Bolstad and Manson, 2022, p.389-393). 

• Coordinate reference system (CRS): it is a coordinate-based local, regional or global system used to locate and precisely measure geographical entities on the surface of Earth. It is also called spatial reference system (SRS). Among the thousands CRS, the two most used are: 1) Geographic coordinate system (or geodetic) (GCS); and 2) Projected coordinate system (or planar, grid) (PCS).

• Albedo: it is the ratio of the energy reflected from a surface to the energy incident on the surface. Dark surfaces have a low albedo, and bright surfaces have a high albedo (Sabins and Ellis, 2020, p. 30).  

• Bidirectional reflectance distribution function (BRDF): it measures the impact caused by different angles of incoming and reflected energy on the interpretation of features in imaginary. Understanding BRDF matters when comparing remote sensing images acquired on multiple dates with different illumination and viewing angles (Sabins and Ellis, 2020, p. 20).  

• Datum: a Geodetic datum or reference frame is an abstract coordinate system with a reference surface (such as sea level) that serves to provide known locations to begin surveys and create maps. Common datums for North America are NAD27, NAD83, and WGS84.

• Datum transformation: it is the calculation of the change in geographic coordinates when moving from one datum to another.

• Earth ellipsoid: it is a mathematical figure approximating the Earth's shape and size, used as a reference frame for computations. The latitude and longitudes are measures on earth ellipsoid in GIS and remote sensing (unlike elevation measurement that is based on the geoid). 

• GDAL: it stands for "Geospatial Data Abstraction Library" and it is the core open-source library that almost all GIS software relies on. 

• Geographic coordinate system (or geodetic) (GCS): it is a popular type of Coordinate reference system (CRS). In GCS, latitude and longitude coordinates are angles measured from the earth's center to a point on the earth's surface and the reference grid. One of the most common GCS projections is EPSG:4326. 

• Geoid: it is the earth's true shape, and is a sort of lumpy ellipsoid. Measurement of elevations in GIS and remote sensing are relative to the geoid (unlike latitude and longitude that are based on the earth elipsoid).

• Global navigation satellite system (GNSS): it is a sysetm for collecting accurate horizontal and vertical coordinates for moving or fixed platforms. There are multiple versions: GPS of the US, GLONASS of Russia, Galileo of the European Uninon, and BeiDou of China (Sabins and Ellis, 2020, p. 15).  

• Great-circle distance: it is the distance between two points on a sphere, measured along the great-circle arc between them. This arc is the shortest path between the two points on the surface of the sphere.   

• Heading (yaw) angle: it is one of three dimensions of movement when an object moves through a medium. It is when the nose moves from side to side. See also pitch and roll angles.

• Lambert conformal conic: it is one of the most common projection types used for spatial data in North America and much of the world. It has a low-distortion band running in an east-west direction between the standard parallels, which makes the Lambert conformal conic projection common for mapping areas that are larger in an east-west direction (Bolstad and Manson, 2022, p.122).

• National Spatial Reference System (NSRS): it is a consistent coordinate system that defines latitude, longitude, height, scale, gravity, and orientation throughout the United States.

• Non-selective scattering: it is one of two types of atmospheric scattering, in which all wavelenghts of light are equally scattered. Dust, fog and clouds cause non-selective scattering. See also selective scattering (Sabins and Ellis, 2020, p. 30).  

• North American Datum of 1983 (NAD83): 

• North American Vertical Datum of 1988 (NAVD88): 

• Overlay: it is the process of combining the spatial and attribute data from two or more spatial data layers. In doing so, the spatial datasets are stacked and merged together. Three ways of applying the overlay functions are clip, intersection and union (Bolstad and Manson, 2022, p.394-402).

• Pitch angle:  it is one of three dimensions of movement when an object moves through a medium. It is when the nose moves up or tail moves up. See also yaw/heading and roll angles.

• Projected coordinate system (or planar, grid) (PCS): it is a popular type of Coordinate reference system (CRS). It is defined on a flat, two-dimensional surface. A commonly used PCS is the Universal Transverse Mercator (UTM).

• Public Land Survey System (PLSS): it divides lands by north-south lines, 6 miles apart, running parallel to a principal meridian. East-west lines are perpendicular to these north-south lines, also at six miles intervals. Each township (=square?) is subdivided into 36 sections, each section approximately one mile on a side. Each  small square is further divided to quarter sections or six-tenth sections (half a mile on a side and quarter a mile on a side, respectively). Sections are numbered in a zigzag pattern from one to 36, beginning in the northeast corner.  (Bolstad and Manson, 2022, p.132). Many road intersections in the US occur at PLSS corner points (same).

• Relief displacement: it is scale variation on aerial photographs caused by variations in terrain elevation.  

• Remote sensing: it is the science of acquiring, processing and interpreting images and related data, that are typically acquired from aircraft and satellites with sensor systems that digitally record the interaction between electromagnetic energy and matter (Sabins and Ellis, 4th edition, 2018, p. 1).

• Roll angle:  it is one of three dimensions of movement when an object moves through a medium. It is thea circular (clockwise or anticlockwise) movement of the body as it moves forward . See also yaw/heading and pitch angles.

• Selective scattering: it is one of two types of atmospheric scattering. It is when the shorter wavelengths of UV energy and blue light are scattered more severely than the longer wavelengths of red light NIR energy. Fumes and gases like nitrogen, oxygen and carbon dyoxide cause selective scattering. See also non-selective scattering (Sabins and Ellis, 2020, p. 30).  Because of this, earth's atmosphere scatters UV and blue wavelenghts at least twice as strongly as red light (p. 31).

• Standard parallels: The cone in the Lambert conformal conic intersects the ellipsoid along two arcs, typically parallels of latitude. These lines of intersection are known as standard parallels (Bolstad and Manson, 2022, p.122).

• State plane coordinate system (SPCS): it specifies positions in Cartesian coordinates systems for each state in the United States. There are one or more zones in most states, with slightly different projection parameters in each State Plane zone. This multiplicity are intended to limit distortion errors due to map projections. Most states have adopted zones such that  projection distortions are kept below 0.0001; with a few states like Monata and Nebraska allowing larger distortions for the sake of having only one state plate zone. There has been multiple versions of this system: NAD27, followed by NAD83, and NATRF2022. For US maps created before January 2023, notice that the foot to meter conversion was different from all other countries: In the US, it was 1 foot = 0.304800609612 meters; while in other countries is it 1 foot = 0.3048 meters. US States have adopted the international convention beginning in 2023  (Bolstad and Manson, 2022, pp.125-126).

• Transverse Mercator: it is one of the most common projection types used for spatial data in North America and much of the world (Bolstad and Manson, 2022, p.122). Because distortions in a transverse Mercator increases with distance from the central meridian, this projection type is most often used with states or zones that have a long north-south axis (Bolstad and Manson, 2022, p.125).

• Universal Transverse Mercator coordinate system (UTM): it is a global coordinate system. It divides earth into zones that are 6 degrees wide in longitude (so each goes from 0 to 60 degrees horizontally), and extends from 80 degrees south to 85 84 degrees north attitude (vertically). Along the X axis, each point lies between 0-60 degrees North or South, depending on locating in the northern or the southern hemisphere. Distances are specified in meters north and east of the zone origin.  The origin of coordinates are different for north and south of equator.  Locations in the southern hemisphere have a false northing, to report positive values of their latitude  (Bolstad and Manson, 2022, p.128). UTM is not always compatible with regional analysis, because coordinate values are discontinuous across UTM zone boundaries. When doing regional analysis in the US, use instead a continental projection like Albers Equal Area Conic or Lambert Conformal Conic.