Embedded Files
Post date: Aug 26, 2012 5:3:4 PM
The concept of validity applies to both whole studies (often called inference validity) and the measurement of individual variables (often called construct validity).
Inference Validity
Inference validity refers to the validity of a research design as a whole. It refers to whether you can trust the conclusions of a study. Generally the two key issues are causality and generalizability. Statistical measures show relationships, but it is the theory and the study design that determine what kinds of claims to causality you can reasonably make and what you can make them about.
1. Internal validity (largely interpretability)
Refers to whether claimed conclusions, especially relating to causality, are consistent with research results (e.g., statistical results) and research design (e.g., presence of appropriate control variables, use of appropriate methodology).
An obvious example of failing internal validity is when a researcher misinterprets a statistical result. For example, in ordinary regression, you want a significant r-square, as this implies that knowing the scores on the independent variables helps predict the dependent variable. But in log-linear models, you want the chi-square test to be non-significant, because it means the model fits -- the predicted values are not significantly different from the observed values. Going from one kind of models to other, it is easy to make a mistake and misinterpret the meaning of the significant chi-square.
Another example is when a researcher hypothesizes a mediation argument but tests a moderation argument. The result is that the conclusions they make about mediation are not based on an adequate test.
Internal validity can sometimes be checked via simulation, which can tell you whether a given theorized process can in fact yield the outcomes that you claim it does.
2. External validity (generalizability)
Refers to generalizability of the results. Does it say anything outside of the particular case? i.e., in your study of 150 workers in a consulting company's IT department, you find that the more central they are in the friendship network, the better they do their jobs. To what extent can you say this is true of other workers?
A carpenter, a school teacher, and scientist were traveling by train through Scotland when they saw a black sheep through the window of the train.
"Aha," said the carpenter with a smile, "I see that Scottish sheep are black."
"Hmm," said the school teacher, "You mean that some Scottish sheep are black."
"No," said the scientist glumly, "All we know is that there is at least one sheep in Scotland, and that at least one side of that one sheep is black."
Three strategies for strengthening external validity:
Sampling. Select cases from a known population via a probability sample (e.g., a simple random sample). This provides a strong basis for claiming the results apply to the population as a whole.
Representativeness. Show the similarities between the cases you studied to a population you wish your results to be applied to, then argue that the correlations you will found in your study will also be similar
Replication. Repeat the study in multiple settings. Use meta statistics to evaluate the results across studies. Although journal reviewers don't always agree, consistent results across many settings with small samples is more powerful than a large sample of a single setting.
Construct Validity
Construct validity refers to the validity of a variable that is being measured. There are many subtypes that have been defined. One should not get too hung up on the exact terminology used because there is a lot variation in usage. The breakdown below is Trochim's version.
Page updated
Google Sites
Report abuse