Embedded Files
Note. From Reliability and validity, by Nevit Dilmen, 2012, Wikipedia (https://en.m.wikipedia.org/wiki/File:Reliability_and_validity.svg). CC BY-SA 3.0 DEED.
What is validity?
What is validity?
Validity measures the accuracy of what you are talking about. Not a matter of yes or no but of the degree to which (i.e. are the shots scattered or clustered closely on the target?).
Reliability in contrast, is the measure of what you intended on talking about or if your constructs are consistent (i.e. are your shots close to the targets bullseye or not?).
Construct validity is an "umbrella" term for the general accuracy of your research (Trochim, 2006, construct validity page). Combining both reliability and validity, "how well a given measurement scale is measuring the theoretical constructs that it is expected to measure" (p.37)
(Bhattacherjee, 2012)
Types of validity
Types of validity
Outside of construct validity several types include:
Internal-related to causality measuring the statement ‘if X happens then Y occurs’.
External- how generalizable is your research to a larger population, other places, or different time.
Statistical conclusion- the degree of relationship between independent and dependent variable measured with a statistical test is correct.
Translational- how well a theoretical construct is interpreted to an operational measure.
Face- an indicator is measuring it's 'face value or 'underlying construct'.
Content- set scale matching relevant content that it's trying to measure.
Criterion-related- based upon empirical observations, degree a construct relates to on or more external factor. Assessed with current or future criteria/ factors.
Convergent- closeness with which a measure relates to the construct it is meant to measure.
Discriminant - degree to which other constructs are not measured
(Bhattacherjee, 2012)
Experimental validity
Experimental validity
Experimental designs use a control group and treatment group(s) to ensure causality and reduce maturation threat (Meltzoff & Cooper, 2018). True experiments use random assignment (Bhattacherjee, 2012) and large populations to ensure saturation of the data is discovered (Meltzoff & Cooper, 2018).
Generally, this design is high in internal and low in external validity as variables are controlled (Bhattacherjee, 2012). If details on how the research was conducted, controlled, or analyzed were missing, then the conclusions would be “highly suspect”, and the research would be unusable (Bhattacherjee, 2012, p. 92). Attempting to control internal and external validity through modifying/ controlling variable, leads to “inevitable tension” (Del Boca & Darkes, 2007, p. 1049). Decreasing the internal validity increases the external validity and vice versa. However, there is a ‘sweet spot’ or rather a sweet ‘cone’.
The Cone of Validity are designs that fall within a range of both high amounts of internal and external validity (Bhattacherjee, 2012, p. 36). “[F]ield experiments, longitudinal field surveys, and multiple case studies or repeated lab experiments fall within this cone (Bhattacherjee, 2012, p. 36; Del Boca & Darkes, 2007). By repeating your experiment and receiving consistent results or following up with your experiment to assess the results a year later, both validate your research accuracy (Del Boca & Darkes, 2007)
Not using a workable theory in your experimental design would also render your research useless as the construct validity could be low or the outcomes have little meaning to real life (Bhattacherjee, 2012, p.92). Experimental designs within social sciences often present several difficulties in experimental design including ethical dilemmas, external threats, complex constructs, multiple treatment modalities, organization or practical concerns, and “constraints opposed by block and stratification” (Del Boca & Darkes, 2007).
Research Methods and Statistics, 2016
CRC Clinical Reseach Center, 2020
By: Rachel Taylor, 2020
Threats to validity
Threats to validity
Regression threat- “statistical phenomenon” if you choose the lowest or highest pretest scoring population and not randomly select, the group you choose would drift closer to the “population mean”, or assumed average which would be false (Bhattacherjee, 2012; Meltzoff & Cooper, 2018;Trochim, 2006)
Multiple group threat- If the groups are not comparable from the start as we assume the probabilistic equivalence of the sample groups without testing (Trochim, 2006)
History threat- factors that occurred in the past or out of your control that influence the sample population (Bhattacherjee, 2012;Trochim, 2006).”General” to all groups involved or “local” to one selected group (e.g. the season you completed each groups testing being different, and the climates or amount of daylight effecting the scores; Meltzoff & Cooper, 2018).
Maturation threat- Normal or natural growth and development of disease, people, or a topic (Bhattacherjee, 2012; Meltzoff & Cooper, 2018;Trochim, 2006)
Testing threat- Occurs with pre and post test designs, where the pretest influences or prepares the study population to the research and alters the outcome (Bhattacherjee, 2012; Meltzoff & Cooper, 2018;Trochim, 2006)
Multiple treatment threat- the first treatment could have affected the second (Meltzoff & Cooper, 2018).
Instrumental threat- If the design is altered during the study (Bhattacherjee, 2012) or the people coding the results start to drift from their code overtime with being board or from bias, then the results are less valid (Meltzoff & Cooper, 2018; Trochim, 2006)
Attrition or mortality threat- People quitting the study reduces your population size and the people leaving may be situational (e.g. drop outs scored low on the pretest leading to regression threat and altering the posttest average) (Bhattacherjee, 2012; Meltzoff & Cooper, 2018;Trochim, 2006)
Social threats
Hawthorne effect- is the situation were participants behaviour changes just by being a part of a study (Click here for more details) referred to as reactive arrangement (CRC Clinical Reseach Center, 2020).
Compensatory equalization of treatment threat- the moral dilemma that the researchers may feel when not giving the treatment to the one group and they may treat the control group differently or assist them in another way (Trochim, 2006)
Resentful demoralization- control group becoming resentful against the researchers of treatments group and giving up risking and effect on the post test results (Trochim, 2006)
Compensatory rivalry threat- when the control group becomes competitive and “compensates” for their lack of treatment so may try harder (Trochim, 2006). Click here to learn about the relatable John Henry effect.
Diffusion or imitation of treatment threat- when the control and treatment groups share knowledge of the treatment or research, and the control group may change their behaviour (Trochim, 2006)
How to avoid threats
How to avoid threats
By blinding the study sample or double blinding both the sample and the researchers you can reduce social threats. Blinding entails not knowing if the person is receiving the treatment or not (this can be difficult with social sciences; Meltzoff & Cooper, 2018)
By adding a control group you can reduce threats like history, maturation, testing, instrumentation, mortality, and regression (Trochim, 2006).
By adding randomization, you can reduce threats of maturation and history (Bhattacherjee, 2012; Meltzoff & Cooper, 2018).
By not pretesting, pretesting all groups, or using the Solomon four group method you can reduce testing threat (Meltzoff & Cooper, 2018)
By not selecting the low or high scorers of your pretest for your sample population will reduce the regression threat (Bhattacherjee, 2012; Meltzoff & Cooper, 2018, Trochim, 2006).
Two group extended repeated measures helps assess the threats of reactive arrangement (Meltzoff & Cooper, 2018)
By using three or more groups, multiple independent variables, or crossover designs you can assess multiple variables without reducing the internal validity (Meltzoff & Cooper, 2018).
By keeping the design the same throughout the study, you can reduce instrumental threat (Bhattacherjee, 2012; Trochim, 2006).
Switching replications design assists with the ethical dilemma of denying one group treatment, or the compensatory equalization of treatment threat (Trochim, 2006)
Using a pretest will compare your groupings and reduce your multiple group threat (Trochim, 2006)
Waiting until the first treatment has worn off before the second treatment is initiated reduces the multiple treatment threat (Meltzoff & Cooper, 2018)
Page updated
Google Sites
Report abuse