Validity and reliability

Miles and Huberman outline five main issues for quality criteria in qualitative research:

• Objectivity/confirmability of qualitative investigations

• Reliability/dependability/auditability

• Internal validity/credibility/authenticity

• External validity/transferability/fittingness

• Utilization/application/action orientation (Miles and Huberman 1994, p. 277)

Validity

Validity in quantitative research is defined as demonstrating that a particular data collection instrument is in fact measuring what the researcher set out to measure. Quantitative research tends to focus on questions of validity through use of sampling, appropriate instruments, and correct use of statistical analysis of the data (Cohen et al. 2007). These will be familiar to physicists from scientific research also.

However in qualitative research, validity is measured in other ways, such as through the integrity, richness, depth and scope of the research (Cohen et al. 2007). Maxwell suggests five forms of validity suitable to qualitative research, descriptive validity, interpretive validity, generalizability, theoretical validity and evaluative validity (Maxwell 1992). He points out however that in qualitative research, unlike quantitative research, threats to validity cannot be ruled out by building in prior design features such as randomization and controls. Qualitative researchers rather tend to eliminate particular threats to their accounts, seeking evidence to rule them out, and therefore these five criteria cannot be ‘applied’ to the research design, but rather detail the ‘modus operandi’ of qualitative research (Maxwell 1992).

If for example you decide to interview your students following an intervention where you had used virtual laboratories instead of face to face laboratories, you could guard against threats to validity in a number of ways. You could ask an independent researcher (a colleague) to conduct the interviews and/or observe the lab classes. If you are in a research team, you could independently code and theme the data from the interviews, to cross-check your interpretations. You could use data from a number of data collection instruments to triangulate findings. In your methodology section of your paper you would outline the context of the interviews, the participants and your relationship to them, accounting for how you overcame differences in power relationships (for example, students may say what they think the lecturer wants to hear).

Reliability

In quantitative research, reliability refers to the consistency of the data measurement over time and across similar samples (Cohen et al. 2007). Reliability in educational research relates to replicability. Reliable research would mean that if students were tested for their understanding of how to build an RC circuit, for example, and then the test was repeated with a very similar sample group, then the same results would be found (Cohen et al. 2007). This is akin to repeating a scientific experiment under the same conditions.

Similar to validity, the suitability of the concept of reliability for qualitative research is debated (Cohen et al. 2007). Miles and Huberman prefer to use the term ‘dependability’ and ‘auditability’. Reliability is difficult to establish in the same way as is possible in quantitative research. Cohen resolve this debate by appealing to the notion of ‘fitness for purpose’:

[W]e need to note that criteria of reliability in quantitative methodologies differ from those in qualitative methodologies. In qualitative methodologies reliability includes fidelity to real life, context-and situation – specificity, authenticity, comprehensiveness, detail, honesty, depth of response and meaningfulness to the respondents (Cohen et al. 2007, p. 149).