Psychometric sample size calculators
Psychometric sample size calculators
Determining the minimum sample size needed for a planned psychometric evaluation of a specific self-reported questionnaire or observational measure is not straightforward and no magic one-size-fits-all guidelines exist. Besides the actual performance of an instrument, the required sample size for such studies depends on many factors, including the type of psychometric properties assessed (e.g., structural validity or intra-rater agreement) and the specific type of analysis used (e.g., confirmatory factor analysis or a simple correlation). Standards for the psychometric quality of instruments, and required types of analyses to demonstrate these, may also be highly dependent of the nature and complexity of the construct of interest, use of the instrument (e.g., high stakes versus low stakes tests), and even the field of study.
Many psychometric studies reported in literature do not report any a-priori sample size calculation at all and have simply used the data available from the application of the measure in some study (such as a clinical trial or survey) that was not specifically powered for the purposes of a psychometric evaluation. Others have used very general ‘rules of thumb’ for a specific analysis, such as those reported for the required number of persons per item needed to obtain robust factor solutions.
Sample size for reliability and correlation with desired precision
In some cases, researchers planning a psychometric evaluation are specifically interested in the number of people needed for a certain reliability estimate (e.g., internal consistency, test-retest or intra-rater reliability) of the instrument or some expected correlation of the instrument’s scores with another variable (e.g., criterion validity or convergent validity). Several methods and (online and offline) calculators are available to determine the sample size required to test a hypothesis regarding the value of such an association. Most of these methods determine the sample size needed to establish whether a certain association is significantly different from zero.
Merely showing that an observed association is significantly different from zero is, however, not very relevant and provides little information about the precision of the observed estimate. In a psychometric context it is much more relevant to calculate the sample size needed to obtain a confidence interval of a specific desired width for an expected reliability or association or to show that the anticipated estimate is significanly different from a minimum acceptable value.
Below you can find some online calculators that can be used to determine the number of people needed to obtain a confidence interval of the desired width around some expected or planned association (interval estimation) or to have sufficient power show that this estimate is significantly different from a minimum acceptable value (hypothesis testing) for several commonly used coefficients.