07. Analysing Data

Errors, uncertainty and outliers

It is important not to confuse the terms ‘measurement error’ and ‘personal error’. Error, from a scientific measurement perspective, is the difference between the measured value and the true value of what is being measured. For the purposes of VCE Psychology, two types of measurement errors should be considered when evaluating the quality of data: systematic errors and random errors. Personal errors should not be included in reporting and analysis of data. Rather, the experiment should be repeated correctly.

·       Random errors: Affect the precision of a measurement and are present in all measurements except for measurements involving counting. Random errors are unpredictable variations in the measurement process and result in a spread of readings. These may be caused by limitations of instruments, environmental factors and slight variations in procedures. The effect of random errors can be reduced by making more or repeated measurements and calculating a new mean, increasing sample size and/or by refining the measurement method or technique.

·       Systematic errors: Affect the accuracy of a measurement. Systematic errors cause readings to differ from the true value by a consistent amount or by the same proportion each time a measurement is made, so that all the readings are shifted in one direction from the true value. They may be observational error, imperfect instrument calibration and environmental interference. The accuracy of measurements subject to systematic errors cannot be improved by repeating those measurements. Most systematic errors can be reduced by being familiar with the limitations of instruments and experience with their correct use.

·       Personal errors: Include mistakes, miscalculations and observer errors when conducting research.

It is also important not to confuse the terms ‘error’ and ‘uncertainty’, which are not synonyms. ‘Outliers’ in data are a separate consideration, and must be further analysed and accounted for, rather than being automatically dismissed.

·       Uncertainty: All scientific efforts are directed towards reducing the degree of uncertainty in the world about observations, relationships and causes. All measurements are subject to uncertainty and may have many potential sources of variation and this uncertainly extends to all inferences and conclusions that depend on uncertain measurements. The uncertainty of the result of a measurement reflects the lack of exact knowledge of the value of the quantity being measured. The nature of human behaviour and mental processes measured in the discipline of psychology often involves psychological constructs which, by their very nature, can increase the degree of uncertainty associated with such measurements. VCE Psychology requires only a qualitative treatment of uncertainty (no calculations). When evaluating personally sourced or provided data, students should be able to identify contradictory (incorrect data) and incomplete data (missing data – questions without answers or variables without observations), including possible sources of bias.

·       Outliers: Readings that lie a long way from other results are sometimes called outliers. These may occur by chance in any given data set, with a small number of outliers expected when there is a larger sample. Their presence alone does not suggest an anomaly, but repeating readings may be useful in further exploring an outlier. Measurement and recording errors, a skewed distribution or data points from a different underlying distribution may be all causes of outliers. VCE Psychology requires students to use data visualisation to recognise whether outliers are present in the data and reflect on how these outliers would affect the testing efforts and validity of the research.

In psychological research, accuracy and precision are essential concepts for evaluating the correctness of measurements and their susceptibility to error.

• Accuracy refers to how close a measurement is to the true value of the quantity being measured. For example, an accurate weight measurement uses a scale that closely reflects the true weight of an object. Accuracy is not quantified numerically in psychology but is described as more or less accurate.

• Precision refers to the consistency of a set of measurements with each other, indicating how repeatable the measurements are. Precision does not consider how close the measurements are to the true value. A precise weight measurement would show the same value every time an item is weighed, even if it's not the true weight.

It's important to note that a measurement can be accurate but not precise and vice versa. Using a dartboard metaphor, an accurate measurement hits the bullseye (true value), while a precise measurement consistently hits the same spot, whether or not it is the bullseye.

Thus, accuracy and precision are separate but critical considerations in evaluating the reliability and validity of measurements in psychological research.

In psychological research, repeatability and reproducibility are critical concepts for evaluating the reliability and validity of findings and investigations. Due to psychology's commitment to the scientific method, particularly hypothesis testing and re-testing, it is essential that experiments can be repeated to ensure findings are not isolated anomalies and can be confirmed or expanded upon by future research. A study that is both repeatable and reproducible is considered more robust.

• Repeatability refers to the extent to which successive measurements or studies yield the same results when conducted under identical conditions within a short period of time. This includes using the same procedure, observer, instrument, instructions, and setting. Repeatability ensures that the same study or measure produces consistent results when repeated in the same conditions.

• Reproducibility refers to the extent to which successive measurements or studies yield the same results when repeated under different conditions, such as with different participants, times, observers, or environmental conditions. Reproducibility indicates the reliability of a study or measure when applied under varying conditions or with different methodologies.

Both repeatability and reproducibility contribute to the strength and credibility of scientific findings in psychology.

Validity refers to the extent to which psychological tools and investigations accurately support their findings or conclusions. It evaluates whether a measurement tool or an investigation as a whole measures what it intends to measure. Validity can be assessed in two main ways: internal validity and external validity.

• Internal Validity is the extent to which an investigation truly measures or investigates what it claims to. If internal validity is lacking, the results may not be true, and no reliable conclusions can be drawn. To achieve internal validity, researchers should consider:

  • The adequacy of measurement tools and procedures: Do they test what they claim to?

  • The adequacy of the experimental design: Did it minimize extraneous variables?

  • The adequacy of sampling and allocation procedures: Was the sample representative and unbiased?

  • Whether the independent variable truly affected the dependent variable: Were there confounding or extraneous variables?

Internal validity ensures that the investigation accurately tests the relationship between the variables it aims to study.

External Validity is considered only when internal validity is present. It refers to the extent to which the results of an investigation can be generalized to similar individuals in different settings, such as at different times or in different environments. To improve external validity, researchers can:

• Use sampling procedures that create a more representative sample, making it similar to the broader population.

• Include broad inclusion criteria to ensure a diverse range of people in the sample, including cultural diversity, to better represent the real world.

• Use a larger sample size to increase the likelihood that the sample is representative of the population.

External validity ensures that the findings of a study can be applied to real-world situations and other populations beyond the study sample.