Experiments (Evidence Based Policing and RCTs)

Bias: An intentional or unintentional systematic error in an estimate. Bias leads to inaccuracy in measurement and usually results in misleading inferences about the relationship between the variables.

Causal inference: Conclusions drawn from a sample or specific study conditions about the causal relationship between two or more phenomena. For valid causal inference, three conditions must be met: (1) a statistical correlation between the two variables, (2) a temporal sequence such that the cause precedes the effect and (3) the absence of an alternative explanation other than the independent variable for the observed change in the dependent variable.

Causality: A relationship between two or more variables whereby change in one variable results in a reaction in the other variables(s).

Control group: A comparison group that is unexposed to the studied treatment effect. In experiments, participants can be assigned to the control group either randomly (by chance) or using statistical matching techniques when randomisation is not possible. Control groups can comprise of no-treatment, placebo or alternative treatments.

Counterfactual: A ‘parallel universe’ with identical conditions to the treatment group but without the treatment applied.

Covariate: A variable associated with the outcome variable that can therefore affect the relationship between the studied intervention and the outcome variable. These extraneous variables are included in quasi-experimental designs to rule out alternative explanations to the observed change in the outcome variable, as well as to increase the precision of the overall causal model.

Dependent variable: The variable affected by the independent variable; the outcome of the stimulus applied in an experiment.

Effect size: The magnitude of the difference between treatment and control conditions following the intervention, expressed in standardised units.

Effectiveness: An expression of the benefit of an intervention measured under ‘real-world’ but controlled experimental settings.

Efficacy: An expression of the benefit of an intervention, measured under ideal and controlled experimental settings.

Evidence-based policy: The use of scientific method to produce policy recommendations.

External validity: The degree to which the study outcomes can be generalised to different people, places, times and contexts; often expressed in narrative rather than mathematical terms.

Implementation: A set of processes that have taken place (or that have been withheld) as an indispensable part of the studied treatment and its effects.

Independent variable: The variable that causes a change in the dependent variable; the stimulus or treatment applied in an experiment.

Intention to treat: An analytical approach in randomised controlled trials where the units are assumed to have been exposed to the condition to which they were assigned; ignores any violations of the allocation sequence or the level of completion of the assigned treatment.

Interaction effect: Situations where two or more variables jointly affect the dependent variable, thus considered a new treatment term.

Internal validity: The degree to which the inference about the causal relationship between the independent and dependent variables is valid.

Matching: A statistical approach in which a balanced comparison group is created based on the pretreatment characteristics of the treatment group participants; unlike randomisation, in which balance is gained through the random allocation of participants into treatment and control conditions. Statistical matching can be performed on measured data but cannot control for differences between treatment and control conditions based on unmeasured data.

Natural experiment: A methodological approach to identify causal inference in which cause-and-effect relationships are observed in their natural settings, without the direct involvement of the researcher in the form of allocating units into treatment and control conditions or applying the intervention.

Null hypothesis: A statement about the lack of a relationship between the variables under investigation. As the causal expression that is tested in the experiment, the null hypothesis serves as the starting point in experimental research.

Observational research: A research design in which phenomena are described without drawing inferences about causal relationships. Observational studies involve only measurement, not manipulation of stimuli.

Participant: Any type of unit that takes part in a study, such as individuals, cases or groups.

Publication bias: The systematic error associated with selective dissemination of results, when findings that reject the null hypothesis are more likely to be published. Also known as the ‘file-drawer’ problem, publication bias often can lead to errors in systematic reviews and meta-analyses because the overall results may erroneously suggest that the treatment is more effective than it is.

Quasi-experiment: A causal design in which participants are not randomly assigned to treatment and control conditions.

Random sampling: The procedure of selecting a group of participants out of the population using chance alone, giving every participant in the studied population the same probability of being recruited into the study sample.

Randomisation: The allocation of units into treatment and control conditions based on chance. Over time and with sufficiently large samples, random allocation creates balanced treatment and control groups before administrating the treatment in one but not in the other group (i.e. at baseline).

Selection bias: A process of selecting units to participate in an experiment that results in unbalanced groups, usually when the allocation systematically favours one group over the other; often leads to invalid inferences about the treatment effect.

Specification error: Bias resulting from an incomplete or imprecise statistical model of causality, usually due to omitted control variables or imprecise measurements.

Statistical power: The likelihood that an experiment will be able to appropriately reject the null hypothesis – that is, the ability of an experiment to detect a statistically significant effect that exists in the population.

Stimulus (plural stimuli): The intervention or treatment that the experimenter manipulates (or observed in natural experimental settings).

Systematic review: A type of literature review that aims to collect information on all published and unpublished studies relating to a particular research question. Unlike narrative reviews that are often subjective, systematic reviews are objective, with greater transparency and systematic methodology about how evidence was collected and synthesised.

Time-series analysis: A statistical approach in which multiple waves of observations of the data are made chronologically. In causal research, time-series analyses often explore how a trend in the dependent variable was ‘interrupted’ by a treatment effect (also known as interrupted time-series analysis).

Trickle flow assignment: A process of randomly allocating eligible units into treatment and control conditions over time, as the units become available – as opposed to batch random assignment, in which all units are randomly assigned simultaneously.