S5 Pilot Studies

A small scale preliminary survey, done BEFORE carrying out a full, large scale random sample survey, is called a PILOT sample or a Pilot Study. This is extremely useful for many purposes. We highlight THREE such purposes here:

1. Selection of sample size -- since the precision depends on p, a rough estimate of p provides a more accurate guide to sample size than setting p at the worst possible value of 50%
2. Determination of length and accuracy of questionnaire. Quite frequently, the questions designed to be asked are ambiguous or confusing, and are interpreted very differently by the person being asked from what the intention of the surveyor was. These problems can often be sorted out by a careful pilot study.
3. Post-Survey Analysis. It often turns out that surveys MISS asking key questions which require analysis. The data required for the analysis is not present in the answers received. This is because of poor design of survey, but this often does not become apparent until AFTER the survey is done. At this stage it is too late to undo the damage. So doing the final analysis on a preliminary pilot survey ENSURES that the analysis that is desired CAN BE DONE with the data to be collected.

One special consideration arises for cases where p is very small, near 0 (or equivalently, very large, with p near 1). For small values, absolute error sometimes is not the relevant consideration -- to say that with 95% confidence, p is less than 5%, conveys no information when p is known to be small, around 1% in the population for example. Achieving higher accuracy in such cases requires huge sample sizes. To avoid this, it is useful to specialize to a subpopulation where the incidence is higher. For example, if one is study people with certain types of handicaps, one might look at a population of hospital patients, where one is more likely to find the desired typed of people being surveyed. By a targetted smaller subpopulation with higher incidence, one can get a higher sampleing rate for the desired characteristic, and thereby get a more accurate survey.

Action without change? On the use and usefulness of pilot experiments in environmental management — Raphaël Billé: One of the most common approaches to environmental action consists in designing and implementing pilot experiments, which aim at testing new practices and organisations.

Sample Size for Pilot Studies — Lengthy discussion of how to choose pilot study sample size.

The importance of Pilot Studies — Edwin R. van Teijlingen and Vanora Hundley

Wiki Pilot Experiment — Brief and not very informative