Objective

To ensure content validity, readability, and brevity of instrument.

Ways to Pre-test Your Survey Before You Send It

1. Respondent Debriefing

• For this approach, you will need to add several evaluation questions to the end of your survey for the respondents to answer. 

• These can be open-ended or closed-ended questions and usually focus on assessing respondent comprehension and interpretation of survey questions. 

• It should also include overall evaluations of the survey content, time, satisfaction and difficulty.

2. Cognitive Interviewing

• “Cognitive interviews” are a good way to really understand what is going on the minds of your respondents when they are answering your questions. 

• These are typically performed face-to-face with a small sample of 5–15 respondents. 

• As the respondents answer each survey question, they are asked to “think aloud,” which can include paraphrasing, providing retrospective thinking or providing judgments of their confidence in what each question means.

3. Expert Evaluation

• Your survey can be dramatically improved by feedback from two types of experts: 

1. Topic experts that have deep knowledge and expertise about the subject matter of your survey, and 

2. Survey methodologists that have expertise in how to collect the most accurate data for your research question. 

• These experts' evaluations can help shape the content and form of your survey and result in better data quality and more valuable insights.

• Researcher should create a rubric to assess the content validity of the measurement. Example of the rubric can be accessed here.

4. Focus Groups

• In the preliminary phases of questionnaire development, it can be very helpful to ask a focus group discuss your survey. 

• These discussions, which are usually semi-structured discussions between 7–15 people led by a moderator, are particularly helpful for clarifying basic concepts in the survey and evaluating perceptions of respondent burden or topic sensitivity.

5. Experiments

• Splitting a pre-test sample of respondents into groups and testing different variations of your survey design and content can be very powerful for understanding the results you will get when you field your main survey. 

• These experiments are particularly useful for understanding how changes in question wording, questionnaire design, visual layout, question order, and many other methodological factors may influence the data you collect.

After the pre-test the researcher should refine the questionnaire based on the results of literature review and expert review, before proceeding to pilot study.

Testing the final version of your survey on a small sample of your target population is critical - it can give you a sense of the kind of responses you will receive and any issues that may arise during the real survey period. Pilot studies often serve as a ‘dry run’ and are typically done just before fielding the survey to the entire sample. 

Objective

To ensure construct validity in terms of items dimensionality and their internal reliability before conducting the real study.

Methods

• Obtain respondents from the real population by using the proper sampling methods. 

• Distribute questionnaire using the real data collection methods.

• It is usually a good idea to include some evaluative questions, such as respondent perceptions of the length or difficulty of the questionnaire, satisfaction with taking the survey, etc. 

• Run exploratory factor analysis and reliability analysis.

• Evaluate and improve the questionnaire.

Pre-test and pilot test are used for measurement development.

Deleting items during the pilot study is not a proper way to follow.

The result of pilot study should be used for further assessment of questionnaire and to improve it further before the real study.

Pilot testing and Pre-testing. Click here.

Hunt, S. D., Sparkman Jr, R. D., & Wilcox, J. B. (1982). The pretest in survey research: Issues and preliminary findings. Journal of marketing research, 19(2), 269-273. Click here. 

Hardy, B., & Ford, L. R. (2014). It’s not me, it’s you: Miscomprehension in surveys. Organizational Research Methods, 17(2), 138-162. Click here.