■ It takes considerable effort to create a well designed questionnaire – data from open ended questions are often difficult to interpret and not advised for large samples. Closed ended questions that have options selected by the researcher can result in a lack in providing contextual meaning and researcher subjectivity

■ Questionnaires are not well suited for issues that require clarification on the part of the respondent

■ Long delays in participants responses, thus requires continuous monitoring and long investment into the research project. Response rates from mail-in questionnaires is lower than with other survey methods

■ Mail-out questionnaires can not be changed through out the process of data gathering, for some studies, this inflexibility can be viewed as a limitation.

■ Questionnaires can skew findings from demographic bias - due to language and cultural barriers; social desirability bias - participants may feel uncomfortable to honestly answer personal or controversial questions

■ Two surveys may have to be used to prevent age bias – older populations may be more likely to respond to a mail-in questionnaire, and a younger demographic may be more likely to respond to an online questionnaire

■ Issues with security can compromise responses, and confidentiality of participants

■ Validity relies on effective wording, formatting, and sequencing – they all have important consequences. Poorly framed or ambiguous questions will likely result in meaningless responses

■ Example, if the majority of the respondents fail to respond to a survey, then legitimate concerns whether non-respondents are not responding due to a systematic reason, it may raise questions of validity of the study's results, i.e., dissatisfied customers tend to be more vocal about their experience than satisfied customers, therefore, dissatisfied customers may be more likely to respond to questionnaire survey requests than satisfied customers. Hence, the respondent sample is likely to have a higher proportion of dissatisfied customers than the underlying population from which it is drawn. Thus, the results will lack generalizability.

Example of Validity issues in a research study

■ In a study titled, Access, Education and Policy Awareness: Predictors of Influenza vaccine acceptance among VHA Health care workers, the researchers used a retrospective design, sent out email surveys, used yes/no /not sure questions about awareness, respondents could select among 11 different local practices, access to influenza vaccine question. For questions related to employer communication respondents could select from 11 common elements.

The original national CDC survey instrument was used to validate questionnaire development, thus it ensured generalizability

The sample size was 321,000 employees, and it represented a diverse occupational and geographical population, a threat to validity occurred because the response rate was only 17.4 %

Selection bias is another threat, it is reported that individuals who respond to surveys as a matter of habit in general may be more likely to accept influenza vaccination

The retrospective design relied on individual recollection, making it impossible to establish causality, and that is an internal threat

■ It is assumed that every question will have a response, however, respondents may not have an answer, therefore every question should have a response category of “don’t know”, “unsure”, or not applicable” , however, you should assume that this data can not be included

■ Assume that if greater than 10% select the alternative “other”, then the other alternatives may not be appropriate for your population under study

■ Avoid errors in making assumptions in the planning stage by clarifying with all the researchers involved, the goals, agreeing on the length of the questionnaire and how it will be administered

■ Do not assume that a valid questionnaire measures what it claims to measure. In reality, many fail to do this. For example, a self completion questionnaire that seeks to measure people's food intake may be invalid because it measures what they say they have eaten, not what they have actually eaten