Strengths Weakness, Validity Issues & Assumptions.
Strengths of research questionnaires
■ Questionnaires have a long history, used in formal research since 1930-40s.
■ Questionnaires can be inexpensively distributed to very large populations, thus a large representation of data allows the researcher to extract data that are near to the exact attributes of the larger population
■ Self administered questionnaires are unobtrusive and allows participants to respond at their own convenience, thereby easier to obtain sample size
■ Group administered questionnaires allow respondents to ask questions, ensures a high response rate, and is convenient for the researcher. This method is effective for organizations
■ Unstructured questionnaires allow subjects to respond using their own words
■ Questions/items may be aggregated into a composite scale or index for statistical analysis. It is easier to find statistical significance from the high representativeness of questionnaires
■ Standardized questions with uniform definitions ensure participants answer questions in the same way. Eliminates researcher bias and results in high reliability results
Limitations of research questionnaires
■ It takes considerable effort to create a well designed questionnaire – data from open ended questions are often difficult to interpret and not advised for large samples. Closed ended questions that have options selected by the researcher can result in a lack in providing contextual meaning and researcher subjectivity
■ Questionnaires are not well suited for issues that require clarification on the part of the respondent
■ Long delays in participants responses, thus requires continuous monitoring and long investment into the research project. Response rates from mail-in questionnaires is lower than with other survey methods
■ Mail-out questionnaires can not be changed through out the process of data gathering, for some studies, this inflexibility can be viewed as a limitation.
■ Questionnaires can skew findings from demographic bias - due to language and cultural barriers; social desirability bias - participants may feel uncomfortable to honestly answer personal or controversial questions
■ Two surveys may have to be used to prevent age bias – older populations may be more likely to respond to a mail-in questionnaire, and a younger demographic may be more likely to respond to an online questionnaire
■ Issues with security can compromise responses, and confidentiality of participants
Validity issues of research questionnaires
■ Validity relies on effective wording, formatting, and sequencing – they all have important consequences. Poorly framed or ambiguous questions will likely result in meaningless responses
■ Example, if the majority of the respondents fail to respond to a survey, then legitimate concerns whether non-respondents are not responding due to a systematic reason, it may raise questions of validity of the study's results, i.e., dissatisfied customers tend to be more vocal about their experience than satisfied customers, therefore, dissatisfied customers may be more likely to respond to questionnaire survey requests than satisfied customers. Hence, the respondent sample is likely to have a higher proportion of dissatisfied customers than the underlying population from which it is drawn. Thus, the results will lack generalizability.
■ The following are several strategies that will help you control validity in your questionnaires:
1.Use a blueprint (a table of specifications) to identify the essential content to be covered by the questionnaires
2.Search the literature for questionnaire items in questionnaires that match your blueprint criteria to facilitate comparison of results between studies, however, poorly written questions can be modified, items may be added or deleted, items from several questionnaires can be combined to develop an appropriate questionnaire
3. Consistency in the way the questionnaire is administered is important to validity, for example using a mix of mail-out and group setting questionnaires is a risk to validity.
4. Sequencing questions is also important: A) the first question should be a general question related to the goals or purpose, should be applicable to all respondents and answerable in a few seconds, B) questions should be grouped according to their content – to help the respondent organize their thoughts and reactions, leading to more accurate responses
5. It is important to keep the respondents sense of commitment, interest and sense of responsibility in completing the questionnaire, therefore, keep the survey as short as possible
Example of Validity issues in a research study
■ In a study titled, Access, Education and Policy Awareness: Predictors of Influenza vaccine acceptance among VHA Health care workers, the researchers used a retrospective design, sent out email surveys, used yes/no /not sure questions about awareness, respondents could select among 11 different local practices, access to influenza vaccine question. For questions related to employer communication respondents could select from 11 common elements.
The original national CDC survey instrument was used to validate questionnaire development, thus it ensured generalizability
The sample size was 321,000 employees, and it represented a diverse occupational and geographical population, a threat to validity occurred because the response rate was only 17.4 %
Selection bias is another threat, it is reported that individuals who respond to surveys as a matter of habit in general may be more likely to accept influenza vaccination
The retrospective design relied on individual recollection, making it impossible to establish causality, and that is an internal threat
Assumptions of research questionnaires
■ It is assumed that every question will have a response, however, respondents may not have an answer, therefore every question should have a response category of “don’t know”, “unsure”, or not applicable” , however, you should assume that this data can not be included
■ Assume that if greater than 10% select the alternative “other”, then the other alternatives may not be appropriate for your population under study
■ Avoid errors in making assumptions in the planning stage by clarifying with all the researchers involved, the goals, agreeing on the length of the questionnaire and how it will be administered
■ Do not assume that a valid questionnaire measures what it claims to measure. In reality, many fail to do this. For example, a self completion questionnaire that seeks to measure people's food intake may be invalid because it measures what they say they have eaten, not what they have actually eaten