Date: Friday, 5/17/2019 10:00 AM

Room: Dominion North Concurrent Session C

Alisu Schoua-Glusberg

Cognitive testing of questionnaire items is increasingly standard practice in social research. Increasingly, more U.S. federal agencies are recongizing the need to cognitively test their survey questions, in particular since the issuing of the 2016 OMB Standards and Guidelines for Conducting Cognitive Interviewing Studies.

There are different approaches to probing in cognitive testing. Some researchers prefer a highly scripted approach with a protocol that lists a number of specific probes.They craft these probes based on their expert review of the instrument and an a priori selection of possibly problematic question formulations. Other researchers prefer to elicit a full narrative for each question that will --by itself and supplemented by spontaneous probes -- reveal how the respondent's answer to the survey question relates to the respondent reality. This, in turn, will show how the respondent interpreted the question and whether their response was appropriately selected to most accurately reflect their reality.

Both approaches, with highly scripted probes or with narrative elicitation and spontaneous probing, include specific probes that are asked to further elucidate the respondent thinking.

In this presentation we will present and discuss the a classification of probes that focuses on whether a probe is asking about the question itself (or some of its features) or is asking about the respondent's answer. This, we believe, may be a better way to think about probes. We will also discuss how this works in the cross-cultural, multi-language context, where the researcher must also look for comparability across languages.