Low Cost Message Testing Guide

SECTION 4

The testing process

STEP 1

Outline your reasons for testing

Know what you are looking for

Research questions help you know what answers you’re searching for in your test. They guide you towards achieving the mission of your testing activity, and it is from this mission where the research questions are crafted from.

What goes into a research question?

When testing your message, a research question needs to show the relationship between two variables. The first variable (called “independent variable”) is the key feature that you are hoping to test. The second variable (called the “dependent variable”) is the outcome you think the key feature will have.

For example, if you want to test whether a positively-framed message can increase the reporting of gender-based violence, your research question can be as follows: “Do positively-framed messages increase the reporting of gender-based violence?” The positively framed message is your key feature and the increase in reporting of gender based violence is the outcome you hope to achieve.

When designing your research question, put the following in consideration:

How you choose to measure your outcomes:

Is a quantitative or qualitative approach better in the questions you are interested about? Which approach is easy and clear for the respondent to answer? For both quantitative and qualitative, how do you choose to operationalise (or measure) the things you most care about, especially concepts like trust, sustained interest, and behavior change?

Channel of communication:

Will it be online, SMS, in-person, or phone call? Depending on your resources and the method used, you could test these four channels and see how they affect your outcome. An example of a good research question here will be: “Do SMS invitations increase the number of people who sign up for the gender-based violence campaign?”

Features of your message:

STEP 2

Choose your testing method

Which approach to use

Now that you know your research question, which method should you use to try to answer it?

There are different methods that you can use to effectively test your messages, depending on what resources you have. Next we provide a snapshot of 4 key methods that you can use, including what the method looks like under a low budget.

In-Depth Interviews

Individual interviews on a one-on-one basis between an interviewer and respondent using a semi-structured conversation on a specific topic.

It provides in-depth qualitative knowledge and experiences that cannot be collected in a group setup.

The respondent might give biased responses due to their stakes in the program.

Have a few guiding questions in the interview guide to discuss one or two themes of interest. Select the purposeful sample based on knowledge or experience with a particular phenomenon. Conduct the interviews in person.

CSO LCM IDI Section.pdf

Focus Group Discussions

Small groups of people, usually 6-8, are brought together to have a semi-structured discussion on a topic under investigation.

It gives an in-depth knowledge of the topic under investigation.

The data collected is more subjective than factual or objective.

Have a few guiding questions to engage with your audience. For sample selection, rely on your networks through a snowballing technique. The basis of the analysis is summarizing the data per each research question.

CSO LCM FGD Section.pdf

Surveys

Surveys use a list of questions to extract specific information, mainly numerical data, about a message.

Data is more objective and can be done remotely over a large sample, which provides rich data.

Does not capture in-depth knowledge on the topic under investigation.

Have a few guiding questions based on your research objectives. Check what other instruments have had for questions around the same topic. Implement via readily available tools such as survey monkey or google forms. Use your network to identify a suitable sample. Draw out general trends that relate to your research objectives.

CSO LCM Surveys Section.pdf

A/B Testing

It tests two, or more different types of messages or communications to identify which message best encourages engagement with the audience.

Suitable for drawing out findings that are generalizable to your target audience.

Requires some specific knowledge, e.g., randomization, to deploy such a test effectively.

Have two or more messages for testing. Use remote channels, e.g., social media, to roll out your test. Have only one iteration. The analysis is mainly to identify general trends without drawing generalizable conclusions.

CSO LCM AB Testing Section.pdf

Page updated

Report abuse