Embedded Files
Welcome to the first August Asynchronous Module
In this session, you will be taking a deep-dive into survey design and other means of collecting qualitative data.
By the end of the session, Data Fellows will be able to
Create a set of questions for a focus group based on a research question their RSSP team would like answered
Use interviews as an addition to survey collection as a way to enhance their top-line report
Materials needed:
Note-Catcher
Headphones (to listen to audio/videos)
RSSP goals and measurement plan for 2023-2024
Time anticipated to complete module: 60 minutes
Part I
Part I
It’s important when we are creating a survey to begin with a draft and then iterate along the way. This can greatly improve the chances that we will get the data we need in order to make critical decisions in our district.
As you are beginning the school year, this is a perfect time to refine your survey creation process and measure change over time. We will be looking closely at an article from the Pew Research Center and applying learning from our first synchronous session in August to inform the work done in this asynchronous module.
Activity 1
Activity 1
In your note-catcher, create a set of 5-7 survey questions based on something your RSSP team may want to collect information on (e.g. teacher confidence with HQIM implementation; student satisfaction with new curriculum; instructional coaches’ confidence with coaching their teachers; administrators comfortability with a new walkthrough form).
Activity 2
Activity 2
Directions: Watch this video on wording. After watching the video, go back to your survey questions and, using the information you learned from the video, make revisions to your questions to make them less biased and more specific.
Acquiescence Bias:
Acquiescence Bias:
Acquiescence bias means that people are more likely to agree with something regardless of how they actually feel. They tend to agree with survey questions statements in order to be agreeable rather than provide their own personal, unbiased views.
Directions: Update your survey questions a second time to make them less prone to acquiescence bias.
Part II: Question Order
Part II: Question Order
Research has shown that the order of questions can drastically impact the way in which respondents answers survey questions. According to Pewresearch.org, “researchers have demonstrated that the order in which questions are asked can influence how people respond; earlier questions can unintentionally provide context for the questions that follow.”
Activity 1
Activity 1
Look at the example from the Pew Research Center (to the right) and reflect in your Note-Catcher:
Why do you think the order in which questions are asked on a survey can drastically change the outcome?
Activity 2
Activity 2
Let’s consider this with an RSSP example:
Lost Pines ISD district leaders, alongside the RSSP team, want to know teachers’ confidence with implementing high quality instructional materials in their classrooms. They are asked a series of questions on a survey. Two of them are below:
Rate the extent to which you feel prepared for Eureka Math lessons on a weekly basis.
How many minutes do you spend internalizing your Eureka math lessons on a weekly basis?
Activity 3
Activity 3
If we ask the questions in the order above, it’s clear that we are hoping to find a correlation between how many minutes are spent on lesson internalization and the extent to which teachers feel prepared to execute on their lessons and therefore may respond in a way that shows that.
If we change the order, or include questions between the two questions, participants in the survey may provide a more realistic picture of where they’re at. If we first ask them how many minutes they spend internalizing lessons, they do not yet have the context for why we are asking.
Head back to your note-catcher and look at the survey questions you have created:
Make edits to your question order to make sure earlier questions won’t influence later questions
Part III: Pairwise Comparison
Part III: Pairwise Comparison
A useful tool to help determine course of action based on survey results is a Pairwise Comparison. A Pairwise Comparison is a process of comparing multiple candidates in pairs through a “head-to-head” comparison. There are several advantages to using a Pairwise Comparison. It’s especially helpful when priorities are not clear (as is often the case in education) and when you are getting competing ideas of what’s important to focus on.
Activity
Activity
Directions: To understand more about pairwise comparison, read through this article, focusing on the sections “What is pairwise comparison,” “how does it work” and “what is pairwise ranking”
Now, let’s consider how we can apply this to our work in RSSP with our example from above from Lost PInes ISD. Let’s say you want to administer a survey to determine the extent to which teachers are confident with implementing HQIM, but you also want to know how the district can best support them with implementation.
There are several ways to support teachers with HQIM implementation:
1:1 coaching with an administrator
1:1 coaching with an academic coach
Time to internalize their lessons each week
District-created Internalization guides
Professional development sessions
PLC planning time
In a pairwise comparison, you would put each option “head-to-head” against the others to see what ends up being most important for your teachers. The winner of each round receives points and at the end, you’ll have a better idea of actions to take to support teachers and what teachers’ receptiveness to each strategy would be.
There are many different tools for creating a Pairwise Comparison. If you are interested in exploring one more, here is an example.
Part IV: Pilot Testing Your Survey
Part IV: Pilot Testing Your Survey
Activity 1
Activity 1
Directions: Watch the video below and reflect in your note-catcher.
Creating a cognitive interview:
Cognitive interview: technique used to provide insight into the other person’s perspectives. When you conduct a cognitive interview, you don’t care as much about their answer, but how they arrived at their answer.
There are three questions to the cognitive interview:
What is this question asking?
How did you arrive at your answer?
How sure are you of your answer?
Activity 2
Activity 2
Directions: Plan to pilot test using the following steps.
Who can you reach out to to pilot test your survey?
Schedule time to do it!
Module Complete!
Module Complete!
Congratulations on completing the module. Please complete the Exit Ticket form by clicking on the link above. We will use the information you submit to track your completion.
Page updated
Report abuse