September 2015
ITComm Virtual Coffee Shop: Survey Design
September 24, 2015
Presenter: Alison Cruess, University of North Florida
Moderator: Allison Oslund, Texas A&M University
Notetaker: Sarah Sharif, St. Edward’s University
Resources
Presentation by Alison Cruess
Presentation Notes
Where do we start when we are creating a survey? The first thing that we tend to do is start writing questions. But before you start writing down questions, think about the why? For a successful survey, you have to always start with the objectives:
Who are you surveying
What you will ask
On common survey objectives shared on the slide, it is important to note that all have clear objectives. Remember, that the first question is “NOT a question” - The introduction should explain the survey.
Who is conducting the survey
The purpose of the survey
Expected time to take the survey
Thank you statement
Confidentiality statement
Don’t “give away the cow” - Providing too much information in the first question can create bias. Writing a leading statement can skew your results.
Sample introductions (on slide) identify: who, the topic, the expected time + thank you statement.
Response rates is critical to a successful survey, we want them to finish it from start to finish (abandonment rate)
Response Rate = #Completed Survey / #ppl asked to take survey
What is good response rate? There is not really a good answer, survey fatigue is becoming a reality to be aware of nowadays
When surveying our student population, over 10% for faculty, over 25%, targeted groups, over 40%
Always include a “way out” for each of your questions, this decreases your abandonment rate, include one of these options:
Not applicable
I don’t know
Not Sure
Other
Open vs. Closed Questions (chart provided in slides):
Closed - From a select group of answers, data can be viewed in charts and graphs
Open - Harder to analyze, increased time in review
Have a balance of both
Tips for Decreasing Abandonment:
Decrease the number of open questions that you use
Try not to make someone scroll through your questions
Use similar scales in all your questions (Disagree/Agree or Important/Unimportant)
Tips for Increasing Response Rate:
Send reminders to those that haven’t completed your survey
Rule of Three (Initial Invite, 1st Reminder, and Final Reminder)
Use “Final Reminder” in your subject line
Question Wording:
Terms such as “always” and “never” will bias the response in the opposite direction
Double-barrel, respondents may need to agree with one and not the other, you are making assumptions can make the data useless.
Questions that are too broad have little use of to you
Question is negative and can broadcast a bad message about the organization
Be Careful of Survey Bias
Make sure to avoid leading questions with no assumptions
Be concise in your survey questions
Order of Your Questions
Random order can be confusing
The order should be intentional and make logical sense
Branching and Display Logic
Don’t ever display a question to someone unless it is applies to them directly, start with a broad question and then ask the specific question to the catered group
Types of Questions:
The Dichotomus Question - Agree/Disagree, Yes/No
Multiple Question - Same question, multiple answers
Rank Order - be careful of click bias has been shown that the first and last are most commonly picked
Rating Scale (Or Matrix Questions)
Open-ended, best used when want to avoiding bias or want to collect details
Demographic
“Mixed” Matrix Questions (Side by Side) - Can be confusing to set up, but you have to be intentional
How many questions in a survey?
The ideal survey is 10 or less and takes no more than 5 mins to complete
Know your audience, each respondent have different motivations
Relationship of # of Questions to Time
Make sure that you are balancing your survey goals
Survey Ready?
Test the survey
Before you send out your survey, look at your test data
Final Thought
Design deliverables are just at the top of the iceberg, design is crucial to a successful survey
Q&A (chat transcript + audio)
Allison Oslund - Texas A&M: Participants who want to call in dial 1-877-944-2300, access code 99281#
99281# 3: hi all, is there audio? i called in and i am on the web and not hearing any audio
99281# 3: i hear some rustling
Stan - NC State: I'm not hearing audio either.
Tam Frager - Oregon State Univ.: I'm hearing no voices - just typing and rustling
Deb Dexter, WPI: i hear typing>
Chris - UMB: same
Melanie Thomas - UTEP: I called in so I hear you.
Nancy Novitski - Univ. of Oregon: Yes, can hear you by phone
Stan - NC State: I've called in and can not hear via phone or web.
Nancy Novitski - Univ. of Oregon: Audio on the computer never works for these
Melanie Thomas - UTEP: I called on the phone too.
Nancy Novitski - Univ. of Oregon: (that was my impression, anyway)
Deb Dexter, WPI: whoever is speaking now, we hear you!
Elizabeth Cline: Please stand by and don't select anything for right now! :)
Alison Cruess - Univ. of North Fla: stand by we are fixing things
Elizabeth Cline: We are reconnecting audio bridge.
Loren Benavente: Please standby
Nancy Novitski - Univ. of Oregon: Yes - can hear you via phone!
Stan - NC State: We hear someone whispering just a bit!
99281 2: Oops!
Stan - NC State: Those on phone who are not presenters, it's helpful to mute your phone.
Loren Benavente: If you are on the phone and are NOT speaking. Please make sure you are on mute
Allison Oslund - Texas A&M: If you CAN NOT HEAR, please let us know
Beth G Lyons, Cornell: Cannot hear
Tam Frager - Oregon State Univ.: I still don't hear voices
99281# 3: i hear the typing that is all
Stan - NC State: I'm still only hearing others on the phone without phones muted.
Chris - UMB: cannot hear
Nancy Novitski - Univ. of Oregon: You have to call to get the audio
Sarah Sharif: My phone has been on mute since the start :)
Allison Oslund - Texas A&M: Everyone should call in on the phone
99281# 3: me too, phone and computer i hear
Tam Frager - Oregon State Univ.: I heard both of those, too
Loren Benavente: You should hear audio over the phone and over computer speakers
Stan - NC State: Yes!, but getting feedback!
Deb Dexter, WPI: there's echo because you need to mute a phone maybe?
Loren Benavente: If you're on the phone - please mute your computer speakers
Deb Dexter, WPI: that did it, thanks!
Carlyn - Rice: We get a very high response rate (over 90%) for our Incoming student technology expectations survey - it is listed in their "things to do" list to matriculate and they don't have survey fatigue yet.
Nancy Novitski - Univ. of Oregon: Wow!
Melanie Thomas - UTEP: yes - big complaint is there is no "out"
Melanie Thomas - UTEP: Sometimes I do closed, but give them a field to enter comment
Melanie Thomas - UTEP: We also use Qualtrics
Stan - NC State: I think whether to scroll or use mulitple pages depends on the total length of survey instrument. If it's short, I think it's better to see all of the questions.
Carlyn - Rice: Yes! Qualtrics is awesome. :-)
Paul Reaves - GaSouthern: At Georgia Southern we don't do many surveys but Qualtrics is what we use when we do.
Deb Dexter, WPI: I like the test response feature in Qualtrics. It allows me to see the answers so I can better analyze the answers.
Carlyn - Rice: Isn't it still a negative question?
Melanie Thomas - UTEP: Trick question - no "out' option. :)
Melanie Thomas - UTEP: yes
Tam Frager - Oregon State Univ.: yes
Bess Walden, CSU: Yes we can here you fine
Bess Walden, CSU: *hear
Nancy Novitski - Univ. of Oregon: Assucming significant barriers exist - isn't that bias? :)
Melanie Thomas - UTEP: i even hung up the phone.
Jacquelynn Hongosh - Oberlin College: I had the same questoin, Carlyn. It seems that option B suggests there definitely is a "significant barrier" while A asks if the respondent thinks there is.
Alison C, ANSWER: A better question would be to ask if they see barriers and what those are. We have a survey that addresses faculty barriers and I am happy to share that.
Allison O, ANSWER: Avoid bias hopefully and making it more positive
Carlyn - Rice: @Jacquelynn - thanks! I think it is a good question and would definitely use it if we had those services
Allison Oslund - Texas A&M: We can bring this question up to Alison during the Q&A period
Jacquelynn Hongosh - Oberlin College: Thanks, Allison. That would be great.
Deb Dexter, WPI: I tend to have difficulty analyzing the data with the matrix questions. Will we be looking at examples of analyzing the data?
Alison C, ANSWER: Matrix questions tend best to be analyzed in a bar chart so that you have every possible choice and you can get a feel for the # of responses you got.
Allison Oslund - Texas A&M: I'll add that to our question list
99281: Recommendations for survey tools, especially free
Alison C, ANSWER: Survey Monkey has limited number of questions, but it is free
Allison O, ANSWER: Survey Monkey does not meet privacy or accessibility reason
Alison C, ANSWER: Qualtrics is a great application
Melanie Thomas - UTEP: I always recommend they start survey using Word.
Melanie Thomas - UTEP: I agree with that...and no out.
Melanie Thomas - UTEP: A less negative
Deb Dexter, WPI: Sometimes starting with the result and working backwards has helped me to determine if the format of the question returned the answer I'm looking for. Hope that makes sense. I didn't know if you had an example data file.
Jacquelynn Hongosh - Oberlin College: That sounds like a great solution, Alison. Thanks!
Stan - NC State: Be careful with Survey Monkey's accessibility....
Nancy Novitski - Univ. of Oregon: @Stan - what problems have you run into?
Beth G Lyons, Cornell: A caution with SurveyMonkey, we found that it could not meet university standards for data privacy
Stan - NC State: Not all question types are accessible for those using screen readers.
Stan - NC State: Unless they have improved recently.
Tam Frager - Oregon State Univ.: In the past, I've worked places that used Google surveys
Stan - NC State: Google Forms used to be completely inaccessible, but they are much better now.
Stan - NC State: Qualtrics is clearly the gold standard!
Nancy Novitski - Univ. of Oregon: Good to know!
Nancy Novitski - Univ. of Oregon: We use Qualtrics here
Susan West - Yale: we use qualtrics as well
Sarah Sharif: Both Qualtrics and Google Forms :)
Beth G Lyons, Cornell: Qualtrics
Paul Reaves - GaSouthern: Georgia Southern uses Qualtrics
Susan West - Yale: our students do use google forms frequently
Nancy Novitski - Univ. of Oregon: How do you distribute your surveys? Through mass email lists?
Susan West - Yale: i cannot speak, unfortunately
Stan - NC State: We used to use Lime Surveys before we had a campus license to Qualtrics
Susan West - Yale: it's requiring me to call in to speak
Melanie Thomas - UTEP: We set it up so students get minimized account but can go through faaculty to request full account for the semester, then back to student account.
Susan West - Yale: google forms is used frequently by students because it's simple, free, and part of the Google package for student email
Nancy Novitski - Univ. of Oregon: @Alison C. - you do all the distribution through Qualtrics?
Alison C, ANSWER: We send out our surveys through emails.
Allison O, ANSWER: We can give different levels of privacy with surveys
Stan - NC State: Google forms provided automated response stats, but also puts the data into a spreadsheet that you can also use for further analysis.
Melanie Thomas - UTEP: Also it is good to ask participants for their email if they want someone to follow up with them - especially on satisfaction surveys.
Alison C, ANSWER: We can’t give out emails due to FERPA, they populate the survey with the student email addresses, but I can’t see the email addresses listed before I send it out
Chris - UMB: regulated
Carlyn - Rice: Pii - personally identifiable information
Susan: I have a question about demographic questions and why you placed those at the end?
Alison C, ANSWER: I like them at the beginning because they are easy to answer, however, I know that some researchers put them at the end and it drives completion. I don’t think that it matters overall.
Susan: We have always put them in the beginning based on their university status. This is a campus wide survey, we would ask tailored questions based on that cohort and limit our questions and data.
Alison C, ANSWER: I think I have read it both ways
Tam Frager - Oregon State Univ.: I put them at the end to ensure the other questions get answered before folks get worried about sharing that info
Melanie Thomas - UTEP: If they don't finish but you have it catch their survey even if they don't finish.
Stan - NC State: Same with us @Tam. Unless you're basing questions on the cohort.
Beth G Lyons, Cornell: Demographics - depends on the survey, we always do them at the end unless we need them for branching. Reason is that some people won't even take the survey if they're at the beginning
Tam Frager - Oregon State Univ.: end (unless needed for branching)
Melanie Thomas - UTEP: iff anonymous, then ask email at end so we can foloow-up
Stan - NC State: At the end. And if we're able, we don't ask them but match respondents with data on file.
Melanie Thomas - UTEP: No questions - really good ideas!
Susan West - Yale: thanks for your thoughts on beginning vs. end
Carlyn - Rice: This was timely, thanks for helping us get a quick handle on the most important tips to remember!
Beth G Lyons, Cornell: Thanks, great discussion!
Deb Dexter, WPI: Thank you!
Nancy Novitski - Univ. of Oregon: Yes, thanks very much for this helpful presentation, Alison et al.!
Tam Frager - Oregon State Univ.: Thanks for this. It's been helpful!
Chris - UMB: Good presentation and questions!