Surveys

Welcome to the UA Survey Guide! -- NEEDS CLEANUP

The first – and probably most important message – is that surveys should be used judiciously and only when necessary.

The purpose of this guide is to provide you with a sense of when surveys are appropriate and how to construct them to not waste your or your survey-takers’ time.

So when are surveys useful? Surveys are necessary in cases where broad student interest needs to be gauged, or when “the answer” is not able to be determined satisfactorily via qualitative methods.

• Qualitative methods are alternatives to surveys: These include simple brainstorming sessions (discussion papers or even more informal), focus groups, or conversations with administrators, staff, or other experts.
• Surveys should be avoided if at all possible. Usually, other means of information-gathering are more efficient and do not have the drawback of “survey fatigue” – more surveys means fewer responses and decreased quality of information.
• Surveys cannot prove causality. Essentially, surveys cannot “prove” things – they can suggest or provide insight into relationships. Be careful about the conclusions you draw and the questions you seek to answer!

Success Stories

• Late Night Dining: Hours/location/food preferences required a large sample size
• The Bike Share Program: The large level of investment required mandated a sure sense of interest

Okay, so I’ve chatted with my committee director, and I am confident that a survey is the right way to proceed.

Now What?

First, you need to select a survey host. Great options include:

• Qualtrics – free for any Whartonite and quite advanced
• Survey Monkey – a basic service available to StudGov members via log-in ExecNEC@lists.upenn.edu and password “electnec”
• Wufoo – the UA has an account with login secretary@pennua.org and password “tangiblechange”

Second, you need a reason! Survey-takers need a concise opening that explains why they are taking the survey, how the results will be used, and any incentives being offered. Please be careful not to influence the survey-taker. For example, the first statement below is inappropriate:

• We are surveying the student body to prove that healthy late-night dining options are very popularly in demand; rather, say
• This survey intends to gauge student interest in dining options at Penn.

Third, always remember that surveys should be as short and sweet as possible. Every item included on the survey should have satisfying and well-developed answers to the questions:

• How will I use these results in my analysis?
• Does this question ask something that is otherwise unanswerable?
• Will my survey takers be confused or fatigued by my question?
• In other words, don’t include things just for fun! For example, it makes sense to include an athlete v. non-athlete demographic question in a survey on dining habits; however, it wouldn’t be helpful in many other cases

Fourth, think about response types. Remember that people are lazy. Writing (and processing!) open-ended responses is time-consuming and inefficient. This should be avoided if at all possible.

Likert scales (1-7 preference), rank-ordering, and simple multiple choice are great starting points

Fifth, think about who is taking your survey. Never forget that Penn is an incredibly diverse place. Demographic questions such as race, gender identity, Greek status, sexual orientation, age, et al. can be used (judiciously) to confirm that a sample is representative of the wider Penn population. However, they are not always necessary.

So I have a survey… how do I release it?

First, have someone else (preferably a committee director) read it over. Then, you should try to discuss it in committee. The smarter you are at this stage the less work you’ll do later! Finally, send it to Exec for distribution.

Data Analysis and Presentation

This is a crucial step in the surveying and decision-making process. You, as the researcher, have to give meaning to the data you collect. Don’t be afraid of Excel! Pie charts and bar graphs are nice, simple ways to convey information. That said, more advanced graphing and statistical techniques can and should be applied on an as-needed basis. You should consult someone familiar with these matters when making these decisions. Or, take any basic statistics or OIDD course to learn more!

Finally, what good is hard work if it goes to waste? Please make sure your data and charts are clean, attractive, and properly labeled and titled. It might sound silly, but you do not want good numbers and analysis to be dismissed because they are difficult to decipher. This happens more frequently than you think! Take pride in your work and spruce it up - it’s well worth the effort.

I want more information on survey design!

• Take a class! Penn has tons of great courses that involve research, surveying, and sampling. One great intro course is MKTG212 (Marketing Research).
• Or check out these tips from Amy Bleakly, an Annenberg professor:
• Below are some tips on constructing surveys from Amy Bleakly, a professor in the Annenberg School:

Question types: A survey can ask three different kinds of questions:

Open ended (“What is your favorite television show?")

• Advantages
  • Permits the researcher to obtain unanticipated answers
  • May more closely describe the views of the respondent
  • Respondents like the opportunity to answer some questions in their own words
  • Appropriate when a list of possible answers is longer than feasible to present
  • Useful for exploratory purposes
• Disadvantages
  • Many people give rare answers that are not analytically useful
  • You have to beat it into shape afterwards to get meaningful data

Unordered close ended (“What is your favorite television show? The OC, Grey’s Anatomy, The Office, or Entourage?”)

• Advantages
  • Respondent can more reliably perform the task of answering the question
  • The researcher can more reliably interpret the meaning of the answer
  • Providing response categories increases the likelihood that there will be enough people giving any particular answer to be analytically interesting

Ordered close ended: “How much do you agree or disagree on a five-point scale with the following statement: TV is evil?”

• Advantages
  • Rapid way of asking a question
  • Special form of balance
  • Can measure different forms using the same question
• Disadvantages
  • Scale sorts people into 5 categories, but most analysis puts people into 2
  • Interpretable only when asked about the extremes of a continuum
  • Strong predisposition for “yeasaying”, where a person will stake out a tepid middle ground when they actually have no opinion.

Tips on use of question types

• Use open-ended questions only when searching for more ideas or suggestions or feedback (you should always have one question of this kind as a deference to the voters). Remember, open-ended answers are almost useless when it comes to drawing conclusions from your data.
• Measure the strength of the attitude or opinion opposed to just asking yes/no questions. (Schuman and Presser 1981, Sudman and Bradburn 1974)
• Provide a “don’t know” option. According to Schuman and Presser (1981), 30% of respondents will offer opinions that they don’t anything about b/c they do not want to confess to mental incapacity. Also, it can reduce the number of values in an item because if people don’t know the answer, they might not answer the question (Schuman and Presser 1981).
• Offer an equal number of response categories for each side of the issue (Schuman and Presser). If you don’t do this, you are potentially biasing the respondent to one side.
• Start your scale with the socially undesirable responses. If the socially desirable are presented first, the respondent is more likely to choose those answers (Sudman and Bradburn 1974).
• Too few categories can hurt the validity of a study and can increase the residual error (Andrews 1984). The more categories you use the better-it is recommended that you use at least 5-7 response categories.
• A multi-item scale is superior to single items (McKennel 1977).
• Use mutually exclusive response categories.

Tips on wording questions

• Poor question wording can negatively affect answers and also result in misinterpretation.
• Use wording that all respondents can understand. Keep wording simple and familiar to all respondents. To do this, researchers must define key terms in a question and avoid jargon.
• Avoid double barreled questions, like “would you prefer late night dining at all and if so after or before 2am?”
• Respondents should be given instructions and definitions as necessary.

Question order

• There should be a logical order to a set of items. You should ask all the questions relevant to one subject before asking questions on another subject in order to not confuse or frustrate the respondent and to keep them focused.
• Start the questionnaire with easy, salient, and non-threatening questions and then get to the more sensitive ones -- helps build rapport and trust between the respondent and interviewer.
• Use chronological order for recall questions on specific events.
• Use a funneling technique—start with the most general questions and then move onto more specific questions-- this aids in the retrieval of information.
• Ask objectionable questions at the end of the survey.
• The first question is the most crucial- it determines whether the survey will be completed or discarded (in mail surveys). This question should apply to everyone and be interesting and easy to understand.