SURVEY AND TEST CONSTRUCTION LAB
1. Choose a topic/variable for a test that you are to construct. The topic/variable should be both interesting and challenging. Below are a few possibilities.
• the anal retentive personality
• attitudes toward academic success
• attitudes toward eating
• masculinity/femininity
• political beliefs
• ability to understand people
• attitudes toward war
• attitudes toward deception
• concern with appearance
• flexibility
• social intelligence
• superstitiousness
• range of stimulation that one can take
• prejudice
• love
• vulnerability to "earworms" (not being able to stop "hearing" a song or part of song play over and over in your head)
• narcissism
• Machiavellianism
• attitudes toward gun control
Please note that you are not limited to these traits. Choose something that interests you and your group. You are going to work hard on this project, so make it worth it. If you can't think of any traits, browse the following sites:
http://www.depts.ttu.edu/hs/research/reifman/qic.htm
http://www.muhlenberg.edu/depts/psychology/Measures.html
http://www.yorku.ca/rokada/psyctest/
The variable we choose to measure was:
Of all the variables and traits in all the world, we chose to measure that variable because
2. Define the trait/characteristic you want to measure.
A. Your informal definition:
B. Dictionary definitions (see http://www.merriam-webster.com/ or http://www.abacon.com/psychsite/dict.html):
C. Make your definition more precise by using a
1. thesaurus (e.g., http://thesaurus.com/),
2. the Thesaurus of Psychological Index Terms
a. (go to http://psycnet.apa.org/index.cfm?fa=subjects.thesaurus&id=C7A789EB-DBD3-69A7-5BE5-B2CA6F24CF88 , enter your term in the “look up term” box, click on the “term hierarchy” tab, and then click “go”) or
b. Go to http://psycnet.apa.org/index.cfm?fa=search.advancedSearchForm , click on “TERM FINDER,” put your term in the “look up term box,” click the red “Go” box, and then click on the “plus” sign next to your term)
3. dictionary (such as http://www.visuwords.com/ or http://www.merriam-webster.com/ or http://www.abacon.com/psychsite/dict.html)
D. To illustrate that you do have a good definition, state the following:
1. Related terms:
2. The characteristic I am going to measure is most closely related to—but different from _____________________ because it is (broader/narrower)
3. The characteristic I am measuring has __ aspects or subtraits. Those aspects/dimensions/subtraits are:
4. The new, improved definition of your trait (if possible, refer to its aspects/dimensions) is:
Note: In the past, the biggest problems that groups have had is not defining their concept carefully. So, know precisely what your trait is and is not. Know whether participants would be aware of the trait you are measuring. For example, if you are measuring self-esteem, are you measuring explicit self-esteem (the person's conscious feelings of self-worth), implicit self-esteem (the person's feelings of self-worth that are not conscious), or both?
Knowing what you what it is you want to measure will help you with content validity because you can write questions that measure the aspects of your concept.
Knowing what you want to measure will also help you with convergent validity: You may be able to find measures of the same construct, and you will know what factors may be correlated with your construct.
Knowing about closely related characteristics that you do not want to measure will help you establish discriminant validity: You will find measures of those related constructs to see whether they correlate too hihgly with your measure.
3. Brainstorm about items (questions) that you think will differentiate between individuals who are extremely high on the trait you want to measure (or have an extremely positive attitude toward the attitude object) and those who are low on that trait (or have a negative attitude toward the attitude object). That is, you want questions that people "high" on your trait for answer one way and people low on your trait will answer a different way. Generate at least 10 questions. (You will probably discard some later.) You may write some items for another group in return for them writing some items for you. Note that most of your questions should use
Likert-scaling. That is, use the following format
STRONGLY DISAGREE DISAGREE UNDECIDED AGREE STRONGLY AGREE
1 2 3 4 5
4. Choose a set of items for the first draft of your test (note this is just your test, not your entire survey). In selecting items,
a. choose the items that you think would do the best job of correlating with the measures/behaviors that you decided on in step #4a.
b. choose/edit items so that they are short, easy to understand, and are not obviously biased (for editing tips, see pages 278-280 of your text).
c. be sure that you have items that cover all the relevant dimensions of the construct you are trying to measure. For example, if you think anal retentives are orderly, stubborn, and cheap, you should have items that represent all those dimensions. This will help increase the content validity of your scale. In addition, if appropriate, have items that refer to behaviors, feelings, thoughts, values, habits, general lifestyles, and attitudes.
d. choose/edit items so that the test is somewhat protected from the “acquiescence” (yea-saying) response set. In other words, make sure that just saying “yes” to every question (or "strongly agree") doesn’t result in getting a high score on your measure.
e. Order your items so that all the questions relating to the same subscale are together.
5. In addition to your scale, you should have the following sections (sections that are not part of your test, but are part of your survey):
a. a section that will allow you to collect data about the convergent validity of your test. That is, think about behaviors, measures, and peer ratings that should correlate positively with your measure as well as items that might negatively correlate with your measure. Start by thinking about collecting information on behaviors that should correlate with your measure. For example, if you are measuring attitude toward academic success, you would probably ask some questions about the time they spend studying and their grade point average. If you are having trouble thinking about behaviors that would correlate with your measure, consult the following source: http://homepage.psy.utexas.edu/homepage/faculty/Gosling/reprints/EnvandBehav05GoslingetalPLSCI.pdf Then, find a scale that is measuring the same thing you are measuring (if you can’t find such a scale, find a scale of a related construct). Be sure to write down an APA style reference for the source of that scale (you will need to reference that paper in your method section). Then, include a few questions from the more established test of your construct in this section. Later, we will see how well these items correlate with your scale.
b. a section that will allow you to collect data about the discriminant validity of your test. Think about behaviors, peer ratings, and measures that should not correlate strongly with your measure. That is, what constructs should be uncorrelated or only modestly correlated with your measure. In other words, you need to figure out what other construct(s) your measure might be accused of measuring--and then collect data to show that your measure is not measuring these other constructs. Often, you will want to show that your measure doesn't correlate with social desirability.
c. any questions that you have to test specific hypotheses that may or may not be related to your scale.
d. demographic information (gender, age, etc.)
6. State at least one hypothesis relating to how people differing on a demographic variable (e.g., gender) will differ on your measure.
A. Hypothesis:
B. Logical explanation for why this hypothesis might be supported:
7. Prepare a draft of your entire survey. (it will contain your scale as well as your other measures).
The beginning of your survey should look like this:
This scale is in the construction stage and cannot and will not be used for evaluation or diagnostic purposes. Thank you for your time and cooperation. They are greatly appreciated.
STRONGLY DISAGREE DISAGREE UNDECIDED AGREE STRONGLY AGREE
1 2 3 4 5
Read each statement below and decide on the degree to which you agree or disagree with each statement. For each statement, circle the rating scale number than best corresponds to your point of view.
1. How strongly do you feel _______________ 1 2 3 4 5
2. How strongly do you feel _______________ 1 2 3 4 5
After your 10-question scale, you should have the following sections:
A. Convergent validity questions or scales
B. Discriminant validity questions or scales
C. Any other questions you have that relate to any hypotheses you want to test.
D. Demographic questions
E. A statement thanking participants for their help (e.g., “Thank you for participating! We appreciate your help!).
8. Be sure your scale has no more than 10 questions and that your survey has no more than 30 questions. Then, fill in and print out the following template:
Our scale is composed of ___ (insert number) questions. They are questions (list question numbers) of our survey. The reversed scored items on our scale are questions (list question numbers). Our scale is composed of (Insert number) subscales. Subscale ___ is composed of questions (list questions). Subscale ___ is composed of questions (list questions). Subscale ___ is composed of questions (list questions).
In addition to our scale,our survey has questions (list question numbers) measuring ________ to help establish convergent validity and questions (list question numbers) measuring ________ to help establish discriminant validity. We also have questions (type in question number and question) and (type in question number and question) to test our hypothesis that ...... Finally, questions ___ and ___ are demographic questions.
We expect that our scale will be internally consistent:(define internal consistency in your own words) ____________________. Specifically, we predict that correlations between individual items and the whole test will be between in +.3 and +1.0 for most items and between -.3 and -1.0 for the reverse scored items. In addition, we expect if we correlate the first half of the test with the second half, the split half reliability will be above .70.
We also predict that our scale will have convergent validity. That is, it will correlate with other measures of the construct. Specifically, we thought that scores on our scale would correlate with ______.
Our scale was designed to measure ______, not _____. To establish that our measure was not measuring _____, (that is, to establish our scale's discriminant validity) we correlated our scale with ______. We expect that our scale will not correlate with _____________.
In addition to providing evidence for the reliability and validity of our scale, we had the following two hypotheses. First, that _____ individuals would score higher on our scale than ___________ because ....
Second, that high scorers on our scale would tend to ____________. The reason for this hypothesis was ....
9. Go over your template with me. Be sure the questions on your survey allow you to establish convergent validity, establish discriminant validity, and test your hypotheses. You may find that you need to add or modify some questions.
10. Proof this version of your survey, have someone in your group take the survey, and then proof it again.
11. Once you have a draft of your survey, we need to get it online. Create a free account in surveygizmo. To create this account, go to
http://www.surveygizmo.com/free-student-account/
12. Type in your survey. Be sure to use the radio button option for questions in which you want participants to choose one of several answers (e.g., rating scale questions and multiple choice questions). Do not use checkboxes for rating scale or multiple-choice questions.
13. Have a textbox for participants to write comments.
14. Have a question that asks participants if they have filled out this survey before.
15.Proof your online version of the survey.
16. Have everyone in your group take the survey using SurveyGizmo's "test" function.
17. Make sure that the downloaded data matches your responses.
18. If necessary, revise your survey.
19. Use the "test data" function to generate data from 100 participants. Then, see if those data allows you to test your hypotheses.
20. Revise your survey, if necessary.
21. Provide a test data link to your classmates to see if they have any problems with your survey.
22. Revise your survey based on class feedback.
23. Launch your survey.
24. Once you have the survey in final form, talk to me to discuss how to have volunteers take the survey.
Analyzing Data From Your Survey