1. Download your data. If you are using SurveyGizmo, “Export” your data. To export your data, double-click on your project in the dashboard. Then, click on the “Reporting” tab. Then, click on “Exports.” Then, choose “Create a New Export” by pressing the “CSV/Excel (Quick)” button. You can choose to either “Open with Microsoft Office Excel” or “Save File” option.
2. In Excel, you may need to use the “Text Important Wizard.” Be careful when you get to Step 2. Although you may be able to get by with just clicking the “Next” button, you may have to do more. For example, in Step 2, you may have to check the “comma” box. Before leaving Step 2, look at the Data Preview box to see whether your data look correct. If the data in the “Preview” looks garbled, play with the “Delimiters” check box until they look the way you want them to look. Once everything looks good, press the “Finish” button.
3. Save a copy of your data. If your survey was on Gender Roles, save it as “MyGenderRolesdatabackup1” or some similar name. Save this file on your USB and e-mail it to yourself and your lab partners.
4. Check to see that your data appear clean. For example, be suspicious if you have several submissions within minutes of each other from the same IP address. If you only have a few missing values, consider deleting participants who have missing values. Otherwise, assign the missing value the average response (the average of the column from which there is a missing value). Be sure to note what you have done. You will need that for your Results section.
5. Be sure you know which question each column represents—and that you know how to code each column. What does a 1 represent? a 0? a 5? If you are unsure, have someone in your group take the survey and answer the questions according to a certain pattern, then download those data to see how they match up. (Of course, you will not use those “test data” from that “participant’) in your analysis.)
6. Label your columns and make notes about any coding issues that you might forget (e.g., is male “1” or “2”?) or get confused (e.g., if you have a reverse scored item).
7. Again, save a copy of your data. If your survey was on Gender Roles, save it as “MyGenderRolesdatabackup2” or some similar name. Save this file on your USB and e-mail it to yourself and your lab partners.
8. Then, to see whether there is an obvious problem with any of the items, look at the means and standard deviations of individual items. You can use Excel (just highlight the column and then click on the triangular button to the right of "AutoSum" and then select "Mean"), or you can export these data into VassarStats (Click on “Miscelanea” and choose “Use the Basic Sample Stats”). Are there any items that have standard deviations near zero? If so, which ones? Why would low standard deviations be a problem?
9. Look at the total scores on your scale: Remember that some of the items on your survey are not part of your scale. Once you account for reversed scored items, you can get the sum of each person on your survey by adding each individual’s score (by hand or with a calculator) or by having Excel add them up for you. To have Excel add them up, first go to the "View" menu in Excel and make sure "Formula Bar" is checked. Then, add a column and ask Excel to fill it in using the formula that you put in the Formula bar near the top of the page. For example, you might enter: “= F2 + G2 + (6 - H2)” Once you have the formula for the first participant, you can drag the corner of the box all the way down the column, and Excel will calculate the right total for each participant. Whether you use Excel or not, be sure to account for reverse scored items. Note that in the example above, “H2” was reverse scored. By subtracting the "H2" value from 6, a “1” becomes a “5”, a “2” becomes a “4,” and so on.
a. Individuals known to be high on the characteristic your test is trying to measure should have a higher average score on your test than those known to be low on that characteristic. If this is the case, your test has known groups validity.
b. There should be a wide range of test scores. A wide range of scores is necessary if your scale is to be sensitive.
1. What was the range of scores? (Just subtract the high score from the low score)
2. What was the standard deviation? (By following the same basic procedures you followed in Step 8 to get means and standard deviations for each individual question, you can have either Excel or VassarStats compute the standard deviation for your total scores.)
c. Correlate each of your individual items with the whole test score. This will give you an idea of your test’s internal consistency. You can get these correlations by using the correlation function in Excel or imputing your data into VassarStats. (In VassarStats, click on the Correlation and Regression link in the left menu, then choose " Matrix of Intercorrelations Version 1 (data-import format)."
1. What is the range of those correlations?
2. What is the mean of those correlations?
10. Again, save a copy of your data. If your survey was on Gender Roles, save it as “MyGenderRolesdatabackup3” or some similar name. Save this file on your USB and e-mail it to yourself and your lab partners.
11. Calculate Convergent Validity Coefficients: To organize your data, model the following sheet (you can do this by moving columns around in Excel)—except that you will not call your criteria "Criterion 1, Criterion 2, and Criterion 3." Instead, you will label your criteria. Your goal is to correlate the participants' scores on your scale (each person's total score on your test) with variables that you predicted would correlate with your scale.
The higher the correlations between your test and these criteria, the better the case for your test’s validity. One reason you may fail to have a high correlation is that you don’t have a wide range of test scores or you don’t have a wide range of scores on your criteria.
12. Calculate Discriminant Validity Coefficients: To organize your data, model the following sheet (except that you will label your criteria with descriptive names rather than as “Criterion 1,” “Criterion 2,” and “Criterion 3”). For example, you might replace "Criterion " with “Score on Social Desirability Scale.” Your goal is to correlate the participants' scores on your scale (each person's total score on your test) with variables that you predicted would not correlate highly with your scale. That is, you are trying to show that your scale is not measuring a different variable than the one you claim it is measuring. Thus, if you were measuring attitudes toward abortion, you would not want your measure to correlate with a social desirability scale--and you would not want it to correlate very highly with a measure of conservatism
You expect these correlations to be positive, but less than the correlations you obtained for your criterion validity. In some cases, you may expect the correlation between your test scores and these criteria to be near zero.
What are those correlations?
14. If you have not already done so, go back and do analyses to test any of your hypotheses.
List your hypotheses:
What do the data say about your hypotheses? You will probably summarize your results by writing something like "As predicted, the more tattoos a participant had, the more likely they were to get a high score on our scale, r (24) = .60, p <.05." Alternatively, you might write something like "As predicted, women (M =72.6) scored higher than men (M = 65.4) on our scale, t(18) = 7.52, p <.05."
14. If we are short on time, skip to 16. If we have time, go back and take a closer look at your test’s reliability.
A. First, assess the test's internal consistency. To get started, make your data conform to the format of the following table. You may have to use formulas to compute the sum of the odd items and the sum of the even items, just like you did to compute the total score. Remember to reverse score your reverse scored items.
Once you have those totals, you can correlate them to get and estimate of the measure’s internal consistency: a split-half correlation. To correlate “odds” and “evens,” you can use Excel or VassarStats.
B. To get an idea about whether your subscales are tapping different dimensions, you might model the following table:
Then, calculate the correlations between your different subscales (here you might expect low correlations) and the correlations between items belonging to the same subscale (the higher the better, since the items are supposed to be measuring the same thing).
15. Again, save a copy of your data. If your survey was on Gender Roles, save it as “MyGenderRolesdatabackup4” or some similar name. Save this file on your USB and e-mail it to yourself and your lab partners.
16. Work on your Results section using the following template. I suggest you copy the template into MS Word.
Results
Of the ____ surveys that were completed, we discarded _____ because of _____. In addition, ___ of the surveys had ___ missing responses. Those missing responses were replaced with the mean response for that question.
Scores on our scale
On our measure, participants' scores ranged from a low of ____ to a high of ____. The average total score was ___ (SD = xx.xx).
Scores on our subscales [skip this section if you do not have subscales]
The average for subscale ___ was ___ (SD = xx.xx). The average for subscale ___ was ___ (SD = xx.xx). The subscales were correlated r = . .
Internal consistency
To determine our scale’s internal consistency, we took several steps. First, we computed a correlation between the individual items and the scale as a whole. We predicted that these correlations would be between +.3 and +1.0 for most items and between -.3 and -1.0 for the reverse scored items.
As predicted, the correlations for the positively worded questions (insert question numbers) were above +.3. Specifically, the correlations between questions (insert question numbers here) _, _, _, … and the entire scale were between ____ and ____. However, contrary to expectations, the correlation between question(s) ___ and the scale were low (r = .__) and the correlation between question(s)__ and the scale was negative (r = - . ).
Also as predicted, the correlations for reversed scored questions (insert question numbers) were between -.3 and -1.0 . Specifically, the correlations between questions _, _, _, … and the entire scale were between ____ and ____. However, contrary to expectations, the correlation between question(s) ___ were low (r = .__) and the correlation between question(s)__ and the scale was positive (r = . ).
[Skip this paragraph if you had to skip #15] To get another view of our scale’s internal consistency, we calculated a split-half reliability. Specifically, we correlated the (insert either “first half” or “odd-numbered items”) with the (insert either “second half” or “even-numbered items”). Calculated this way, the scale’s split-half reliability was ____.
Convergent validity
We expected that our scale would correlate with other measures of the construct. Specifically, we thought that scores on our scale would correlate with ______. (Insert either “As expected,” or “Contrary to expectations,”) the scale correlated with _____ (r= ____).
Discriminant validity
Our scale was designed to measure ______, not _____. To establish that our measure was not measuring _____, we correlated our scale with ______. Consistent with (Contrary to) our prediction, the correlation between our scale and ____ was low (high) (r= ____).
Hypotheses
We also hypothesized that ______________. To test this hypotheses, we (did ____ analysis comparing ____ with ______). We found that (insert either correlation between variables or means of the groups). This (insert “correlation” or “difference between means”) was (not) in the predicted direction and (but) was but was (not) statistically significant (insert values of statistic, degrees of freedom, and p; e.g., t (44) = 12.60, p < .05; r (50) = .80, p < .05 ). Thus, the hypothesis that ____ was (supported, not supported, refuted).
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
16. If you have time, go through these tutorials.