Analyzing data from your observational study
1. See whether your observations were reliable.
• If your ratings involve placing people into unordered categories (e.g., you have nominal data), calculate the degree to which your ratings agreed with the other rater(s) by going to an online kappa calculator, such as
• http://faculty.vassar.edu/lowry/kappa.html ,
• http://justusrandolph.net/kappa/ , or
if you have “occurred”/"didn’t occur" data (e.g., "stopped" versus "didn't stop"), by going to
Enter the number of raters who noted that a certain case belonged in a certain category (the instructions say number of raters who “agreed” but that is not really what they meant).
From that analysis, report the percent of overall agreement and the free-marginal kappa
• If, on the other hand, your ratings are at least interval data (e.g., time, ratings on a 1 to 5 scale), use the Medical Education Online inter-rater reliability calculator, available at
or
just compute the correlation between your ratings using any program that computes correlations, such as the one at Vassarstats.
2. See whether your data supported your hypothesis. If you have nominal data (e.g., you counted how many times people did a given behavior), you will probably do a chi-square analysis.
• If you have a 2 X 2 contingency table like this
Did not stumble Stumbled
Not a Cell Phone User 20 3
Cell Phone User 2 16
Then, use the 2 X 2 contingency calculator available at this site
http://faculty.vassar.edu/lowry/tab2x2.html
If you have a 2 X 3, click here.
If you have a 2 X4, click here
If you have a 3 X 3, click here. http://faculty.vassar.edu/lowry/fisher2x4.html.
If you have more cells (e.g., a 4 X 4), use Kris Preacher’s chi square tool available at
http://people.ku.edu/~preacher/chisq/chisq.htm
• or
• http://www.physics.csbsju.edu/stats/contingency_NROW_NCOLUMN_form.html
• or
• http://faculty.vassar.edu/lowry/newcs.html (nice, because it gives you a correlation coefficient.
If you are only trying to see whether people deviate from a certain percentage (e.g., the percentage of people using computers to study is less than 50%), use the binomial probabilities table at http://faculty.vassar.edu/lowry/binomialX.html
If you have interval data, you can use the correlation calculator at Vassarstats.