For this second test, my goal was, again, to obtain statistics relating to the first part of my design criteria. The only difference this time was I tested more people. Essentially, I tested a small group of students and teachers (48 total responders) and asked them a couple of questions relating to the number of clicks to complete each method, the number of highlights to complete each method, and how long it took them to complete each method while filling out a randomly selected academic planner.Â
Responder Groups of People Chart:
The overall percent difference from this data, as shown in the picture on the right, from students, was about 29%.
Looking at the picture on the left, it's clear that the overall spread of data was very spread out among staff and students. I then needed to calculate a percent difference for this data compared with all of GMHS.
In terms of percent difference from the teacher side of the respondents, there isn't one, as I only added teachers into the mix, to give me a bigger variety of users. With a larger variety of users, it would help me user-proof the website for future users as your age/technical know-how wouldn't hinder your ability to complete this new website method
Random Academic Planner Groups Chart:
Now, since each person who completed test #2 was given a random academic planner to use, I also calculated a percent difference for the overall spread of 9th, 10th, 11th, and 12th, grade planners used. This time, the percent difference was 14.06%.
Now, considering that this survey was completely optional, and not everyone had to participate in each class that I tested, I would argue that a percent difference of just above 20% is pretty good for this test.
61.25% net decrease in total clicks (Criteria Met)
100% net decrease in total highlights (Criteria Met)
20.48% net decrease in total time (Criteria Not Met)
The margin of error in this set of data is tough to decipher. I say this because all of the data inputted by one person could have been vastly different than what someone else inputted. This is because I didn't specify a margin by which people needed to enter their data. This theoretically wouldn't be an issue, however, since everyone who completed this test was supposed to complete both the document method and the website method for their given transcript. Then, since they would complete both methods, their rounding methods would also be the same for both methods, reducing the issue of whether or not their inputs would skew my data in the end. Whether everyone still did both methods is still, unfortunately, left up to the tester. Long story short, the margin of error for the data could be anywhere from 1 click to hundreds of clicks, or 1 second to a couple minutes. If I were to do this test again, I would specify a margin on which I want the testers to round their numbers to.Â
Some possible experimental errors are:
Incorrectly counting the number of clicks while filling out either method of pre-check
Incorrectly stopping/starting stopwatch when beginning or ending each method of pre-check
Misunderstanding how to use either method of pre-check and taking additional time than what is needed
Incorrectly counting the number of highlights used in the document method of pre-check
Misunderstanding instructions in the survey and giving false data
Due to lack of motivation, not giving accurate data in survey response
Due to missing classes in the website database, not understanding to skip these classes, and instead giving up
The idea of random chance not being fully removed as a culprit when discussing my data and not keeping my "p-value" below the 0.05 margin.
Even though I did manage to meet two of my criteria, when it comes to the timing of things, there were a couple of things that people still seemed to struggle with. These things were:
Understanding that you had to enter all classes from your high school career
Understanding that you had to enter both semesters of classes, as opposed to just one, and the other class will add itself.
Understanding what certain classes were named
Knowing that if you couldn't find a class just skip it
Knowing that the academic planner was your best friend for completing this website method of pre-check
How to effectively search for classes and use the search bar as a whole.
After completing this test, it seems clear to me that instructions were still the number one issue of my new method of pathway precheck. If I were to give better and more straightforward instructions or adjust the way people enter classes to a sectioned-off set-up, people might have been able to reach that 50% decrease in the total time mark.
Document Method
Among the 8 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
76.5 total clicks
10.8 highlights
9 minutes
& 2 of the 8 people gave up
Website Method
Among the 13 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
55 total clicks
0 highlights
4.71 minutes
& No one gave up
9th - Overall
28.1% decrease in total clicks
100% decrease in total highlights
47.67% decrease in total time
9th - "2-Samp T-Tests" (Statistical Proof)
These tests were conducted a check step to ensure that my data was not caused due to random chance.
The results of this data show how the "p-value" for both clicks and total time is greater than 0.05. This, unfortunately, means that we can't statistically prove that this data was not skewed by random chance. This, however, does not invalidate my data, but is rather an experimental error.
Document Method
Among the 5 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
76 total clicks
17.5 highlights
8.17 minutes
& 1 of the 5 people gave up
Website Method
Among the 9 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
21.89 total clicks
0 highlights
5.55 minutes
& No one gave up
10th - Overall
71.2% decrease in total clicks
100% decrease in total highlights
32.07% decrease in total time
10th - "2-Samp T-Tests" (Statistical Proof)
These tests were conducted a check step to ensure that my data was not caused due to random chance.
The results of this data show how the "p-value" for both clicks and total time is greater than 0.05. This, unfortunately, means that we can't statistically prove that this data was not skewed by random chance. This, however, does not invalidate my data, but is rather an experimental error.
Document Method
Among the 5 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
100 total clicks
12.5 highlights
7.52 minutes
& 1 of the 5 people gave up
Website Method
Among the 9 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
33.78 total clicks
0 highlights
8.42 minutes
& No one gave up
11th - Overall
66.22% decrease in total clicks
100% decrease in total highlights
11.97% increase in total time
11th - "2-Samp T-Tests" (Statistical Proof)
These tests were conducted a check step to ensure that my data was not caused due to random chance.
The results of this data show how the "p-value" for both clicks and total time is greater than 0.05. This, unfortunately, means that we can't statistically prove that this data was not skewed by random chance. This, however, does not invalidate my data, but is rather an experimental error.
Document Method
Among the 7 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
268.2 total clicks
25.6 highlights
12.3 minutes
& 2 of the 7 people gave up
Website Method
Among the 10 responders who filled out the website method and tested for time, clicks, and highlights, the averages were...
55.1 total clicks
0 highlights
10.56 minutes
& No one gave up
12th - Overall
79.46% decrease in total clicks
100% decrease in total highlights
14.15% decrease in total time
12th - "2-Samp T-Tests" (Statistical Proof)
These tests were conducted a check step to ensure that my data was not caused due to random chance.
The results of this data show how the "p-value" for both clicks and total time is greater than 0.05. This, unfortunately, means that we can't statistically prove that this data was not skewed by random chance. This, however, does not invalidate my data, but is rather an experimental error.
At the end of my survey, I asked for any additional feedback that the user might have. The main things that were listed were:
Some honors/AP classes were missing
It was tedious to not be able to find classes by just typing in a few characters from each word in name of the class
It was a little confusing on what classes everyone was supposed to enter
High school career, vs. current year
Some classes didn't seem to count when user's clicked to the next page
Most people appear to like the idea of SHERPA and that with a little more refinement, it can be a very powerful tool for students at GMHS