Scores Calculation
Scores Calculation
Post-comfort questionnaire
Simulator Sickness Questionnaire, developed by Kennedy et al. (1993) is a self-reported checklist of 16 symptoms categorised into three subscales:
Nausea (e.g. general discomfort, increased salivation, sweating, nausea, difficulty concentrating, stomach awareness, dizziness, digestive disorders)
Oculomotor (e.g. general discomfort, fatigue, headache, eyestrain, difficulty focusing, difficulty concentrating, blurred vision)
Disorientation (e.g. difficulty focusing, nausea, fullness of head, blurred vision, dizziness with eyes closed, dizziness with eyes open).
Participants indicate for each symptom to what degree it was possibly experienced, using a 4-point scale: "none", "a little", "quite a lot", "intense" which in the calculation of the values will be worth 0, 1, 2, 3 respectively.
Points analysis expressed by the participants will produce a sub-score for each of the three categories (nausea, oculomotor, disorientation) given by the sum of the item scores multiplied by the weight of the relevant column.
Scores calucation:
Nausea = [1] x 9.54
Oculomotor = [2] x 7.58
Disorientation = [3] x 13.92
Total Score = ([1] + [2] + [3]) x 3.74
The three subscales are combined to produce a total severity score (TS) ranging from 0 to 235.62 . The TS score is an indicator of the overall distress experienced by the subjects during the simulation (Johnson, 2005).
VR system usability evaluation
System Usability Scale questionnaire, by John Brooke [Brooke 2013], to provide users with statements describing the usability of a product [Brooke 1996] and evaluate:
Effectiveness : can users successfully achieve their objectives?
Efficiency : how much effort and resource is expended in achieving those objectives?
Satisfaction : was the experience satisfactory?
In order to answer the questions in the questionnaire, users have to respond to a Likert scale that consists of 5 choices. The order of the questions is not random and cannot be changed without compromising the effectiveness of the test.
The 10 System Usability Scale Questionnaire (SUS) questions:
1. I think that I would like to use this system frequently.
2. I found the system unnecessarily complex.
3. I thought the system was easy to use.
4. I think that I would need the support of a technical person to be able to use this system.
5. I found the various functions in this system were well integrated.
6. I thought there was too much inconsistency in this system.
7. I would imagine that most people would learn to use this system very quickly.
8. I found the system very cumbersome to use.
9. I felt very confident using the system.
10. I needed to learn a lot of things before I could get going with this system.
After the participants have answered the questions, a SUS score is then calculated to evaluate the system:
Considering questions separately: Odd (1, 3, 5, 7 and 9) and Even (2, 4, 6, 8 and 10)
Add up the total score for all odd-numbered questions, then subtract 5 from the total to get (X).
Add up the total score for all even-numbered questions, then subtract that total from 25 to get (Y).
Add up the total score of the new values (X + Y) and multiply by 2,5 to obtain the final SUS score. This gives a score ranging from a minimum of "0" to a maximum of "100".
Scores below 68 point to issues with the design that need to be researched and resolved, while scores higher than 68 indicate the need for minor improvements to the design.
Sense of presence evaluation
Witmer and Singer questionnaire [Witmer, Jerome, Singer - 2005] to quantitatively measure the degree of presence experienced within a virtual environment by each subject and, moreover, how specific elements of the experience might have influenced the intensity of the experience.
The scale is among the most widely used [Pianzola, 2020] and offers 7 choices with different descriptive labels.
Each question of the questionnaire corresponds to one of the following items: "Involvement", "Sensory Fidelity", "Adaptation/Immersion", "Interface Quality", “Auditory”, “Haptic”.
Presence Questionnaire item Stems (Version 3.0) questions:
1. How much were you able to control events?
2. How responsive was the environment to actions that you initiated (or performed)?
3. How natural did your interactions with the environment seem?
4. How much did the visual aspects of the environment involve you?
5. How natural was the mechanism which controlled movement through the environment?
6. How compelling was your sense of objects moving through space?
7. How much did your experiences in the virtual environment seem consistent with your real world experiences?
8. Were you able to anticipate what would happen next in response to the actions that you performed?
9. How completely were you able to actively survey or search the environment using vision?
10. How compelling was your sense of moving around inside the virtual environment?
11. How closely were you able to examine objects?
12. How well could you examine objects from multiple viewpoints?
13. How involved were you in the virtual environment experience?
14. How much delay did you experience between your actions and expected outcomes?
15. How quickly did you adjust to the virtual environment experience?
16. How proficient in moving and interacting with the virtual environment did you feel at the end of the experience?
17. How much did the visual display quality interfere or distract you from performing assigned tasks or required activities?
18. How much did the control devices interfere with the performance of assigned tasks or with other activities?
19. How well could you concentrate on the assigned tasks or required activities rather than on the mechanisms used to perform those tasks or activities?
20. How much did the auditory aspects of the environment involve you?
21. How well could you identify sounds?
22. How well could you localize sounds?
23. How well could you actively survey or search the virtual environment using touch?
24. How well could you move or manipulate objects in the virtual environment?
25. How completely were your senses engaged in this experience?
26-28_Revised (removed from the study 3 - Witmer et al. 2003)
29. How easy was it to identify objects through physical interaction, like touching an object, walking over a surface, or bumping into a wall or object?
30. Were there moments during the virtual environment experience when you felt completely focused on the task or environment?
31. How easily did you adjust to the control devices used to interact with the virtual environment?
32. Was the information provided through different senses in the virtual environment (e.g., vision, hearing, touch) consistent?
In order to calculate a category's total score, e.g. "Interface Quality", all questions in the questionnaire that are related to that factor are added together. This produces the "Interface Quality" category score for a given participant. Similarly, the rest of the sums are calculated for all categories according to the 4-factor model proposed by Witmer et al. in the third study (2003).
Factor 1=INVOLVEMENT (1,2,3,4,6,7,8,10,14,17,18,29)
Factor 2=ADAPTATION/IMMERSION (9,20,21,24,25,30,31,32)
Factor 3= SENSORY FIDELITY (15,16,5,11,12,13)
Factor 4= INTERFACE QUALITY (19,22,23)
AUDITORY = 5,11,12
HAPTIC= 13,17
The total sum of the averages of all the subcategories listed is the final Presence score.
Training effectiveness evaluation
Errors and task completion, training score
Passing and scoring exams/certifications
Repeating the experience after some time
Task execution evaluation and final exams (comparison of VR training and real).