Simuli: Spring 2018
Authors: Jill Salembier and Bill Fischer
See video screen captures and download the VR application here
Overview
- Type of study: Focus Group Analysis.
- January 30th & February 1st .
- 34 students – 7th Graders age: 12-13, from Grand Rapids Public Museum School.
- 18 Boys / 16 Girls.
- Only scenario 1 (of 3) was tested, (scenarios 2 and 3 were still in development).
- No preliminary instruction or information was provided to the students prior to the test.
Simuli is a work-in-progress. This Unity produced, Google Cardboard app aims to provide a VR experience for middle, high school and college students that simulates, in the first person, the feeling of some of the challenges that neurodiverse students face in a classroom setting. The goal is to develop empathy and understanding that will lead to improved interpersonal, social, learning and community relationships across all neuro spectrums (including "neurotypical"). The app focuses on the social aspects of experience neurodiverse challenges in a school setting.
The room was set up with an area for four students to view the scenario at one time. Each student was sitting in a chair and provided View Master brand VR goggles that used mid-range Android phone to run the app. Once the scenario had finished, students were escorted to two separate tables to be interviewed about what they had just participated in.
Publish Quality Settings
During testing the it was found that the following Unity publishing settings were required for smooth playback.
- Mid-range quality phones (from $150 - $250 price range): "Beautiful"
- High-end phones (above $250)): "Fantastic".
It will be difficult to predict which version will playback smoothly on any particular phone, so two versions have been made available for download (HiRes, LowRes)
Summary of Results
In order to isolate the effectiveness of the simulation, no instruction or introduction was provided before the students experienced the simulation. Also, only one of the three simulations was tested (the other two were still in development). This test was administered to acquire mid-development feedback.
- All subjects were all able to understand what the purpose of the app was after experiencing the simulation.
- It's unclear from the test what the students were looking at and when during the scenarios. Future tests will need to address that issue.
- Approximately 50% of the subjects indicated that they may have more empathy for others in the future. Their actual behavior was not tested at a later date, however.
- The limitations posed by a combination of a short development cycle and the Google Cardboard technology led to lower production values than high-end VR apps. But, that did not seem to get in the way of the messaging or the immersion into the narrative. The audio played a significant role in making up for the lack of animation and visual realism.
- 17% of the subjects stated that the simulation was stressful, when asked. However, no observable adverse reactions were triggered.
- Ongoing technology management during the testing was required. This included: making sure the phones were charged (the batteries drained quickly), the volume was set correctly (we used headphones with volume controls so the subjects could adjust them) and the app was set back to its starting point for the next subject. This required two KCAD students at first. But with a little practice, one could handle it.
Observations During Testing
- 100% of students paid attention.
- 50% sat still the entire time looking primarily in one direction (at the action in the scenario).
- 50% were spinning and looking around the VR classroom while viewing the scenario.
- There was sometimes Giggling.
- About half were actively looking around the room with the VR goggles long after the experience was over.
Responses To Post Test Questions
Have you ever experienced other VR simulations or games?
- Yes – 16 (47%)
- No – 7 (21%)
- No response - 11 (32%)
Where and What kind?
(percent taken from 16 total responses to having VR experience).
- Whale VR at museum (using Oculus Rift) - 5 (31%).
- Iphone (1-location was a park on rides, 1-haunted house) - 2 (13%)
- Hunting game - 1 (6%)
- Has own VR headset - 1 (6%)
- No response regarding their previous VR experience - 7 (44%)
How does this experience compare to your previous VR experiences?
(percent taken from 16 total responses to having VR experience).
- Visuals were cooler and more realistic in other VR experiences - 5 (31%).
- Like our VR project better than museum whale because it teaches something and they could look around more and felt part of a story or scene - 3 (19%).
- No responses from remainder of the group - 8 (50%).
Was the experience stressful?
- A Little - 9 (26%)
- Yes - 6(17%)
- No - 8 (24%)
- No response or comment (33%)
Was the scenario realistic?
Of those that responded (approx. 50%), 1/2 the students said Yes, and other 1/2 said No.
Of the students that responded "yes", they indicated that they responded to the feelings of the event, the sounds and the room environment. Some said the scenario (story) was realistic even though the graphics were not.
The reasons given for not being realistic are:
- The artwork is not realistic and too cartoony to be realistic.
- Characters mouths were not moving.
- Felt the characters were creepy.
- Thought it strange that all the tables were pushed together in the classroom.
Do you know anyone that taps or something similar? And, why do you think they do it?
- Of those that responded (approx. 50%), 90% said YES, they know someone that taps or something similar.
- Some students feel that other kids do this because they are bored or are just doing this to be annoying or funny. A few students felt that tapping stems from stress which they see when taking a test (eg: math).
- A couple of students thought the app was simulating some type of autism and that they know, or have a family member with autism.
What might you do differently in the future?
(percent taken from 29 total responses to question above)
- To be nicer to people; don’t bully and judge others - 9 (31%).
- Not sure - 4 (14%).
- Should ask others if they are ok - 5 (17%).
- No responses from remainder of group - 11 (38%).
What did you learn?
#1 Response - Why some kids may pencil tap and how tapping can be distracting to others.
- Students felt some kids react this way because they have anxiety and get nervous (possible because they want to share ideas and are ignored).
- Kids can feel left out.
- They are not comfortable in class.
- They have a hard time socializing.
- People need attention.
Will you be more understanding in school now?
- Yes – 17 (50%) Will be more understanding and ask others that seem to be struggling if the are OK).
- No – 9 (26%) Said they thought pencil tappers do it to be annoying. (1 student said they don’t want to elicit drama by saying something).
- Probably Not - 3 (10%).
- No response - (23%).
Responses to "How can we improve our app?"
#1 Response
Characters need improvement - 10 (29%).
- make realistic art - higher quality models .
- more texturing.
- mouth movements, movement overall, detailed animation.
- characters all looked the same and felt creepy the way they look and stare.
#2 Response
More interaction is desired - 5 (15%).
- Ability to socialize with people in VR.
- Ability to get up and move.
- Interactive controls with buttons and options like a video game.
#3 Response
Demonstrate ways to handle the situation in the explainer video so they learn what they should do in this situation - 4 (12%).
#4 Response
Other: three or less responses to each bullet.
- Desired the ability to see your own legsand not just chair - 3 (10%).
- Desired more scenarios (eg: fidgeting, kicking chairs + stimming) - 3 (10%) .
- The visuals were blurry - 3 (10%).
- The audio was hard to hear - 2 (7%).
#5 No Responses
- From remainder of the Group - 17%.