Hearing Based Needs

• Provide subtitles for all important speech

• Provide separate volume controls or mutes for effects, speech and background / music

• Ensure no essential information is conveyed by sounds alone

• If any subtitles / captions are used, present them in a clear, easy to read way

• Keep background noise to minimum during speech

• Provide subtitles for supplementary speech

• Ensure subtitles/captions are or can be turned on before any sound is played

• Provide captions or visuals for significant background sounds

• Provide a visual indication of who is currently speaking

• Allow subtitle/caption presentation to be customised

• Support text chat as well as voice for multiplayer

• Provide visual means of communicating in multiplayer

• Allow a preference to be set for playing online multiplayer with players who will only play with / are willing to play without voice chat

• Ensure that all important supplementary information (eg. the direction you are being shot from) conveyed by audio is replicated in text / visuals

• Provide a stereo/mono toggle

• Ensure that subtitles/captions are cut down to and presented at an appropriate words-per-minute for the target age-group

• Provide signing

• Use symbol-based chat (smileys etc)

1. Picture Alterations - Some picture alterations can be used to help direct users within the virtual experience. The following categories are standard adjustable options within the VR/AR/AI platforms:

• Highlighted outline option for 3D objects and pictures - Option to enable a thin outline around important 3D objects or pictures. This attribute points the user to key objectives in the scene.

• Guideline - The technique to add a color or symbol to a spot of interest for the user to observe. Possible uses: Giving a quick way for users to detect key points in the narrative. Can define features that would otherwise be missed by the user, and could point to features such as the user controls, and options.

• Peripheral Remapping - Communicating key points of interest can be achieved by mapping out the peripheral vision during an experience. By briefly moving the picture peripheral to key points throughout the experience, the player can visually understand the objective without the reliance of text or audio communication.

2. Live Captions: Real-time print-out of useful information that is displayed in the field of view of the user. Described video is the audio definition of core scene elements, and includes features such as dialogue, important audio, and environmental cues.

3. Curved UI: A technique used to present visual data in a head mounted display format. It involves wrapping user interfaces such as text or pictures in a cylindrical, hemispherical, or spherical representation. This technique is important, as a flat screen in a moving 3D world is not possible, and UI must be situated in the real space.

4. Clear Sound Acoustics - Some users have trouble with the associated direction of a heard sound to a sound source. Depending on the user’s auditory acuity(ability to perceive sounds of low intensity), sound can be perceived from different locations. When possible, having simple, and clear audio sources as a baseline will help all users enjoy the audio experience. This technique can be summarized within the two points of environmental acoustics, and simulated acoustics.

• Environmental Acoustics: Within physical surround sound setups such as projector virtual reality setups, sound can bounce everywhere in the room. This will cause the situation of sound distortion, which will confuse users and overall reduce the audio fidelity of the scene. This is why it is important to install physical sound absorbing infrastructure (acoustic panel for example), which reduces the amount of times a sound wave will bounce in the real room space.

• Simulated Acoustics: Within virtual worlds, there exists the ability to mimic real world soundscapes. A rule of thumb is to have at most 3 active audio sources at once, where there is only one primary source active at a time. It is important to space the sound sources to about 45 degrees apart if they are direction dependent. It is also important to keep background noise or music at minimal during important primary dialogue. In conclusion, sound that emulated in the 3D world needs to maintain the same rules that in the real world, and keeping a simple soundscape is recommended for providing an accessible digital sound experience.

5. Pictorial and Artistic Representation of Music - It is important that each audio experience is paired with a visual experience. Music conveys much of the emotion for virtual media, and information is lost for users who cannot rely on the audio cues. A way to alleviate this is to make the representation of the music itself pictorial or artistic within a visual media. Through designing the VR/AR experience in a way that includes some form of music visualization, all users will benefit from a compelling experience.

Example of this method 1: Artistic Representation of Music

Example of this method 2: Pictorial Representation within VR

1. Non-Occluding (Bone Conducting) Smart Headphone - Some users require the volume of certain sounds to be amplified to be audible. This feature is standard for personal head mounted VR/AR headsets, however this accommodation for increased volume can not be made rationally in group experiences such as projector VR/AR setups.

• This issue can be alleviated with the use of a personalized smart headphone that tracks the user’s head orientation, and position in the projector area. When the user is within the play area of the projector VR/AR experience, the headphone will activate and provide both amplified, and directional audio to the user.

Example of this technology

• An important component of this headset (particular in the low vision and partial deaf communities) is to be a bone conducting model. It allows the user to still experience their natural environmental audio perceptions with the addition of the overlapped digital layer. This will make the net experience for the user more comfortable, and allow them to still interact with non- VR/AR environments in a nominal way. For more information, please visit the following project link:

HearThere Project Link

• Additionally in the field of augmented reality(AR), Facebook is developing a new AR glasses that specialize particularly in the real sound space. With the use of artificial intelligence, in real time it shifts out the important sounds over just background noise, giving clear, and if needed amplified audio to its users. The device being AR, can provide an experience.

2. Vibrotactile Wearables - Through an array of vibration coils woven into the material, the user can experience audio in a physical experience. The amount of vibration felt is typically represented based on the frequency (position of where vibration is felt) and amplitude (amount of vibration felt) of sound.

Example of this method 1: Sound Shirt : This suit was designed to directly cover real sounds into instantaneous vibration feedback. This suit helps the user experience sounds that are not digital, and provides a way for people with hearing loss to interface with highly audio experiences. This suit is highly useful for converting audio for projector virtual surround sound systems.

Example of this method 2: Tesla Suit: This suit was designed to interface with digital based audio experience such as those for head mounted display headsets. This set transfers audio within a digital world into tangible audio cues. Additionally this suit can be programmed to perform feats such as body tracking, which allows for to accurately represent vibrations for when the body interacts with objects in the virtual space.