Project 2- Sound in Virtual Reality

Unity Project for HTC Vive

Important Links

How to Run

Videos

Sound Placement Scene Video

Sound Shapes Scene Video

Important Links

Code Base

Sound Placement Scene Video

Sound Shapes Scene Video

How to Run

The project is developed on Unity Version 2019.2.11f . Download the version of unity mentioned.
Download Steam VR
Download or clone the project from the Code Base
Once the project is download go to the following path: CS528Project2/Assets/Scenes/ and open the SoundPlacement Unity Scene File.
Run the SoundPlacement Unity Scene File.

Note: By default the project should run on the desktop using the simulator. If the HTC Vive is connected make sure Steam is running and the project should run on the HTC Vive.

Videos

Sound Placement Scene Video

Sound Shapes Scene Video

Overview

This project was created as a part of the course work requirement for CS 528 taken at University of Illinois at Chicago in Spring 2020. The project was developed using Unity and allows the user to explore and manipulate sounds in virtual reality via the HTC Vive platform.

The application has two scenes: Sound Placement and Sound Shapes each of which enables the Vive user to manipulate and experience sounds in different ways.

Sound Placement

Sound Shapes

Sound Placement

On launching the application the user is shown the Sound Placement scene. The main goal of the Sound Placement scene is to allow the user to place sounds and manipulate sounds properties in a virtual exhibition center environment in order to customize the sound experience in the exhibition.

The Sound Placement scene shows a exhibition center with 6 exhibits and a central area. Each exhibit has sounds in it which are represented by different objects. The convention center by default has three background sounds which the user can manipulate too. Each background sound is also associated with an object.

The sounds in the exhibition center's exhibits are by default 3D sounds and the three background sounds are 2D sounds.

The main aim of the Sound Placement scene is to allow the user to customize the sound experience in the exhibition center according to user preference.

Objects which Represent Sounds aka Sound Objects

Throughout the Sound Placement scene there are clearly distinguishable objects which represent sounds. The objects which represent sound are of three types.

Type 1: Sound Objects which have a red or green sphere on top.
Type 2: Sound Objects which have a black sphere on top. These are all the microphones objects placed throughout the scene.
Type 3: A Red Cube. The Red Cube object used to manipulate the volume of all the 2D background sounds in the scene.

Type 1 Sound Objects: Sound Objects with Red/Green Sphere

Type 1 sound objects can represent 3D sounds or 2D background sound throughout the scene.

The background sounds are playing by default when the scene is launched. The other 3D sounds are not playing by default when the scene is launched and are stopped.
The red sphere on the sound object indicates that the sound is stopped.
The green sphere on the sound object indicates that the sound is playing. If the green sphere is oscillating the sound is looped.
On touching these sound objects with the left controller, a Radial Menu' opens up which can be used to manipulate the sound and sound properties.
Pressing the grip of either controller can be used to move these objects anywhere within the space when the controller is touching the sound object.

Radial Menu for Type 1 Sound Objects

The 'Radial Menu' as represented in the left image pops up when the left controller is touching these Type 1 sound objects. The Radial Menu disappears when the left controller stops touching these sound objects.

This radial menu has 4 menu options. The menu options can be selected by hovering the left controller over the menu option. The selection can be confirmed using the left controller trigger press.

The 4 menu options are:

Play and Pause: Used to play and pause the sound associated with the sound object.
Stop: Used to stop the sound associated with the sound object.
Hamburger Menu: Toggles between opening and closing the 'Sound Properties' Menu.
Loop: Toggles between looping and not looping the sound associated with the sound object.

Type 2 Sound Objects

These sound objects are all microphones and are placed in front of characters standing in front of each of the six exhibits in the exhibition.
When you walk near these microphone sound objects,the sounds associated with them plays.
These sounds are only played when the player walks near the characters who has the mic sound objects in front of them.
These sound objects cannot be moved and placed anywhere in the scene. Their placement remains static in the scene. However, the sound properties of sound associated with these sound objects can be manipulated.
On touching these sound objects with the left controller opens up the Radial Menu to manipulate the sound and sound properties associated with that object.

Radial Menu for Type 2 Sound Objects

The 'Radial Menu' as seen in the left image pops up when the left controller is touching these Type 2 sound objects. The Radial Menu disappears when the left controller untouchables these sound objects.

This radial menu has one menu option. The Hamburger Menu option can be selected by hovering the left controller over the menu option. The selection can be confirmed using the left controller trigger.

Following is the menu option for this radial menu

Hamburger Menu: Toggles between opening and closing the 'Sound Properties' Menu. The Sound Properties menu is used to manipulate sound properties.

Red Cube

The Red Cube can be interacted with to manipulate the background sounds volume.
The red cube is a Type 3 sound object.
The red cube is placed in the central area.
On touching the red cube with the left controller, a 'Radial Menu' opens up through which the 'Audio Mixer' menu can be opened to manipulate the volumes of the background sounds.
Pressing the grip of either controller can be used to move the red cube sound object anywhere withing the space when the controller is touching the red cube sound object.

Radial Menu for Red Cube

The 'Radial Menu' as seen in the left image pops up when the left controller is touching the red cube sound object. The Radial Menu disappears when the left controller untouchables the red cube.

This radial menu has one menu option. The Hamburger Menu option can be selected by hovering the left controller over the menu option. The selection can be confirmed using the left controller trigger.

Hamburger Menu: Selecting this menu option toggles between opening and closing the 'Audio Mixer Menu' used to control background sounds volumes.

Sound Placement Menus

The sound placement scene has the following menus:

Sound Properties Menu
Audio Mixer Menu
Scene Management Menu.
Radial Menus. (Described in the previous section)

Sound Properties Menu

The Sound Properties menu can be opened by selecting the Hamburger Menu Option on any of the radial menus associated with the sound objects, except that of the Hamburger Menu Option of Radial Menu of the 'Red Cube' object.
The Hamburger Menu option associated with the Red Cube sound object's Radial Menu opens the Audio Mixer menu.
This Sound Properties menu provides options to manipulate the sound properties of the sound.
User can manipulate the sound properties of the sound associated with the sound object from whose radial menu triggered the Sound Properties menu.

Sound Properties Menu Details and Menu Options

The sound properties menu has the following details and menu options.

Name

Name of the sound whose properties is being manipulated.

Volume

The user can adjust the volume of the sound.

Current volume of the sound is shown when the menu is opened up. The user can use the volume slider to set the volume of the sound between the range of minimum and maximum value. The volume has three labels: 1. minimum value, 2. maximum value 3. current volume.

The default volume value for each sound is 1.

Priority

The user can adjust the priority of the sound.

Current priority of the sound is shown when the menu is opened up. The user can use the priority slider to set the priority of the sound between the range of minimum priority value and maximum priority value. The priority has three labels: 1. minimum priority value, 2. maximum priority value 3. current priority.

The default stereo priority for each sound is 128.

Pitch

The user can adjust the pitch of the sound.

Current pitch of the sound is shown when the menu is opened up. The user can use the pitch slider to set the pitch of the sound between the ranges of minimum pitch value and maximum pitch value. The pitch has three labels: 1. minimum pitch value, 2. maximum pitch value 3. current pitch.

The default pitch value for each sound is 1.

Stereo Pan

The user can adjust the stereo pan of the sound.

Current stereo pan of the sound is shown when the menu is opened up. The user can use the stereo pan slider to set the stereo pan of the sound between the ranges of minimum stereo pan value and maximum stereo pan value. The stereo pan has three labels: 1. minimum stereo pan value, 2. maximum stereo pan value 3. current stereo pan.

The default stereo pan value for each sound is 0.

Spatial Blend

The user can adjust the spatial blend of the sound.

Current spatial blend of the sound is shown when the menu is opened up. The user can use the spatial blend slider to set the spatial blend of the sound between the ranges of minimum spatial blend value and maximum spatial blend value. The spatial blend has three labels: 1. minimum spatial blend value, 2. maximum spatial blend value 3. current spatial blend.

If the spatial blend is zero the sound becomes a 2D sound. If the spatial blend is one the sound becomes a 3D sound.

The default spatial blend value for each sound is 1. Default spatial bend for each background sound is 0.

Reverb Mix

The user can adjust the reverb mix of the sound.

Current reverb mix of the sound is shown when the menu is opened up. The user can use the reverb mix slider to set the reverb mix of the sound between the ranges of minimum reverb mix value and maximum reverb mix value. The reverb mix has three labels: 1. minimum reverb mix value, 2. maximum reverb mix value 3. current reverb mix.

The default reverb mix value for each sound is 1.

3D Sound Properties

If the sound is a 3D sound the user can manipulate the following 3D sound properties too.

Doppler Level

The user can adjust the Doppler level of the sound.

Current Doppler level of the sound is shown when the menu is opened up. The user can use the Doppler level slider to set the Doppler level of the sound between the ranges of minimum Doppler level value and maximum Doppler level value. The Doppler level has three labels: 1. minimum Doppler level value, 2. maximum Doppler level value 3. current Doppler level.

The default Doppler level for each sound is 1.

Spread

The user can adjust the spread of the sound.

Current spread of the sound is shown when the menu is opened up. The user can use the spread slider to set the spread of the sound between the ranges of minimum spread value and maximum spread value. The spread has three labels: 1. minimum spread value, 2. maximum spread value 3. current spread.

The default spread value for each sound is 0.

Minimum Distance

The user can adjust the minimum distance of the sound.

For each sound the minimum distance value is limited between 0.1 to 19. The default minimum distance value for each sound is 1 on application startup.

Current minimum distance of the sound is shown when the menu is opened up. The user can use the minimum distance slider to set the minimum distance of the sound between the ranges of minimum min distance value and maximum minimum distance value. The minimum distance has three labels: 1. minimum min distance value, 2. maximum minimum distance value 3. current minimum distance.

Maximum Distance

The user can adjust the maximum distance of the sound.

For each sound the maximum distance value is limited between 0.2 to 20. The default maximum distance value for each sound is 20 on application startup.

Current maximum distance of the sound is shown when the menu is opened up. The user can use the maximum distance slider to set the maximum distance of the sound between the ranges of minimum maximum distance value and maximum max distance value. The maximum distance has three labels: 1. minimum maximum distance value, 2. maximum max distance value 3. current maximum distance.

Volume Roll Off

The user can adjust the roll off type of the sound.

The current roll off of the sound is shown when the menu is opened up. For each sound the user can select the drop down values select the volume roll off for each sound.The roll off values are

Linear
Logarithmic
Custom

By default the Liner Roll off is selected for each sound. The user cannot adjust the roll off but only can select between the type of roll off.

Reset Values

The user can hit the reset values button to reset all the sound properties to default.

Audio Mixer Menu

The Audio Mixer menu is used to adjust the volumes of the three background sounds all together or individually.

The three background sounds are: 1. Background Music, 2. Background Chatter and 3. Air Condition Vibrations.

The three background sounds are all connected to a audio mixer group. This audio mixer group has a sub group for each background sound.

The Hamburger Menu option associated with the Red Cube sound object's radial menu opens the Audio Mixer menu.

The Audio Mixer menu has the following menu options

Master Slider

The master slider is used to manipulate the volume of all the three background sounds together. The audio mixer takes care of adjusting sound intensities of the individual sounds. When the menu opens up the current intensity value of Master audio mixer group is set to the slider.

Background Music Slider

The BG Music slider is user to manipulate the volume of the BG music. When the menu opens up the current intensity value of BG Music audio mixer subgroup is set to the slider.

BC Chatter Slider

The BG Chatter slider is user to manipulate the volume of the Background Chatter. When the menu opens up the current intensity value of BG Chatter audio mixer subgroup is set to the slider.

Air Con Slider

The Air Con slider is user to manipulate the volume of the Air Condition Vibration sound. When the menu opens up the current intensity value of Air Con audio mixer subgroup is set to the slider.

Scene Management Menu

The right controller button 2 is used to toggle on and off the 'Scene Management' menu.

The Scene Management menu has two buttons: 1. Refresh Scene and 2. Sound Shapes.

Refresh Scene

The Refresh Scene button refreshes the current Sound Placement scene.

Sound Shapes

The Sound Shapes button takes the user to the Sound Shapes scene.

HTC Vive Controller Controls for Sound Placement Scenes

Left Controller

The left controller when touching an object which represents a sound triggers a 'Radial Menu' having menu options to manipulate the sound and properties. When the left controller stops touching the sound object the 'Radial Menu' disappears.

The left controller can hover over the radial menu options to highlight the menu option to select on the radial menu.

7) The left controller trigger is used to confirm the selection of the 'Radial Menu' option chosen on the 'Radial Menu'.

8) Left controller grip is used to grab and move the sound objects when the left controller is touching the sound object.

Right Controller

The right controller when touching an object which represents a sound triggers a 'Radial Menu' having menu options to manipulate the sound and properties. When the right controller stops touching the sound object the 'Radial Menu' disappears.

The right controller can hover over the radial menu options to highlight the menu option to select on the radial menu. The right controller trigger press can confirm selection if radial menu option.

1) The right controller button 2 is used to toggle on and off the 'Scene Management' menu.

2) The right controller Touch pad is used for teleportation throughout the scene and for pointing at menu UI elements such as slider, buttons and checkbox on the 'Sound Properties', 'Scene Management' and 'Audio Mixer' menus. This does not include radial menu.

7) The right controller trigger press is used to select and manipulate menu UI elements such as slider, buttons and checkbox on the 'Sound Properties', 'Scene Management' and 'Audio Mixer Menu'.

8) Right controller grip is used to grab and move the sound objects when the right controller is touching the sound object.

Exhibition Center Exhibits

The exhibition center has a central area and six exhibits all having sounds which the user can manipulate and experience. These are as follows:

Central Area

The central area has an information center and is surrounded by the six exhibition exhibits. The central area has three background sounds. All the background sounds are 2D sounds by default and are represented by Type 1 sound objects in the scene.

The volume of all the background can be manipulated together by interacting with a Red Cube sound object. The red cube is also placed in the center of the central area.

The three background sounds are:

Chatter of People Talking (Represented by Loud Speaker)
Air Condition (Represented by AC Vent)
Background Music (Represented by Black Speaker)

Information Area

The central area has an information area which is denoted by the 'Information' text. If the user walks near the 'Information' text, the information panel comes up on the glass wall giving the user information about the application.

On walking away from the 'Information' text the information panel disappears.

Central Area Sounds and Sound Objects

The central area has 3 background sounds. All the the background sounds are represented by sound objects which are Type 1 sound objects. The central area also has the red cube sound object which is a Type 3 sound object.

Following are the sounds in the central area

Background Music

The background music sound is represented by the Black Speaker . This sound is a 2D by default. The sound is playing by default when the scene is launched.

Background Air Condition

The air condition sound is represented by the AC Vent. This sound is a 2D by default. The sound is playing by default when the scene is launched.

Background Chatter

The background chatter sound is represented by the Loud Speaker. This sound is a 2D by default. The sound is playing by default when the scene is launched.

Red Cube

The volume of the three background sounds can be manipulated by using Audio Mixer menu. The red cube's radial menu opens the 'Audio Mixer' menu with the hamburger menu option on it.

Auto Exhibit

This exhibit has is an area in the exhibition center for cars and bikes. This exhibit has three sounds associated with it.

Auto Exhibit Sounds and Sound Objects

The auto exhibit has three sounds associated with it. Two of these sounds are represented by Type 1 sound objects and one is represented Type 2 sound object.

Following are the sounds in the auto exhibit.

Car Engine Running

This sound is represented by the red car key and is a Type 1 sound object.

Bike Engine Running

This sound is represented by the black bike key and is a Type 1 sound object.

Auto Exhibit Welcome (Self Created)

The sound is represented by the microphone and has a black sphere on it. The microphone is a Type 2 sound object. The sound welcomes you to the auto Exhibit.

Electronics Exhibit

The electronics exhibit showcases laptops and mobile phones. The electronics exhibit has four sounds associated with it.

Electronics Exhibit Sounds and Sound Objects

The electronics exhibit has four sounds associated with it. Three of these sounds are represented by Type 1 sound objects and one is represented Type 2 sound object.

Following are the sounds in the electronics exhibit.

Mobile Phone Ringing

This sound is represented by the phone which is a Type 1 sound object.

Mobile Phone Message Tone

This sound is represented also by the phone which is a Type 1 sound object.

Laptop Startup Tone

The sound is represented by the laptop which is a Type 1 sound object.

Electronics Exhibit Welcome (Self Created)

The sound is represented by the microphone and has a black sphere on it. The microphone is a Type 2 sound object. The sound welcomes you to the electronics Exhibit.

Toys Exhibit

The toys exhibit showcases soft toys.

The toys exhibit has six sounds associated with it.

Toys Exhibit Sounds and Sound Objects

The toys exhibit has six sounds associated with it. Five of these sounds are represented by Type 1 sound objects and one is represented Type 2 sound object.

Following are the sounds in the toys exhibit.

Monkey Sound

This sound is represented by the monkey soft toy which is a Type 1 sound object.

Bear Sound

This sound is represented by the bear soft toy which is a Type 1 sound object.

Penguin Sound

The sound is represented by the penguin soft toy which is a Type 1 sound object.

Pig Sound

The sound is represented by the pig soft toy which is a Type 1 sound object.

Sheep Sound

The sound is represented by the sheep soft toy which is a Type 1 sound object.

Toys Exhibit Welcome (Self Created)

The sound is represented by the microphone and has a black sphere on it. The microphone is a Type 2 sound object. The sound welcomes you to the toys Exhibit.

Guitars Exhibit

The guitar exhibit showcases guitars.

The guitar exhibit has three sounds associated with it.

Guitar Exhibit Sounds and Sound Objects

The guitar exhibit has three sounds associated with it. Two of these sounds are represented by Type 1 sound objects and one is represented Type 2 sound object.

Following are the sounds in the guitar exhibit.

Guitar Music 1

This sound is represented by the yellow guitar pick which is a Type 1 sound object.

Guitar Music 2

This sound is represented by the purple guitar pick which is a Type 1 sound object.

Guitar Exhibit Welcome (Self Created)

The sound is represented by the microphone and has a black sphere on it. The microphone is a Type 2 sound object. The sound welcomes you to the guitar Exhibit.

Music Exhibit

The music exhibit showcases music played on the piano and drums.

The music exhibit has three sounds associated with it.

Music Exhibit Sounds and Sound Objects

The music exhibit has three sounds associated with it. Two of these sounds are represented by Type 1 sound objects and one is represented Type 2 sound object.

Following are the sounds in the music exhibit.

Piano Music

This sound is represented by the music note which is a Type 1 sound object.

Drum Music

This sound is represented by the drum stick which is a Type 1 sound object.

Music Exhibit Welcome (Self Created)

The sound is represented by the microphone and has a black sphere on it. The microphone is a Type 2 sound object. The sound welcomes you to the music Exhibit.

Robot Exhibit

The robot exhibit showcases robots of various types.

The robot exhibit has three sounds associated with it.

Robot Exhibit Sounds and Sound Objects

The robot exhibit has three sounds associated with it. Two of these sounds are represented by Type 1 sound objects and one is represented Type 2 sound object.

Following are the sounds in the robot exhibit.

Robot Talking

This sound is represented by the human robot object which is a Type 1 sound object.

Robot Sounds

This sound is represented by the sphere robot object which is a Type 1 sound object.

Robot Exhibit Welcome (Self Created)

This sound is represented by the microphone and has a black sphere on it. The microphone is a Type 2 sound object. The sound welcomes you to the robot Exhibit.

Sound Placement Scene Assets

Sound Placement Character Assets

Sound Placement Other Assets and Materials

Sound Shapes

The sound shapes scene takes place in space. You are in a room on a spaceship with glass walls looking out into space. You have the ability to create various shape objects of different colors in the room. Each shape object is associates with a music tune. On creating the shape object the music tone associated with the shape object plays. The user can destroy the shape objects too.

Once a shape is created by the user, the user can pause, stop, play, loop and unloop the music tone associated with the shape object. The user can also manipulate the volume of the music tone associated with the shape object. If the player destroys a shape object then the music tone associated with the shape gets destroyed too.

The main aim of this scene is to mix audio tunes by creating various shapes objects and experiment with audio mixing and creating shape object patterns. You can create the shapes objects of various sizes anywhere in the room. As you progress all the created the shape objects will form a interesting patterns or clusters of shapes objects.

In the scene, the user is surrounded by an audience of spaceships and astronauts who have gathered to watch the shape patterns you create and listen to the music you create while mixing the music tunes.

Shape Creation and Destroy

Creation

The user can create 8 unique shape objects of two shapes having different colors. Each shape object is associated with a music tone.

The user can select the shape object to create by selecting the menu options the radial menu associated with the left controller touch pad. By pressing the trigger of the left controller the shape selected in the radial menu can be created. The user can determine the size of the shape object created by holding down the trigger of the left controller for a longer time. Longer the trigger press of the left controller the larger the shape object gets created. Once a shape object is created the music tone associated with the shape object plays.

The music tone associated with the shape object is by default looping and its volume is 1.

Destroy

The user can destroy any of the shape objects created. The shape object can be destroyed using the right controller trigger press when the right controller is touching a shape object. The object being touch touched by the right controller is destroyed. Once the player destroys a shape object, the music tone associated with the shape gets destroyed too.

Manipulation

The user can move any created shape objects by grabbing the shape using the grip press of either left or right controllers. The shape can be moved anywhere within the transparent walled room.

Sound Shape Menus:

The Sound Shape scene has the following menus:

Radial Menu
Music Tone Menu
Scene Management Menu

Radial Menu

The radial menu is present on the touch pad of the left controller. The radial menu has 8 shape objects which the user can select to create.

The radial menu has the following 8 menu options:

Red Sphere
Yellow Sphere
Purple Sphere
Blue Sphere
Green Cube
Blue Cube
White Cube
Purple Cube

Music Tone Menu

With the left controller, the user can launch the Music Tone menu associated with a shape object when the left controller is touching the shape and the user presses the button 2 on the left controller.

The Music Tone Menu can be closed if it is open and the user presses the button 2 on the left controller while touching the shape object.

The Music Tone Menu allows the user to manipulate the music tone associated with that shape.

The Music Tone Menu has the following menu options:

Volume Slider: Allows the user to adjust the volume of the music tone associated with the shape object.
Loop Check Box: Allows the user to loop and unloop the music tone associated with the shape
Pause/Play Button: Allows the users to pause the music tone if playing and play the music tone if paused or stopped.
Stop Button: Allows the user to stop the music tone if the music tone is playing.

Scene Management Menu

The right controller button 2 is used to toggle on and off the 'Scene Management' menu.

The Scene Management menu has two buttons: 1. Refresh Scene and 2. Sound Placement.

Refresh Scene

The Refresh Scene button refreshes the current Sound Shapes scene.

Sound Shapes

The Sound Placement button takes the user to the Sound Placement scene.

Shape Objects

The user can create the following 8 shape objects. Each shape object is associated with a different music tone. User can repeat the creation of the same shape object many times.

Red Sphere

Yellow Sphere

Purple Sphere

Blue Sphere

Green Cube

Blue Cube

White Cube

Purple Cube

Music Tones

Each shape object has its own unique music tone. The music tones have been created by myself with the help of a friend using the Noise app on the apple application store.

Created Music Tones are as follows and they are found in the 'Music' folder in the source code.

Music1BaseMelody.wav associated with Red Sphere
Music1Sound2.wav associated with Yellow Sphere
Music1Sound3.wav associated with Purple Sphere
Music1Sound4.wav associated with Blue Sphere
Music2Sound1.wav associated with Green Cube
Music2Sound2.wav associated with Blue Cube
Music3Sound3.wav associated with White Cube
Muisc4Sound4.wav associated with Purple Cube

Left Controller

The left controller is used to create the shape objects. The left controller touch pad is associated with a radial menu with 8 menu options, each menu option indicates the shape object to create.

2) The left controller touch pad is used to select the the radial menu options to indicate which shape object. Touching the touch pad makes the menu appear.

7) The left controller trigger press creates the shape selected in the radial menu. Holding the trigger longer creates a larger shape object.

8) Left controller grip is used to grab and move the shape object created when the left controller is touching the shape object.

1) When the left controller is touching the shape object, the user can presses the button 2 on the left controller to launch the 'Music Tone' menu. If the 'Music Tone' menu if already open, it can be closed by pressing the button 2 while touching any shape object.

Right Controller

1) The right controller button 2 is used to toggle on and off the 'Scene Management' menu.

2) Touch pad is used for teleportation throughout the scene within the room with transparent walls and for pointing at menu UI elements such as slider, buttons, and checkbox on the 'Music Tone' menu and the 'Scene Management' menu. This does not include radial menu.

7) Trigger is used to select and manipulate menu UI elements such as slider, buttons and checkbox on the 'Music Tone' menu and the 'Scene Management' menu. The trigger is also used to destroy shape objects when the tight controller is touching a shape object.

8) Right controller grip is used to grab and move the shape objects when the right controller is touching the shape object.

Sound Shape Assets

Report

Interesting Things, Lessons and VR Implementation

While developing the application I learnt various things. Developing each scene had its challenges. I believe these learning can help me in the future. There were also many interesting things to note during the development process. In this section I will give a brief overview of the learning, findings and interesting things I came across for each scene in my application

Sound Placement Scene

The main aim of the Sound Placement scene is to allow the user to customize the sound experience in the exhibition center according to user preference. To do this, the conceptualization of the scene was a process. I wanted to give the user an option to play with both 3D sounds and 2D sounds. The representation of the sounds, the number of sounds and coming up with controls mappings to manipulate the sounds in a meaningful way was a challenge.

I realized that 2D background sounds can overpower the 3D sound especially when you want to adjust 3D sound properties. The Min Distance, Max Distance, Roll off, etc. make the 3D sound less prevalent in front of the 2D sounds especially if they have the same volume. It is necessary to allow the user to instant access to control the background 2D sound volume while the user is playing with adjusting 3D sound properties to get the sound right. To make this possible I introduced the Red Cube sound object which the user can take with them everywhere throughout the scene and it can be used to adjust the 2D background sound volumes via a audio mixer.

Another thing I explored and which was new to me was the use of an audio mixer which allows you to group sounds together. With the audio mixer I could create sound groups, sub groups and snapshots which allowed me to control the volume of the 2D background sounds all together and individually as well.

Secondly during testing of this scene I realized that some sounds are louder than others even at maximum volume. The quality of the sounds are also different. For example the car engine running sound is less louder than the phone message tone when both are at maximum volume. In order for the user to get a better understanding of what sound properties to set for each sound, it is essential that the all the sounds be recorded with the same quality and format which was not the case in all the sounds I searched and found online ti use in this project.

I also want to point out that although I am allowing the user to adjust the sound properties associated with the audio source for each sound, the application still does not take into consideration the impact of the environment on the sound. For example the sound properties can change due to the material of the walls, presence of objects in between the user and the sound source. As a future work this project can implement Google's Resonance Audio SDK for Unity plugin.

Sound Shapes Scenes

The main aim of this scene is to mix audio tunes by creating various shape objects which have audio tunes associated with them. I faced some difficulties in while implementing this scene. Initially I wanted to have a base tune and give the user the ability to put individual music notes of a particular instrument on the base tune so as to create a new sound tune which will overlay on the base audio tune. However, this idea became a disaster as creating the shape object such that the music note associated with it syncs with other notes to create a new tune which is harmonious is difficult. Getting sound notes to play in sync is a implementation challenge considering the time I had allotted for the Sound Shapes scene. This idea also requires some knowledge of music and sounds which I lack to create unique tunes out of individual sound notes.

Since the individual notes idea did not work, I went ahead trying to find music tunes which could mix well with each other so that the user can play music tunes on top of one another to create a mix of music tunes. I spent a while searching for new free music tunes. In the end I approached a friend to help me create tunes and we created some customized music tunes using the Noise application which is there on the Apple app store.

In this scene I wanted to use the Unity's post processing stack's bloom feature to introduce glowing effects into the scene when the sound shape is created. I implemented the and set up the post processing stack accurately but I did not see a big difference while using it. I found it better and simpler to just use the emission materials. I would still want to implement and take all look at all the features of the Unity's Post Processing Stack as a enhancement to this scene.

An interesting thing I found while using the application is using the VRTK pointer to manipulate a UI canvas menu with many menu controls can become challenging. The user can end up selecting a different UI control than the one intended. In order prevent the user from frustration of resetting all the menu control values in case of error, I provided the user with a 'Reset Values' button.

VR Technologies

The main aim for this project was to interact, play and manipulate with sound in VR through two different ways of sound placement and manipulation within the Sound Placement scene and sound creation and mixing within the Sound Shapes scene. In order to develop the application I considered a few VR options.

HTC Vive
CAVE 2
Quest

One of the major reasons I chose to develop the for the HTC Vive was the higher immersion over the CAVE 2 and single user experience. I also wanted the user to have more interaction capabilities to manipulate the sound objects, sound properties, create and delete shapes, interact with different types of menus. VRTK provided options to use the HTC Vive controllers in a variety of ways for interacting with different menus, grabbing, using objects and locomotion in the application.

Another reason for choosing the HTC Vive is access to the Vive. I was able to borrow the HTC Vive form my friend for a day or two during setting up and exploring the features of VRTK to use in this application. I also wanted to explore using the Quest for this application but I did not have access to the Quest headset expect for in the EVL. Due to the current lock down the Vive was more accessible.

Before staring a application development I downloaded the latest stable version of VRTK and went through the various example scenes and legacy example scenes in order to figure out menus and interaction with menus, possible interaction with objects and locomotion options provided via VRTK.

One of the major challenges was to design and map the controllers inputs to the variety of actions the user can perform within the two scenes. The design process of deciding which controllers inputs interacted with which menu, which menus to use and what should be the menu options along with incorporating teleportation and object interaction took some figuring out.

As for the code as I figured that I am the same menus can be to manipulate the sounds and the sound properties throughout both the scenes. Hence, In order to avoid repeating of the code, the code is written in a very generic way and gets executed based on the sound object passed to the functions. The code is not repeated for each sound object creating duplication and complication. Optimizing of the code to make it generic took a little more time but the code is more manageable and code maintenance is easier.

I do not think the use of VRTK in my application will allow the user to port this application to the CAVE. In order to run on the CAVE a lot of code changes would need to be make to implement the some of features of VRTK used in this application.

I want to thank professor Andy Johnson for all this guidance during the conceptualization and proposal of the project. Without his guidance I would have ran into unforeseen issues and and some of the issues I faced I was already expecting.

Page updated

Google Sites

Report abuse