"Musical Tiles" is an immersive VR experience that combines dynamic visuals, interactive spatial audio, and creative gameplay, showcasing the Oculus Quest's high-fidelity 3D sound and intuitive user interaction in a uniquely artistic environment.
Development Process
Creating "Musical Tiles" was an exciting journey into blending audio, visuals, and interactivity. The app's central concept revolves around balls moving across a plane, triggering piano notes when they collide with tiles. Each ball also leaves a mesmerizing particle trail, creating a dynamic and artistic environment that evolves with every interaction.
The development process began with the core mechanics:
Ball Spawning and Movement:
A script manages the spawning of balls at regular intervals, setting their initial position and velocity to ensure a steady stream of interaction points.
Each spawned ball has an initial velocity applied, allowing it to move across the plane naturally.
Tile Audio Playback:
Each tile is assigned a specific piano note. A script plays the corresponding note through an AudioSource whenever a ball collides with the tile.
The piano notes are pre-generated audio clips, and the script dynamically assigns the correct sound to each tile.
Particle Trails for Balls:
Balls generate trails as they move, creating a visually dynamic effect. This is achieved using a LineRenderer that updates each frame with the ball's position. The particles were initially considered but were not implemented; the LineRenderer provided a simpler and effective solution for creating visually appealing trails.
The trail fades over time by removing older positions, maintaining performance while enhancing visual appeal.
Ball Destruction:
To keep the environment clean, a script monitors the vertical position of each ball. If a ball falls below a certain threshold, it is automatically destroyed.
Using the XR Rig:
The XR Rig used in the app was imported directly from the Unity demo scene provided with the XR Interaction Toolkit. This saved time by providing a functional base for VR interaction.
Hierarchy Overview
The following key GameObjects were used in the scene hierarchy:
Plane:
Acts as the main environment where the balls and tiles interact.
Includes a Collider to ensure balls bounce and interact correctly.
SphereSpawner:
Responsible for spawning the balls at regular intervals.
Includes logic to set the initial velocity of each ball, ensuring consistent movement across the plane.
TileSpawners:
Multiple spawners, each responsible for spawning a specific tile associated with a unique piano note.
This setup allows for clear organization and customization of tile behavior and placement.
XR Rig:
The player’s rig imported from the demo scene, equipped with controllers and spatial audio capabilities.
Ensures smooth navigation and interaction within the VR environment.
Examples of Core Scripts (Pseudocode)
Sphere Spawn Script:
Initialize a timer to control spawning intervals.
When the timer reaches zero, spawn a sphere at a designated location.
Apply an initial velocity to the sphere for horizontal movement.
Tile Collision Script:
Detect collisions between tiles and balls.
Play the corresponding piano note using an AudioSource component.
Ball Trail Script:
Track the ball’s position in real-time.
Add each new position to a trail while removing the oldest positions to limit the trail length.
Update a LineRenderer to display the trail dynamically.
Destroy Ball Script:
Monitor the ball’s vertical position.
If the ball falls below a specified threshold (e.g., y < 0), destroy it to optimize performance.
Spatial Audio and Immersion
While the mechanics and visuals were satisfying to implement, the standout feature of "Musical Tiles" is the spatial audio. When you walk around in the Oculus Quest, the spatial sound changes dynamically based on your position. Piano notes sound louder and clearer as you approach specific tiles, creating an immersive, 3D soundscape that feels alive. This element of the app truly highlights the power of VR audio.
Lessons Learned
However, I was initially disappointed when I recorded a video of the app. The magic of spatial audio and immersion was lost in the flat, 2D recording. Experiencing the app firsthand in VR is a completely different experience. The spatial audio and visual interplay bring the environment to life in a way that a video simply cannot convey.
Wrist Menu Features:
Select note for tiles.
Add new ball spawn locations and configure their speed.
Adjust user vertical movement (up/down).
World Boundaries:
Prevent users from falling off the plane with invisible barriers or repositioning.
Resizable Tiles:
Allow tiles to be resized by grabbing both sides.
Add visual feedback (e.g., handles or glowing edges) during resizing.
Trails and Grid:
Add optional particle trails or transparent effects for moving balls.
Implement a configurable, transparent grid system (aligned to music beats and divisions of whole notes).
Optional Enhancements:
Add snap-to-grid for tiles for precise alignment.
Include save/load layouts for user configurations.
Provide undo/reset options in the wrist menu.