At A Glance

A tool I developed in Unity that can translate audio information into a visually representing spectrum graph, in a neonwave style. It accepts input from any audio source connected to the computer, including microphones as well as instruments with an input jack.

It can also detect the intensity of the input being received in order to trigger events to emphasize moments of higher intensity during the musical performance.

Below is some footage of the tool being used by myself (apologies in advance to your ears for my very mediocre musical performance lol):

   Git Repo Here   

The goal I had in mind when programming this tool was actually to visualize audio from a microphone, not an instrument. Originally, I started with a snippet of code capable of translating the internal audio output of the computer itself into audio visualization data. I decided I wanted to see if I could get the audio of a microphone to come through to Unity to create a sort of Karaoke machine.

I eventually figured out how to expose the microphone's audio to Unity, however, just when I wanted silver, I struck gold instead! Not only did I expose my microphone's audio to Unity, I found that my webcam could be used as an audio input, as it also has a microphone on it, and Unity recognized it.

This got me thinking that the way Unity connects to microphones is flexible enough to allow me to try connecting some other audio source instead. I bought myself an audio interface (pictured below), connected that into my computer, and plugged my guitar in it. Just like that, the interface got recognized by Unity, and the audio from the guitar came through! 

This tool works by combining 3 different scripts. I will break each of them down in detail (lots of text incoming!):

1 - Device Identifier

Inside of my project, I actually called this "Voice Detection", because as I said before, I originally wanted to use this script to just connect Unity to my microphone, not instruments, and I ended up leaving it with that name.

Each instrument you would like to visualize requires this script to be attached, as well as an Audio Source to go along with it.

The simplest yet most crucial script of all! As seen below, it simply has two fields. The first one, "Microphone", is a string that correlates to which specific device from your computer you wish for this object to draw audio input from. The array of strings "Options" contains every audio device that your computer currently has connected and available to draw information from.

As seen below, using the context menu function within my script, I exposed a function called "Detect Available Inputs", which will automatically check the computer's audio devices and populate the array with every input available:

Once the array is populated, you can then copy whichever audio device you wish to use into the "Microphone" field. You need to ensure this object also has an Audio Souce component to it, because the final step is entering play mode, which will make the script automatically assign itself as the "AudioClip" of the Audio Source of this object, as pictured below:

This is where the main functions are executed to actually translate the audio being received into visual information. In a nutshell, it enables the user to choose the frequency ranges they desire to display, and how this frequency range will look in terms of colour and size, and what events it triggers.

For ease of understanding and succinctness, I shall clarify one term which I will use repeatedly about this script: "intensity of the instrument". What I mean by this is "how much stimulus the frequency range of this particular instrument is receiving on average". If said average is high, then it means the user is playing the desired range of this instrument quite powerfully, whereas if the average is low, it means they are either playing it softly or not at all.

Each instrument you would like to visualize requires this script to be attached, in the same way as the Device Identifier script.

It has a lot of exposed fields, and I will break down what each of the main ones do:

Spectrum - It's a massive array which includes the entire audio spectrum being captured, from frequencies 1 to 1024, so you can see the value in each individual frequency. Purely for debug but very helpful to ensure you know for sure the frequencies you are interested in actually have values being received.

Skybox Shader Properties - The skybox has a fragment shader attached to it, which creates the circumcentric circle pattern (which is actually a simple sine wave). The movement and visibility of this pattern is dictated by the instrument's intensity. If you are playing very quietly, the circles will be less visible and move slowly, whereas if you play notes with more strenght, it will move fast and shine bright. The variables here control the impact your instrument has on the speed and intensity of in the skybox.

Instrument Events - Whenever you reach a certain amount of intensity in this particular instrument, you can determine something to happen in response. I leveraged Unity's event systems to be able to trigger pretty much any public function of any component in any object, which is why this is formatted as an array of arrays, where each individual item is a Unity event. In the above example, if you reach the intensity required, you will cause the 2 specified particle systems to trigger.

Event Freq Trigger - This is the number that determines how much intensity is needed to trigger the events into action. A lower number means less intensity is required, and the opposite is also true.

Randomly Pick One Event - If false, it will trigger every event in the array at once when the required intensity is achieved. If true, it will choose only one at random and trigger only that.

Instrument Band Obj Properties - A "band" is the name given to an individual bar that shrinks and grows in accordance to a frequency range. Here you can define if they spawn automatically when you enter play mode, how large they are (span), and you can define the prefab that will be spawned per band.

Each instrument being captured must be a child of a parent object, which will have this component attached to it. This component is similar to the Audio Visualizer in that it is capable of translating input being received from instruments into visual effects. The difference in this case is that this component considers input from every instrument simultaneously, as opposed to each individually. The main thing that it is capable of doing I will highlight below: