Waveforms

Live video coding: animating a waveform

Hi, Today I will show you how to write a waveform that will danse with the music. I prepared a small skeleton that is just composed of an audio element with a mp3 file that will be streamed when I press the play button.The sound is captured in an audio graph using a media element source like we explained in a previous lesson. So let's start from the beginning and look at this example. When the page is loaded we go to the onload listener, we create an audio context because we are going to work with the Web Audio API, and then we are just using the standard way for using a canvas.We get the canvas, and we get the canvas context, so we are ready to draw things in the canvas here. Then we build an audio graph and then we start an animation... and for the moment the animation just draws an horizontal line 60 times per seconds. We will look at it later.Let's have a look at the audio graph. The audio graph is made of a source node that is the media element source. It corresponds to the audio element.Then we create an analyser, this is a special node that will provide on demand the time domain and frequency domain analysis data, and this data will be useful for drawing a waveform or for drawing frequencies that will dance with the music, or for drawing volume meters, or for doing beat detection. We will see examples in the next lessons.When you create an analyser node, you specify the size of the Fast Fourier Transform (FFT), do not worry if you do not know exactly what this technique is. It is just the size that will have an influence on the number of data you will have to draw.Here for waveform, classical values are 1024 or 2048. They must be powers of 2.The number of data we will get depends on this size: it is exactly the FFT size divided by 2. In this example I set the FFT size to 1024 so I will get 512 data... So here is how we can declare a buffer that will get the data. It is called dataArray here and we will use it in the animation loop.So, I've got a source node that corresponds to the audio stream, an analyser node that will analyse the stream, and then I connect the source to the analyser, and the analyser to the destination, and the destination is the speakers.Let's have a look at the animation loop! We clear the canvas, and in the end we call again the animation loop, so that the animation will be done 60 times per seconds... and the way we will draw the waveform is just a set of connected line that we call "a path".The "path" is a way of drawing that we presented during the HTML5 Part 1 course.So here, for drawing an horizontal, flat waveform, we just do a loop on the number of data. Here, we are not using the real data but we will do a loop 512 times and we compute an increment in x, so that depending on the width of the canvas and on the number of data we have got to draw, we will add this increment to the x coordinate.And the y coordinate here is just faked because we use height/2.I can add a random element here if you like, and we will see that I can just fake some data to be drawn. So, instead of drawing this, I will use the real values. How can we get the data? In the animation loop, 60 times per seconds, we call the analyser.getByteTimeDomainData and we pass the dataArray of the correct size.And just after this call, the dataArray will contain the data we want to draw. And what is interesting here, is the value of each data... and we will compute the y coordinate depending on that. I'm just declaring a variable that will get the value. This value is between 0 and 255 because we are working with bytes, and bytes are 8 bits encoded data and the value is between 0 and 255. I will normalize it. So now it's between 0 and 1. Then, in order to compute the y value, I just have to scale it to the height of the canvas, like this...And now if I play the sound, I've got the waveform that is animated.These 3 lines are very straightforward, just in order to transform a value between 0 and 255 and scale it to the height of the canvas...

Do try on and study the code in this demo shown in the video:

HTML:

<head>

<title>2D audio vizualization: waveform</title>

</head>

<body>

<h2>2D audio vizualization: waveform</h2>

</audio>

<p>

</body>

</html>

CSS:

canvas {

border:1px solid;

}

JavaScript:

let canvas;

let audioContext, canvasContext;

let width, height;

let analyser, bufferLength, dataArray;

window.onload = () => {

// we get the audio context

audioContext = new AudioContext();

// prepare eveything for the canvas

canvas = document.querySelector("#myCanvas");

width = canvas.width;

height = canvas.height;

canvasContext = canvas.getContext('2d');

buildAudioGraph();

// start the visualization

requestAnimationFrame(visualize);

};

function buildAudioGraph() {

let player = document.getElementById("player");

let source = audioContext.createMediaElementSource(player);

// fix for autoplay policy

player.addEventListener('play',() => audioContext.resume());

analyser = audioContext.createAnalyser();

// set its properties

analyser.fftSize = 1024;

// its size is always the fftSize / 2

bufferLength = analyser.frequencyBinCount;

dataArray = new Uint8Array(bufferLength);

// connect the nodes

source.connect(analyser);

analyser.connect(audioContext.destination);

}

function visualize() {

// clear the canvas

canvasContext.clearRect(0, 0, width, height);

// We will draw it as a path of connected lines

// First, clear the previous path that was in the buffer

canvasContext.beginPath();

// draw the data as a waveform

// Get the data from the analyser

analyser.getByteTimeDomainData(dataArray);

// values are between 0 and 255

// slice width

let sliceWidth = width / bufferLength;

let x = 0;

for(let i = 0; i < bufferLength; i ++) {

let v = dataArray[i]; // between 0 and 255

v = v / 255; // now between 0 and 1

let y = v * height;

if(i === 0) {

canvasContext.moveTo(x, y);

} else {

canvasContext.lineTo(x, y);

}

x += sliceWidth;

}

// draw the whole waveform (a path)

canvasContext.stroke();

// call again the visualize function at 60 frames/s

requestAnimationFrame(visualize);

}

WebAudio offers an Analyser node that provides real-time frequency and time-domain analysis information. It leaves the audio stream unchanged from the input to the output, but allows us to acquire data about the sound signal being played. This data is easy for us to process since complex computations such as Fast Fourier Transforms are being executed, behind the scenes.

Example #1: audio player with waveform visualization

codepen.io/w3devcampus/pen/WNRvjRZ

HTML:

<head>

<title>2D audio vizualization: waveform</title>

</head>

<body>

<h2>2D audio vizualization: waveform</h2>

</audio>

</div>

</body>

</html>

CSS:

div audio {

display: block;

margin-bottom:10px;

margin-left:10px;

}

#myCanvas {

border:1px solid;

}

.main {

margin: 32px;

border:1px solid;

border-radius:15px;

background-color:lightGrey;

padding:10px;

width:320px;

box-shadow: 10px 10px 5px grey;

text-align:center;

font-family: "Open Sans";

font-size: 12px;

}

div.controls:hover {

color:blue;

font-weight:bold;

}

div.controls label {

display: inline-block;

text-align: center;

width: 50px;

}

div.controls label, div.controls input, output {

vertical-align: middle;

padding: 0;

margin: 0;

font-family: "Open Sans",Verdana,Geneva,sans-serif,sans-serif;

font-size: 12px;

}

JavaScript

let canvas;

let audioContext, canvasContext;

let analyser;

let width, height;

let dataArray, bufferLength;

window.onload = () => {

audioContext= new AudioContext();

canvas = document.querySelector("#myCanvas");

width = canvas.width;

height = canvas.height;

canvasContext = canvas.getContext('2d');

buildAudioGraph();

requestAnimationFrame(visualize);

};

function buildAudioGraph() {

let mediaElement = document.getElementById('player');

// fix for autoplay policy

mediaElement.addEventListener('play',() => audioContext.resume());

let sourceNode = audioContext.createMediaElementSource(mediaElement);

// Create an analyser node

analyser = audioContext.createAnalyser();

// Try changing for lower values: 512, 256, 128, 64...

analyser.fftSize = 1024;

bufferLength = analyser.frequencyBinCount;

dataArray = new Uint8Array(bufferLength);

sourceNode.connect(analyser);

analyser.connect(audioContext.destination);

}

function visualize() {

// clear the canvas

// like this: canvasContext.clearRect(0, 0, width, height);

// Or use rgba fill to give a slight blur effect

canvasContext.fillStyle = 'rgba(0, 0, 0, 0.5)';

canvasContext.fillRect(0, 0, width, height);

// Get the analyser data

analyser.getByteTimeDomainData(dataArray);

canvasContext.lineWidth = 2;

canvasContext.strokeStyle = 'lightBlue';

// all the waveform is in one single path, first let's

// clear any previous path that could be in the buffer

canvasContext.beginPath();

let sliceWidth = width / bufferLength;

let x = 0;

for(let i = 0; i < bufferLength; i++) {

// normalize the value, now between 0 and 1

let v = dataArray[i] / 255;

// We draw from y=0 to height

let y = v * height;

if(i === 0) {

canvasContext.moveTo(x, y);

} else {

canvasContext.lineTo(x, y);

}

x += sliceWidth;

}

canvasContext.lineTo(canvas.width, canvas.height/2);

// draw the path at once

canvasContext.stroke();

// call again the visualize function at 60 frames/s

requestAnimationFrame(visualize);

}

Do things in order!

First, select the audio context and the canvas context, then build the audio graph, and finally run the animation loop.

Typical operations to perform once the HTML page is loaded:

window.onload = function() {

// get the audio context

audioContext= ...;

// get the canvas, its graphic context...

canvas = document.querySelector("#myCanvas");

width = canvas.width;

height = canvas.height;

canvasContext = canvas.getContext('2d');

// Build the audio graph with an analyser node at the end

buildAudioGraph();

// starts the animation at 60 frames/s

requestAnimationFrame(visualize);

};

Step #1: build the audio graph with an analyser node at the end

If we want to visualize the sound that is coming out of the speakers, we have to put an analyser node at almost the end of the sound graph. Example #1 shows a typical use: an <audio> element, a MediaElementElementSource node connected to an Analyser node, and the analyser node connected to the speakers (audioContext.destination). The visualization is a graphic animation that uses the requestAnimationFrame API presented in teh W3C HTML5 Coding Essentials and Best Practices course (Module 4).

Typical code for building the audio graph:

HTML code:

<audio src="https://mainline.i3s.unice.fr/mooc/guitarRiff1.mp3"

id="player" controls loop crossorigin="anonymous">

</audio>

JavaScript code:

function buildAudioGraph() {

var mediaElement = document.getElementById('player');

var sourceNode = audioContext.createMediaElementSource(mediaElement);

// Create an analyser node

analyser = audioContext.createAnalyser();

// set visualizer options, for lower precision change 1024 to 512,

// 256, 128, 64 etc. bufferLength will be equal to fftSize/2

analyser.fftSize = 1024;

bufferLength = analyser.frequencyBinCount;

dataArray = new Uint8Array(bufferLength);

sourceNode.connect(analyser);

analyser.connect(audioContext.destination);

}

With the exception of lines 8-12, where we set the analyser options (explained later), we build the following graph (picture taken with the now discontinued FireFox WebAudio debugger, you should get similar results with the Chrome Audion extension):

Step #2: write the animation loop

The visualization itself depends on the options which we set for the analyser node. In this case we set the FFT size to 1024 (FFT is a kind of accuracy setting: the bigger the value, the more accurate the analysis will be. 1024 is common for visualizing waveforms, while lower values are preferred for visualizing frequencies). Here is what we set in this example:

analyser.fftSize = 1024;

bufferLength = analyser.frequencyBinCount;

dataArray = new Uint8Array(bufferLength);

Line 2: we set the size of the FFT,
Line 3: this is the byte array that will contain the data we want to visualize. Its length is equal to fftSize/2.

When we build the graph, these parameters are set - effectively as constants, to control the analysis during play-back.

Here is the code that is run 60 times per second to draw the waveform:

function visualize() {

// 1 - clear the canvas

// like this: canvasContext.clearRect(0, 0, width, height);

// Or use rgba fill to give a slight blur effect

canvasContext.fillStyle = 'rgba(0, 0, 0, 0.5)';

canvasContext.fillRect(0, 0, width, height);

// 2 - Get the analyser data - for waveforms we need time domain data

analyser.getByteTimeDomainData(dataArray);

// 3 - draws the waveform

canvasContext.lineWidth = 2;

canvasContext.strokeStyle = 'lightBlue';

// the waveform is in one single path, first let's

// clear any previous path that could be in the buffer

canvasContext.beginPath();

var sliceWidth = width / bufferLength;

var x = 0;

for(var i = 0; i < bufferLength; i++) {

// dataArray values are between 0 and 255,

// normalize v, now between 0 and 1

var v = dataArray[i] / 255;

// y will be in [0, canvas height], in pixels

var y = v * height;

if(i === 0) {

canvasContext.moveTo(x, y);

} else {

canvasContext.lineTo(x, y);

}

x += sliceWidth;

}

canvasContext.lineTo(canvas.width, canvas.height/2);

// draw the path at once

canvasContext.stroke();

// once again call the visualize function at 60 frames/s

requestAnimationFrame(visualize);

}

Explanations:

Lines 9-10: we ask for the time domain analysis data. The call to getByteTimeDomainData(dataArray) will fill the array with values corresponding to the waveform to draw. The returned values are between 0 and 255.

Below are other examples that draw waveforms.

Example #2: video player with waveform visualization

Using a <video> element is very similar to using an <audio> element. We have made no changes to the JavaScript code here; we Just changed "audio" to "video" in the HTML code.

Example #3: both previous examples, this time with the graphic equalizer

Adding the graphic equalizer to the graph changes nothing, we visualize the sound that goes to the speakers. Try lowering the slider values - you should see the waveform changing.

Page updated

Google Sites

Report abuse