Computer science has had an impact on every field, and music is no exception. Computer software now plays a crucial role in the ways we create music, edit music, share and distribute music, and even listen to music.
Music visualization has been used to enhance the music listening experience in music videos, live performances, and even in education.
The Cooper Union in NYC is using music visualization to teach deaf children about sound. They have developed an interactive light studio in the American Sign Language and English Lower School in NYC. This consists of an interactive wall display that shows digital output created by sound and music. Children can trigger the playing of instruments with their movement, and they can watch the visual feedback from this music. (source: How Visuals Can Help Deaf Children 'Hear', 2014)
Nowadays, people create all kinds of music visualizations using code. Just search "#creativecoding music visualization" on Twitter to see some of the awesome creations people are making.
First thing's first: let's write a program that plays some music.
To play an audio file in your program, we'll need 2 things:
To play your own audio file, Fork this program to your sandbox and use the More > Upload tab to upload your own audio file. Use the resulting URL to create your own Audio object on line 3.
Great! We have our music playing, but how can we visualize that music?
The big idea here is that the music playing through your speakers is just DATA.
All information stored in a computer is stored as digital data. Digital means stored as digits, aka numbers.
When a computer plays a music file, it's simply reading all the numbers in that file, and making noises through the speakers based on the numbers it reads.
So what do these numbers look like? We can picture a music file as a series of frames. A frame represents a single moment in the song, and each frame is a list of numbers that describe the notes being played at that moment.
The frame is just a list of numbers. The first numbers in the list represent how loud the low notes are being played. The middle numbers represent how loud the mid-range notes are being played. And the final numbers in the list represent how loud the high notes are being played.
Let's start off with a program that just prints out each frame in the audio file:
Let's break down what's happening here.
The key to step 4 is the use of audioChangeMethod. audioChangeMethod is an event listener that listens for a special type of event - the given audio file changing frames.
mouseMoveMethod(myFunction) - calls myFunction anytime the mouse is moved moveClickMethod(myFunction) - calls myFunction anytime the mouse is clicked mouseDragMethod(myFunction) - calls myFunction anytime the mouse is dragged mouseDownMethod(myFunction) - calls myFunction anytime the mouse button is pressed down mouseUpMethod(myFunction) - calls myFunction anytime the mouse button is released
Here we are introducing a new event listener - specifically for audio events!
audioChangeMethod(song, myFunction) - calls myFunction anytime the given song changes to a new frame
Our job is to write a more interesting visualize function that draws shapes based on the given frame, rather than just printing Text based on the given frame.
Let's write a program that draws a vertical Rectangle for each number in the frame. The height of each Rectangle will be equal to each value in the frame, so quieter notes will result in shorter Rectangles, and louder notes will result in taller Rectangles:
But the visualization feels sort of "upside down"
Let's see if we can get the bars to rest at the bottom of the canvas, rather than the top.
Instead of using 0 for each bar's y-position, we want the y-position to be the bottom of the canvas MINUS the height of the bar:
From here we can make any customizations we'd like. The goal is to make shapes that are based in some way on the numbers in the given frame. Each shape's height can be based on the frame, or the width, or the color, or the position, or a combination!
Let's add one more customization:
The color of each bar will be based on the value in the frame. A large number will result in a more blue color, and a small number will result in a more red color.
Now it's your turn!
Try forking these examples and creating your own visualizations. Here are a few ideas to start with: