You might have encountered audio visual art in various forms, not at least on YouTube or in the world of computer gaming. Music visualization refers to systems which convert music or sound into film, video or computer graphics.

Music visualization, as a tool for multisensory learning, often occurs in pedagogical discussions. We are happy to have a chance to talk to audio visual artist Andy Fillebrown about his Musical Sculptures, which are now available on YouTube.

Rimsky-Korsakov: Flight of the Bumblebee

Piano Street: When did you get in touch with the piano?

Andy Fillebrown: My earliest introduction to the piano was pretty painful. I was running through the house when I was a child and as I rounded the corner into the hall, I ran smack into the side of an old upright. Ouch! Probably not the best way to start playing, but it’s a funny story considering how much I love the instrument.

I was one of those kids that likes to take things apart to see how they work but I didn’t like getting my hands dirty so I read a lot of car repair manuals, instead. I grew up listening to my mother’s ballet class music. She would take us to the dance studio so she could keep an eye on my brothers and me while she was teaching. Initially, I took her beginner classes, but the old pianos in the building were a lot more interesting. I loved opening up the cases and looking at the inner workings while I pressed the various keys, and the fact that they were designed to make noise was awesome!

My father was a self taught guitarist and was starting to take piano lessons. He had beginner’s piano books around and he started teaching me from them. I took to it pretty quickly and formal classical lessons started soon after. I think was around 7 or 8 years old.

PS: How did you discover the connection between music and moving images?

AF: I had childhood dreams of being the next Beethoven but by the time high school came around I wanted to be the next Billy Joel. So I applied to Berklee College of Music instead of Juilliard. When I got there, I was blown away by how good the musicians were. Most of them were much better performers than I was, so I decided to switch to a film-scoring major. That was a lot of fun but the best part was learning about the physics of sound and the process of how the notes on the score make their journey to the final mix. It was like peeking under a piano’s hood all over again!

After graduating, I worked at a local music studio, taught myself how to write software so I could pursue microtonal composing more easily, and started working on converting AutoCAD into a 3D music sequencer. With AutoCAD, I was working with notes in 3 dimensions and experimenting with different scoring ideas in an immersive environment for the first time. Looking at scores that way, I started seeing things I had not noticed before and I started thinking of sound not as just symbols on a page, but as physical objects that could be shaped and molded.

PS: Which effects did these discoveries have on your work?

AF: It was exciting! I felt like I was an explorer discovering a new world, but I was having difficulty communicating the advanced compositional possibilities that would become available with the software and I was going broke from not pursuing a traditional career path. Necessity being the mother of invention, I taught myself how to use the 3D graphics software Blender and put together a home-made DVD to give people for Christmas. Everybody loved it so I started selling it, too. To advertise, I put three of the videos up online. They didn’t lead to many DVD sales and I was not getting the “wow” response I was looking for, so I went back to the drawing board, refined the concept, and put the camera in the score, instead.

PS: When you choose pieces for your musical sculptures, which musical qualities are you looking for when planning for a new animation?

AF: I believe beauty is divine, so it’s the only constant I use when choosing pieces. Everything else is in flux because I’m refining the selection process to reach the widest possible audience without making myself crazy working on music I don’t want to hear.

Right now I’m choosing pieces based on how much eye candy they will generate in the animation, so compositional complexity and virtuosity are big factors. Long scale runs, contrary motion, and notes jumping all over the place are mainly what I’m looking for right now. Licensing is also a consideration. I’m working with a zero budget, so anything in the public domain is great.

Early on I wanted to make the videos extremely technical with lots of annotations regarding melody, chord structure, and performance, but I decided it is too difficult to automate right now and too time consuming to do by hand. I may make fully annotated videos in the future, but it’s not a priority at the moment.

J.S. Bach: Contrapunctus I from The Art of Fugue

PS: We are thrilled by the space-ship-traveling-in-time experience, opening up for a 3D experience of music! How can your musical sculptures help us to experience music?

AF: The 3D aspect of the visualizations is cool and people are enjoying the immersive quality, but I don’t think it’s really helping the musical experience beyond what can be done in 2D, yet. Fortunately, the animations are only the tip of the iceberg. As things progress, the 3rd dimension will become indispensable and I’ll be able to apply the designs I’ve been working on to interactive forms easily since a lot of the details have already been worked out.

With the rise of noise pollution and visual overstimulation, people are struggling to stay focused on the purely audible long enough to discover the genius in the details. This makes tying the sense of sight in with the sense of sound incredibly important for music in my opinion, because deeper understanding of sound is getting lost in the din, especially for the general public. With the 3D aspect making music more engaging, I’m hoping people will be more likely to explore compositions they’ve never heard before and listen to pieces again and again. Moving forward, this is the only way I can see general musical knowledge returning to the state it was in when popular music could only be distributed in sheet music form and a larger percentage of the population owned, and knew how to play, a piano.

PS: Do you have plans for a software release enabling the public to try your concept?

AF: Eventually, yes, but not anytime soon unless other people join the project. The source code is freely available, though, so if someone were feeling adventurous they could probably compile it on their own without too much difficulty. It might be tough to learn how to use since there is no documentation, but it’s do-able.

A few people have expressed a desire to make similar kinds of visualizations, but they lose interest when they discover how long it takes me to generate a full length animation after the modeling and effects are finished. If I’m having a good week I can render about 30 minutes of video, but that’s only because I’ve got 12 old computers networked together running 24 hours a day. The same process would take over a month on one computer. When I start rendering with graphics cards instead of CPUs, it will speed things up dramatically, but I’m not at that point, yet…

PS: What can you tell us about your future plans?

AF: Eventually, I’d like to start working with full orchestral scores and live performances like Stephen Malinowksi does with the Music Animation Machine, but I’m also excited about the possibility of making videos based on the Open Goldberg Project‘s Bösendorfer CEUS recordings performed by Kimiko Ishizaka.

Down the road I’ve got some ideas for making interactive apps, and I like to eventually get around to polishing up the sequencer I’ve been coding so others can use it, too. There’s a lot more to come, God willing. Stay tuned!

Related reading:
What Your Ears can’t See – The Music Animation Machine

Music Visualization – Wikipedia article