Last year I was contacted by my good friend Jürgen Rienow of Kiel University in Germany to ask if he could use an image from Vortex for their 2014 Yearbook of Immersive Media. The topic for the book was sound and music within immersive spaces so he also asked if I’d write an article on my experiences working on the spatial sound for Vortex to be included in the book. I of course said yes and today I received a copy in the post. Super proud. Unfortunately the rest of the book is in German, and my German isn’t as good as thought it might be when I left school. Anyway, uber high fives and danke schöns to Jürgen Rienow and Patrick Rupert-Kruse.
Below is the article I wrote.
Vortex – Spatial Sound
The idea for Vortex is an amalgamation of many things that inspire me. Some of these things are hard to identify as they surface through the subconscious whereas some are conscious influences, woven together to create a unique and abstract entity. Some of the more important inspirations are Cold War era engines, user interfaces, mandatory and postage labelling and armillaries as well as artists such as Delta, Syd Mead, Bradley G Munkowitz and Jordan Belson. The biggest inspiration however is the music it is set to: Mind Drift by Flavio Martines.
I don’t really consider Vortex to be a music video. A very loose version of the visual elements had already formed before I heard the track and already knew the kind of music that was required. Soundcloud is one of the many streams of music I digest constantly in my infinite quest of musical discovery. I can’t actually remember how I came across the track initially but when I heard it I knew it was the music of Vortex.
Music is a big inspiration behind all of the films I make. When listening to music I find it easy to feel movement and experience scenarios unfold and develop in my mind, like a conscious form of synaesthesia. Mind Drift instantly creates visions of fractured geometry and engine parts. It spins around and penetrates your personal space. It creates visions of a force so strong that it engulfs itself. So frame by frame I began to break down the music into all of its constituent parts then started designing and building every single musical element into a machine.
As with every project I work on, Vortex pushed a number of technical and creative boundaries beyond what I had achieved before. It’s mastered in fulldome format at 8192 x 8192 pixel resolution, 60fps, in full panoramic stereoscopic 3D and has a spatial sound mix. All of these things came with a unique set of challenges but the one I had no previous experience with was spatial sound.
It’s not complicated to understand the basic principles of 3D sound as it’s closer to our usual experience of sound in the world; sounds can appear to emanate from any point in space. How this is achieved however, is complicated. The format requires mind bending concepts like wave field synthesis, huge arrays of speakers, bespoke software and, of course, a spatial sound engineer. Fortunately I knew one of the world’s leading developers of wave field synthesis, Rene Rodigast from Fraunhofer IDMT.
I first met Rene at the International Planetarium Society Conference in Chicago in 2008. He has a powerful passion for audio, specifically within the dome environment. The first of his presentations I attended really got me thinking about how sound works in the dome and highlighted its importance within immersive media. After meeting Rene and discussing his field of work at several conferences and festivals, it made sense to approach him regarding a spatial mix for Vortex. He was very keen to be a part of it, along with his colleague Silvio Kuehm.
Rene’s studio at Fraunhofer IDMT in Berlin
Rene’s studio is in Fraunhofer IDMT in Berlin. Its walls are lined with a multitude of speakers that create the array required to produce 3D sound. Due to having zero budget, neither Flavio (Brazil) nor I (UK) could visit Rene’s studio to hear the mix. This meant we had to use email and Skype to describe what we were trying to achieve and hope that nothing got lost in translation. This was particularly challenging due to the complex language required to explain such abstract concepts and Flavio and Rene using English to do so (their second language).
Perhaps the most fascinating aspect of the 3D sound mix for Vortex is that it is made from music rather than real world sounds. The animation is completely driven by the music and many of the structural elements are visual manifestations of specific instruments or sounds within the music. One example is the electric arc that coils around the dome in time to the musical element I would describe as a bass buzz. The bass buzz was used to influence the shape, colour and movement of the electric arc and in turn we attached the bass buzz sound to the electric arc to control the sounds movement through space. The effect is that both the visual and audio senses work together to convince you that the object you see and hear moving around you is actually there.
The main task was to define what sounds should have a unique location, where that location would be and where and how it would move. One of the main approaches to this task was to extract sound elements from the music that apply to visual elements and then use the visual position of those elements to define the location of that sound in space, such as the electric arc and bass buzz. Another approach was to be less literal with placement and create cognitive dissonance between what you see and where you hear it. This was used for the power-up sequence where the engine is formed whilst spinning directly above the viewer. The sound that drives the visual element is positioned a distance away from the engine and spins at speed around the perimeter of the sound space. The physical detachment of the sound from the visual produces an interesting sensation of uneasiness or confusion, which adds to the initial shock of Vortex as it begins.
Positioning musical elements from a stereo track into three dimensional space can change the original mix to the extent that it no longer has a similar feel. When Rene created a stereo conversion of an early spatial mix it sounded very different to the original track. Some stems had to be repositioned, made less dynamic or altered to try and retain more of the original composition. Certain audio effects that had been applied to musical layers in the original piece had to be removed to extract the individual sounds. This also posed a challenge when aiming to produce a similar audio experience to the original stereo version. At some point we decided to treat the spatial sound mix as a new version of the track rather than just an alternative output. This allowed Flavio to be more creative with the individual elements he was extracting.
The idea of positioning music in 3D space opens up new and expanding possibilities to the musician, composer or anyone involved in immersive media. It’s perhaps a daunting prospect to think about positioning specific elements to produce dynamic movements through the listener’s space. An instrument is no longer constrained by stereo or even surround positions; it is given freedom to move towards and away from a listener in any direction.
For immersive media, spatial sound makes perfect sense but content creation is impeded by the current lack of production tools, production facilities and spatial sound venues. On top of this, there aren’t any standards for hardware or software, so even if a facility does have a speaker array capable of playing spatial sound, it may not be able to play back the files that another facility has produced. Vortex won the Spatial Sound Award at Jena Fulldome Festival in 2013: this remains the only place where it has been heard in this format. It seems like a lot of work for such a small amount of people to experience but as with most purely creative ventures, audience numbers are rarely the driving force behind the effort.
Vortex project online: