Love + Immersion
This is an extended version of my presentation at Fulldome UK in November 2014. The presentation is an insight into the cinematic techniques I used to make Vessel and some of the thought processes involved with making an immersive, narrative based film.
Vessel is an immersive film I’ve been working on over the past 2 years. Not continuously, but its beginnings were in the tail end of 2012 following my previous immersive film Vortex. I’ve been wanting to work on a narrative based immersive film for some time but a 360 degree visual field comes with a diverse set of challenges when it comes to storytelling. Just understanding some basic principles in immersion can take some time and it was only really after several years working in immersion at NSC Creative and creating Vortex that I felt that my personal techniques and ideas had evolved enough to tackle narrative in such an arena.
More than 100000 chemical reactions happen in your brain every second. So what about the moment you saw the girl of your dreams, and she saw you? If we could capture those few seconds what would the data look like? How would we represent the different messages being sent and received in the various areas of the brain? Vessel is a document of this moment. A poem written within data. A visualisation of love.
Narrative
The initial concept of Vessel was about chemicals colliding in the brain during the moment of falling in love, and of course being some kind of metaphor for two people falling in love. This is my description of the film I wanted to make that I wrote in an email to Jon Hopkins on December 4th 2012:
“I’m trying to let the idea evolve as I go along but at the moment it’s (very briefly) a journey of chemicals, culminating in a collision that forms new connections. I guess it’s like a love story between two chemicals in the brain.”
And from that, the idea did evolve, and the narrative started to form. But the narrative writing process didn’t begin with words, at least not as a traditional script.
Vessel is a piece of music by Jon Hopkins that audially represents this moment of falling in love. It has the swells and bursts that tell the story. I didn’t exactly know this when I first heard the track back in 2009, but when I came up with the concept for my film, I knew this was the right track to tell this story. I can only really describe the way I shaped the narrative as an extraction process. I extracted the story from the music, like it had been buried within it the whole time. Hidden within the sweeping builds and behind every vibrating string. This process took some time, literally carving and reshaping until there was almost nothing left. Just the essential elements. Just the moments that define the structure of a 4 second story. Looking at it now, the story is so obvious to me I’m amazed it alluded me for so long.
The whole story is 4 seconds of captured data stretched over 4 minutes, to see all the neural processes taking place during this moment. But within this narrative there are really 3 scripts running in parallel throughout the film.
- Poetic document
- Neural processes
- Machine data
Poetic document
This forms the backbone of the narrative.. These are perhaps the only words that it’s important to read as they carry the story. For this reason they are bigger, shorter, more obvious, and appear within a small field of view from one to the next. Each statement appears as the subject experiences it within the 4 seconds. Refining these words was also a lengthy process. Something I came back to several times over the 2 years. Writing has never come naturally to me. It takes a lot of time and effort to commit to words.
Neural processes
This represents the neural activity inferred from analysing the data captured by the machine. This includes information about activity spikes in areas of the brain and what chemicals are being released. There’s a lot of references to dopamine which, when released in certain areas of the brain makes us feel pleasure. All the best drugs such as heroin and cocaine cause a huge surge of dopamine in the brain. Another common chemical is oxytocin, which is the primary neurotransmitter involved with the feeling of empathy. By simply hugging people, your brain releases oxytocin and makes you more likely to lend someone money. The research in this area was vast. Most of this script represents real neural processes, which relate exactly to the narrative at that moment. I have documents filled with information about what’s happening in the brain for every moment of the narrative; when you smell something, what part of your brain is activated, what chemicals are released, what chain reactions occur, how the information is perceived. This information was then condensed into basic statements that in some deep and complex way support the narrative. Although I’m pretty sure this level of storytelling will never really be followed or understood by anybody, it’s something I obsessed over for months. On its own it’s possibly the least emotive story of love ever told but it highlights the incredible way our conscious minds translate emotions from a series of chemical reactions.
Machine data
If you imagine an advanced EEG machine attached to someone’s head, this would be all of the diagnostic information about sensors and data handling. To create the script for this I installed a number of EEG data analysis applications. By downloading free data-sets I was able to simulate the capturing process and use tools to visualise the data. Most of the machine data script is taken directly from this analysis software and is placed in appropriate areas to reflect what is happening in the narrative. Most of it makes little sense, so it’s used mainly as a visual element to add realism and complexity. All three of these scripts run alongside each other visually and tell the same story from a different perspective.
When we see ‘Breath rushes from his lungs’ we also see a release of the neurotransmitters glutamate and nitric oxide as these are important in the control of ventilation.
When the subject gets overwhelmed with chemical reactions, we see errors in the data stream from the machine.
Colour
Colour has a profound effect on our perception. Blue pills are more effective at easing anxiety than red pills. Purple can be used to lower blood pressure and you’re more likely to win stuff when wearing red. The colours in vessel change over time to follow the emotional state of the subject. These colours along with the audio and text hopefully work together to produce the appropriate emotion within the viewer.
This image represents the timeline of the film and this is the predefined palette I worked from. Longer-wavelength colors like red are experienced as arousing, and shorter-wavelength colors like green are experienced as calming. So initially the palette is limited to greens as this is the subjects stable baseline… before he has seen the girl. It’s only as the subject becomes more mentally engaged do we start to see different colours.
This is part of a fascinating data visualisation project by Orlagh O’Brien showing associations of colour with emotions. The above image shows what colours people associate with fear, and I used some of these colours for a part of the narrative where the subject feels apprehension and self-doubt.
This shows the general perception of love. It’s predominantly pinks and reds, but even with all the other colours I still don’t feel that this is enough to portray the dizzying moment of falling in love.
Chimerical colours don’t appear within the colour space of human vision. They are a construct of the mind. They can be created by inducing a natural process of the eye called colour fatigue. If you stare at a colour for a long time your eye will temporarily displace the colour space by the opposing colour. So if you stare at yellow, then black, for a short time you will perceive that black to contain blue. The colour you are seeing is out of the range of visible colours. It is a pitch black saturated with blue, which is impossible.
These are the colours that are used on some of the text panels during the narrative moment of falling in love… To me they sum up that surreal moment. An explosion of colours beyond the visible spectrum.
Stereoscopic Depth
Stereoscopy is something I find myself defending within a creative space more than discussing its creative potential. So rather than do that here, I’ll go through the creative process of stereography and how it works alongside the narrative and other cinematic devices in Vessel.
This image again represents the timeline of the film showing how depth is manipulated over time. The quarter circle to the bottom left represents the dome from above and the line cutting through the centre represents the front edge of the dome. So anything above the line has positive parallax, which would be perceived as being outside the dome, and anything below the line has negative parallax and is perceived inside the dome. The top of the graph represents the furthest point or infinity and the bottom represents the viewers position.
This script was created relatively early on and developed alongside the narrative to emphasize the story being told. The changes in depth are actually more detailed than this in the final setup, with smaller peaks and troughs to keep everything comfortable. To begin with, everything is almost flat and just as the colours change to represent the emotional state of the subject, the depth increases, creating a richer space. Generally, depth is used to mirror the emotional stimulation felt by the subject in the story. Flatness accentuates the lack of emotion in the early parts of the story. Most of the achievable depth isn’t used for the first three quarters of the film to allow the peak to have as much impact as possible. Although some elements do come closer to the viewer in the final stereoscopic setup, it’s not until the moment she returns his gaze that objects stretch out to infinity and enter the audience space to a distance that is tangible. This exaggeration of depth takes the moment into the realm of fantasy, appearing almost dream-like. Most people wont notice any of this happening and this is quite important. Slow changes in depth are important to allow easy accommodation and maintain comfortable viewing.
Most of these changes in depth work at a subliminal level, tugging our feelings around without us realising. Just as a subtle shift of colour can nudge our perception of a scene, subtle changes in stereoscopic depth can alter how we feel. The stereoscopic language isn’t quite as developed as colour, but the more we experience it and manipulate it, the more we will understand how and when to use it.
Immersion
Immersion is something that gets talked about a lot in the fulldome world. It’s what makes fulldome different to cinema. It’s something I think and write about a lot and the area I’ve been most keen to explore and experiment with whilst making Vessel.
Understanding immersion is something I’ve struggled with for a while. Even working in the immersive field of fulldome for so long I still haven’t really cracked it. I think this has something to do with a limited vocabulary. The dictionary definition of immersion has a few entries. The first two are interesting as they are both experiences that are quite different. The first relates to a physical experience, and the second is a mental experience.
Books vs VR… This is something I’ve been thinking about a lot whilst making Vessel. I’ve been hearing a lot of people within the industry talk about it and I’ve been asking a lot of people outside the industry about their feelings on it. Richard Birkin is a writer, a musician and an avid book reader. He’s not working in what we might consider to be an immersive industry but the word immersion means something very specific to him in relation to reading.
When you exchange a few words, as I have done in red, you get similar responses from people talking about films. But with books there is no sensory manipulation. There’s no retinal resolution displays, no 360 degree video projection, no high-frame-rates… Just some words on paper… Yet still a high level of immersion can be achieved. But what about that sense of falling you get when the stars roll around the sky in a planetarium? You feel so immersed in that world of stars that it feels like you are falling. That’s not something I’ve ever felt from reading. I’ve never had to hold on to something as the words shift the world around me… At least not physically. This reaction is perhaps closer to what you might experience on a roller-coaster as you are literally span around. And how does the immersion experienced from a fictional literary journey relate to the same immersion we feel from the roller-coaster?
This statement from James Iliff relates to gaming, where the word immersion is commonly broken up into 4 separate terms. All of which are considered very different forms of immersion that require different game mechanics to induce.
Immersion is perhaps one of those words that we’ve used to bundle together a few things that relate to each other. And maybe it wasn’t so relevant in the past to distinguish between them. But the more I discuss immersion and attempt to understand the basic principles involved in creating immersive experiences, the more I feel that there is something not quite right with my understanding at the foundations of the language I’m using to describe those experiences. This constant questioning lead me to breaking immersion into two distinct areas. Maybe there are more, but within the field of immersive cinema, two subsets of immersion made a lot of sense to me. These two subsets are mental immersion and physical immersion.
Perhaps this is obvious to some people already, but to me this was a breath of fresh air. It made Vessel make a little more sense. For a while I felt like Vessel wasn’t really immersive. Or at least it wasn’t immersive for the first three quarters. Really I was just thinking in physically immersive terms.
This image shows how the level of physical immersion changes over time. What’s interesting about this is that most of the film has little immersion in it. Even though it’s designed for a 360 degree experience, most of the rendered image has nothing in it for the first third of the film. But what’s more important for the overall experience isn’t to feel physically immersed for the whole duration, but to feel mentally immersed and then use the physically immersive nature of the dome to accentuate specific narrative moments.
A fisheye still from earlier in the film, showing how little of the frame and field of view is used.
A fisheye still from later in the film when the level of physical immersion peaks
The reduced field of view and sedated movements at the start of the film focus the attention of the viewer and make it easier to follow the story. It’s the words that form the story that start the journey of mental immersion. As the story progresses, objects in the peripheral pull the audience further into the world until mental and physical immersion meet in a big explosion of love.
Spherical Composition
Vessel is designed as a full 360 experience. Understanding how people view immersive environments is crucial if you want to tell a meaningful story. Many principles from the disciplines of design, fine art, photography and cinematography can be used in spherical composition, much the same way they are used in traditional flat image composition. Colour, contrast, shape, tone, movement etc can be used to draw the viewers attention, but can be much harder to control when part of the image will always be outside the visible field of view. I’ve been calling this process domeography for a few years whilst working primarily in fulldome but perhaps a more appropriate term for 360 degrees would be spherography. Although a limitless field of view allows the viewer to explore the whole environment, it’s important to know how to control where they look to make sure they don’t miss key parts of the story. Previously my workflow has been to watch videos in the dome at work and then make amendments and occasionally get some colleagues to join me and watch them as they look around to see where they go.
Since getting hold of an Oculus Rift, my review process has completely changed. It’s great for quickly previewing videos without having to wait for the dome to be free and having it at home means that I’m able to iterate ideas constantly. Creating camera moves, testing them and reworking them, rather than a daily cycle of test at home and review in the dome. Because Vessel has a text based narrative it’s crucial for the viewer to be able to follow the story. So the rift came in useful for me to track peoples eyes for both the 360 and fulldome versions and understand when they were following something I didn’t want them to.
Love + Immersion
It’s easy to focus on (physical) immersion alone when creating an immersive film. Stories can’t be told through (physical) immersion alone. Immersion is really just one of many tools that can be used to carve a story. It makes sense to use it within an immersive environment but it’s important to think about how it fits into the story and when it can be used to project that story to the next level. In Vessel it’s used to represent that moment of falling in love, to create that head over heels moment. When two souls collide and find themselves spinning in infinity. For this moment I don’t think there’s a better tool for the job.
6 Comments
Xiangzhen Lu
May 30, 2015I must say I’m truly impressed by your work, Aaron! I saw this very one in Fulldome Festival 2015 in Zeiss-Planetarium Jena, Germany. Very nice graphics, animation and sound. I’m studying interface design, and I can say I will learn a lot by looking many more times into this piece!
Aaron
June 23, 2015Thanks for your kind words. I’m fascinated by interface design myself 🙂
Devon Copley
July 3, 2015Hi Aaron –
I stumbled across your website earlier today, and then tonight was wandering around Littlstar and lo and behold there was LoVR so I booted it up…
Aaron, I have to say, it’s an extraordinary thing, honestly one of the most beautiful and compelling experiences I’ve seen in VR to date. I’m personally a lot less interested in games — I want story and narrative, and you really delivered it in this terrific piece. I kind of wish I had seen the thing (film? video? experience? we need a name for VR things) before I’d read your blog post, since I kind of knew where it was going … but all the same the build and climax of it was really well constructed and the thought and planning you put into the arc of the experience really paid off.
If it’s not obvious, I *loved* LoVR. Looking forward to more work from you. In the meantime, if you’re ever in San Francisco, look me up and let me buy you a beer 🙂
Best of luck –
Devon
Aaron
July 4, 2015Hi Devon,
This is by far the best comment I’ve had on my blog. Thank you.
Narrative in immersive media in general is very sparse. Every time I see something that has a story it really stands out and I’m always keen to see how other people approach narrative with an endless canvas. It’s something I’ve been trying to figure out for a while now.
I also struggle to find the right word to describe immersive films. ‘Films’ doesn’t seem to quite fit… ‘Experience’ does, but it feels too broad. I use all kinds of weird terms to describe immersive cinematography also. It’s hard to communicate without a common language. I have an unfinished blog post somewhere on this topic. I probably should finish it sometime.
I’m certain I will be in San Francisco at some point in the future. It’s been high on my list for some time as it’s home to the Morrison Planetarium at Calacademy. It has a rich history of immersive art and I have a good friend who works there. So if I do visit, I will take you up on that beer. I’ll go to great lengths for a free beer.
Thanks again.
Rednapx
January 8, 2017I enjoyed thoroughly the VR experience (given that also i’m a fan of IDM Music) and this document shows that is not only a work on aestethicism, but is also a very well documented and researched video, enjoyed it very much.
Chris
November 15, 2018What software did you use?