Love + Immersion

Posted by on Feb 2, 2015 in Fulldome, LoVR, Thoughts, Vessel | 6 Comments

This is an extended version of my presentation at Fulldome UK in November 2014. The presentation is an insight into the cinematic techniques I used to make Vessel and some of the thought processes involved with making an immersive, narrative based film.

Vessel is an immersive film I’ve been working on over the past 2 years. Not continuously, but its beginnings were in the tail end of 2012 following my previous immersive film Vortex. I’ve been wanting to work on a narrative based immersive film for some time but a 360 degree visual field comes with a diverse set of challenges when it comes to storytelling. Just understanding some basic principles in immersion can take some time and  it was only really after several years working in immersion at NSC Creative and creating Vortex that I felt that my personal techniques and ideas had evolved enough to tackle narrative in such an arena.

chemical-reactions

More than 100000 chemical reactions happen in your brain every second. So what about the moment you saw the girl of your dreams, and she saw you? If we could capture those few seconds what would the data look like? How would we represent the different messages being sent and received in the various areas of the brain? Vessel is a document of this moment. A poem written within data. A visualisation of love.

Narrative

The initial concept of Vessel was about chemicals colliding in the brain during the moment of falling in love, and of course being some kind of metaphor for two people falling in love. This is my description of the film I wanted to make that I wrote in an email to Jon Hopkins on December 4th 2012:

“I’m trying to let the idea evolve as I go along but at the moment it’s (very briefly) a journey of chemicals, culminating in a collision that forms new connections. I guess it’s like a love story between two chemicals in the brain.”

And from that, the idea did evolve, and the narrative started to form. But the narrative writing process didn’t begin with words, at least not as a traditional script.

AudioScript 10-01

 

Vessel is a piece of music by Jon Hopkins that audially represents this moment of falling in love. It has the swells and bursts that tell the story. I didn’t exactly know this when I first heard the track back in 2009, but when I came up with the concept for my film, I knew this was the right track to tell this story. I can only really describe the way I shaped the narrative as an extraction process. I extracted the story from the music, like it had been buried within it the whole time. Hidden within the sweeping builds and behind every vibrating string. This process took some time, literally carving and reshaping until there was almost nothing left. Just the essential elements. Just the moments that define the structure of a 4 second story. Looking at it now, the story is so obvious to me I’m amazed it alluded me for so long.

script

The whole story is 4 seconds of captured data stretched over 4 minutes, to see all the neural processes taking place during this moment. But within this narrative there are really 3 scripts running in parallel throughout the film.

  • Poetic document
  • Neural processes
  • Machine data

Poetic document

caresses-her-face

This forms the backbone of the narrative.. These are perhaps the only words that it’s important to read as they carry the story. For this reason they are bigger, shorter, more obvious, and appear within a small field of view from one to the next. Each statement appears as the subject experiences it within the 4 seconds. Refining these words was also a lengthy process. Something I came back to several times over the 2 years. Writing has never come naturally to me. It takes a lot of time and effort to commit to words.

Neural processes

chemical-data

This represents the neural activity inferred from analysing the data captured by the machine. This includes information about activity spikes in areas of the brain and what chemicals are being released. There’s a lot of references to dopamine which, when released in certain areas of the brain makes us feel pleasure. All the best drugs such as heroin and cocaine cause a huge surge of dopamine in the brain. Another common chemical is oxytocin, which is the primary neurotransmitter involved with the feeling of empathy. By simply hugging people, your brain releases oxytocin and makes you more likely to lend someone money. The research in this area was vast. Most of this script represents real neural processes, which relate exactly to the narrative at that moment. I have documents filled with information about what’s happening in the brain for every moment of the narrative; when you smell something, what part of your brain is activated, what chemicals are released, what chain reactions occur, how the information is perceived. This information was then condensed into basic statements that in some deep and complex way support the narrative. Although I’m pretty sure this level of storytelling will never really be followed or understood by anybody, it’s something I obsessed over for months. On its own it’s possibly the least emotive story of love ever told but it highlights the incredible way our conscious minds translate emotions from a series of chemical reactions.

Machine data

eeg-data

If you imagine an advanced EEG machine attached to someone’s head, this would be all of the diagnostic information about sensors and data handling. To create the script for this I installed a number of EEG data analysis applications. By downloading free data-sets I was able to simulate the capturing process and use tools to visualise the data. Most of the machine data script is taken directly from this analysis software and is placed in appropriate areas to reflect what is happening in the narrative. Most of it makes little sense, so it’s used mainly as a visual element to add realism and complexity. All three of these scripts run alongside each other visually and tell the same story from a different perspective.

breath-rushes

When we see ‘Breath rushes from his lungs’ we also see a release of the neurotransmitters glutamate and nitric oxide as these are important in the control of ventilation.

eeg-error

When the subject gets overwhelmed with chemical reactions, we see errors in the data stream from the machine.

Colour

quote_wassily-kandinsky

Colour has a profound effect on our perception. Blue pills are more effective at easing anxiety than red pills. Purple can be used to lower blood pressure and you’re more likely to win stuff when wearing red. The colours in vessel change over time to follow the emotional state of the subject. These colours along with the audio and text hopefully work together to produce the appropriate emotion within the viewer.

ChromaScript 09

This image represents the timeline of the film and this is the predefined palette I worked from. Longer-wavelength colors like red are experienced as arousing, and shorter-wavelength colors like green are experienced as calming. So initially the palette is limited to greens as this is the subjects stable baseline… before he has seen the girl. It’s only as the subject becomes more mentally engaged do we start to see different colours.

Orlagh-O'Brien-Fear

This is part of a fascinating data visualisation project by Orlagh O’Brien showing associations of colour with emotions. The above image shows what colours people associate with fear, and I used some of these colours for a part of the narrative where the subject feels apprehension and self-doubt.

Orlagh-O'Brien-Love

This shows the general perception of love. It’s predominantly pinks and reds, but even with all the other colours I still don’t feel that this is enough to portray the dizzying moment of falling in love.

Chimerical colours don’t appear within the colour space of human vision. They are a construct of the mind. They can be created by inducing a natural process of the eye called colour fatigue. If you stare at a colour for a long time your eye will temporarily displace the colour space by the opposing colour. So if you stare at yellow, then black, for a short time you will perceive that black to contain blue. The colour you are seeing is out of the range of visible colours. It is a pitch black saturated with blue, which is impossible.

chimerical

hyperbolic

luminous

These are the colours that are used on some of the text panels during the narrative moment of falling in love… To me they sum up that surreal moment. An explosion of colours beyond the visible spectrum.

Stereoscopic Depth

quote_martin-scorcese

Stereoscopy is something I find myself defending within a creative space more than discussing its creative potential. So rather than do that here, I’ll go through the creative process of stereography and how it works alongside the narrative and other cinematic devices in Vessel.

iX_NSC_depth-script

This image again represents the timeline of the film showing how depth is manipulated over time. The quarter circle to the bottom left represents the dome from above and the line cutting through the centre represents the front edge of the dome. So anything above the line has positive parallax, which would be perceived as being outside the dome, and anything below the line has negative parallax and is perceived inside the dome. The top of the graph represents the furthest point or infinity and the bottom represents the viewers position.

This script was created relatively early on and developed alongside the narrative to emphasize the story being told. The changes in depth are actually more detailed than this in the final setup, with smaller peaks and troughs to keep everything comfortable. To begin with, everything is almost flat and just as the colours change to represent the emotional state of the subject, the depth increases, creating a richer space. Generally, depth is used to mirror the emotional stimulation felt by the subject in the story. Flatness accentuates the lack of emotion in the early parts of the story. Most of the achievable depth isn’t used for the first three quarters of the film to allow the peak to have as much impact as possible. Although some elements do come closer to the viewer in the final stereoscopic setup, it’s not until the moment she returns his gaze that objects stretch out to infinity and enter the audience space to a distance that is tangible. This exaggeration of depth takes the moment into the realm of fantasy, appearing almost dream-like. Most people wont notice any of this happening and this is quite important. Slow changes in depth are important to allow easy accommodation and maintain comfortable viewing.

Most of these changes in depth work at a subliminal level, tugging our feelings around without us realising. Just as a subtle shift of colour can nudge our perception of a scene, subtle changes in stereoscopic depth can alter how we feel. The stereoscopic language isn’t quite as developed as colour, but the more we experience it and manipulate it, the more we will understand how and when to use it.

Immersion

immersion-definition

Immersion is something that gets talked about a lot in the fulldome world. It’s what makes fulldome different to cinema. It’s something I think and write about a lot and the area I’ve been most keen to explore and experiment with whilst making Vessel.

Understanding immersion is something I’ve struggled with for a while. Even working in the immersive field of fulldome for so long I still haven’t really cracked it. I think this has something to do with a limited vocabulary. The dictionary definition of immersion has a few entries. The first two are interesting as they are both experiences that are quite different. The first relates to a physical experience, and the second is a mental experience. book-vs-rift

Books vs VR… This is something I’ve been thinking about a lot whilst making Vessel. I’ve been hearing a lot of people within the industry talk about it and I’ve been asking a lot of people outside the industry about their feelings on it. Richard Birkin is a writer, a musician and an avid book reader. He’s not working in what we might consider to be an immersive industry but the word immersion means something very specific to him in relation to reading.

quote_richard-birkin

When you exchange a few words, as I have done in red, you get similar responses from people talking about films. But with books there is no sensory manipulation. There’s no retinal resolution displays, no 360 degree video projection, no high-frame-rates… Just some words on paper… Yet still a high level of immersion can be achieved. But what about that sense of falling you get when the stars roll around the sky in a planetarium? You feel so immersed in that world of stars that it feels like you are falling. That’s not something I’ve ever felt from reading. I’ve never had to hold on to something as the words shift the world around me… At least not physically. This reaction is perhaps closer to what you might experience on a roller-coaster as you are literally span around. And how does the immersion experienced from a fictional literary journey relate to the same immersion we feel from the roller-coaster?

This statement from James Iliff relates to gaming, where the word immersion is commonly broken up into 4 separate terms. All of which are considered very different forms of immersion that require different game mechanics to induce.

quote_james-iliff

Immersion is perhaps one of those words that we’ve used to bundle together a few things that relate to each other. And maybe it wasn’t so relevant in the past to distinguish between them. But the more I discuss immersion and attempt to understand the basic principles involved in creating immersive experiences, the more I feel that there is something not quite right with my understanding at the foundations of the language I’m using to describe those experiences. This constant questioning lead me to breaking immersion into two distinct areas. Maybe there are more, but within the field of immersive cinema, two subsets of immersion made a lot of sense to me. These two subsets are mental immersion and physical immersion.

quote 06

Perhaps this is obvious to some people already, but to me this was a breath of fresh air. It made Vessel make a little more sense. For a while I felt like Vessel wasn’t really immersive. Or at least it wasn’t immersive for the first three quarters. Really I was just thinking in physically immersive terms.

ImmersionScript 09

This image shows how the level of physical immersion changes over time. What’s interesting about this is that most of the film has little immersion in it. Even though it’s designed for a 360 degree experience, most of the rendered image has nothing in it for the first third of the film. But what’s more important for the overall experience isn’t to feel physically immersed for the whole duration, but to feel mentally immersed and then use the physically immersive nature of the dome to accentuate specific narrative moments.

immersion-early
A fisheye still from earlier in the film, showing how little of the frame and field of view is used.

immersion-lateA fisheye still from later in the film when the level of physical immersion peaks

The reduced field of view and sedated movements at the start of the film focus the attention of the viewer and make it easier to follow the story. It’s the words that form the story that start the journey of mental immersion. As the story progresses, objects in the peripheral pull the audience further into the world until mental and physical immersion meet in a big explosion of love.

Spherical Composition

Vessel is designed as a full 360 experience. Understanding how people view immersive environments is crucial if you want to tell a meaningful story. Many principles from the disciplines of design, fine art, photography and cinematography can be used in spherical composition, much the same way they are used in traditional flat image composition. Colour, contrast, shape, tone, movement etc can be used to draw the viewers attention, but can be much harder to control when part of the image will always be outside the visible field of view. I’ve been calling this process domeography for a few years whilst working primarily in fulldome but perhaps a more appropriate term for 360 degrees would be spherography. Although a limitless field of view allows the viewer to explore the whole environment, it’s important to know how to control where they look to make sure they don’t miss key parts of the story. Previously my workflow has been to watch videos in the dome at work and then make amendments and occasionally get some colleagues to join me and watch them as they look around to see where they go.

Since getting hold of an Oculus Rift, my review process has completely changed. It’s great for quickly previewing videos without having to wait for the dome to be free and having it at home means that I’m able to iterate ideas constantly. Creating camera moves, testing them and reworking them, rather than a daily cycle of test at home and review in the dome. Because Vessel has a text based narrative it’s crucial for the viewer to be able to follow the story. So the rift came in useful for me to track peoples eyes for both the 360 and fulldome versions and understand when they were following something I didn’t want them to.

 

Love + Immersion

It’s easy to focus on (physical) immersion alone when creating an immersive film. Stories can’t be told through (physical) immersion alone. Immersion is really just one of many tools that can be used to carve a story. It makes sense to use it within an immersive environment but it’s important to think about how it fits into the story and when it can be used to project that story to the next level. In Vessel it’s used to represent that moment of falling in love, to create that head over heels moment. When two souls collide and find themselves spinning in infinity. For this moment I don’t think there’s a better tool for the job.

6 Comments

  1. Xiangzhen Lu
    May 30, 2015

    I must say I’m truly impressed by your work, Aaron! I saw this very one in Fulldome Festival 2015 in Zeiss-Planetarium Jena, Germany. Very nice graphics, animation and sound. I’m studying interface design, and I can say I will learn a lot by looking many more times into this piece!

  2. Aaron
    June 23, 2015

    Thanks for your kind words. I’m fascinated by interface design myself 🙂

  3. Devon Copley
    July 3, 2015

    Hi Aaron –

    I stumbled across your website earlier today, and then tonight was wandering around Littlstar and lo and behold there was LoVR so I booted it up…

    Aaron, I have to say, it’s an extraordinary thing, honestly one of the most beautiful and compelling experiences I’ve seen in VR to date. I’m personally a lot less interested in games — I want story and narrative, and you really delivered it in this terrific piece. I kind of wish I had seen the thing (film? video? experience? we need a name for VR things) before I’d read your blog post, since I kind of knew where it was going … but all the same the build and climax of it was really well constructed and the thought and planning you put into the arc of the experience really paid off.

    If it’s not obvious, I *loved* LoVR. Looking forward to more work from you. In the meantime, if you’re ever in San Francisco, look me up and let me buy you a beer 🙂

    Best of luck –
    Devon

  4. Aaron
    July 4, 2015

    Hi Devon,

    This is by far the best comment I’ve had on my blog. Thank you.

    Narrative in immersive media in general is very sparse. Every time I see something that has a story it really stands out and I’m always keen to see how other people approach narrative with an endless canvas. It’s something I’ve been trying to figure out for a while now.

    I also struggle to find the right word to describe immersive films. ‘Films’ doesn’t seem to quite fit… ‘Experience’ does, but it feels too broad. I use all kinds of weird terms to describe immersive cinematography also. It’s hard to communicate without a common language. I have an unfinished blog post somewhere on this topic. I probably should finish it sometime.

    I’m certain I will be in San Francisco at some point in the future. It’s been high on my list for some time as it’s home to the Morrison Planetarium at Calacademy. It has a rich history of immersive art and I have a good friend who works there. So if I do visit, I will take you up on that beer. I’ll go to great lengths for a free beer.

    Thanks again.

  5. Rednapx
    January 8, 2017

    I enjoyed thoroughly the VR experience (given that also i’m a fan of IDM Music) and this document shows that is not only a work on aestethicism, but is also a very well documented and researched video, enjoyed it very much.

  6. Chris
    November 15, 2018

    What software did you use?

Leave a Reply