Being With Picture

I don’t remember exactly when but at some point in my earlier years I started imagining myself entering the pictures in my picture books. Someone would be reading to me and I would zone into the picture and wish I could step in. I would look at every detail and wonder what else there was. Franfreluche, a clown storyteller from a children’s tv show, did exactly that and I would watch her enviously. I wanted those giant storybooks that I could step into.

This curiosity about “being with” picture still animates me. To be clear I didn’t want to have an interactive picture world. I wasn’t interested in a game. I was interested in the sense of space, the colours, the world the characters were in. I wanted to see them do their thing while fully sensing their environment. A modest wish. Theatre and other live performances do this to a degree. We share a space and it is partially determining the performance. A crucial dimension to the pleasure of liveness is that we feel entangled with the performers through a shared space. Still with theatre there is a conceptual distance imposed by the stage. The characters’ world is not wholly accessible. In a way, films and books do a better job of immersion because they specifically ask us to dim some senses in order to merge on others. With live performance, our senses are awake and the merge or resonance happens through a more physical entanglement.

What I seek then is something in between theatre and movie, or between theatre and book. I want the merge/resonance of a book with the feeling of physical entanglement with all my senses awake. I do want to lose myself in the story. But I want to do so as a fully engaged observer.

Enter VR. The promise of the merge. In all its incarnations, we talked about immersion, like sinking into water, being surrounded by our imagination. And yet…not quite. Something is missing. The worlds are empty somehow, happening “out there” even as they surround us. We have readily adapted the gaming paradigm to the current VR platforms but so far have largely stayed away from the book or the theatre. But I have recently encountered two examples that are steering towards a “being with” picture. They have both made me happy, like discovering a new land that just feels interesting and that I always knew existed.

Cave is a piece showcased at SIGGRAPH 2018, by NYU’s Future Reality Lab (FRL). In this piece an audience of 30 people is positioned in two seated groups of fifteen pointed towards a front centre “stage” area. Each audience member dons a headset with headphones (or specialized Bose personal speakers). As soon as I put mine on I could see the other audience members as a virtual audience, their heads and bodies moving. We are not the usual avatars. We are semi-transparent, with large skeletal masks on our heads. The audience heads are tracked to their physical movements so I can look at my neighbour and if they are looking at me I would know to start a conversation. There was enough time before the show that I could do this, and I could also observe the world in which we had all been plunged: a prehistoric cave 12,000 years ago in Northern Europe. A young girl is illuminating the cave wall as (presumably) her mother is drawing on the wall. A soft humming song plays. It feels dim. It doesn’t feel cold though I suppose it must have been. I feel like there is a warm glow of a fire. It feels good to know that I am experiencing this shared space with others and that I can see them looking at different details of the cave. A few minutes later, the characters in the scene fade, and the show starts. The girl comes down from a ladder into the cave. She lights a fire and explains that her people are in grave danger and that they need a new shaman. Her mother was the last great shaman and she doubts if she possesses the same powers. As the story proceeds she transitions from doubt to confidence and activates her abilities to imagine and animate cave paintings. Along the way there is a magical mammoth, a visit from her mother, and handfuls of fairy dust. There were no cuts.

I saw Cave twice and the first time I was analyzing it from the perspective of someone who had just discovered a place where anything could happen. I looked around, I looked at the other people, I observed people as the “play” started, I looked for the intricacies of the movements of the characters, the voice, the sound quality. I wanted to sense the world I was in. How much detail was there? How many times could I watch this and still be enthralled? Why was I even asking these questions? The container was interesting and it was separate but wholly entangled with the story unfolding in front of me. Like theatre. But also like film. And like comic book. Not like a game. fascinating. The second time I saw Cave I decided to focus on the story. Could I be with the story and feel like we had been transported to her world as ghost witnesses? It turns out that the answer to that is “sometimes but not always”, and I think this is because it borrowed a bit too much from theatre at times, and a bit too much from film at others. There were amazing moments like a hug that was so well animated you could almost feel the brush of the hands against cloth and the gentle envelop of an embrace. We had time to linger and appreciate. In general, the timing of the piece was of theatre, with a slower more deliberate delivery (in fact the dialogue followed a Shakespearean iambic pentameter). It could have been even slower given the capacity of the space to speak for itself. I would have liked more detail in the movement or more expressiveness. It felt a little too much like theatre in the seating configuration. I wish I had been cross-legged on the cave floor with others strewn around me also seated on the ground. Though I recognize the desire to match physical situation (seated on a chair) with the virtual representation. I felt taken out of the moment when the music became too orchestral, too Disney-like. In a film, the music would have been appropriate for the moment but in this context it overwhelmed the space which was already providing many emotional cues. In that moment, focusing on complex environmental sounds might have served the same purpose as big orchestral sound in film. Still there were moments where tropes from film were very well integrated including magical visual effects. I think I could watch it a third time and still find something interesting. This is the first time I have felt this way about a VR piece.

To me, Cave is about “being with” picture. And yet it’s not moving pictures, it’s dancing space. I can see so many ways forward to explore the story space that FRL has opened. Many questions come to mind. What exactly is essential to what they created? Co-located audience? How many people before it’s too few or too many? Could some of the audience be less physically present (networked in)? Could the audience be moving around?  The possibilities for the space are also interesting. The audience could disappear. There could be cuts into different spaces including internal conceptual spaces. Once a world is created, it could be re-used for a number of stories, details filled in, characters introduced. This then starts to feel more like a comic book world: episodic and vast. I can also see the possibility of this kind of theatre/film/comic book mix in Mixed Reality. The questions then centre around how site-specific the story becomes, how much “liveness” is introduced into the story either through AI or other performative means.

The other piece that has changed the way I look at VR and space is Dami And Falian, a VR graphic novel being created by Edward Madojemu in the Basically Good Media Lab at Emily Carr University of Art + Design. Edward is creating a completely hand-drawn virtual world (drawn in AnimVR and Quill) where the world progresses in slices of time like a graphic novel. There are speech bubbles and portals in space and time. Within a slice of time you can explore the world, walk around. In that sense it feels a bit like Myst. There remains many questions to answer with regards to the design of a VR graphic novel including when a speech bubble appears/disappears, the orientation of speech bubbles, the use of sounds, the integration of object interaction. Yet the feeling of entering a world which exists regardless of the actions being enacted within it, is strong and interesting much like Cave. Edward has chosen to also integrate networked and social elements to Dami And Falien. You can leave notes in the environment and explore with someone else. This work is nascent and it’s unclear where it will land in the mix of theatre, film, comic, and game. Edward is in Generation Z. I am curious to see where his vision takes him and us.

As Dr. Ken Perlin of the Future Reality Lab reminded me in conversation, our generation has been heavily influenced by film and it is hard to see how much of what we do is driven by those tropes and visual conventions. I see Cave as an integration of past cultural loves and Dami And Falian as an integration of niche media, towards the creation of truly immersive spaces.

Links:

Future Reality Lab (their blog has followed the development of Cave as seen through the eyes of the production team)
Cave XR
Dami and Falian
Basically Good Media Lab