The Swayze Effect

Oculus Story Studio blog
Posted by Matt Burdette
18 November 2015

The gospel around here at Story Studio is to deliver a compelling, immersive, evocative VR experience by telling a great story. As a studio founded by former Pixar team members, we believe in a simple mantra: “Story is King!” We even put it up on the wall. We even put it in the studio name!

But there’s another mantra we believe in: “Presence is VR Magic.” We knew telling a story where you feel present would be a significant technical challenge, but we were surprised by how much it is a creative problem as well.

As we moved through production on the first Story Studio experience, Lost, the more we added to enrich the feeling of presence, the more the narrative felt disconnected. It seemed inarticulable; the best way any one of us could describe it was that “we didn’t care about what was going on.”

During one review, we huddled together and began spitballing questions about what was wrong. Maybe it’s the animation? It runs a little long when the Hand is being sad and lonely; maybe we can move that along quicker? It’s awfully dark; should we increase the light around the viewer? Does the audio feel like it’s coming from actual physical sources? The conversation gradually went into deeper, far more existential territory. Is it the story itself? Can you even tell stories in VR? What if you can’t? What are we even doing?

We thought “perhaps presence and narrative cancel each other out.” It made sense in a way: if you’re paying attention to the narrative in front of you, you’re far less inclined to be absorbed by your surroundings and allow yourself to be transported to somewhere else. Conversely, if you’re too swept up in the environment and the spectacle of virtual reality, you find yourself paying a lot less attention to the narrative action. Is it not possible to have both at the same time? How can one be present in a moment and also have a level of investment in the characters and the story? We considered how live theater addressed this. You’re present in a seat in the audience while real live actors perform roles in a story in front of you. Why is this any different? Do the rules of theater apply in VR cinema? Is the existence of a “fourth wall” a necessity in the scope of telling a story? If so, then why even do storytelling in VR? There is no fourth wall in VR.

We started trying a number of small experiments to adjust the balance between narrative and presence to see if we could smooth out this disconnect. We strengthened presence by bringing the foliage in tighter, thinking that if we had to peer around objects, that might give a stronger sense of being in the world. But that was just annoying. Obstructing the view made following the action feel like work. Plus, objects that exist in your immediate proximity are really fun to look at in VR. If we weren’t paying attention to the story before, we were paying even less attention now.

We noticed many people who entered the experience often missed the first couple minutes because they were gazing at the moon, or inspecting the leaves. In response, we extended the opening sequence by a minute to give the viewer time to absorb the surroundings and help acclimate to VR. Better, but something was still off.

Finally, we tried a moment where the Hand directly acknowledges the viewer by coming up and “sniffing” them. Suddenly, we noticed viewers seemed more connected with the character and reacted to the Hand’s closeness. The reactions viewers were having began to reflect curiosity instead of ambivalence. Many even leaned forward as though they were looking at another person. Now, this felt like a step in the right direction!

The team discussed two different schools of thought:

A. The mind has a limited capacity to interpret and process stimuli. The narrative occurring in front of you and the stimulating immersive environment compete for your brain’s focus. Maybe storytelling in VR is too much of a challenge for the average attention span? Why does it matter how the main character feels about his dog if you can’t get over this coffee cup sitting right here in front of you… it’s right here and ohman, look at it. It’s right THERE, you can just reach out and pick it up and there you go. You’ve lost the narrative.

B. The absence of viewer agency creates an invisible wall between you and the virtual environment. The actors ignore you, the world remains indifferent to your presence, and yet you are so undeniably there. Turns out, there could be a fourth wall in VR after all and it doesn’t feel right.

We thought Option A had merit to it: there is definitely a honeymoon period when new viewers are so enamored with VR that it floods their mind. But Option B seemed much more likely to me. Whatever was going on felt more specific than just a sensory overload, and at this point we had all seen enough VR (and enough of Lost) that we weren’t losing our minds every time we put the headset on. We were feeling a lack of connection to the characters, environment, and in turn, the story. I started calling it the “Swayze Effect”.

The Swayze Effect (or just Swayze, in the adjective form) describes the sensation of having no tangible relationship with your surroundings despite feeling present in the world. Much like the experiences and struggles of Sam Wheat, the protagonist in Ghost, the 1990 hit crime-romance film starring Patrick Swayze. Basically, it’s the feeling of yelling “I’m here! I’m here!” when no one or nothing else around seems to acknowledge it.

In Lost, we have a huge robot that stomps onto the scene with little to no acknowledgement of our presence and no indication of how he would react to us if he knew we were there. Up until he looks down and greets us, all we have is an enormous machine of ambiguous moral intent at a very unsafe distance from us. We started thinking that maybe things felt off not because we were too wrapped up in looking at leaves and the moon, but because we were too wrapped up in the whole “GIANT ROBOT. GIANT ROBOT HERE. GONNA KILL ME. GONNA DIE.” element. While probably not a factor that everyone experienced, this shed light on a critical element: the viewer is an entity in the scene.

With Henry, the problem presented itself in an entirely different way. The inclusion of Henry’s “look at” behavior (Henry glancing over and locking eyes with you during emotional moments) wound up being one of the more popular aspects of the experience. It showed us that, despite our hypothesis, many people didn’t need to have their identity in the world explained. Lots of people are just happy to be there and to take part in the experience. Despite this, there was still contention among team members of whether the “look at” behavior was the right move to make. There was still a dissonance between presence and story: why is Henry so lonely if I’m sitting right here with him? For this reason, many members of the team still consider Henry a flawed narrative.

We also discussed adding more interactivity. Maybe if the viewer had more of an ability to directly affect the world, it might help anchor the sense of presence in the space? However, this came with an all new circus of issues. If there are interactive elements, how do we convey that to the viewer? How do we convey a visual language that doesn’t feel hokey or obvious? Furthermore, how do we demonstrate the limits of interactivity? How do we keep them from constantly trying to rustle the bushes and pet the dog when there’s a story being told in front of them?

Giving the viewer the ability to knock things off shelves or alter their environment, while definitely cool, isn’t really solving the right problem. The problem we want to solve is to establish relationships between the narrative, environment, and the viewer that creates a sense of being there. This was one of the motivations behind having the Hand come close and “sniff” you in Lost. That fleeting moment where a character acknowledged your existence is what allowed presence and the narrative to come together.

While many technical hurdles we encountered were not all unexpected, the Swayze Effect was one that really surprised and challenged us. It was one of the first obstacles we hit while trying to tell a story and relate it to our perception of self and the subtle ways we test the legitimacy of the virtual world we’re in.

Honestly, we’re still trying to determine what the takeaway is. Lost showed us that not acknowledging the viewer can create a considerable gap in connecting with the story and action. Henry showed us that acknowledging the viewer is powerful but can contradict the intent of the story being told.

What we do know, though, is that the Swayze Effect is important and something we need to keep investigating and consider early on in the development process. Armed with this knowledge, we hope to keep pushing VR as a space to tell great stories with the added dimension of your presence being an element in the world.