Tackling Telepresence: ‘Spatial’ Delivers Collaborative Computing on Oculus Quest

Oculus Blog
September 3, 2020

In 2020, we’ve seen an explosion in remote work, with an increasing number of people and companies turning to productivity apps to more deeply connect with their coworkers and work in ways not possible through conventional video conferencing. Add in VR and its ability to engender social presence—the feeling that you’re sharing a virtual space with someone else—and you have a recipe for successful collaboration at a distance. And it just got a whole lot easier with the launch of Spatial on Oculus Quest.

With realistic 3D avatars, screen sharing, and hand tracking, Spatial lets you collaborate with others in immersive virtual environments. We sat down with Co-Founder & Chief Product Officer Jinha Lee to learn more.

How did you get your start in the tech industry?

Jinha Lee: Growing up, I was more passionate about art and architecture than tech and often found computer interfaces really frustrating to use. A 2D monitor and mouse didn’t feel like the right tools for creatives.

With a goal of changing how people interact with computers, I went to MIT Media Lab where I developed SpaceTop, one of the first AR desktop computers that lets you reach into the screen with your hands, and ZeroN, a levitating physical pixel. These projects went viral on YouTube which led me to speak at TED in 2013. There I met my co-founder Anand Agarawala, who created BumpTop, a pioneering physics-based 3D multi-touch user interface, which had been sold to Google. We shared the vision that the future of computing should live in the 3D space, instead of on a 2D screen.

What was the original inspiration behind Spatial?

JL: I have always been fascinated by the fact that, when computing becomes spatial, people can finally use it “together” in the same space. While at Samsung, I designed an experimental UI that allows people to use multiple connected screens to explore their content together.

I used to think of AR and VR as the ultimate platform for building this vision, but it had seemed too far out—until I tried out Oculus Rift and Microsoft HoloLens in 2016, which completely blew my mind. That was around the time I reconnected with Anand who had a passion for building a tactile UI that allows people to stay in a creative flow state. We both re-convinced each other that computing will happen in 3D space and knew it was time to start a company together.

How has the application changed during the course of development?

JL: Despite our initial passion for the collaborative aspect of VR, it wasn’t until later that we started building the remote collaboration platform. Instead, we started building a variety of experiments such as VR / AR launcher and Media Browser, which later became important building blocks of Spatial. Along the way, we found that the most magical moment of VR exists when more than two people share a scene, which led us to focus on collaboration and productivity. We thought there was an opportunity for VR and AR to solve a huge real-world problem of remote collaboration, where people and companies are spending tremendous time and money on business travel as they value in-person meetings over video calls.

Tell us about your avatar system. How did you go about developing that?

JL: Having compelling telepresence is key to successful remote collaboration. We started off by representing people as a box with googly eyes (which resembles Minecraft blocks) and moved on to cartoony avatars. But when we set our focus on productivity, we learned that people don’t take it seriously unless the avatar is a realistic representation of their colleague.

Today we use a neural network engine to turn a 2D selfie into a photorealistic 3D avatar in just a few seconds. The avatars are also brought to life through directional audio that attracts attention to the person speaking, hand tracking for real-life gestures, fist bumps or high fives, as well as simulated eye tracking, blinking, and lip-syncing to mirror real-life interactions as closely as possible.

How did you go about integrating hand tracking on Quest? Did you run into any technical challenges, and if so, how did you overcome those obstacles?

JL: Hand tracking adds a lot to telepresence by making avatars more expressive with much finer gestures. It enables quicker and finer interactions, such as typing on a virtual keyboard or drawing on a note.

It was sometimes challenging to deal with its sensitivity since certain gestures can get triggered even when people don’t intend to. Discoverability of gestures such as teleport was also a challenge. We overcame these issues by lots of iteration and prototyping across many layers from sensor data to new interaction mechanics.

What sorts of use cases have you seen for Spatial thus far? Any future use cases you can imagine that would be particularly effective?

JL: Pre-COVID, we mainly worked with large enterprises. Since March, we’ve seen an insane uptick in interest and usage—1,000% in fact. But interestingly this is not just from our typical Fortune 1000 companies but also from small and medium-sized businesses and prosumers, all looking for new ways to connect with coworkers and remain productive while working remotely from teams.

Spatial is being used across all industries, from consulting to pharma, automotive to medical, and everything in between. As consumer usage has grown, we are seeing usage on Quest skyrocket—becoming our most widely-used device to access Spatial today.

The main use cases are:

  • Immersive presentations: Ideas are expressed more creatively and clearly, and users pay much closer attention and retain learnings more deeply.

  • Team planning sessions: Users can fill their entire room with information as they coordinate projects and assign tasks.

  • Product reviews: Being able to work natively in 3D allows users to feel like they’re in the same room each holding the product in their hands.

Organically, we’re seeing that panels or town hall meetings work well in Spatial, and even virtual events with “booths.” Training and educational sessions are proving popular with customers, as well as 3D design and data review work sessions. It seems likely that we’ll see more and more meetups take place in Spatial that don’t qualify as a formal work meeting, but rather just an opportunity for everyday people to connect, build relationships, and share information.

What can you tell us about your future product roadmap? Anything that users should be on the lookout for?

JL: We have some exciting updates to come that play to our vision of becoming the most accessible collaboration tool, regardless of platform or device. Being part of the Oculus Store breaks down barriers on the most widely-used VR headset today. Additionally, in the fall we’ll be announcing our iOS and Android native apps, giving people access to a full holographic experience from a device they already own. With approximately 3.4B smartphone devices in the market today, that’s a pretty broad pool of potential users!

Many of our power users are using Spatial for over half the day, underlining that we are tracking towards a “holo office” experience where users are not only hopping in Spatial for meetings and collaborative sessions, but also as a whole new way of working throughout the day, expanding their workstation beyond the limits of their screen and into the room around them. In VR you break away the limits of space. More to come there!

How do you think VR and AR will continue to change the world of enterprise in the future?

JL: We think it will be the OS of the future—3D computing will become ingrained in the way we interact with computers and each other, across any line of business, in any industry, and accessible through any device. It’s how work should be.

When this kind of technology evolves further, we may live in a world where physical space and distance are no longer relevant, since you can work from anywhere and with anyone, reducing geographic and economic divides. Businesses will be able to source talent from anywhere.

A beautiful side effect of this is enabling human beings to build deeper connections in spite of physical distance. Imagine being able to feel like you’re working alongside someone throughout the day, even though that person is physically sitting on the other side of the world. Deep relationships will cross borders and oceans, and effective collaboration depends on mutual trust.

What kind of response have you seen while demoing Spatial?

JL: It’s one of our favorite parts of what we do—showing the world the result of years of hard work! The response is more often than not “Wow!” Our favorite quote (yes, we wrote it down) was “Incredible. I need a cigarette” (we don’t encourage smoking).

Users are blown away by the photorealism, the crisp images, and the intuitive nature of the app. We also get a lot of comments about the quality of our avatars—how it transports them into a virtual space where they actually feel they are together again with those they are collaborating with. This has been especially important right now while many of us are feeling the loneliness and isolation brought about by COVID-19.

The other heartening thing is that users find the experience extremely comfortable, intuitive, and easy to use. We pay huge attention to this. It’s essential that VR feels like a more intuitive way to interact with information. And it’s critical that it embeds into how people already work. Thankfully, user feedback has vindicated these efforts. Our attention to minimizing friction and enabling workflow integration has made adopting Spatial seamless.

What’s next for you? Any exciting updates in the works?

JL: Plenty! We have several key launches upcoming (some noted above) as well as a host of new hardware and software/connectivity partners in the pipe.

We are focused on our mission from the start: to empower anyone to be more connected, creative, and productive through a truly device-agnostic AR and VR platform that can be used by anyone, across any device—whether headset, desktop, or phone.

Anything else you’d like to share with our readers?

JL: We are pumped by our partnership with Oculus! Quest continues to be the No. 1 used headset for Spatial and is only growing in importance as our platform is being used more and more beyond the enterprise space.

Being on the Oculus Store breaks down the barrier to entry for so many of our existing power users and opens us up to a huge pool of new Quest users who can now simply search and start using Spatial. It opens big doors, and we are hugely grateful for our strong partnership with the Oculus team.

Check out Spatial on Quest today.