Building Mixed Reality with Presence Platform: A New Dimension for Work and Fun

Oculus Blog
May 31, 2022

Imagine sitting at a virtual table with two friends. One is in the same room as you, and you can actually see her. The other is many miles away, but you can see his avatar, and all three of you can converse as if you’re in the same space. When you touch the table, you can actually feel it, and there’s a game board there that everyone can see. When you challenge them to play, any of you can move the pieces by reaching out with your hands.

Or picture sitting with colleagues who live on the other side of the world and feeling truly present with them while you brainstorm together at a whiteboard, each of you able to see your real-life desk in full color and use your real keyboard to take meeting notes.

In a fully realized metaverse, these and a wide range of other new experiences will be possible. But this isn’t some vague, far-off potential. This is already becoming real: hundreds of developers have been hard at work using our suite of machine perception and AI capabilities called Presence Platform to build apps that blend the physical world with the virtual — a kind of experience commonly called mixed reality — for our current-generation VR headset, Meta Quest 2. And many of them are available today.

Starting today, developers can build and ship apps that use the full set of Presence Platform capabilities we revealed at Connect last fall. This suite of tools supports developers building mixed reality experiences, and it’ll only be getting better, as we’ll be adding even more capabilities to the platform, including richer social presence, this fall. And today, we’re also releasing a mixed reality demo app on Meta Quest built with all the capabilities Presence Platform has to offer—The World Beyond, which users can try out themselves and developers can use as a sample app for building.

Later this year, we’ll also take mixed reality experiences to the next level with color Passthrough on Project Cambria, the first in our new class of VR headsets packed with cutting-edge technologies that will be a major leap forward for VR hardware. Cambria will unlock ​​a new dimension of immersion and presence, so people can transport themselves into work and entertainment experiences that combine the physical environment with virtual content. As an early look at this future, we recently showed the first sneak-peek of The World Beyond running in rich color on Project Cambria.

Whether it’s collaborating with coworkers on 3D designs in Gravity Sketch, playing an escape-room experience like the forthcoming mixed reality demo I Expect You To Die: Home Sweet Home, or tossing balls with a playful creature in The World Beyond, mixed reality will open up entirely new ways to bring our real environments into immersive virtual worlds.

“With mixed reality since you can see your world around you, you can afford to be more ambitious in your movement,” says Mathieu Castelli, the Chief Creative Officer at Resolution Games, which has developed the mixed reality experience, Fish Under Our Feet, for Meta Quest. “This opens up a lot of opportunities to take games into more places than just your living room.”

New Presence Platform Capabilities to Build for the Metaverse, Available Today

With today’s VR headsets, games are a primary use case and we know people will play them in the metaverse as well. But like today’s computers, the next generation of VR devices will bring many new use cases that take us closer to the metaverse, including productivity, communication, collaboration, and social interaction. And they will go much further by allowing you to visit deeply immersive places with friends, family, or co-workers that blend the real and virtual worlds in rich, full color.

While we’re a long way from having the technologies that enable the complete capabilities of a rich metaverse, VR is already pointing the way towards that future. With Meta Quest 2, we’ve built a platform that scratches the surface of mixed reality on today’s hardware. Meta Quest features outward-facing cameras that can show a representation of the world around us inside the headset via the Passthrough feature, and many Meta Quest apps are already incorporating physical spaces into immersive experiences.

When we first unveiled Presence Platform, its capabilities included Passthrough, Spatial Anchors, Scene, Interaction SDK, Voice SDK, and now our advanced Hand Tracking API.

Today, we’re expanding the power of Presence Platform by making our Scene understanding capability available to developers. Scene provides geometric and semantic representations of people’s spaces, and works with Passthrough and Spatial Anchors to help developers create room-scale mixed reality experiences. This means apps can be aware of objects in space, like your couch or desk, and know how and when to blend virtual objects with the physical environment so they can co-exist.

For example, imagine looking down and seeing a large handle sticking out of your living room carpet — which just happens to look like a frozen pond. When you grab the handle, you can smash the ice and then use a virtual rod to catch the carp that appear to be swimming below your floor. These are some of the things that Resolution Games has been able to create in their project Fish Under Our Feet that are enabled by Scene, which allows an app to create virtual objects that interact with the environment around you, including surfaces like your floor.

With Presence Platform, we’re giving developers an easy way to address these extremely challenging machine perception problems with simple APIs that hide all the complexity. “So if you’re trying to create content using color Passthrough,” says Meta Product Manager Sarthak Ray, “you don’t have to deal with things like depth re-projection or color alignment. Developers will get an easy-to-use API that will do this work for them."

Blending Virtual and Physical Worlds Together

At the same time, Presence Platform makes it possible to place persistent virtual elements in the spaces of your home. Imagine using an art app to craft your masterpiece. “Let’s say you paint your version of a Matisse,” says Meta Product Manager Rangaprabhu Parthasarathy. “The app can query the room and allow you to put that picture in a virtual frame and not only mount it on your actual wall, but also be able to distinguish between the walls—and also be persistent between sessions so that your paintings are still hanging on the same walls every time you put on the headset.”

Persistence, powered by Spatial Anchors, will help people be more productive, too. Picture standing in your home office surrounded by numerous virtual monitors spread around you in space, each of which you’ve customized to display exactly what you want, and this magic desk setup appears exactly the same way every time you put on the headset so you can get right to work. “Productivity experiences are a category that mixed reality will really open up,” says Ray. “By taking advantage of both Spatial Anchors and Passthrough, users will be able to create a persistent virtual workspace that also lets them interact with their physical environment and productivity devices, like their mouse, their keyboard, their stylus, or even with people co-located around them.”

Or think about playing the role of a secret agent trapped at home but under attack from outside enemies. That’s the premise of I Expect You To Die: Home Sweet Home, which Schell Games is developing for Meta Quest 2, and which utilizes Passthrough and Scene to blend the digital elements of a spy’s life into your home. “Just floating virtual objects over a real scene isn’t very interesting, but by using Spatial Anchors, we can really connect virtual objects to real ones in ways that are really compelling,” says Jesse Schell, the CEO of Schell Games. “The promise of mixed reality is to bring magical virtual objects into your real life, and Presence Platform really makes that happen.”

Finally, picture being able to create fantastical virtual scenes that transform the walls and ceiling of your study so that it looks like you’re standing in a forest, and then playing with a little creature that chases a ball when you throw it. You can see your hands, and you can see the furniture in the room as you move around. You can even bounce the virtual ball off your real couch.

Those are some of the things you can do in The World Beyond [LINK], which incorporates all of Presence Platform’s capabilities and which we’re releasing today for Meta Quest. Seeing is believing—step foot into The World Beyond and you’ll see the far-reaching potential of blending elements from physical environments into virtual experiences. We hope it will help spark developers’ imaginations about what’s possible when building for mixed reality.

Mixed Reality on Project Cambria Will Be Even Richer

Even as we’ve been working to ensure that Meta Quest owners can enjoy mixed reality, we’re also getting ready for a step change in VR hardware that will enable developers to build the kinds of advanced mixed reality experiences that will take us much closer to the metaverse.

Project Cambria will be our first in a new class of innovative, high-end VR headsets that furthers our journey towards the kinds of computing the metaverse calls for. Among the hardware advancements that Cambria will bring to enable next-level mixed reality:

  • Cambria will feature an all-new sensor architecture, including high-resolution outward-facing cameras (both RGB and monochrome) that have 4X the number of pixels as Meta Quest 2’s cameras, enabling the headset to recreate the environment around you in greater fidelity.
  • Cambria will also feature active depth sensing that uses an IR (infrared) projector. This will be a significant upgrade over the passive stereo depth sensing currently available on Meta Quest 2, which relies only on cameras for sensing depth. Cambria’s depth projector creates an invisible IR pattern on surfaces in the room, which helps reconstruct surfaces (especially textureless ones, which can't be as easily seen by the headset's cameras) with higher fidelity and accuracy. This works together with advanced reconstruction algorithms that triangulate the location of 3D points in the room that are sensed by the cameras. All of this technology helps reconstruct the space and perspective around you, inside the headset.

Together, these advanced technologies are key components for building more realistic mixed reality apps. And for developers, there’s no need to wait until Project Cambria launches to start building mixed reality experiences: If you create an experience for Meta Quest 2 using Presence Platform, it’ll also run on Project Cambria— in higher fidelity and with color Passthrough for a richer experience, thanks to its advanced hardware specs. “Color passthrough is a unique capability that's available on Project Cambria. And in the future, devs will be able to add more bells and whistles to their apps that are only available on this device,” says Parthasarathy.

Building Mixed Reality on the Meta Quest Platform Together with Developers

We know many developers in the industry are building mixed reality experiences today, and we want to encourage these dedicated enthusiasts to build more new experiences on the Quest platform with us, from productivity to gaming and entertainment, and even more that we’ve yet to imagine. That’s why we’re investing to support developers as they build mixed reality apps for both today’s devices and tomorrow’s. “Being able to bring virtual objects into the physical world opens up a huge array of new ways to interact, especially socially,” says Schell. “The feeling you are in the same room with a distant friend is incredibly powerful.”

Beginning today with mixed reality apps built for Meta Quest with Presence Platform, and looking ahead at the even richer experiences that will take advantage of Project Cambria’s advanced technologies, we invite you to come along for the ride. Together, we’ll help lay the foundations for the metaverse.