While graphic fidelity goes a long way toward creating realistic, immersive, and compelling experiences in VR, sound also plays an important role. We recently introduced some significant breakthroughs in spatialized audio to the Rift SDK, including Near-Field HRTF and Volumetric Sound Sources. Today, we’re excited to share what these changes mean and introduce you to members of the team that’s making it all possible.
Spatialized Audio: Hear the Difference
Sound is crucial when it comes to creating a believable VR experience. Spatialized audio replicates how sound waves interact with the environment as well as your head and ears, so that you really feel like you’re in the virtual world.
While video games and conventional film present audio on a plane, VR uses head-related transfer function (HRTF) to simulate how sound would reach the ears, which allows for a full 3D audio experience. The Rift headset is designed so that the sound you hear comes from the same direction as a given visual stimulus. Thanks to built in headphones, that means designers can place sounds behind you as a prompt to turn toward something beyond your field of view.
All of this gives developers the tools they need to make their experiences more realistic, expressive, and immersive. And that’s just the beginning.
Brave New World
Film and traditional game development are largely known fields. With VR, there are new problems to solve that no one has had to tackle before. Rather than coming from a screen, sounds in a virtual environment can come from behind you, above you, below you—practically anywhere. This presents some interesting challenges—and exciting opportunities.
Spatialization is good at representing some things and conveying a sense of direction and distance. The farther away something is, the quieter it should sound. As you move toward it, or it moves toward you, the sound should grow progressively louder. But when something’s close to your head, all of this changes dramatically.
With the launch of Touch controls, there’s a lot of action taking place really close to you. That makes it all the more important to get the sound just right—otherwise, you risk breaking the sense of immersion and pulling people out of the experience. While Rift’s existing audio filters provided good directionality, they were designed to place sounds at least one meter away from the listener’s position in space.
With our recently introduced Near-Field HRTF, developers can model sounds much closer than one meter away with a greater degree of accuracy. Now, if you’re holding an object that makes sound (like a ringing telephone) and bring it closer to your head, we’re able to replicate that experience in VR in a more believable way.
Volumetric Sound Sources
In the real world, not all sounds originate from a single point in space—sometimes they come from a larger volume, like a waterfall or the ocean.
Volumetric Sound Sources let sound designers model objects of virtually any size in a way that sounds realistic. Rather than trying to pinpoint the source of a sound, designers can give a sound a radius—the larger the assigned radius, the larger the sound’s source.
As you get closer to the radius, the sound will get progressively louder. Once you step inside the radius, you’ll hear the sound all around you. While Volumetric Sound Sources alone don’t provide a sense of scale, they’re an important tool along with reverb balance and dynamics to help designers get the right mix.
Meet the Dream Team
Prior to joining Oculus, Software Engineering Manager Pete Stirling worked on audio tech for console and mobile games at FMOD. He was drawn by the increased potential for audio when working in VR. “Because VR headsets deliver such a consistent listening environment, you can deliver a lot more,” he says. “There’s more opportunity there.”
Audio Design Manager Tom Smurdon shares a background in game audio, with industry credits including Halo and Guild Wars—although he got his start as a recording engineer in Seattle working with bands like Foo Fighters and Soundgarden during the ’90s grunge wave.
“Once I heard what Oculus was doing, I jumped at the chance to be involved,” Smurdon says. “VR audio is new and super exciting—it’s an entirely different way to work,” he adds. “We get to test new theories of how to do music.”
Hearing Is Believing
“When audio’s done right in flat media, you don’t even notice it,” explains Stirling. “VR is a multiplying factor—when you’re actually in the space, your mind naturally accepts sounds as natural.”
Somewhat surprisingly, the believability of spatialized audio can even work against your overall sound design if you neglect to take the corresponding visuals of a scene into account. Stirling points to Oculus Dreamdeck as an early example—while one particular scene was meant to show off spatialized sound as an object moves around you, the effect was largely lost of audiences who followed the animated object with their head.
For developers entering the VR space, Smurdon recommends involving audio much sooner in the development cycle than might be the norm with more conventional forms of media. “Slapping it on at the end was never a good idea, but it’s even worse in VR,” he explains. “Since you have a 360° view, audio is really helpful in directing the player to look in the right direction. Audio plays a bigger role than ever given the absence of traditional directorial cues.”
The Road Ahead
Stirling, Smurdon, and the larger team have been hard at work for about three years, and they’re still finding new problems to solve. Right now, they’re tackling sound reflections—enabling VR experiences that better match the environment and making it easier for sound designers to get them to work automatically.
“The work we’re doing in this area makes what you perceive in VR feel like the real world without having to actually model the real world,” explains Stirling. This should make it easier and faster than ever for developers to bring realistic VR experiences to life.
We want people to step inside VR and experience new worlds. If it doesn’t sound real, the whole thing feels fake. Our goal is to blend visuals and audio into a cohesive, realistic experience—without you ever noticing we were there.
“All these little cues that you’re used to having in real life—they aren’t there in VR,” Smurdon adds. “We’re slowly adding them in. Every little piece is cumulative, and it makes a huge difference.”
You can hear more from Stirling and Smurdon next month when they present “2017 Breakthroughs in Spatial Audio Technologies” at OC4. Click here for more information.
— The Oculus Team