VR and AR will converge, says Unity XR lead

The gaming industry may be conflicted on the future of VR and AR, but as director of XR at Unity, it's Timoni West's job to remain optimistic.

And it's a job she seems to find effortless, especially as she's constantly working with the developers who are pushing innovation into the technology.

"Since the advent of what people are calling the third wave of virtual reality, Unity's numbers have been [strong]. I think 60% of all VR content is made with Unity," she says. "The number is much higher for augmented reality, with 90% of all Hololens content, for example. And obviously, we're always in the mobile space. But the cool thing about virtual reality specifically is that when it came time to make these 3D interactive worlds, Unity had been doing that for years, but people just hadn't had the headset on which to do it."

West speaks with pride as she tells me that Unity has been involved with VR from the very start in its support of developers working on every headset on the market. And it plans to keep that level of involvement with AR as the technology blossoms, a plan that led to the announcement that AR Foundation was moving into preview at GDC 2019.

"The reason why AR only came about in the last three years is because we finally made small, portable devices that were fast enough to take in vast quantities of information...and do something about it"

AR Foundation is a Unity engine package that allows users to develop for AR across both ARCore and ARKit, with plans to add any other future platforms. This means that developers making AR games for one platform don't need to completely rework their creations for a second platform - they simply work across both.

West says this kind of ease is key at this particular moment in time, as AR is just now beginning to come into its own thanks to the advent of mobile devices capable of using the technology.

"When it comes to augmented reality, the key difference between it and virtual reality today is how much 'world data' that you need to take in in order for augmented reality to work," she says. "When it comes to virtual reality, you're tricking the human brain into thinking it's somewhere else. But when it comes to augmented reality, you need to have your digital experience work in the real world. The reason why augmented reality only came about in the last three years in a real way is because we finally made small, portable devices that were fast enough to take in those vast quantities of world information from camera feeds or wherever, and be able to process it, get semantic information out of it, and do something about it."

With both VR and AR still in their early stages, much of the development being done in either technology is still experimental and often surprising. Part of West's job is to stay on top of what developers need as they continue to make strides in either one, which she does by listening to Unity's developer partners, and also by keeping an eye on what major platform holders and fellow AR and VR rivals seem to be up to. A recent example she offers is that of 3D markers.

"We already have 2D image markers that have been around for a long time, like QR codes," she says. "The difference between a 2D marker and a 3D marker is that if an object is at an angle, you still need to have it work no matter what angle the device sees it at. ARKit is coming out with 3D object recognition.That's a thing we knew that we needed, and although Apple kept things very close to the chest and we didn't know if they'd have it, it was an obvious use case. So it made it easy for us to say we needed to support it across the board and make sure our developers have access to it in a consistent way.

"There are some things that are further afield that we're not sure how many different platforms will offer. Here's a weird example. You can extrapolate out heartbeat from eye tracking by measuring how the veins in the eye change. That's weird, right? Are we going to support heartbeat? I'm not sure. That's a use case where we know you can do it, but it's a question where, if we hear that the Hololens team is doing it as well, and then if we hear that a studio that is working with a hospital is also supporting this, or that Apple wants to put it in their health app, then we can say it's a compelling use case and make this a key feature we support."

"We got a Nintendo when I was eight, and that thing did not work 90% of the time. People like to make fun of virtual reality, but new manufacturing is always hard"

Keeping an eye on trends and industry conversations also means that West is part of the discussions revolving around challenges in XR development that no one has quite solved. One major one she highlighted was that of object persistence and relocation in AR. Put simply, AR technology still hasn't perfectly solved the problem of 'remembering' digital spaces, such as if you place a virtual object into a room, leave, and then return.

"If I put a giant digital teddy bear in my conference room and I want it to face the door, I need to make sure all the information about the room orientation, my device's orientation, and the orientation of the object in the room are stored, so when I go back in I can pull that information back from the cloud or device or wherever. Object persistence isn't a thing that can exist on its own without a lot of other stuff, which includes information about the room, you, your device, and where the object's last state was.

"There are companies specifically working on this problem, and we can very much assume that a large part of the big partners like Google, Apple, and Facebook are working on this problem independently. A big problem then is how we get all that information to relocalize across all platforms, but that's a discussion that's happening both in the community, internally at companies, and with all our partners."

Problems aside, West's vision of the future isn't just one where AR and VR simply function slightly better than they do now. She's untroubled by cynicism about either market, comparing VR in particular to the boom of consoles.

"I am 38 years old, and we got a Nintendo when I was eight, and that thing did not work 90% of the time. Sometimes it just wouldn't start! People like to make fun of virtual reality headsets today, but come on! New manufacturing is always hard.

"The first home computer console came out in 1968. It had twelve games. That was almost 20 years before Nintendo hit the consumer market of America. And just before Nintendo, there was a big dip in the console market. Computers were taking over. Then Nintendo comes out and they have this fabulous D-pad and some great games and they just totally killed it. But that took almost 20 years. That's a really significant amount of time. I think we're going to catch up a lot faster this time around."

In fact, West believes that XR development is about to start moving even faster. VR took its time to get there, she says, but there are now consumer-priced headsets on the market. AR isn't quite there yet, but because VR already has a lot of the mechanisms needed for AR to work, West believes a combination of the two technologies (at consumer-level pricing, too) is on the horizon.

"Augmented reality has all the challenges of virtual reality, and it has to have a bunch of sensors on it so it can take in the real world information," she says. "That's why it's lagging a bit behind. The cool thing though is that even the virtual reality headsets are starting to get sensors of their own, forward-facing cameras so they can do live feeds, so I think we'll start to see a convergence, possibly not in 2019, more in 2020, where these devices truly become more XR-ish."

"People want computers to be different than they are. That's why we're making virtual and augmented reality headsets"

But more than that, West thinks the true uses for XR are still to come. Rather than viewing them as just gaming devices or just industry tools, West's work with a wide variety of developers for a broad spectrum of projects has given her a vision of vast potential for the technology. The future of XR, she says, isn't as a headset you buy for one specific purpose. Rather, it's more akin to how buying a PC is now, with XR technology one day replacing typical screens by virtue of one day being far, far better at what they already do.

"You might spend a lot of time playing games if you're at a certain age, but most of what we do on computers now is really boring," West says. "We type emails to each other, we tweet to each other, Facebook. And right now text is still pretty shitty on lower-end VR headsets. The resolution just isn't that great. Until we hit that point, we might see that people just don't want to spend as much time in VR.

"I'm not that worried about it though, because in the end, people want computers to be different than they are. That's why we're making virtual and augmented reality headsets. We know what we have right now isn't good enough. We don't even like it that much. It's not ergonomic. It's not that accessible. People use computers every day, and they don't get to stretch their imagination, they don't get to express what they want, they don't get to use them to the limits of their intellectual or artistic curiosity. They're constraining little boxes.

"Even if it doesn't take off this year, we're always going to be trying to push this medium forward to that type of computer where anyone can pick it up and do what they want to with it. I truly believe that's the future we're heading towards."