Cloudhead Games' VR sleight of hand

Since the early days of the Oculus, VR studio Cloudhead Games has been a dab hand at... well, hands.

Co-founder Denny Unger was dabbling in VR around the end of 2012 when he met Oculus Rift creator Palmer Luckey on an online forum and began experimenting with the headset tech; he even designed the first Oculus logo. In the process, Cloudhead Games was formed and began working with the headset prototypes, but always with an emphasis on hand interaction.

Cloudhead designer Antony Stevens tells GamesIndustry.biz that, back then, the headsets they were working with were limited to just head motions. That required extra equipment like the Razer Hydra for hand presence, something that people getting in on the ground floor of VR might not necessarily have on them. But the limited technology and audience did not deter Cloudhead.

"Hands were a big thing because they were the next step," Stevens says. "The endgame of VR was never going to be just our head. And hands are such a tactile part of the human experience in general and in gaming, too. You game with gamepads and controllers and control sticks, so the fine dexterity you get with hands, the manipulation, is all super important for experiences, and VR is trying to emulate that."

"The endgame of VR was never going to be just our head"

Antony Stevens

Cameron Oltman, Cloudhead Games senior programmer, adds: "The very first VR headsets out of the recent wave of VR stuff gave you the ability to stick a thing on your head and look around and look at an environment in every direction, but you didn't get that thing you get in real life where you can lean around and look under stuff. You could look in any direction, but it was basically stereoscopic 3D video.

"There's this thing that completely changes the nature of the experience when you get into room-scale, where you move your head and the world feels like it stays still around you, just like the real world. That's hard to describe if you haven't been in it, but it's a very transformative experience. And as soon as you're doing that in an environment, you want to reach out with your hands and grab stuff. You can't not do it, basically."

Cloudhead Games' fascination with hands in VR paid off fairly early on, when Valve took notice of its work. Valve had been working with HTC and its Vive headset, offering support through SteamVR, but it was also internally developing its own technology that just recently came to market in the form of the Valve Index. Key to the Index is its controllers, which not only track motion in VR space like most modern VR hand controllers, but also can detect individual finger motion and grip, allowing players to let go of and pick up objects.

That technology was a perfect fit for Cloudhead, which has been involved since long before the Index was officially announced, when its controllers were still referred to as "Knuckles" controllers. Cloudhead was approached by Valve in 2016 to create a demo for the Knuckles after its work on a cinematic fantasy adventure game called The Gallery, which prominently featured hand interactions.

"Lots of VR games in 2016 focused on what Owlchemy Labs [Job Simulator, Vacation Simulator] calls "tomato presence," which is where your controller turns into the object you're holding, rather than your hand holding an object," says Stevens. "We went in the opposite direction -- we had hands, and hand animations, and all that stuff, and Valve was working on these hand controllers, so we used The Gallery as the demo for the original Knuckles controllers, and we've been involved with Valve as the Knuckles evolved and as Index evolved."

Stevens says that the pitch for Aperture Hand Labs was essentially a "big book of ideas for Valve" that spanned multiple properties, but the Aperture theme was what took. The result is a five-minute tech demo for the Valve Index that takes the player through multiple hand gestures that make specific use of the Valve Index controllers and their individual finger tracking.

"Most of the interaction you do [in Hand Labs] is not what you think of as a normal interaction mechanic"

Cameron Oltman

The demo clearly takes place at Portal's Aperture Science lab, and it's guided by various robot personalities reminiscent of some of the characters from the Portal games. And though Portal writer Erik Wolpaw and Portal 2 writer Jay Pinkerton put their "signature twist" on the writing and dialogue, Stevens says Cloudhead had a fair amount of freedom to build what they wanted from the ground up -- a brief experience that gives just a taste of what the Valve Index's motion-sensing technology might one day be used for in a full game.

"On previous controllers, your gross movements are basically floating in space and [your controllers] are your hands," says Oltman. "All the interactions beyond that were a button press or a direct translation of something you do on a computer already.

"But the Index controllers were very different from that. There were all those buttons, but the compelling thing about them was the interpretation of hand pose and intent. Part of the difference for the experience of building for that was there was this large space to explore that didn't involve the sort of things you normally think of as game interactions. Most of the interaction you do [in Hand Labs] is not what you think of as a normal interaction mechanic.

"There was a general shift in thinking that was freeing and a challenge at the same time, because you have to come up with new ways to think of what an interaction is."

Both Stevens and Oltman say that they feel the Valve Index is a meaningful step forward, not just because of its controllers, but also because of its display clarity. Stevens recalls a time when Valve internally referred to the Index as the next generation of VR, but thinks the final product is more of a "1.5." But from both him and Oltman, that's a compliment.

They do acknowledge that the high cost of the headset ($1000 for two base stations, the headset, and controllers) is a bit prohibitive for developers, but the existence of a high-end, high cost product like the Index shouldn't surprise anyone.

"Anything that leads the pack in any direction is going to have a smaller development base just by nature"

Cameron Oltmans

"A lot of the thing that keeps coming back is, what is the gameplay of these Index controllers?" Stevens says. "And we try to show an example of what that will be with Hand Labs, and you haven't really seen too much of that with other developers, and that's likely because to get into the system is a bit more expensive than usual."

Oltman adds: "This is one of those things that happens with any kind of market fragmentation, where the majority of what people develop for is the common set of capabilities that a broad install base can make use of. Anything that leads the pack in any direction is going to have a smaller development base just by nature of how many people there are that are going to have it."

Because Cloudhead has a history of pushing forward on VR technology and capabilities, I asked the two what they saw as the next step for the kind of work they do with hands and immersive VR experiences in general. Stevens suggests that new narrative strides might be made in the coming years as AI improves, which could combine with motion-sensing technology for interesting and unique interactions with NPCs and characters in VR.

Oltman's suggestion is, unsurprisingly, very hand-specific.

"There isn't anything on the broad market right now that gives any kind of meaningful, physical, tactile feedback," he says. "Very fine motor control interactions are still basically all fakery. If you think about the things you do without even thinking about it in your day-to-day life, there are all kinds of very subtle, physical cues that are still impossible in VR, and we just kind of fake around the edges of them.

"AA is what's going to sustain the platform, and we don't have a solid system there yet"

Antony Stevens

"For example, there's this cup on the table in front of me. I can reach out and grab it quickly, and I'm not going to slap it across the room unless I'm really distracted, because I get these very subtle cues about exactly where it is and when my fingers are touching it. In VR, you don't have that. That's one of the things that would be really cool."

With the support of Valve and other big names in VR, Cloudhead has seen success in a tricky segment of the industry that's still struggling to win the same kind of mainstream support as consoles, PC, or other devices. Oltman maintains that cost has been a huge factor in this, but that systems such as the PSVR and newly-released Oculus Quest may soon break down the barrier.

For Stevens, it all comes down to content.

"I personally think there's an issue with how [VR] content is distributed, in terms of how it is curated," he says. "Sometimes it's overly curated, sometimes it's not curated enough. I don't think we've hit that sweet spot where we are with marketplaces like the PlayStation store or the Xbox store.

"We're seeing a lot of big-budget experiences coming from existing 2D teams who aren't looking at VR from the ground up. They're looking at it from their existing genres and ethos and games and IPs and that kind of thing. And then we're seeing lots of amazing experiences from tiny, tiny teams, but those aren't going to be big enough experiences to sustain the ecosystem.

"Collectively, there's a big issue with the middle tier in VR in terms of companies like that getting funded. There are some exceptions, but largely there are not too many of your average AA game in VR. There's lots of indies, a few AAA, but the AA is what's going to sustain the platform I think, and we don't have a solid system there yet."