The eyes have it: Creating believable digital humans

With video games well and truly in 4K territory, creating believable characters is more important than ever to immersing players in your world and your story.

Even AAA games are under increasing scrutiny when it comes to their characters -- you need only look at the derision 2017's Mass Effect: Andromeda received to see how high a standard gamers expect.

Perhaps standard is not the right word, however. Countless smaller-budget projects -- especially in the indie space -- have proven you can forge an emotional connection between player and digital characters without the resources of BioWare. Instead, it comes down to believability -- but what makes a digital human a believable one?

"A character is believable if it conveys emotion and performance effectively," says Steve Caulkin, CTO at animation specialist Cubic Motion. "It doesn't matter if the character is photo-real or stylised - the animation needs to be clean, coherent and hit the correct emotional notes."

There are two key aspects to human interaction on which the entire digital performance of your characters depends: the eyes and the lips. Get these wrong, or create inconsistencies, and it will immediately push the player out of the experience. But getting these right is something of a challenge.

"A lot of subtle emotion is conveyed through the eyes, with the gaze direction and point of focus showing where the character's attention is, and subtle eye-widening and squinting hinting at emotion," says Caulkin.

"The lips need to look completely consistent with the character's speech, and all of the beats need to be hit and completely synchronised with the audio. And you can't neglect the rest of the face and body: eye movement needs to be synchronised with head movement, and everything needs to be coherent."

Caulkin also warns of several things that can make a character's animation jarring or unbelievable. This ranges from the obvious -- such as sharp discontinuities between animations or poorly represented facial expressions -- but there are also more subtle behaviours that can be off-putting.

"Very small errors in eyelines can have a huge negative impact on realism, regardless of how well the character is animated and rendered"

"If a character does not realistically transition between different emotional and idling states, it will be jarring," he says. "Very small errors in eyelines and the point the character is focused on can also have a huge negative impact on realism, regardless of how well the character is animated and rendered."

The CTO goes on to say it doesn't matter if your digital human is capable of inhuman things. Players will accept everything from Spider-Man's uber-flexible web-slinging acrobatics to Kratos' Herculean power in combat, because "people will go with exaggerated elements and accept them in context." It's mostly in terms of story and more personal moments where that scrutiny over characters increases.

That said, animations around combat can also prove to be jarring if they become repetitive. Even the most realistic swing of a sword will remind players the action before them is virtual if it's executed perfectly every few seconds. The key, Caulkin says, is variety.

"The simplest way to address this is to draw from a wider range of variants to avoid repetition," he says. "More sophisticated approaches take advantage of physical and statistical models to generate fluid animations within a range of variation and the constraints of each specific situation. Adding physical effects like gravity and external forces to animation will better reflect the real-world and add variation to pre-defined sequences."

Expectations for believable characters are only likely to increase as more players upgrade to 4K consoles, and whispers of the next generation looming on the horizon suggest another visual leap approaches. Caulkin predicts more realistic rendering and more detailed assets will be central to this leap, with subtle details already demonstrating where animation might be heading.

"We are already seeing things like light transmission through skin, refraction of light in the eye and even ray tracing in real time," he says. "Machine learning technology has the potential to learn how to apply physical simulations in real time and add a further level of realism."

He continues, "The visual quality of 4K gaming demands a more VFX-like approach to animation. The challenge for gaming is to do this on a larger scale than a movie, with all the additional procedural and interactive elements involved.

"Animation will need to move more towards virtual production [motion capture with a simulated environment appearing on screens during filming], with real-time integration, to do this effectively. This will allow animators to better visualise the final quality of their work in context. Animation technology needs to innovate across the board, from performance capture through to motion tracking, solving, and rendering."

But this doesn't necessarily mean motion capture is mandatory when creating convincing digital humans. Caulkin acknowledges that "believable animation can be done with skill and use of reference in many cases", although warns that this is "very difficult to produce consistently and at a scalable cost."

"For large projects, motion capture provides a consistent basis on which to build the animation and makes it a lot easier to retain believability, because it is tied to a real performance," he continues. "There are some effects which are very difficult to achieve by hand, such as preservation of volume in the facial geometry and the appearance of fine details like wrinkles. It's much better to model and simulate this behaviour procedurally and build these constraints into the character assets."

While the prospect of motion capture may be daunting, particularly for smaller studios without the luxury of a AAA budget, Caulkin notes that it is possible for these developers to utilise such technology, as ably demonstrated by Hellblade: Senua's Sacrifice developer Ninja Theory.

"By carefully deciding where to focus internal efforts and where to bring in external expertise, they were able to produce one of the best digital performances to date with a relatively small team," says Caulkin. "There is no escaping from the need to invest in things like quality scans, performance capture and animation expertise, but many of these components are becoming more accessible and affordable."