A beginner's guide to performance capture by Epic Games

One of the ways game developers add realism to character motion is through the use of motion capture technology. An actor dons a specialized suit and performs motions, and these motions are captured as a data stream and retargeted to a digital avatar. The result is more realistic character animation for both in-game loops and cutscenes.

In game development, the earliest uses of motion capture, or mocap, go all the way back to 1994. Since then, mocap has been part of the process for hugely successful franchises such as Gears of War, Grand Theft Auto and Mortal Kombat.

But in this process, there's traditionally been a disconnect between body and facial animation -- a common workflow involved using mocap for the character's body motions, and then having an artist animate the face manually.

With recent advances in motion capture technology, this disconnect is no longer necessary. Facial capture systems -- once clunky, stationary, and affordable only to big-budget studios -- are now more sensitive, less expensive and portable enough for an actor to wear during a mocap session. Some systems simply use an iPhone X attached to a helmet.

The combination of body motion capture and facial capture means that now, an actor's entire performance, complete with all its physical nuances, can be captured and transferred to a digital character. This type of capture, known as performance capture, enables actors to use their full range of skills to bring a character to life, and also gives creatives a much fuller set of data to work with.

The impact of performance capture on game cinematics is already being felt in titles like Ninja Theory's Hellblade: Senua's Sacrifice, where Senua's emotionally wrenching dialogues with her inner demons came directly from performances by actress Melina Juergens. The raw sense of Senua's turmoil in these cutscenes speaks to the power of performance capture in bringing a high degree of humanity to a digital character.

Watch on YouTube

Why real-time performance capture?

Another interesting aspect of Juergens' performance is that it was retargeted to Senua in real time during the performance capture session, with the animation of Senua in her environment projected on a large screen right in the mocap studio as Juergens was performing it.

The raw sense of Senua's turmoil speaks to the power of performance capture in bringing a high degree of humanity to a digital character

But if the animation was just going to be used for cutscenes, what's the point of setting up real-time capture? Couldn't they just record the capture data and retarget it to the character afterward?

The advantage of using a real-time system was that the performance could be tweaked right in the mocap studio, while the actor was still suited up. During the session, director Tameem Antoniades was able to instantly see how Juergens' performance worked with both Senua's rig and the environment, so he could direct Juergens as if on a film set -- each new take could be informed by the director's vision for the character and the scene, and by what worked (or didn't work) on the previous take. And when the last take was done, they knew they had what they needed.

Not surprisingly, the real-time graphics for the projection came through a game engine, in this case Unreal Engine. Unreal Engine's Live Link plugin is designed to stream data directly from mocap systems into the engine for real-time preview. Ninja Theory's work in the performance capture space has continued with the game's sequel Senua's Saga: Hellblade II.

Setting up your own performance capture system

Maybe you've dabbled in motion capture, or even just researched a suit or two. Maybe you've even looked at real-time mocap as a possibility for your cinematics. But where do you start?

Epic Games recently published "Choosing a real-time performance capture system", a white paper that explores the different types of systems for both body and facial capture. The paper goes into detail about how to choose a system based on your production needs, such as the use of props or interaction between multiple characters.

There's also an explanation of how facial capture systems work, which is different from body capture in several significant ways.

While body capture essentially looks for skeletal motion, facial capture looks at skin shape. The only moving bone in the face is the lower jaw, and most facial expression comes from facial muscles moving under the skin to cause relatively small bulges and divots. The possible changes in skin shape for all human beings fall into a certain range, so the system only has to look in this range. The Animojis on iPhone X and later take advantage of this technology to retarget any face's expressions and speech to an animated character in real time.

The paper includes case studies that show how different productions picked different systems based on their very distinct needs.

Simple real-time system for Bebylon

But not all performance capture aims to mimic human actors. Los Angeles-based Kite & Lightning, creators of the upcoming Bebylon Battle Royale, put together a relatively simple system for animating the game's stylized baby characters.

Cory Strassburger, one of the company's founders, developed a real-time performance capture system that included an iPhone X attached to a helmet. He demonstrated the system at SIGGRAPH 2018, and took home first prize at that year's Real-Time Live! competition.

LEGO minifigure real-time performance capture

Shortly before Warner Bros. Pictures released The LEGO Movie 2, they tapped digital studio Animal Logic to team up with HQ Trivia, the hit mobile game that runs live trivia contests every day.

The result was an animated LEGO minifigure of Scott Rogowsky, well known to HQ Trivia players as an energetic host. Rogowsky himself usually appeared on screen as he referred to questions, answers, and players' comments live during the game.

In a special episode of HQ Trivia timed for the film's release, HQ Trivia replaced Rogowsky with a LEGO minifigure, and Rogowsky donned a mocap suit and faced an iPhone to drive the minifigure's speech and movements in real time. You can see more about it in this video.

With performance capture now possible with affordable off-the-shelf hardware and free software like Unreal Engine, this technology is poised to take off in a big way, not only for games but also in filmmaking. Check out the white paper "Choosing a real-time performance capture system" to learn more about what it takes to enter this exciting, growing field.