How can machine learning be applied to game development?

Google talks somewhat regularly about the "next billion users," specifically about how the products and features it designs will help bring them in to the company's ecosystem. Speaking with GamesIndustry.biz recently, Stadia research and development creative head Erin Hoffman-John says her group is more focused on the "next billion gamers."

If Google is going to reach that audience, Hoffman-John says it's going to start with the tools it provides creators.

"We figured that in terms of getting games that can reach really wide audiences and empowering all kinds of new developers to come into the process, we have to make game development easier and smaller teams more effective," Hoffman-John says.

To that end, Hoffman-John's team has been working on machine learning to help address some common pain points and bottlenecks developers have. The team is composed primarily of game developers with a number of engineers in the mix to help apply some of Google's existing technology to game prototypes.

Hoffman-John says they look at the work with "a relatively long horizon." It may take two to five years to even prove the tech they're exploring works, much less to bring it to market in a shipped game.

She shows us a project called Chimera as an example of the what the Stadia research and development team is working on. The pie-in-the-sky idea is that someday machine learning tools would allow a 20-person development team to create a game as large and complex as World of Warcraft. But Hoffman readily acknowledges that's a bit far off, so the idea was to start with using machine learning to streamline development on something smaller, like a collectible card game (CCG) along the lines of Magic: The Gathering.

Hoffman-John notes that for many actual CCGs, the bulk of the work and budget rests on contracted artists who paint and design the various cards. Hoffman-John said on such strategy games, about 70% of development time and investment is spent on generating content in a repetitive way, like making small variations of monsters to populate the world.

"It's not the creative stuff that game developers really want to be doing," Hoffman-John said. "It's the stuff you have to do to fill out a content pipeline so the world seems rich."

So with Chimera, the Stadia team wanted to have machine learning create those monsters for it. The team took inspiration from generative adversarial networks of the sort used by This Person Does Not Exist, which produces fake photographs using a machine learning model trained on photos of actual people.

Hoffman-John says Chimera works on a similar principle. The artists created a number of animal models and broke down the way a CCG card is typically composed to establish some rules: the scene is lit from above, the creature is in frame and dynamically posed, and the camera angle comes from below to make it look powerful. Then they used a machine learning model trained to recognize quality poses and another that would find photographic landscapes for the background of each card and apply a style filter to give them a hand-painted appearance.

With all that working together, Chimera would generate dozens of possible cards for the developer. At that point, the artist could pick out elements they liked from the various options presented to them and tell the program to produce a new card merging those traits. The tool also gave them the ability to fine-tune the creatures, which was absolutely necessary.

"It turns out if we just let machines stick animals together, you get what my team calls nightmare fuel," Hoffman-John says. "They're horrible. They're amusingly horrible, but they're not what we're going for. If you just let the machine do its thing, it will give you something wildly outside of the artist's intent. So if what we want to do is empower [developers], we have to let them be very specific about how they're directing the AI."

The tools allow developers to tweak the mixed-and-matched animal parts that make up Chimera's, well, chimeras. They can tell the system to add wings to various parts or remove them, to make one part more like a bird, or another more like a fish.

"With this system, it allows me to try some wild stuff, and then the machine will tell me what's wrong and what the win probabilities of using those abilities are"

Hoffman-John calls the process "having a conversation with the machine," and it's not the only part of the project that works like that. Beyond machine learning's applications for asset creation, Chimera also examines its potential for game design. In addition to using machine learning to help create the visuals for cards, Hoffman-John relied on it to inform the mechanics of the card game itself.

It's not uncommon for competitive games to discover balance issues only after a title is released to a wide audience. It's one thing to have a game appear balanced to an experienced group of developers and play testers; it's another for it to appear balanced to hundreds of thousands of gamers all looking for an edge on launch day.

Hoffman-John says machine learning can help there, as it can playtest a game millions of times using a multitude of strategies and shine a light on the ones that might be more powerful than the designer intended. For Chimera, she deliberately created a problematic gameplay system with abilities that would likely be over-powered or difficult to test and used the machine learning model to help refine it.

"With this system, it allows me to try some wild stuff, and then the machine will tell me what's wrong and what the win probabilities of using those abilities are, and then we'll just nerf them back," Hoffman-John says. "And we can do all of that without releasing it to players. Ordinarily, you'd have to put it out in the wild and you'd piss people off because it's like, 'What's wrong with this ability.' And it's because the system was so complex that I couldn't predict that was going to happen."

This is not an entirely unique application of machine learning to game development -- Ubisoft was talking about using machine learning to playtest For Honor in 2018 -- but it is in keeping with the research and development team's goal of creating tech that can streamline the development process.

"I'm not sure if in the net it would reduce costs; it's just those costs would be redirected to doing other things"

The Chimera project has the most obvious implications for artists and game designers, but Hoffman says part of the Stadia research and development team's goal is finding out which disciplines machine learning is best suited for across all aspects of game development.

"As we go along, we're trying to distill [discoveries] into principles about the use of machine learning itself," she says. "I think we're not completely there yet because we're still feeling things out. But generally, when you have situations where you want hundreds of thousands of possibilities, and then you want some help curating those possibilities into what you want, that's where machine learning is helpful. You can kind of think about it as the next step beyond procedural generation. With games like No Man's Sky, you're seeing the limits of procedural generation."

So if experiments like those of the Stadia research and development team come to fruition and machine learning empowers even small groups of developers to make the sort of games that used to require small armies of developers to produce, what impact does that have on game budgets?

"That raises interesting macro questions about all of game development," Hoffman-John says. "You could assume that we could just flip the dial and do that. I think that's probably true, but what will tend to happen is when you give developers more power, they just do more stuff. I'm not sure if in the net it would reduce costs; it's just those costs would be redirected to doing other things. And hopefully the net is that games get better."

Regardless, she emphasizes that the full impact of machine learning on the industry won't be known for some time, and the work her team is doing is still in the earliest stages.

"There's a lot we don't know and I think what excites us is putting this stuff in the hands of developers to see what they do with it creatively," Hoffman-John says. "Which also means all this stuff is pretty far out. Even we don't know exactly what will be done with it. We're just hoping to hand these tools to developers and see what they want to do with them."