Balancing Hearthstone
Metrics help the team understand what's happening, but designer Dean Ayala says most changes in the Blizzard card game are made due to player perception
In the games-as-a-service world, metrics are key. When the business model relies on continuous player engagement, it's crucial to understand exactly what effect the ongoing changes to a title are having on the player base.
However, as important as the metrics are, there's still an art to game design. Hearthstone game designer Dean Ayala made that clear in a discussion with GamesIndustry.biz at the Game Developers Conference in March. The topic first came up when Ayala was asked about the team's criteria for judging whether any given card succeeded for the developers or not.
"A lot of it is feel," Ayala said. "It's hard to decipher a lot of that information through data. There are so many variables happening over the course of a set, different events and everything, it's really hard to gauge if a specific card 'succeeded.' There's a lot of ways [to define success]. Did people craft it? Did people play it? Was it the right win rate?"
Sometimes the goal for a card is straightforward, like encouraging people to play with a new style of deck introduced in an expansion. But they don't just judge it on how many people build decks with it; they also look at how much fun it is to play, and whether it's being used in a diversity of decks or solely as part of one fixed strategy.
"It doesn't matter if something's a 65% win rate if everyone's having a good time"
"The real goal at the end of the day is people having more fun playing Hearthstone," Ayala said. "And a lot of times metrics can inform that. Game balance is not necessarily that you want all the classes to have a 50% win rate. Game balance is more that you want there to be a balance of strategies for people to play, a balance of fun things for brand new players to do versus fun things for master level players to do."
One limitation of analytics is that in order to track player behavior at scale, you need to have released it to players first. And as much as the learnings from those analytics can be applied to future expansions, many of the changes they will precipitate will be made to cards already out in the wild. And that's something Blizzard is loathe to do to Hearthstone without good reason.
"We try to do it as sparingly as possible," Ayala said. "We don't do it to shake things up. We change things because we think there's a problem. One of the reasons for that is because Heathstone has a metagame where sometimes there's a really powerful deck. And in a healthy metagame, there are decks you can play against that and the win rate of that powerful deck goes down, and the cycle continues."
Ideally, that means players are always thinking of new and interesting ways to defeat the top decks in the game using their own cards, Ayala said. But if Blizzard were to change things too frequently, players would be just as likely to stop trying those new approaches, content to simply wait until the next inevitable nerf solves the problem for them.
Ayala said the developers still use metrics to help them understand what's going on, but they generally aren't the driving force when the team decides to tweak released cards.
"We do most changes based on perception," he said. "Because perception's really the most important thing. It doesn't matter if something's a 65% win rate if everyone's having a good time. So a lot of the nerfs we'll make are on a deck that's like a 48% win rate, but people just really don't like it. Because that's making the experience worse, and that's the thing you change. The metrics don't really matter, but you should still use them to understand more."
It's clearly not all about perception, however. Ayala said if players are complaining that a certain card or strategy is too powerful, but the metrics say it's not nearly as dominant as the complaints make it seem, the team is likely to stay on the sidelines.
"It leads us to believe that maybe it'll be a problem that solves itself," Ayala said. "We can see the data that says this is countered by these five things, and the players will probably figure out the problem on their own. "
On the other hand, if the data backs up the players' complaints, that is taken as a sign that some intervention is necessary to preserve the evolution of the metagame.
"Sometimes you reach a point where the cycle sort of stops, where there's nothing that beats this thing," Ayala said. "When we look at data, player feedback, and use our own expertise and feel like we get into a situation where something's either so powerful that it's not going to go away or there's a strategy that's not necessarily powerful, but just really not fun and it makes the experience worse, that's when we go in and make a change."
As reluctant as the Hearthstone team is to make changes, they also understand it's essentially inevitable. After all, each expansion to the game is created in 28 weeks. During the first 14 weeks, one team handles the initial design work: core concepts, theme, and mechanics. Then it's sent over to Ayala's team for the second 14 weeks, where a design team of world-class Hearthstone players fine tune things for the desired effect upon launch. While they understand the game and high-level play as well as anyone, there's only so much they can predict about how players will ultimately use the new cards.
"100% of the time, it's slightly different than we thought it would be," Ayala said. "In the first 30 seconds of an expansion, people at large have played 100 times more than we did in 14 weeks."