Devs can't "create magic" with a spreadsheet in front of them

It's well established by now that the use of metrics and advanced analytics can have a profound impact on a game's business. But it's less examined what kind of impact it has on a game's developers. Sociologist Jen Whitson, a researcher at Technoculture, Arts and Games, addressed the issue in a presentation today at the Montreal International Game Summit.

Whitson prefaced her talk by stressing that she wasn't going to advocate abandoning metrics entirely. It's a tremendous gift to know how the games we make impact their audiences, but there are significant dangers to it. Whitson is a sociologist who left the surveillance studies field to examine the game industry, but noticed some of the big data trends from her previous field creeping into her new one.

She began by talking about the Ho Chi Minh trail in 1968, years into the Vietnam War. The trail was a key supply route, and the US sought to break those supply lines in an interesting way. It air-dropped camoflauged sensors along the trail that tracked everything from trucks roaring along the trail to people peeing on the roadside. When the sensors picked these up, a napalm strike would be dropped into the jungle corridors to take the trucks out. However, the Vietnamese figured out what the Americans were doing and began intentionally confusing the sensors with recordings of truck sounds, and even launching water balloons filled with urine into desolate areas. The Americans dropped untold bombs based on their sensors, devastating jungles, accomplishing nothing of strategic import, and wasting incredible amounts of money.

Whitson said the lessons can be applied to social and mobile games, where data-driven design has become increasingly important.

"In many ways, I see devs as canaries in a coal mine," Whitson said.

The difficulties the industry is facing today, the studios shutting down and difficulties finding work, are not unique to gaming. They are instead another example of the evolution of capitalism. Over time, society has transitioned from family-run business to giant companies using assembly line work. While the benefits of the large company were clear, the work was numbing after a while, and people didn't enjoy a life spent as a drone doing one thing over and over. That has since evolved to a leaner process, where workers move from company to company and project to project, learning a multitude of skills and challenging themselves at every step. This stage, which Whitson said should be familiar in the industry, is a meritocracy, but one that makes job security a thing of the past, where developers need to constantly update their skillset.

The gaming industry is undergoing a massive change after decades of console dominance, Whitson said, an era characterized by inequal power relationships between manufacturers, publishers, and players.

"Game development is the very epitome of precarious labor," Whitson said.

In any given week, a quarter of developers will work more than 50 hours a week, Whitson said. But typical weeks aren't always so typical, with crunch lasting as long as a year or more, and some studios operating in perpetual crunch. At the same time, profit margins in the console world are constantly going down, with only one out of every 25 games that starts production ever being profitable. The whole AAA scenario is similar to working the assembly line on the factory floor, Whitson said.

But things are changing. Whitson pointed to Angry Birds as an example. The game was made originally for a couple hundred thousand dollars, and has since ballooned into a half-billion dollar empire. It's an encouraging counterpoint to the AAA market, but big problems remain.

"Rovio made 51 games and almost went bankrupt before winning the lottery with Angry Birds," Whitson said.

The way such a hit could be made was with metrics, Whitson said. Through easily trackable metrics and A-B testing, it was easy for casual game developers to use real-time feedback to maximize revenue around whatever happened to work best. The results can be difficult to argue with, especially in the absence of other explanations for a game's success.

"The game industry can't understand why s***** games like Famrville came out of nowhere to make money," Whitson said. "It couldn't be the game itself, so it had to have something to do with the data under the hood."

The big change of metrics is that the design of the game is no longer dictated from above, Whitson said. Now, it's effectively "crowdsourced," determined by the users. One problem with this is that it drives success to the local maxima, the highest peak in the immediate vicinity. If there is a larger peak that can be reached, but only by going through a valley, reliance on metrics will keep games from reaching it.

"Metrics are really, really seductive, especially when they're seen as a lifeline for the industry," Whitson said.

There is also a belief that players don't know what they want, and aren't to be trusted over the data. When Battlefield Heroes suggested making powerful weapons available in the game, EA found that those who were the most vocal against the change were some of the biggest spenders on the new weapons and revenues doubled almost overnight. So while the casual and mobile revolution promised a change from the assembly line development of the AAA market, it has actually eliminated creative autonomy. Instead of being controlled by assembly line managers, developers now effectively have robot overlords in the form of metrics.

Metrics, by their nature, stifle innovation, Whitson said. They are excellent for optimization, but terrible for innovation. If a studio is all live operations and metrics-driven, the developers on the team will be hungry for more creative outlets. If a studio doesn't offer that, the talent will go elsewhere.

Secondly, Whitson said once developers get in a metrics-driven mindset, one where minor changes are made with clear evidence to say what was right or wrong, they are limited from considering other ways of thinking. They stop considering whether the game's secret sauce is its sense of humor, the real-world experience shared by players enjoying the game with one another. It's difficult, if not impossible, to hold both of those mindsets at the same time, like patting your head and rubbing your belly at the same time.

"If you want your devs to create magic, you can't also put the spreadsheet in front of them," Whitson said.

Third, stats can lie. They can tell you what your players are doing, but not why. And that makes them tell only half the story. On top of that, they can be massaged to make whatever point is wished. And since numbers are convincing, whichever person in a design argument has the numbers is likely to win. Basing something on numbers instead of intuition seems more rational. However, those who hate metrics the most should familiarize themselves with them so as to better make the case for their own gut instincts, Whitson said.

Fourth, analytics must be tempered by ethics, Whitson said. Developers need to talk openly within their companies about where they will draw the line. The decisions to cap spending or hunt for whales has repercussions for studios, possibly leading to layoffs and other such measures, but they should be made together.

Finally, metrics are a symptom of managing a risky industry, Whitson said. They can be escaped by running a Kickstarter campaign or focusing on niche markets, but it's risky and unpredictable. Whitson knows that metrics aren't going anywhere, but she stressed the need for people to consider them as just one tool in a set.

"We need to focus on making our industry a place to live in rather than a place to pass through before we burn out," Whitson said.