Devs must be careful with "massaged" metrics

Last month at the Montreal International Game Summit, PhD researcher Jen Whitson spoke about the negative impacts data-driven design can have on developers. The message was pulled from years of research as well as her own experience as an "embedded ethnographer" with start-up incubator Execution Labs, but as she told GamesIndustry International after her talk, it's not always one developers are eager to hear.

"Of course there has to be some sort of defensive quality," Whitson said of developers' reactions to her work. "You're saying that how you make your money has negative consequences. Obviously, we don't want to hear that in the industry. So pushback is natural, and I don't expect everybody to go out and drastically change how they work."

Instead, Whitson said her goal with the talk was to provoke discussion, to open up a list of possibilities for people to think about their workplaces in different ways. In some ways, the issue parallels that of crunch in the development world, and progress in addressing either could be slowed by people who equate them with financial gain. There's no shortage of successful games produced by teams that crunched, just as it's not hard to find wildly successful games with a reliance on data-driven design.

"We don't want to face some of these ugly things about our industry head on because the way we face them may mean that we're putting other people out of work, which is hideous to think about," Whitson said.

"My fear is that metrics are used to turn people into data just because they're much more tractable than actually dealing with real humans."

Jen Whitson

The problem, as she explained in her talk, is that developers are using metrics not because they're the best tool for the job, but because they're a substitute for player feedback that can be difficult, time-consuming, or even ego-bruising to collect and sift through.

"My fear is that metrics are used to turn people into data just because they're much more tractable than actually dealing with real humans," Whitson said.

There's also a danger that advocates of analytics put too much faith in the numbers, not just in guiding their design decisions, but in trusting where they came from.

"It's important to educate ourselves and to know how the metrics and analytics systems work, and to pay attention to how the messiness of actual player behavior is cleaned up for the data and then transported back to us," Whitson said. "We're taking that 'raw data' as the truth, but in some sense it's cleaned up and massaged along the way. It's almost like a telephone game."

Whitson isn't the only one with misgivings about data-driven design. Her impression on the situation is that it's not going anywhere, even though the "vast majority" of developers (including those who use it in their own games) dislike it. But given how sharply the industry has turned toward the practice, familiarizing oneself with it isn't just a design concern, it's job security.

"There's an idea that metrics and analytics technology is progressing so quickly, and there's so many third-party service vendors and different hooks you need to add in that if you don't get in on the ground floor and learn and test and know how to do these things now, the industry may move on without you," Whitson said. "You won't have picked up these skills about dealing with metrics and dealing with big data in the industry. And then you'll be truly f*****."