"You can't trust people": How to stop your playtesters lying to you

Playtesting is often the most frustrating and least rewarding phase of game development, but only the naive and foolhardy would consider forgoing it entirely.

Seasoned game developers will tell you how important it is to playtest your game because, as a creator, you're so enmeshed with the project it's impossible to take a step back and experience it in the same way a new player would.

However, by placing themselves at the mercy of the player's honesty, developers have been effectively hamstrung when it comes to testing.

That's according to Dr Paweł Strojny, head of research and development at Nano Games, and assistant professor at Jagiellonian University in Poland.

GamesIndustry.biz caught up with Dr Strojny at Digital Dragons in Krakow recently, where the experimental psychologist shared his thoughts on why developers should be using psychological tools and the scientific method to make better games.

During his presentation beforehand, Strojny displayed an ad for playtesters which, to his mind, perfectly sums up the nature of the problem.

"We are looking for free testers who will play, test, and give honest opinions about our game," the ad reads. "We really want to improve our game, so the more opinions and suggestions from the players the better, that's why we're looking for a large number of people. Tests will be carried out at [our] office."

Therein lies the issue with standard playtesting practices: it relies on honesty. This factor is only worsened when the testing takes place at the studio, often with snacks and drinks on offer. Buttering people up with treats to entice more positive coverage is a trick as old as time in the world of public relations, but in playtesting you're after constructive criticism, not favourable preview pieces.

"It's the rule of reciprocity: if you are kind to me, I am going to be kind to you," says Strojny. "Anonymity definitely helps... If you don't know who I am, I don't need to be kind... People always try harder during research. It doesn't matter if we are going to show them the results, if they believe we are going to find out who is who. You should definitely anonymise the data because it helps, [even if it] doesn't rule out all of the problems."

"It's the rule of reciprocity: if you are kind to me, I am going to be kind to you"

Ultimately, when speaking face-to-face with a developer, people are reluctant to provide negative feedback, even if it's honest.

"I'm a psychologist. I don't believe in honest opinions at all," Strojny adds.

However, using operationalisation (a process of defining the measurement of a phenomenon that is not directly measurable, though its existence is indicated by other phenomena) to measure something like commitment, frustration or engagement, Strojny and his team at Nano Games were able to highlight the damaging disparity between self-assessment and fact.

By applying behavioural observation study methods to motivational intensity theory -- the commitment to any given task in relation to difficulty and potential motivation -- Strojny and his team studied player engagement on puzzle platformer Icy Tower using space-bar hits as a metric for engagement.

The results? A huge discrepancy between self-assessment and behaviour. According to participant feedback, they grew more and more engaged in the game the harder it became. The chart, however, tells a very different story, showing a clear plateau and sudden drop-off in engagement while playing on insane difficulty (which was impossible to complete).

The reason, according to Strojny, is simply that people want to be helpful, and that people are polite. If you make it an enjoyable experience to take part in the playtesting, the game itself becomes secondary and so your data is compromised.

"You can't trust people," is the message Strojny drills home throughout the presentation and interview.

This type of behavioural observation study is arguably the most effective for developers looking to improve playtesting; it's mostly-objective, precise, easy to conduct, and relatively inexpensive.

"I would at least say that [self-assessment] has merit... The proof I showed you doesn't mean that everything else is trash"

Although it comes with the downside of producing large amounts of useless data, and requiring laborious analysis, Strojny says that even smaller indie studios can vastly improve their processes by bringing on board academics like himself.

"I don't know about the US, but in Poland and Europe there are plenty of researchers which are working in universities, are low paid, and want to do research," he says. "First time I met with [Nano Games] it was for free, or almost for free because for me it was an opportunity to make some practical, interesting research.

"My advice would be for the small teams to look around nearby universities, and find somebody who is publishing, who is interested; even a PhD student might be enough because they are trained, and highly competent... and the majority of them are eager to research."

Psychophysiology -- which tracks cardiac output, the heart's electrical action, and electrodermal activity -- is another avenue to consider for playtesting, but comes with some fairly major drawbacks. While objective and hard to tamper with, it's also very expensive, difficult to apply, and is unable to distinguish the causes of alterations in the subject's mental or physical state, only that they happened.

Self-assessment is the most straightforward, but deeply-flawed means of measuring player engagement. Even so, it is not entirely worthless, and can be used with caution.

"I would at least say that [self-assessment] has merit... The proof I showed you doesn't mean that everything else is trash," says Strojny. "You just need to be very careful, and you have to know the limitations. Psychologists test their methods and research papers, so if you want to conduct proper play testing you should try to understand the tools, but there are plenty of tools that can be used.... It's not always true what people tell you."