The problem with Metacritic

In a Develop panel titled Metacritic Killed the Video Game Score Splash Damage's Paul Wedgewood, Mastertronic's Andy Payne, Guardian writers Simon Parkin and Keith Stuart and ex-Future Publishing boss and PCGamesN founder James Binns all came out against the score aggregation site.

"Metacritic is Metacritic and nothing we say or do is going to change its role or change its presence or purpose or anything else," said Wedgewood early on.

"The truth is that where journalists really help is with discovery, particularly now we have this fantastic rise of indie gaming again and games that have much lower advertising budget, they don't have the opportunity to be discovered, I can rely on journalists to spend all their time doing that thing so they can find that for me."

He described it as an incredibly important role, but that ultimately what mattered was the opinion and quality criticism, the score wasn't important.

In a tense discussion much of the argument settled on the use of the numeric score, which Payne called "leading the witness," and Parkin suggested made scores seem like something scientific and empiric, when in fact they're entirely subjective.

"It gives the implication that there's some maths behind this, and that's why they go into spreadsheets, because there's some mathematical thing going on here," he said.

"When really scores are basically signifiers of how good you think the game is. It's not like a toaster or a camera."

Binns pointed out that trade media list Metacritic scores, and Steam even had them embedded into its store, when actually what consumers want is opinions from reviewers they know and trust. He also warned the audience, more than once, about the dangers of trusting early review scores.

"Always be suspicious of the first review that goes up, because somebody just wants to get the Google news result and get some cheap page views," he said.

He later added to that when asked about the publisher practice of blacklisting reviews and publications because a low score had dragged down their Metacritic average.

"The thing that is interesting is how a Metacritic score will change over time, if you look at the first reviews that go up generally they're fairly positive because friendly, smaller sites that are just are just excited to get a copy of the code may go early with a good review. And if you mapped it over time you'd see something more interesting than just journalists who are on message."

Stuart added that the biggest problem caused by Metacritic was the commodification of the relationship between the reviewer and the game and the reviewer and the industry.

"There's certainly times that I've been in trouble or writers for the Guardian have been in trouble because we've marked lower than lots of other people, and you get the developers on the phone complaining because you've drawn their Metacritic score down below a threshold where they were going to get a bonus."

He described it as troubling for reviewers to see those consequences, and suggested that inexperienced reviewers looked at Metacritic before deciding on a non-controversial score.

Suggestions to solve the problem included switching to a five star system, or, said Binns, using the other metrics available from games today to make judgement on a game's quality, like how many hours are people playing it for. Not everyone agreed.

"If you used that metric then Farmville would be the greatest game of all time," interjected Parkin.

Binns countered that on a business level, Farmville is a great game, leading Parkin to call him a monster.

"That's why critics are still important," he said. "Because they're judging it through a slightly different lens than just 'is this going to be profitable?'"