Is Bethesda's reviews policy really harming game sales?

Last year, Bethesda launched a new reviews policy for its games.

It decided that it wouldn't let critics have pre-release copies of its titles until a day before the game goes live. The publisher wrote: "we want everyone, including those in the media, to experience our games at the same time."

The policy began with Doom, and has since been adopted for Skyrim Special Edition, Dishonored 2 and Prey.

The question that many were asking at the time was: why? It seemed like an unusually anti-consumer move for Bethesda, as it forced gamers to take a gamble on the quality of the releases. The assumption was that the firm was trying to protect its games from negative reviews, yet all of the above titles received Metacritic scores in the mid to late-80s. All of them were good.

Now with the release of Prey - which has had a slow start in the UK - it's worth pouring over the data, to understand what effect this unusual policy has had.

"The data suggests that although the lack of reviews may have had a negative impact on Doom's week one, the sales were merely delayed"

Let's begin with Doom. In the UK, the boxed version of the shooter suffered a poor first week on sale. But in its second week, where you'd typically see these sort of games lose between 65 - 75% in sales, Doom's physical sales drop was just 35% - helped (no doubt) by the delayed arrival of the reviews. It suggested that although the lack of reviews may have had a negative impact on week one, the sales were merely delayed. Indeed, Bethesda says that Doom hit 1m sales on PC by August (worldwide), and US sales tracker NPD said that the game sold more in its first month than any previous title in the series (although it was released on more platforms).

Armed with that information, Bethesda adopted the policy for its subsequent titles, and the results were more mixed.

Skyrim Special Edition was up next, and it did very well in the UK, including in its first week. Of course, the game was a known quantity to fans, so it's not the best example of the policy in action. Physical sales dropped by just 53% in its second week, which again is higher than you'd typically get (although it actually wasn't too unusual during a rather unpredictable Q4). During a period where game sales often underperformed, Skyrim Special Edition emerged as one of the few real success stories. It debuted at No.6 in the US for the month of October, and fell slightly to No.9 in November.

The same was not true of Dishonored 2 in November. The game was absent from NPD's charts for the month, whilst in the UK its first week sales were lower than its predecessor. During its second week, as the reviews were going live, the game dropped 52% in sales. Again, although a fall of 52% is good for a second week, it was actually quite normal for Q4 last year. The game didn't pick up significantly until Bethesda lowered the price at retail.

Now onto Prey. It's still early days, but the first week's sales were disappointing (for the UK boxed market). However, like Doom, the second week drop-off was just 32%. Overall, the game hasn't performed as well as it deserved, but there are signs that it might enjoy a longer-tail at retail (again, like Doom).

So here's the next question: would these titles have performed better in total if the reviews were available from the start? And it's difficult to say.

Dishonored 2 and Prey are the notable under-performers here, but the bigger issue for these titles is the general consumer apathy for the genre. These games are called 'immersive sims' and another recent game in the category, Deus Ex: Mankind Divided, also struggled last year, despite the (generally positive) reviews going live before launch.

There are so many different facets in how a game performs that to pinpoint anything specific - like reviews - is to overly simplify the challenge that publishers face. Doom had a slow first week on console, but how did it fare on PC (its spiritual home)? The critics loved the single-player, but Bethesda focused on multiplayer before launch (with the Beta) - did that have a negative effect on day one?

"There are so many different facets in how a game performs that to pinpoint anything specific - like reviews - is to overly simplify the challenge that publishers face"

Did Dishonored 2's proximity to Black Friday hurt the game? Or the fact it was sandwiched between Call of Duty and Watch Dogs 2? And what about the reception to Prey's demo (which was released the week before it launched), did that help or hinder things?

In addition, Bethesda may also feel that it's better, from a PR perspective, to see the reviews spread out over a period of time. Kotaku wrote that the absence of a pre-release embargo meant that some critics would rush-review Prey (something that isn't recommended for that game). Yet Bethesda may feel it's a small price to pay so that it gets a more steady stream of articles, as opposed to every review hitting at the same time. The longer a game is being talked about, the better.

Yet there's no real evidence that Bethesda's review policy is making a huge difference either way. Sales do seem to be spread out more evenly between the first and second weeks, and are not just concentrated around the first few days. But that doesn't mean these games are losing or gaining sales - all it says is that some consumers would rather wait for the reviews before taking the plunge.

So we end up returning to that first question. If it's not making much of a difference, why do it? It's not exactly winning Bethesda many friends, nor do any of the three parties (Bethesda, the media or the consumer) seem to benefit.

Indeed, Bethesda is a company that prides itself on its consumer-first nature. That's why its preferred marketing activity is to release demos or betas - why should the press be the only ones to preview a game? That's why its E3 conferences are really consumer shows, with the industry tagging along (rather than the other way around).

So with that in mind, it's worth looking at how consumers feel about this reviews policy. And judging by the sales trends, plenty of them want to hear from the critics before they buy.