"Are we all just fooling ourselves to some degree?"

Let's talk about market research. We cover a lot of it at GamesIndustry.biz, and like any media content you consume, it goes well with a grain of salt.

In many cases, the research can be boiled down to a few simple ideas. People are going to spend lots of money on virtual reality in the next five years. People are going to spend lots of money on eSports in the next five years. Emerging markets are going to spend lots of money on games (perhaps even virtual reality and eSports) in the next five years.

I'm pretty confident that all of these things are true. Anyone who's been paying attention to the industry likely feels the same way. We should all be able to agree that these things are very likely going to happen; we're just going to disagree on exactly how much money we're talking about, and whether or not that number would constitute failure. And we may disagree wildly on those issues because we're all measuring performance against our own differing expectations, and we're talking about a future with an incredible number of variables at play.

Will VR find its killer app? Will motion sickness give the tech a bad reputation, or will a high cost of entry keep people away? Will eSports evolve into a wide field supporting a plethora of games, or will it be League of Legends, Starcraft II, and Counter-Strike: Global Offensive for the next 15 years? Will efforts to televise competitions break through to mainstream audiences? Will the Latin American market settle on a frictionless payment standard for mobile users? Will the hassles and costs of doing business in China finally cool enthusiasm for the massive potential audience it offers?

"It's not a popular thing to say that things aren't growing. There's this emphasis on the economics of permanent growth. And that's a mindset. That's what people want to hear."

Joost van Dreunen

Anyone can guess at the answers, and they may even have reasonably sound justifications behind them, but they're still just guesses. Oftentimes, they're guesses that analysts at market research firms are making with limited information. People already deeply entrenched in virtual reality development, eSports organizations, or emerging markets probably understand their fields better than industry generalists trying to quickly bring themselves up to speed on complex niches to make the best guesses they can on their latest assignment.

Regardless, people are paying for those guesses, often to the tune of thousands of dollars per report. It sounds like an exorbitant amount, but when you have an industry where people need to justify every action with numbers and often have a research budget to help them obtain those numbers, market research suddenly becomes a very viable field.

The catch here is that people looking to justify a course of action typically have a course of action in mind. So if you were a start-up looking to attract venture capital for your VR eSports project, would you be buying the research that says, "This stuff is a fad," or would you be buying the rosiest research you could find, the one that tells prospective VCs your idea is likely to make a sickening amount of money?

"It's not a popular thing to say that things aren't growing," Superdata Research CEO Joost van Dreunen told me when asked about the subject. "There's this emphasis on the economics of permanent growth. And that's a mindset. That's what people want to hear. You want to make your boss happy by saying, 'This is amazing. Look at this cool thing I found. Give me a budget and I'll build this because we have all these reports that say it'll be fantastic.' That's a much more interesting thing for people to do than saying, 'We should be cautious.'"

Van Dreunen said that he received a number of angry emails from companies last year for suggesting that the mobile market is saturated.

"The moment you start saying, 'We need to watch out for this because it looks like the costs of production and marketing are going up tremendously the next 18 months,' now you're one of the suits and you're ruining all the fun," van Dreunen said, adding, "If somebody's paying you to write a report on how awesome something is, does that affect your findings? It's a tricky thing for researchers to walk that line."

Newzoo CEO Peter Warman also acknowledged the abundance of positive research in the field, but added that there was a lot of positive stuff to write about in gaming these days. And it's not necessarily that these research firms never met a trend they didn't love; there just might not be a market for research that questions the conventional wisdom.

"Everybody has actual data as sources, but it's a different situation from the past where 99 percent of the market was just counting the boxes times the prices. Over the past years, all figures have become estimates, you could say."

Peter Warman

Take VR, for example. Warman said that the technology will be huge "a generation from now," but Newzoo research is focused on software revenues, and its initial research into VR suggests there aren't tremendous opportunities in that field in the short term. And to hear him tell it, the reason that research hasn't been offered for sale is explicitly because it's not positive enough.

"We don't feel ready to report on potential game revenues because it's zero now, and we think it will actually remain in the millions or tens of millions for the coming year and a half or so, for software," Warman said. "We tell this story to press a lot when we get asked about VR, but we have not chosen to write down that opinion as an opinion piece, which would counterbalance the super positive articles a little bit. So I do agree that we should feel responsible to do that as well."

The issue is compounded by a lack of clarity in the industry, partly from the abundance of players in the market proclaiming their own set of numbers to anyone who will listen (both van Dreunen and Warman seemed particularly skeptical of analytics firms calling their numbers research a marketing tool), and partly from the ever-shifting business models and distribution methods in gaming.

"The world is completely digital and there are no actual box counts anymore," Warman said. "Everybody has actual data as sources, but it's a different situation from the past where 99 percent of the market was just counting the boxes times the prices. Over the past years, all figures have become estimates, you could say. And that has led to the problem, as you call it, that we see today. It's a lot more complex to [track] that in a sound way."

Even the lingo of the industry hurts. It's hard to make apples-to-apples comparisons when everyone is using different classifications for terms like 'social gaming,' 'casual gaming,' and 'MMOs.' Earlier this month, I even had one research company explain a discrepancy in its forecast numbers by explaining that it was including physical copies in the number given for 'digital games software sales.'

"This is somewhat the bane of my existence," van Dreunen said. "I'll present a number and somebody will say, 'Well I have [a different number]. How does it line up?' It's like, I don't know. I don't know their methodology. I don't know who worked on this. So everybody feels like they have access to information, but there's no real actual knowledge in this... It's really, really confusing."

"[E]verybody feels like they have access to information, but there's no real actual knowledge in this... It's really, really confusing."

Joost van Dreunen

As Warman said, that sort of confusion can be damaging, not just to the research companies involved, but to the industry itself. He said that several years ago when NPD dominated the market of games reporting and Superdata and Newzoo were just starting up, he and van Dreunen discussed the NPD's attempts to report on digital sales and the ways they could improve on them.

"When I think back to my discussions with Joost from Superdata, one of our objectives was to make sure that our high-level data would not be... Well, we would sort of check with each other. Because it doesn't help the market if global figures or US figures in total are very different. It confuses people and it doesn't help anyone."

As an example, he brought up eSports as one sector where Newzoo and Superdata came to different conclusions. In May, Superdata released a free market brief putting 2015's worldwide eSports revenues at $612 million. The next month, Newzoo released its Global Growth of eSports, which only projected a $250 million worldwide market in 2015 and cost $4,900.

"That made us really mad," Warman said. "There was no product behind it; it extrapolated $200 million revenue in Korea from 2007 to the current day, completely disregarding the crash that happened in Korea. That was an example where it really hurt us. We had to explain a lot. I hope they had to explain a lot. And it showed how confusing it can be for a market if the high-level numbers are so different."

So not only is a confirmation bias involved in research, there's also an incentive to release reports that essentially echo each other. If your findings are too negative, there might not be a market for them. If your findings are different from other companies' findings, expect to get some backlash.

And that doesn't even begin to get into the challenges inherent to studying an industry that changes as quickly as games. There's a lag between the time a trend starts picking up steam and the time it takes researchers to identify it as a viable subject for a report, collect data on it, analyze the data, articulate their findings, and finally offer them to the world. And then customers need to get the report, evaluate the information inside, and adjust their own plans in line with the report's findings. Even if the report is fundamentally sound, the information within might be considerably less useful when it's actually put into action than it was when it was collected.

"It's too strong to call it a lie, but you join the party to some degree on your own terms. Ultimately, everybody to some degree is sipping from the Kool-Aid because you don't want to be the one that rains on the parade."

Joost van Dreunen

The crazy thing here is that these shortcomings are well understood, but almost everybody benefits from the system the way it is currently set up. The press gets some attention-grabbing headlines. The industry as a whole benefits from positive buzz. People wanting to explore these new fields are given a way to justify spending money to do so. The people who control the purse strings (and often need to invest in something) get reassurance that those enthusiastic explorers are not completely off the mark. And even the investors whose money is ultimately being spent are investing in a field people are confident has loads of growth potential.

"It's sort of an ecosystem on its own," Warman said. "It maintains itself."

Van Dreunen expanded a bit more on the idea, saying, "It's too strong to call it a lie, but you join the party to some degree on your own terms. Ultimately, everybody to some degree is sipping from the Kool-Aid because you don't want to be the one that rains on the parade. You don't want to miss out, and I think that's a big driver. Ultimately, you're dealing with humans. There's a rational way of approaching markets, industries, and opportunities and a methodology behind it, but then there's the sort of sticky, weird, human factor that just doesn't want to miss out on an opportunity, or is in a hurry to leave and rush to the next thing because that's where growth is going to be, and that's what's going to solve all our problems as a company, or whatever. People have very irrational reasons to make decisions, and once that catches momentum...

"People were talking about VR, but once a company like Facebook put down $2 billion, all of a sudden everybody's an expert and everybody's totally pro-VR. I never heard about it from all these people in my life, but all of a sudden after Facebook bought in, now it's the best thing after sliced bread. So you kind of have to be skeptical about that. Are we all just fooling ourselves to some degree?"

None of this is to dismiss the value of research out of hand or suggest that they are working backward from positive conclusions first and only later figuring out how to support them. But people need to exercise media literacy about these reports the same way they would the rest of the news. Is the methodology based on straight-forward point-of-sale data, or numbers extrapolated through a complex formula that the firm doesn't want to detail? Is the research based on a survey of 10 people, or 1,000? Do the people involved really know the field they're writing about? Do they need to? Is the research citing hard data from a game's past performance, or projecting industry-wide trends on a worldwide basis five years into the future?

Because without that critical evaluation and context, we're all just huddled around a Ouija board, asking unseen forces to show us the way, each one swearing that we aren't the ones nudging the planchette to the answer we all want to hear anyway.