Analysing the Analysts

Feature: Just how often do they get it right?

We all make predictions - but some people get paid for it. Just because of that we put the microscope on the world of videogame market research and analysis to see if their bar charts are worth the graph paper they're written on.

In many ways an analyst is in an unenviable position: it's a damned if they do, damned if they don't situation. While we might baulk at the Gordon Gecko salaries they get paid (though by all accounts these are exaggerated) everyone always remembers the things they get wrong, not the things they get right.

Let's take the industry's most outspoken crystal-ball gazer, Michael Pachter (analyst at Wedbush Morgan) as an example. Back in 2005 in the pages of New York Times he made the extraordinary claim that World of Warcraft was going through a bit of a fad. Over the previous ten months Blizzard had notched up over 3 million subscribers for its game but this didn't impress Pachter.

"I don't think there are four million people in the world who really want to play online games every month," he said while grinding some salt and pepper over his hat. "World of Warcraft is such an exception. I frankly think it's the buzz factor, and eventually it will come back to the mean, maybe a million subscribers." World of Warcraft now commands over nine million subscribers worldwide.

While it's true to say that WoW caught most industry pundits by surprise few would have predicted a reverse in subscriber numbers given its ability to woo and absorb. But we'll be coming back to Pachter a little later and revealing some of his other whoppers.

At this point we should note an important distinction between analysts and market research groups. While both attempt to predict future events analysts usually report directly to one company and generally use their knowledge of a key area to inform financial institutions. Market Research groups systematically gather, record and analyse info in an attempt to report on, and predict future trends.

Market research groups sell their information, graphs and predictions for a premium, but are they any more accurate? Well... in 1995 Forrester Research projected that the Internet would have a worldwide user base of 34.9 million people by 1998. Notice the precision. Not 35 million, but 34.9 million.

The real number ended up being in excess of 100 million users. Forrester also grossly underestimated the size of the electronic commerce marketplace. In 1995 Forrester predicted $2.3 billion. The real number turned out to be more than $13 billion.

Yes, it's easy to rake over old ground and nit-pick but what's startling is how much companies pay for this kind of information. We contacted one independent market research group in the US for a quote.

They said they'd be able to process a 15 question survey to their database of 500 clients (videogame enthusiasts in this case) 'tabulate' the information and get it back to us for a measly $15,000. Makes you want to start up your own company doesn't it?

One worrying feature of videogame analysis is how out of touch some of the commentators seem. Recently Richard Newboult, analyst at Panmure Gordon, made this extraordinary statement: "Last year 70-75 percent of all SCi's profits were generated by two hit games - Lara Croft's Tomb Raider Anniversary and Hitmen Bloody Buddy." Whoops!

And if that wasn't bad enough listen to this brilliant piece of advice from Nick Williams of GamerMetrics: "The Wii Balance Board, for example, could provide a completely fresh experience for all kinds of gamers. Could you imagine a Grand Theft Auto game where you actually run out of breath while trying to outrun the cops?" Probably not the kind of GTA game most fans would want to imagine.

  1 2