Over the last few years, Metacritic has become a popular whipping boy within the games industry. A recent example would be Adam Sessler’s bit at GDC’s journalist rant session. At the risk of beginning to sound like a reactionary contrarian, I feel a case needs to be made for Metacritic. Unlike my argument for used games (or, rather, for thinking critically about what we are trying to sell consumers for $60), I feel much less conflicted in this case, so let me state my thesis very clearly: Metacritic has been an incredible boon for consumers and the games industry in general. The core reason is simple – publishers need a metric for quality.
What should executives do if they want to objectively raise the quality bar at their companies? They certainly don’t have enough time to play and judge their games for themselves. Even if they did, they would invariably overvalue their own tastes and opinions. Should they instead rely on their own internal play-testers? Trust the word of the developers? Simply listen to the market? I’ve been in the industry for ten years now, and when I started, the only objective measuring stick we had for “quality” was sales. Is that really what we want to return to?
Yes, I know translating all ratings onto a 100-point scale distorts them – a C is not a 60 is not three stars – but we need to not let the perfect be the enemy of the good. What are the odds that we can get every outlet onto the same scoring scale? Not likely. Can Metacritic improve the way it converts non-numeric ratings into scores? Absolutely. However, the whole point of an aggregator is that these issues come out in the wash. When 50 opinions are being thrown into the machine, a 74 is actually different from a 73.
Critics allege that Metacritic is being unfairly used to judge games’ success, and potentially penalize developers.
I use Metacritic all the time, and I love it. It’s changed my game-buying (and movie-watching and music-listening) habits for the better, which of course funnels money into the pockets of deserving developers and encourages publishers to aim for critically-acclaimed products. Have we gotten so jaded that we have lost sight of what a wonderous thing this is? Metacritic puts an army of critics at our fingertips. Further, consumers are not morons who can’t judge a score within a larger context. We all realize that, due to the tastes of the average professional reviewer, some games are going to be overrated and some will be underrated.
Ultimately, the argument against Metacritic seems to revolve around whether publishers should take these numbers seriously. Some contracts are even beginning to include clauses tying bonuses to Metacritic scores. Others are concerned that publishers are too obsessed with raising their Metacritic averages. Actually, let’s think about that last sentence in detail. Note that when I just wrote “others,” I was referring to journalists, not to investors. As John Riccitiello famously said, “I don’t think investors give a *** about our quality.” How bizarre is it that once the game industry starts taking journalists’ work seriously, they complain about it?
I’ll give my own perspective on this issue. Over the years, I have seen many great ideas shut down becomes someone in charge thinks they won’t impact sales. However, when I am in an EA meeting in which we talk about the need to raise our Metacritic scores – and the concrete steps or extra development time thus required – I’ll tell you what I feel like doing. I feel like jumping for joy. How incredible is it to work for a publisher who cares about improving the quality of our games in the eyes of critics and uses an independent metric to prove it.
Opinions can wildly differ: Does an aggregate review score really reflect a game’s performance accurately?
As for the remuneration issue, isn’t it a good thing that there is a second avenue for rewarding developers who have made a great game? Certainly, contracts are not going to stop favoring high game sales, so – hopefully – Metacritic clauses can ensure that a few developers with overlooked but highly-rated games will still be compensated. Now, if a game doesn’t have high sales and also doesn’t get a good Metacritic score, well, there’s a name for that type of game, and these developers should not be protesting. Further, developers also need to stop complaining that a few specific reviews are dragging down their Metacritic scores. Besides the fact that both good and bad reviews are earned, in a world without Metacritic, one low score from GameSpot, GameSpy, 1Up, or IGN becomes a disaster. Score aggregation, by definition, protects developers from too much power being in the hands of one critic.
Journalists also need to have the guts to give games a score and stick by it. Putting a score on a review doesn’t take away the ability to add nuance to one’s criticism. My favorite music book is the Third Edition of the Rolling Stone Album Guide. As the reviews were written by just four critics, I have learned to understand the exact difference between five and four-and-a-half stars (or, for that matter, between two-and-a-half and three stars). If you are a great reviewer, the score you give a game helps me place it in context with everything else you have rated. Moreover, your score lets you contribute, via Metacritic and all the other aggregators, to the meta-critique of games on the Net. What exactly is the problem here?
* Editor’s Note: For more articles by Soren Johnson, be sure to read his Designer Notes blog.
Loading ...
Scott Steinberg is the CEO of video game consulting firm TechSavvy Global, and founder of GameExec magazine and Game Industry TV. Hailed as a top technology and video game expert by dozens of publications from USA Today to Forbes and NPR, he’s covered the field for 400+ outlets from Playboy to Rolling Stone. A frequent on-air analyst for networks like ABC, CBS, NBC and CNN, he’s also the author of Video Game Marketing and PR.
Hi Soren,
While I agree with the ideas behind metacritic I don’t think it works perfect in all instances.
The enumeration is a big issue for me, namely because I have worked on many smaller games which get no press and no major attention, but they are still approached with the intention of hitting a minimum metacritic score. But because the projects don’t garner a lot of attention many sites don’t bother to review them. So in essence you only get a fraction of the media reviewing it, and sometimes most of the reviews are by a few blogs or bottom of the barrel review sites. There isn’t a Supreme Court type of panel where 9 reviewers are always reviewing every game coming through the door. So for a small game which only gets 4 reviews, when 1 guy is a hobbyist trying to make people laugh by being as mean as he can, it can really hurt a metacritic score.
Also, the enumeration really bugs me because, despite not liking Adam Sessler on TV, I have to agree the renumeration is broken. As a designer you of all people should know that something which works in the initial design process, and works on paper, can at times not feel right in the game. But that’s just me trying to make a horrible analogy.
The real problem with the renumeration is the nature of all scoring systems in general. Speaking as an American who grew up in the American school system with the A-F grade scale, when I see a %50 in a score, I think F. Now while some review sites may intend a %50 to be “average”, it isn’t common for the general audience to think the same way. When people see a 8.1, they imagine a B- in their head. But on metacritic, a B- is a 67. There is a huge disparity between these scores and your average person will see that 67 and think it is in the “D” category, not “B”. This can become very compounded if different sites are reviewing different games.
I agree with having a site like metacritic, but I feel the only way to have a true to source renumeration would be to get a conversion from the review source. Now I can’t claim to know how review standards are kept at a game magazine or site, but I would imagine they have some kind of scale to work off of so that two reviewers at the same review source don’t see an 8/10 as two completely different things. And assuming these standards ARE in place, I feel a site like metacritic should have some kind of conversion. So when Adam Sessler gives a game a 2/4 the numbered score matches the “out of 10″ score he would want. But even then the reader may see a 50 and think it is horrible and not average (even though metacritic even says it should be considered “mixed reviews”).
But I feel the real way to resolve all this would be to create a site like a metacritic which puts everything into letter scores. Review sites submit their conversions (if their 50 is a “F” or a “C”) and a letter grade is put up. With a letter grade there is less left open to interpretation for the standard reader.
I think your enthusiasm and optimism are great, but I don’t think the standard person looking to metacritic for a quick review looks for the nuance that you do. We live in a different world than we did 20 years ago when there were less than a dozen outlets for game reviews, and those reviews came weeks before the release of a game so you and your friends could debate the scores in EGM vs Gamepro on the playground. In the age of instant digital gratification, people are quick to get their info and quick to judge, and a site like metacritic needs to bend its reviews around peoples interpretations, not try to remanipulate them, because not many people will actually make the effort to understand. I think it is better to have a strong aggregate that matches the true intention of the reviewer so that people are better informed at a glance, but they are still given the opportunity to delve deeper.
I guess for me that is the problem. Hopefully I was somewhat clear and didn’t come across as a total nut-job whacko, and welcome further back and forth on the topic.
Cheers
“However, when I am in an EA meeting in which we talk about the need to raise our Metacritic scores – and the concrete steps or extra development time thus required – I’ll tell you what I feel like doing. I feel like jumping for joy.”
I agree in principle, Soren, but you’ll find no jumping for joy down in the trenches of 3rd party developers. In our muddy waters, Publishers frequently set benchmarks for quality based on Metacritic metrics, but they almost never fudge their dates to accomodate “extra development time”. No, indeed. Its more likely that they will ask the developer to dredge up more resources for a few more peanuts (if that). But “extra people” is quite often no substitute for “extra time”. I wonder if you have suffered the frustration of working on a 9 month concept-to-gold XBox 360 title with a publisher who pays around 1.5 million and expects a 75 or better. It does wonders for morale, I tell you … especially when you can see your game approaching the desired quality, just as you’ve run out of time. You KNOW you need 2 more months, but you won’t get it.
Meaning, Metacritic is a double-edged sword … it’s a great tool, but publishers need to stop poring over it while indulging fantasies of high scores, and get real about the sort of quality they can expect for the small budgets and schedules they continually offer. Ah, I could tell you some blood curdling stories, if only my NDA allowed it….