I agree in principle, Soren, but you’ll find no jumping for joy down in the trenches of 3rd party developers. In our muddy waters, Publishers frequently set benchmarks for quality based on Metacritic metrics, but they almost never fudge their dates to accomodate “extra development time”. No, indeed. Its more likely that they will ask the developer to dredge up more resources for a few more peanuts (if that). But “extra people” is quite often no substitute for “extra time”. I wonder if you have suffered the frustration of working on a 9 month concept-to-gold XBox 360 title with a publisher who pays around 1.5 million and expects a 75 or better. It does wonders for morale, I tell you … especially when you can see your game approaching the desired quality, just as you’ve run out of time. You KNOW you need 2 more months, but you won’t get it.

Meaning, Metacritic is a double-edged sword … it’s a great tool, but publishers need to stop poring over it while indulging fantasies of high scores, and get real about the sort of quality they can expect for the small budgets and schedules they continually offer. Ah, I could tell you some blood curdling stories, if only my NDA allowed it….

While I agree with the ideas behind metacritic I don’t think it works perfect in all instances.

The enumeration is a big issue for me, namely because I have worked on many smaller games which get no press and no major attention, but they are still approached with the intention of hitting a minimum metacritic score. But because the projects don’t garner a lot of attention many sites don’t bother to review them. So in essence you only get a fraction of the media reviewing it, and sometimes most of the reviews are by a few blogs or bottom of the barrel review sites. There isn’t a Supreme Court type of panel where 9 reviewers are always reviewing every game coming through the door. So for a small game which only gets 4 reviews, when 1 guy is a hobbyist trying to make people laugh by being as mean as he can, it can really hurt a metacritic score.

Also, the enumeration really bugs me because, despite not liking Adam Sessler on TV, I have to agree the renumeration is broken. As a designer you of all people should know that something which works in the initial design process, and works on paper, can at times not feel right in the game. But that’s just me trying to make a horrible analogy.

The real problem with the renumeration is the nature of all scoring systems in general. Speaking as an American who grew up in the American school system with the A-F grade scale, when I see a %50 in a score, I think F. Now while some review sites may intend a %50 to be “average”, it isn’t common for the general audience to think the same way. When people see a 8.1, they imagine a B- in their head. But on metacritic, a B- is a 67. There is a huge disparity between these scores and your average person will see that 67 and think it is in the “D” category, not “B”. This can become very compounded if different sites are reviewing different games.

I agree with having a site like metacritic, but I feel the only way to have a true to source renumeration would be to get a conversion from the review source. Now I can’t claim to know how review standards are kept at a game magazine or site, but I would imagine they have some kind of scale to work off of so that two reviewers at the same review source don’t see an 8/10 as two completely different things. And assuming these standards ARE in place, I feel a site like metacritic should have some kind of conversion. So when Adam Sessler gives a game a 2/4 the numbered score matches the “out of 10″ score he would want. But even then the reader may see a 50 and think it is horrible and not average (even though metacritic even says it should be considered “mixed reviews”).

But I feel the real way to resolve all this would be to create a site like a metacritic which puts everything into letter scores. Review sites submit their conversions (if their 50 is a “F” or a “C”) and a letter grade is put up. With a letter grade there is less left open to interpretation for the standard reader.

I think your enthusiasm and optimism are great, but I don’t think the standard person looking to metacritic for a quick review looks for the nuance that you do. We live in a different world than we did 20 years ago when there were less than a dozen outlets for game reviews, and those reviews came weeks before the release of a game so you and your friends could debate the scores in EGM vs Gamepro on the playground. In the age of instant digital gratification, people are quick to get their info and quick to judge, and a site like metacritic needs to bend its reviews around peoples interpretations, not try to remanipulate them, because not many people will actually make the effort to understand. I think it is better to have a strong aggregate that matches the true intention of the reviewer so that people are better informed at a glance, but they are still given the opportunity to delve deeper.

I guess for me that is the problem. Hopefully I was somewhat clear and didn’t come across as a total nut-job whacko, and welcome further back and forth on the topic.

Cheers