So, this is what you get when you have a heavy metal obsessive and a data compulsive rolled into one person.
Put simply, I take the scores given to metal albums by various publications, average them out, and put together year lists of the top rated overall.
This whole thing is a pet project I've been working on for a while now, stemming from curiousity about which albums by certain bands were the best received. Over time it grew and grew (there's that obsessive thing I mentioned), and it currently has 63,841 review scores for a total of 7406 albums, covering 1610 different artists.
I'm continually updating it, with the scores coming from over two dozen different sites and magazines (though some are currently defunct, but I will still access them where I can), and covering everything from the genre's beginnings to the present day.
A few things to clarify about my process:
1.) The sites I used are simply those that I felt had some reasonable understanding of heavy metal music. Yes, some are less professional than others, but personally I value someone's opinion based on their experience of the topic rather than what publication they write for.
2.) My standard for inclusion here is 4 reviews or more from the sites below. I feel fewer than 4 isn't a proper aggregate score. Same sort of thing for genres: there are some albums in here which (in my opinion) aren't metal, but this isn't about my opinion, it's about the opinion of these sites. If 4 or more metal-based sites consider a release significant enough to review it, then it goes in. Oh, and this is full-length albums only, no EPs, singles, compilations, DVDs, live albums, etc.
3.) Current sites/zines used: About.com, Blabbermouth, Brave Words, Chronicles of Chaos, Independent Voice, Live-metal.net, Lords of Metal, Metal.de, Metal Asylum, Metal CD Ratings, Metal Crypt, Metal District, Metal Exposure, Metal Hammer, Metal Ireland, Metal Injection, Metal Judgement, Metal Mayhem, Metal Reviews, Metal Sound, Metal Storm, MetalSucks, Metal Temple, Metal Underground, Metal Archives, Metal Observer, Metal Rules, RevelationZ, Rock Hard, Sound of Metal, Sputnik Music, Terrorizer, The Gauntlet and Thrash Hits.
4.) Most reviews give a score. On occasions when one isn't given, I will usually read the review and try to interpret fairly. Also, reissue reviews are only used if the text is reviewing the album itself, not just the quality of the reissue.
5.) This is about review opinion, not mine, so I will of course include scores even if I strongly disagree with them. However, that doesn't mean there's no standard of quality at all. I don't have time to read all the reviews, but if I do see one that's just absurd, I will sometimes leave it out. For clarification, I don't just mean I think it's wrong, I mean something which utterly misses the point or puts zero effort into actually reviewing it.
6.) Rankings are based of course primarily on overall score. Where the scores are equal, I just do it based on number of reviews/my opinion/randomness. Doesn't mean anything at that point.
7.) The point of this is to help myself and others who might be interested see what the metal community thought of albums. Already it's helped me find a number of albums I'd previously over-looked. I hope it might for you too.