Updated January 4, 2008
Thanks to MCHB, I have now found and added the UK GT Heat results. Along with some other late year events, I offer a rankings update.

No real shifts in rankings. Daemonic Legion, Bretonnians and Wood Elves still hold a solid lead. Dark Elves, Orcs & Goblins and Dwarfs are stuck at the bottom.

Anything that fell under 1% of the entries I didn't list here, although they are in the spreadsheet calculations.

Totals:



I'll try breaking out the numbers slightly differently now. GW GTs and non-GW GTs. All the GW events ignore any form of composition scoring. I think all the non-GW events here have some enforcement of 'friendly' lists, even if not via explicit comp scoring.

Now-GW events (871 total entries)



GW events 1257 entries
(mostly the 2006 and 2007 UK GT)


* The Daemonic Legion was only allowed in the US GW GTs, so has an absurdly low frequency.

The top three armies hold position either way, but some armies do shift. Tomb Kings go from slightly below average to fighting Wood Elves for 3rd at GW events. Vampire Counts do noticably worse at GW events. Dwarfs go from the bottom dregs to merely 'poor'. Hordes of Chaos and Ogre Kingdoms drop. The Orc & Goblin rating score makes them show up higher in this listing, but their ABM score remains the worst.


Conclusions
I remain comfortable saying that Daemonic Legion, Bretonnians, and Wood Elves are top tier power level army books.
I remain comfortable saying that Dwarfs and Orcs & Goblins are low on the power scale. And the poor Dark Elves desperately need a new army book.


Methodology
Looking at just winners does not produce enough numbers for any useful statistics. It also ignores number of entries and tournament size. To increase the statistics and account for size, instead I look at broader measures of placement.

As this was initially started by a post by Jervis (GW guy) on their Games Development forum in 2005, I started out following his "ABM" (Army Brokeness Modifier) method. The concept is to look at the number of entries for each army and the number that finish in the top 1/3 (or another fraction, 1/4 changes little). That is then rated as a percentage of expectation. An army with 12 entries would, on average, expect 4 in the top 1/3. Taking the expected result as 100%, this creates an "army brokenness modifier" based on percentage. For example, if 6 of those 12 finished in the top 1/3, that would be an ABM of 150%. I tried some slight variants on that methodolgy (see other posts in this thread) without seeing much of interest.

The North American Warhammer Rankings (of players) give another method of calculating a rating. Probably a superior rating, as it doesn't have a 'critical point' like my 1/3. At first I couldn't get Excel to calculate that easily, but now that I solved that, I think it is a better indicator. This involves giving each tournament finishers a rating of what percentage of that tournament players they beat. The person that finishes last gets a 0 and the winner get almost 100 (you never beat yourself, so 59/60). Average that score over everything. Here a nice 50% would be perfectly average.

Data
Data sources are limited to 5 game GT style tournaments where I could find results on the web including battle points and army type. Many tournaments do not include this information in posted results and some have taken it down since originally posting it. I focussed on US Indy tournaments, but I also present totals including US GW GTs and UK GTs.
There isn't enough data. There is never enough data. I don't feel even all the US Indys I can find (631 currently) provides great stats because removing one tournament can shift one army noticably. Adding in the US and UK GW GTs helps with that, but I'd be happier with another couple thousand entries. That said, I do think these rankings have some legit meaning. Certainly a ton more than any one tournament or the bursts of anecdotal evidence you can find on the internet.

Error Factors
The biggest absolute error in this data is matching what a tournament posts as the "army" with an actual army on my list. That alone subsumes some SoC lists into their parent army book. The Chaos numbers are by far the least accurate, because of splitting into three armies (mortals, beasts, daemonic legion). Basically, I put any reference to demons as "daemonic legion", any reference to beasts as "beastmen" and everything else as "hordes of chaos".

The only 'player skill bias' I can come up with would be with Daemonic Legions. Because of the conversion effort, the most experienced (and thus typically most skilled) players likely make a larger percentage of Legion players. You can speculate on others, but that is the only one I personally think is significant enough to maybe matter.

Every tournament uses its own scoring system and scenarios. This data is based purely off 'battle' points, or whatever I could uncover that seemed closest to that. Conceivably some scoring systems and scenarios favor one army over another.

Every tournament exists in its own composition environment. As I understand things, the UK is basically no comp scoring and "bring the pain" while the central US tends to be the "friendliest". Additionally, most (all?) US Indys in this data use some form of composition scoring and typically ban or restrict special characters. To what extent that has an impact, I have no idea.


See also the thread on 6th edition rankings.

Very, very interesting...


As you've mentioned though, the inabilityt of most graphs and the such in showing an accurate depiction of an Army without measuring the player's skills the play them is hard.


I love my Daemonic Legions, and think they're tough Army, but not nearly as annoying as Bretonnians or Wood Elves...but that's just my opinion.


Thanks much for sharing this info!

Interesting. Bretonnians have done a lot better than last time, which I find strange - you'd think that people would be more used to them by now.

I might have lost the tables, but if I find them and the time, I might do the analysis again. It would be interesting to compare them.

In order to get a fairer idea, I'd try discounting the top AND bottom 25% of the results, thus taking out the best and worst gamers. This should show how different armies handle in the hands of gamers who have similar performances.

Other than Daemonic Legion (perhaps), no army has an obvious bias in player skill one way or the other. In theory, player skill really should average out, if you get enough data. I do not claim to really have enough data, just more data than anyone else has systematically examined.

I'll try Aardvark's idea later this week, even though I doubt the validity of the concept.

First things first – props to dabber for doing all the number-crunching.

That being said, don’t suppose you could think of a way to include two factors?

1) Scenarios – yes or no
2) Special characters – yes or no

As for the Brets, my theory is that the increased number of Wood Elf lists actually helps the Brets out since, generally speaking, Wood Elf lists have a harder time against Bret lists. More Wood Elves = better odds for a Bret player to draw Wood Elves.

The Green Knight probably helps a fair bit as well.


On the subject of the Daemon Legions, well, player skill does play a part, though it would depend a great deal on which Legion – imho, a Khorne/Slaanesh Legion would take considerably more skill compared to a Nurgle/Tzeentch Legion.

I finally tried Aardvark's idea. It didn't change anything much (all under 20 shift), except for totally nuking the rating of Wood Elves, who went down more than 80.

Chuckles, special characters are difficult to measure. For example, some of the Indys technically allow them (or some of them, "no Morghur!" has been heard), but they rarely show up. Presumably people fear for their composition score. So should that count as a yes or a no?

If it is the case, in any situation, that players are selecting their armies based on something other than their performance on the tabletop, then the whole analysis becomes invalid. You'd be measuring the abilities of armies to perform with restrictions placed on them, which would not reflect the natural ability of each army.

very interesting dabber, thanks for this

i was surprised to see hordes of chaos so low though

That is interesting- does this mean WE are exceptional in the hands of a good general and decided average to poor in the hands of those with less experience?

That is indeed a difficult call.

Though, the more I think about it, the more I suspect that special characters are a bit of a wash, since they tend to cancel each other out.

(Though I must admit I've got a bit of a bias against special characters, and I generally tend to assume that most of the people who actually field them in tournaments tend to be mediocre players who rely too much on the special characters).

Is this strictly W-L results? That answer will definitely alter the results.

The Demonic Legions could benefit from scoring in other departments. An amazing, well converted/painted army may score more composition points, even though they don't deserve it in that particular area (I've seen it a lot....). On the other hand, I know many veterans, who are exceptionally skilled in converting/painting, but are not truly skilled players.

What is a 'balanced' WE army? Maybe they are scoring high in comp, when it really is the same list as everyone else, and it's somewhat abusive.

I'm not picking on anyone, I'm just setting these as example for potential deviations in scoring. Things to consider when looking at the raw data.

No, its pure battle points. Or as close to that as I could get, sometimes they only posted 'best general' rankings, which might have included composition or some other factor. It is NOT overall rankings. Follow the link to the 6th edition rankings for many paragraphs on methodology.

Of course, this method has an internal weakness, due to only being measured in battle points. If one army has a slight advantage on the battle field, then a larger amount of competitive players will select it, thus increasing the seeming effectiveness of it.

Rankings updated. Previous first thread (now destroyed by editting) placed here for reference.



This examines the percentage of entries that finish in the top 1/3. If an army has half the top finishers, but also had half the entries, it didn't actually do spectacularly well. An ABM of 100 means they did exactly as expected - 1/3 of the entries finished in the top 1/3. For my full explanation of methodology, see the thread on 6th edition rankings.

Anything that fell under 1% of the entries I didn't list here, although they are in the spreadsheet calculations.


US Indy results
445 total entries



Everything I can find
(Includes US, UK, and a couple Canada)
1281 total entries



Comments:
Daemonic Legion, Bretonnians and Wood Elves are the top three overall, as expected. Clear gap in the ratings after them. Bretonnians didn't dominate as much in the US Indys, but still were above average.
Orcs & Goblins, Dwarfs, and Dark Elves hold the bottom, which cannot be considered a surprise. Some of us would like to think the Dwarfs new book helped them out significantly, but apparently move 3 still kills them.



Alternative Method
One potential weakness with this method of calculating is it cannot remotely account for player skill. With just GT results, it is impossible to actually account for player skill, but I thought it was worth trying something. My guess was two armies would be particularly subject to that. One is Daemonic Legion, which is almost entirely played by experienced veterans because they take the time to do the conversions. Another might be Orcs & Goblins, which often attract younger and newer players.
So I decided to try simply dropping the bottom half of the results, and then looking at the top half of the rest, or the top 1/4 overall. (This proved really easy to do in Excel, which is part of why I tried it.) The results didn't change all that much.

Ignoring the Bottom Half
658 total entries


Comments:
The biggest changes are Daemonic Legion (yeah) and Bretonnians, bothing looking a lot worse this way. The biggest positive change is Dark Elves, and Empire goes up noticably also. The others don't change much.
However, my take on conventional wisdom would justify these shifts entirely differently. Daemonic Legion looks worse this way because the veterans that build that army don't finish in the bottom half whatever they play. Bretonnians look worse this way because the army is so easy to play it doesn't finish in the bottom half much. This causes me to conclude the change in method doesn't accomplish much.

I've only quickly read through the methodology but on that basis it makes TK the most normal army out there (closest ABM & full ABM to 100%). Though not being a tournament player I have no base on the "average skill level" of people playing the army.

Great work on the stats, liking what it shows/confirms.

WOOT! WE'RE STILL NUMBER ONE!!! GO LEGION! GO LEGION!

Of course, Beasts are still in the middle and Hordes are languishing at the bottom, but oh well, can't have it all.

Thanks for all the hard work and number munching, Dabber.

Hmmm. And people have told me that I was just a whiner for saying that the new O&G book took a non-competative army and made it worse. Numbers would tend to bear out my gut feelings.

Also, this sort of blows the "Dwarfs are too good" calls you hear all over the net.

But in th true spirit of Chaos you are everywhere.

As it would seem.

Oddly enough I expected the HE to do a little bit worse that they did. Makes me wonder what 7th ed will be like.

End of year update now in the first post!


Archiving previous first post here for later reference:

October 10, 2007
With UK heat 1 happening last weekend, it is now one complete year of my tracking 7th edition tournament results by army. That year saw two army books released (Orcs & Goblins and Empire), although the new O&G book was present in almost all of the tournaments.


Anything that fell under 1% of the entries I didn't list here, although they are in the spreadsheet calculations.

Overall rankings (1512 entries)


US Indy rankings (631 rankings)

Comments:
Daemonic Legion, Bretonnians and Wood Elves are the top three overall, as expected. Clear gap in the ratings after them.
Orcs & Goblins, Dwarfs, and Dark Elves hold the bottom, which cannot be considered a surprise. Some of us would like to think the Dwarfs new book helped them out significantly, but apparently move 3 still kills them.

After the GT heats this year, I am very surprised that the Empire list hasn't moved up further. They really were the army to beat in all 3 heats and I really don't see that changing for the final.

Well done, dabber.

The % entry numbers seem reasonably stactic in some cases, so am i right in thinking it's the same players who are winning it for DL (and losing it for Dwarfs)?

Statistically, I didn't keep player information. Not entirely sure how I would get Excel to accumulate it or adjust for it if I did have it. If you can offer detailed suggestions that way, I may consider it for future data.

Anecdotally, a number of the best players bring a variety of armies to GTs. The ultimate example is the GW US Circuit Champion - in the 9 circuit events, Jordan Braun took 8 different army books.

No idea. I was just making an observation.

I don't think the statistics prove anything about Dwarfs being balanced. They have the highest percentage of entries but a low rating because they are attractive to less tactical players (by GT standards).