"Analyzing the Chapel" or Lies, Damned Lies and Statistics

This article explores ways to gather insight from strategies. Why does a particular strategy do well? I’m developing statistics analyzers for the Dominator, and want to put them in action in an actual case. To make it interesting, I examine a Chapel strategy competing with Big Money. To be precise: I run Single Chapel canonical 1 against Big Money canonical. Now I know both are not competitive strategies, but they are equal in the sense that they both don’t buy Duchies and Estates. Including these would complicate things too much for this early analysis. I will investigate real Chapel strategies later.

But first: pop quiz

If you play “Single Chapel canonical 1″ against “Big Money canonical”, what are your win chances?

A) Playing Chapel gives you roughly an 80% win chance.
B) Playing Chapel gives you roughly a 65% win chance.
C) Playing Chapel gives you roughly a 50% win chance.

The Chapel has been hyped a lot, even having been called the most powerful card in the game on more than one occasion. For instance on dominionstrategy.wordpress.com. My guess was that it is one of those cards that is very strong on its own, not needing any other kingdom cards to make it work.

So I ran the simulation and the results were not what I expected at all. Garion even didn’t believe them, so I wanted to fully understand them before writing about them. And in order to do this I added some victory points statistics logging. I’m recording the amount of victory points each player has in each turn, to get an average number of victory points for each strategy per turn. This resulted in the following graph:

What does this graph tell us? There is the expected drop in VP of the Chapel strategy in the early turns as the Estates get trashed. Then at turn 12 the Chapel overtakes the big money in average VP. Since only 1 game in my 10,000 runs ended in turn 12, this graph tells you that the Chapel strategy must win most of the games.

What about the actual results?

Chapel wins: 4886 (49%)
BigMoneyCanonical wins: 5103 (51%)
ties: 11

First of all I’ll bet good money that very few people actually choose answer C off the bat. Playing a single Chapel does not improve your odds?

And if the simulation is right -which I believe it is, because I checked against using another simulator- then my initial average VP graph must be flawed. Well indeed I did not tell the whole story about the way the average is calculated. Calculating that average the big question is: how do you combine the data from games that do not last the same number of turns. Say that you have a normal game that lasts 16 turns and one that lasts 17 turns. Everything is easy for calculating the average number of points for turn 16, but what to do with the data for turn 17. Consider two games where game 1 ends in turn 16, and game 2 last until turn 17:

turn	chapel strategy score game 1	chapel strategy score game 2
16	30	26
17	n/a	26

I thought it would be acceptable to extrapolate that the first game has 30 points in turn 17 as well.

Average points in turn 16: 30 + 26 / 2 = 28 points
Average points in turn 17: 30 + 26 / 2 = 28 points.

But as we have seen the resulting averages are misleading and of no use. So I tried again, this time not extrapolating scores from games. I needed to keep track of how many games ended in each turn, and this allowed me to calculate the average scores using only the games that provided data for it.

Average points in turn 16: 30 + 26 / 2 = 28 points
Average points in turn 17: 0 + 26 / 1 = 26 points.

You immediately see that this can have strange consequences: the average number of points drops. To me this was counter intuitive because (apart from trashing Estates) the actual number of points only rises during games. Still, this method of averaging gives a much less misleading graph:

Here we see that the Chapel strategy is only superior in games that end between turn 12 and 16. If the Chapel strategy take longer than that, then -on average- the big money strategy is superior. It may be that in those cases the Chapel strategy is hurt by the extra Provinces.

Anyway if we look at the histogram of the number of turns played we see that the whole story starts to look plausible:

These totals also add up nicely to the win percentages of the strategies:

The total numbers for games ending up to turn 16 is: 4943
The total numbers for games ending after to turn 16 is: 5057

OK, some more graphs to see if we really understand why the Chapel does so poorly compared to expectations. Let’s look at the average value of each card. The basic idea of this version of the Chapel strategy is to raise the average value by trashing low value cards.

It is nice to see that the graph for Chapel drops flat after turn 10. From that point onwards the influx of Provinces balances the influx of money. I mentioned previously that I expected that once the Chapel deck start buying Provinces it would starts degrading. This does not seem to happen, so there must be another reason that the Chapel strategy is better in shorter games.

Let’s take a quick side step. One theory suggests that you need to have at least one Gold before buying your first Province. This should be very visible in the average card value graph, and so it is:

Now we do see a drop in average value after turn 9. Unfortunately the extra Gold does not actually improve the win-rate of this strategy, which stays at roughly 50%. So trading a Province for a Gold does as much damage as it does good.

But I’m still a bit at a loss. We saw the average VP of the Chapel strategy decline for longer games, but we also see the buying power steadily increase, especially for longer games. Maybe that average buying power does not translate into turn where you can actually buy a Province. One turn with 12 money plus one with 6 money only gives you one Province, even though the average is a whopping 9 money. The graph below shows the percentage of the time that a player had 8 or more money to spend:

This graph is final proof that the Chapel strategy is capable of outbuying Big Money when it comes to buying Provinces. So how on earth can it be that the Big Money strategy still is just as good?

My last stab at solving the mystery is by thinking about the victory conditions. A Chapel deck will always have less Estates than Big Money. So once Big Money buys its 4th Province, the Chapel deck can no longer win. Nonetheless my Chapel strategy implementation will never buy a card that will cost it the game. And it is easy to count how many times that happens for the 10k games simulated. The numbers:

Games where the Chapel deck did not buy a Province: 1801
Total number of Provinces not bought: 3498

Since there is no other way to gain victory points, in those games the Chapel deck will still lose, just a couple of turns later, judging by the numbers by about 2 turns. Now back to the win chance. Given that the Chapel deck will need 5 Provinces to win, and the big money only needs 4, we can relate the actual number of Provinces bought by these targets:

Chapel buys 42973 Provinces for an adjusted number of 42973 / 5 = 8594
Big Money buys 37027 Provinces for an adjusted number of 37027 / 4 = 9256
(Sanity check: 42973 + 37027 = 80,000 Provinces sold in total.)

In this respect Big Money is actually doing better than the Chapel strategy. So although the Chapel is generating more buying power than Big Money, it just is not enough…

Conclusion
The Chapel will increase the buying power, but in this simple form it also increases the number Provinces it needs to win. These effects compensate for each other, making this simple single Chapel strategy no better than canonical Big Money.

Next up is adding Duchies and Estates into the mix. And to give you a sneak preview: my best Chapel strategy so far (that does not use any other kingdom cards) scores 54% against Bmu. More on the esteemed Chapel later…

[Edit: I decided that card names should be capitalized. And thx Willie for pointing out a very basic calculation error.]

About these ads