Home > Simulation > “Analyzing the Chapel” or Lies, Damned Lies and Statistics

## “Analyzing the Chapel” or Lies, Damned Lies and Statistics

This article explores ways to gather insight from strategies. Why does a particular strategy do well? I’m developing statistics analyzers for the Dominator, and want to put them in action in an actual case. To make it interesting, I examine a Chapel strategy competing with Big Money. To be precise: I run Single Chapel canonical 1 against Big Money canonical. Now I know both are not competitive strategies, but they are equal in the sense that they both don’t buy Duchies and Estates. Including these would complicate things too much for this early analysis. I will investigate real Chapel strategies later.

But first: pop quiz

If you play “Single Chapel canonical 1″ against “Big Money canonical”, what are your win chances?

A) Playing Chapel gives you roughly an 80% win chance.
B) Playing Chapel gives you roughly a 65% win chance.
C) Playing Chapel gives you roughly a 50% win chance.

The Chapel has been hyped a lot, even having been called the most powerful card in the game on more than one occasion. For instance on dominionstrategy.wordpress.com. My guess was that it is one of those cards that is very strong on its own, not needing any other kingdom cards to make it work.

So I ran the simulation and the results were not what I expected at all. Garion even didn’t believe them, so I wanted to fully understand them before writing about them. And in order to do this I added some victory points statistics logging. I’m recording the amount of victory points each player has in each turn, to get an average number of victory points for each strategy per turn. This resulted in the following graph:

What does this graph tell us? There is the expected drop in VP of the Chapel strategy in the early turns as the Estates get trashed. Then at turn 12 the Chapel overtakes the big money in average VP. Since only 1 game in my 10,000 runs ended in turn 12, this graph tells you that the Chapel strategy must win most of the games.

What about the actual results?

Chapel wins: 4886 (49%)
BigMoneyCanonical wins: 5103 (51%)
ties: 11

First of all I’ll bet good money that very few people actually choose answer C off the bat. Playing a single Chapel does not improve your odds?

And if the simulation is right -which I believe it is, because I checked against using another simulator- then my initial average VP graph must be flawed. Well indeed I did not tell the whole story about the way the average is calculated. Calculating that average the big question is: how do you combine the data from games that do not last the same number of turns. Say that you have a normal game that lasts 16 turns and one that lasts 17 turns. Everything is easy for calculating the average number of points for turn 16, but what to do with the data for turn 17. Consider two games where game 1 ends in turn 16, and game 2 last until turn 17:

 turn chapel strategy score game 1 chapel strategy score game 2 16 30 26 17 n/a 26

I thought it would be acceptable to extrapolate that the first game has 30 points in turn 17 as well.

Average points in turn 16: 30 + 26 / 2 = 28 points
Average points in turn 17: 30 + 26 / 2 = 28 points.

But as we have seen the resulting averages are misleading and of no use. So I tried again, this time not extrapolating scores from games. I needed to keep track of how many games ended in each turn, and this allowed me to calculate the average scores using only the games that provided data for it.

Average points in turn 16: 30 + 26 / 2 = 28 points
Average points in turn 17: 0 + 26 / 1 = 26 points.

You immediately see that this can have strange consequences: the average number of points drops. To me this was counter intuitive because (apart from trashing Estates) the actual number of points only rises during games. Still, this method of averaging gives a much less misleading graph:

Here we see that the Chapel strategy is only superior in games that end between turn 12 and 16. If the Chapel strategy take longer than that, then -on average- the big money strategy is superior. It may be that in those cases the Chapel strategy is hurt by the extra Provinces.

Anyway if we look at the histogram of the number of turns played we see that the whole story starts to look plausible:

These totals also add up nicely to the win percentages of the strategies:

The total numbers for games ending up to turn 16 is: 4943
The total numbers for games ending after to turn 16 is: 5057

OK, some more graphs to see if we really understand why the Chapel does so poorly compared to expectations. Let’s look at the average value of each card. The basic idea of this version of the Chapel strategy is to raise the average value by trashing low value cards.

It is nice to see that the graph for Chapel drops flat after turn 10. From that point onwards the influx of Provinces balances the influx of money. I mentioned previously that I expected that once the Chapel deck start buying Provinces it would starts degrading. This does not seem to happen, so there must be another reason that the Chapel strategy is better in shorter games.

Let’s take a quick side step. One theory suggests that you need to have at least one Gold before buying your first Province. This should be very visible in the average card value graph, and so it is:

Now we do see a drop in average value after turn 9. Unfortunately the extra Gold does not actually improve the win-rate of this strategy, which stays at roughly 50%. So trading a Province for a Gold does as much damage as it does good.

But I’m still a bit at a loss. We saw the average VP of the Chapel strategy decline for longer games, but we also see the buying power steadily increase, especially for longer games. Maybe that average buying power does not translate into turn where you can actually buy a Province. One turn with 12 money plus one with 6 money only gives you one Province, even though the average is a whopping 9 money. The graph below shows the percentage of the time that a player had 8 or more money to spend:

This graph is final proof that the Chapel strategy is capable of outbuying Big Money when it comes to buying Provinces. So how on earth can it be that the Big Money strategy still is just as good?

My last stab at solving the mystery is by thinking about the victory conditions. A Chapel deck will always have less Estates than Big Money. So once Big Money buys its 4th Province, the Chapel deck can no longer win. Nonetheless my Chapel strategy implementation will never buy a card that will cost it the game. And it is easy to count how many times that happens for the 10k games simulated. The numbers:

Games where the Chapel deck did not buy a Province: 1801
Total number of Provinces not bought: 3498

Since there is no other way to gain victory points, in those games the Chapel deck will still lose, just a couple of turns later, judging by the numbers by about 2 turns. Now back to the win chance. Given that the Chapel deck will need 5 Provinces to win, and the big money only needs 4, we can relate the actual number of Provinces bought by these targets:

Chapel buys 42973 Provinces for an adjusted number of 42973 / 5 = 8594
Big Money buys 37027 Provinces for an adjusted number of 37027 / 4 = 9256
(Sanity check: 42973 + 37027 = 80,000 Provinces sold in total.)

In this respect Big Money is actually doing better than the Chapel strategy. So although the Chapel is generating more buying power than Big Money, it just is not enough…

Conclusion
The Chapel will increase the buying power, but in this simple form it also increases the number Provinces it needs to win. These effects compensate for each other, making this simple single Chapel strategy no better than canonical Big Money.

Next up is adding Duchies and Estates into the mix. And to give you a sneak preview: my best Chapel strategy so far (that does not use any other kingdom cards) scores 54% against Bmu. More on the esteemed Chapel later…

[Edit: I decided that card names should be capitalized. And thx Willie for pointing out a very basic calculation error.]

1. January 11, 2011 at 01:20 | #1

You misinterpreted the dominion strategy post. Chapel does not require any specific card, it doesn’t have a strong bias toward any one action in particular. It helps very many them all a lot. If your simulation isn’t showing that a chapel based strategy is significantly outperforming a straight big money strategy, you probably need to do a lot of work on making your simulations more realistic. Otherwise, your experiments do not match the view of many of the players who do very well on the isotropic dominion server.

Take a look here, for example. http://boardgamegeek.com/thread/586435/i-still-love-dominion-but-im-getting-annoyed-by-th/page/4

Wodan held your thesis, that chapel is not much better than big money because of the results of simulations. theory, the author of the claim that chapel is best card in the game, played 3 games against wodan. Wodan bought never bought a chapel and lost all 3 games.

2. January 11, 2011 at 23:10 | #2

I was quoting the nice “This is the best card in the game.” phrase from dominionstrategy, so I cannot see how I am misinterpreting it. I do not entirely agree with it, but that is another matter.

The BGG topic is great by the way, thx for that. Shame I did not read it earlier.

But I stand by my simulations. Especially in this simple case where no other cards, no duchies or estates are bought. Allow me to recap the essence of this article in fewer words.

Since Chapel trashes estates, it needs 5 provinces to win. Big Money only needs 4 province.

Big Money (no duchies/estates) needs 16.8 turns to get 4 provinces.

Chapel (no other cards, no duchies/estates) needs 16.93 turns to get 5 provinces.

Therefore, in this simple case: big money is on par with chapel. Other simulations agree with these numbers. Try: http://www.westpark-gamers.de/index.html?/Reviews/bericht291.php&wpgref=none

3. January 12, 2011 at 06:59 | #3

The problem is that chapel + straight big money is not why chapel is so great.

Chapel is so great because it lets you consistently hit card combos.

I am not saying your numbers are wrong, just that your are spending way too much effort analyzing an overly simple model. I think that less analysis on a more realistic strategy is a lot more fruitful in terms of relevance to actual reasonable high level Dominion play.

• January 12, 2011 at 12:53 | #4

Ghe ghe, as a matter of fact you did. But indeed this post is just about base stuff. I needed to do this to develop analyzers that allow me to understand more complex situations. I am still very curious how the numbers will stack up for realistic chapel combo’s.

But let’s turn things around: give me a combo (preferably one that requires few decisions to play) and I will see if can give you some numbers on a realistic chapel case. To allow comparison with Big Money you cannot include attack cards in the combo.

4. January 12, 2011 at 22:10 | #5

Here are first two solitaire strategy games that I played with one nice combo available, chapel/quarry/grand market. I hit 8 provinces in 14 turns and 13 turns respectively, with King’s Court/Grand Market in the first, and City/Upgrade/Grand Market in the second.

I did branch out a little and not use exclusively the named cards, but this is just me playing quickly and without much thought as a fairly skilled human.

5. January 14, 2011 at 19:40 | #6

Choke! Those hardly represent easy to simulate strategies, I counted 7 kingdom cards in one game. I’m sure chapel contributes to a strategy that already uses King’s Court, Grand Market and Quarry (not to mention City, Laboratory and Remodel) but I think those cards represent a power combo all by themselves.

6. January 17, 2011 at 06:30 | #7

Thank you for this. Very interesting presentation. This indicates how important unwanted estates can turn out. Keep it up!

7. January 19, 2011 at 01:05 | #8

I think you have a math error or typo here:
“One turn with 12 money plus one with 6 money only gives you one province, even though the average is a whopping 10 money.”

• January 19, 2011 at 06:38 | #9

Doh! That should have read 9 money obviously! Thx for pointing that out.

8. December 8, 2011 at 19:10 | #10

i’m curious did you simulate trashing provinces with chapel. Against straight big money Chapel should have an early lead on the province race. Trashing provinces keeps your deck from over greening while keeping big money from winning. just a thought

• December 8, 2011 at 20:05 | #11

Hi Seamus,

thx for the suggestion. Personally I don’t think this is a good idea for I don’t think the single Chapel will build up a large enough lead to justify this. And since I’m not very active with simulations at the moment, I don’t think I will simulate this. However: if you want to get your hands dirty, I can provide you with a copy of the code.

1. No trackbacks yet.