Math is more reliable than intuition

Sam considers himself an exception to the rule because of all developers he is the only one that lacks a math or a sciences background. A lesson that he learned very early on is that mathematics is reliable and numbers don't lie. They record data and that data is a goldmine. They can see through the data which cards are more liked, which are more disliked, what colors are more played, what colors are least played and all sorts of statistics to help shape the game. More often than not, the numbers contradict the intuition. He uses the phrase that describes to be standing on the shoulders of giants.

I'd like to comment that statistics is not properly taught at school at all. The sole exception depends on really exceptional teachers at school that know statistics very well. One thing about science in all fields is that it requires data and the data has to be reliable. There is a whole field that is to make sure that the data is reliable because one of the greatest pitfalls of science itself is when the data is unreliable or biased. I can't discuss bias here because it's a broad topic with many books covering it. The point that Sam makes is that intuition is often biased and by relying on statistics we can eliminate the bias. I agree with that.

I'd add that mathematics and statistics are prone to mistakes too. I'm not trying to disprove Sam. Mark Rosewater talks a lot about the emotional aspect of designing cards and emotions are not math. Emotions are the opposite of numbers and rational thinking. I'm trying to say that bias is dangerous and numbers can bring in bias too. For example: there is something called the survival bias, which happens when we make assumptions based on who or what survives something, ignoring the population that didn't survive. One of the dangers of blindly believing in numbers is when the numbers themselves are biased. For example: Suppose that a company is hired to play test a game but that company lies or miscommunicates, running playtests that have only women or players of a certain age group. If you look at the data and think that it represents your target audience when it doesn't, we have a problem. I'm not saying that numbers can lie. What I'm saying is that, sometimes, people can lie and use false or biased data to misguide others. Politics is a great example of this and there are numerous examples regarding companies and governments that lie about diseases, catastrophes and so on.

Be careful with blindly following numbers because they aren't immune to mistakes or lies. To give a quick example: suppose that you are measuring the wind speed at the ground level with a sensor. If you have data from a whole month and some days show winds with speeds of 1000 Km/h, there must be something wrong because no place on Earth has winds with speeds of 1000 Km/h. It can be some error in the sensor, some unexpected interference, some unit error, etc. But our intuition tells that something is wrong with the excessively high wind speed.

Don't try to fix everything at once

They have a limited amount of time to create a new set. They have about half a year to playtest the cards before finishing it. They playtest about one or two times per week during development. The most important time is during the beginning, because your first concern is to fix the cards and the set before going on to finish it. From Sam's personal experience he says that there was often an urge to finish the set as soon as possible. There is the problem, you can't rush things. There is always room to improve. One measure that he has had a hard time figuring is the right number of changes from playtest to playtest. Too many changes and you get lost because then it becomes hard to track what changes are working and which ones aren't. You gotta keep the number low to avoid losing track of your process to improve.

I can see that the same thing happens with level design. Trying to fix too many things at once is not going to work. By extension, the same applies to game design as well. We often want to finish the job by burdening ourselves with all problems at once. We have to practice patience here and this applies to life itself. Now comes the question: Which problems to solve first? How many of them first? My answer would be: try to see which of them are core issues and which ones are easier or faster to solve. We certainly have a complex web of interconnected problems, but there is one skill that everyone should have at some degree that is to try to decompose things into smaller pieces. One of the most valuable lessons of college is that any type of problem, no matter how complex or big, can be tackled from multiple viewpoints. Mark Rosewater himself likes to see magic from an holistic viewpoint. Try that, try to see a problem from different perspectives and try to see how smaller problems compound into larger ones.

I can say that I've had the same experience as Sam. You make a level and then wan to finish it as soon as possible. You desperately want it to shine and you don't want to waste too much time polishing it. The problem here is that if you forget the initial plan or don't follow a schedule, you are most likely going to waste more time remaking things, restarting things, making last minute changes, over and over. We have to reach a point where enough is enough and the process to reach it is more important than being obsessed with perfection, which is what Sam is saying.