Embedded Files
Findings
'Personal Rank' (as is commonly used in 5-star and other sentiment ranking systems) is weakly correlated with what actually made the most sense to buy after factoring in cost and degree of enjoyment (aka ordinal scoring is garbage). Only the worst and best ranked item are meaningful, i.e. a 2 star rating system -- but how useful is that?
Looking at package cost and cost per oz (as is common when shopping) is stacked against the consumer and not a reliable measure of what makes sense to buy.
The slight "betterness" of a coffee can be acceptably outweighed by the pricetag cost as well as the net expense if the coffee requires more dry grams in order to make 1 good cup compared to other coffees that may require less product for a good cup.
Methodology
Recorded coffee price and volume information into a google sheet.
Added 3 columns to capture my subjective valuation of the coffees after taste testing over time.
Recorded the amount of dry coffee (g) required for an optimal cup after trying lower/higher amounts of each coffee.
Some coffees required more grounds to be flavorful, presumably because they were old, roasted differently, etc.
I also added a "Personal Rank" score so that I could compare that common method to what I'm about to show you.
I wrote a function to translate a 3 point estimates into to beta distribution parameters. I learned the math of how to do this from David Vose's textbook (Risk Analysis: A Quantitative Guide) then googled my way to implementing the math in Google Script speak.
Transposed the 3-point estimates into a new sheet and used the custom function above to give me the alpha and beta parameter I need for a beta distribution based on the 3 point estimate values for each coffee.
^ This shows the rows naming the product transposed across the top with the 3-point-estimates characterizing how much I value them.
The 7th and 8th rows labeled "a" and "b" are calculated by taking the 3 point estimate into account to define a distribution shape. The output of that function appears in rows 7 and 8.
Implemented a monte carlo simulation using the gsheets native Beta function and giving it the alpha and beta values I generated.
Those a and b outputs are used to shape a Beta distributed "dataset" which is then randomly sampled, starting at row 18, for each coffee product. i.e. "Hey Google, give me a random number between 0 and 11... with a most likely of 3... following a beta distribution ... with alpha of 2.2 and beta of 3.8..."
That row of Beta sampling formulas is copied down 10,000+ times. This creates a synthetic dataset of 10,000 coffee valuations which is shaped by the 3 point estimates I had entered. Each separate coffee product gets its own synthetic 10,000 point dataset that approximates my feels.
The average of this dataset is used to provide a single number that is effectively weighted by my uncertainty in the form of the range between my min and max estimate. That is further weighted by the Mode (most likely) estimate causing more values to appear closer to the Mode.
NOTE: You don't need to do simulation if all you're going to use is the average (you tell the Beta function what average you want from the beginning, the simulation will yield the same average). I opted for simulation just so that I could chart distributions and derive probabilistic statements later on.
Returned the output summary values of my estimation to the original sheet. i.e. The uncertainty/variability and Mode weighted average of how much I enjoy the coffee.
Calculated the value of the coffee factoring in cost, enjoyment, and amount-of-grounds required in order to determine the greatest "bang for your buck" coffee.
This enables more sophisticated analysis and decision support like factoring in:
My personal valuation of the coffee by taste
The actual price of the coffee
The number of dry grams of coffee required to make a good cup
This enables enables me to calculate the Return on Investment (ROI) of the coffee given the above factors.
This revealed a stark contrast from traditional perception ranking methods that fail to capture uncertainty and the "distance" between your top 1 and top 2 choice of a product. Because I provided estimates in the form of dollar amounts, I was also able to calculate ROI. This enabled me to base my coffee purchase decision on factors difficult for most people to keep in mind when comparing several products. The tool also only requires building once. Only the price of the coffee and my valuation need updating over time to generate the same insights.
Most of the time when I'm at the store I think I'm pretty clever by only paying attention to "cost per oz" and I compare products by weighing that against my personal rank of how much I like once thing over another. But if I systematically measure my valuation of a product, my personal rank varies way more than I thought after cost is factored in.
Notice how the Personal Rank was unable to differentiate between the top 3 and completely lost integrity after the top 4.
Other factors to consider adding 3-point-estimate columns for:
Ethical preferences for Coffee distributors like Fair Trade
How valuable a coffee being Organic is to me
Carbon footprint of the distributors such as due to distance in shipping
Caffeine balance
etc
You can make a copy of the gsheet here and put in your own values.
Page updated
Google Sites
Report abuse