Cattle Weight Estimation

Sir Francis Galton was a celebrated British scientist. See, http://en.wikipedia.org/wiki/Francis_Galton

He found that farmers, butchers, and non-experts could as a voting group accurately guess the weight of an ox.

http://galton.org/essays/1900-1911/galton-1907-vox-populi.pdf

"In these democratic days, any investigation into the trustworthiness and peculiarities of popular judgments is of interest. The material about to be discussed refers to a small matter, but is much to the point.

A weight-judging competition was carried on at the annual show of the West of England Fat Stock and Poultry Exhibition recently held at Plymouth, A fat ox having been selected, competitors bought stamped and numbered cards, for 6d. each, on which to inscribe their respective names, addresses, and estimates of what the ox would weigh after it had been slaughtered and " dressed." Those who guessed most successfully received prizes. About 800 tickets were issued, which were kindly lent me for examination after they had fulfilled their immediate purpose. These afforded excellent material.

The judgments were unbiased by passion and uninfluenced by oratory and the like. The sixpenny fee deterred practical joking, and the hope of a prize and the joy of competition prompted each competitor to do his best. The competitors included butchers and farmers, some of whom were highly expert in judging the weight of cattle; others were probably guided by such information as they might pick up, and by their own fancies" 

"The average competitor was probably as well fitted for making a just estimate of the dressed weight of the ox, as an average voter is of judging the merits of most political issues on which he votes, and the variety among the voters to judge justly was probably much the same in either case. After weeding thirteen cards out of the collection, as being defective or illegible, there remained 787 for discussion. I arrayed them in order of the magnitudes of the estimates, and converted the cwt., quarters, and lbs, in which they were made, into lbs., under which form they will be treated."

"According to the democratic principle of "one vote one value," the middlemost estimate expresses the vox populi, every other estimate being condemned as too low or too high by a majority of the voters (for fuller explanation see, One Vote, One Value," Nature, February 28, p. 414). Now the middlemost estimate is 1207 lb., and the weight of the dressed ox proved to be 1198 lb.; so the vox populi was in this case 9 lb., or 0.8 per cent of the whole weight too high."

Galton


Galton continues with these conclusions:

"It appears then, in this particular instance, that the vox populi is correct to within 1 per cent of the real value, and that the individual estimates are abnormally distributed in such a way that it is an equal chance whether one of them, selected at random, falls within or without the limits of -3.7 per cent and +2.4 per cent of their middlemost value.

This result is, I think, more creditable to the trust-worthiness of a democratic judgment than might have been expected.

The authorities of the more important cattle shows might do service to statistics if they made a practice of preserving the sets of cards of this description, that they may obtain on future occasions, and loaned them under proper restrictions, as those have been, for statistical discussion. The fact of the cards being numbered makes it possible to ascertain whether any given set is complete."

Discussion On Galton's Conclusions

Galton's cattle weighing study is cited by many researchers TBD so it is worthwhile to consider them.

Galton says he had access to 787 weight-judging competition tickets. He provides the raw data in the image above from his vox populi article however I do not see any numbers amounting to 787. Galton labels his data "Distribution of the estimates" but does not seem to provide the full underlying data.

Galton picks out on a post-hoc basis the "middlemost value" of 1207 lbs that he says "expresses the vox populi". He uses what he says is the median value, by definition the middlemost value, and does not use the average or the weighted median. In another article he explains why he does not use the average value:

http://galton.org/essays/1900-1911/galton-1907-vote-value.pdf

"How can the right conclusion be reached, considering that there may be as many different estimates as there may are members? The conclusion is clearly not the average of all estimates which would give a voting poser to "cranks" in proportion to their crankiness. ... I wish to point out that the estimate to which is least objection can be raised is the middlemost estimate."

In my opinion, the weighted median value is probably the best estimate - not Galton's median value.

Additional Problems With Galton's Conclusions

  • Many people who write about the wisdom of the crowds cite Galton's 1907 ox-weighing research. Where are the replicated studies? There should be at least several research articles verifying what Galton purported to show but I don't think these studies exist. Plenty of state fairs have cattle weighing contests so there should be many more studies. 
  • Galton says that farmers and ranchers and the average non-farmer who visited the West of England Fat Stock and Poultry Exhibition were all about equally expert in judging the weight of cattle. Farmers weigh their livestock with scales and tape measures and they also go to livestock auctions. There they gain a lot of expertise on visually estimating the weight of cattle.
  • Galton equates guessing the weight of an ox with the ability to assess the qualities of a politician. That seems to be an intellectual stretch.
  • Cattle have ascertainable weights - put them on a scale and you can get a scientific metric such as: This ox weighs 1207 pounds. Politicians have a lot of undefinable characteristics.

My Conclusions

Galton's research is faulty for several reasons including:

  • His findings need to be replicated.
  • His use of the median value appears to be post-hoc and he doesn't discuss why he didn't choose the weighted median which to me seems a better metric.
  • His relating ox-weight guessing to democratic voting needs support.

Critical Review of the Best-Seller: The Wisdom of Crowds, by James Surowiecki

The Wisdom of Crowds, by James Surowiecki, published 2005 by Doubleday; Anchor

This book is widely praised, for example:

Deseret Morning News

"The book is deeply researched and well-written, and the result is a fascinating read."

I do not have this book, but some material is available as a preview on Amazon here:

http://www.amazon.com/Wisdom-Crowds-James-Surowiecki/dp/0385721706#reader_0385721706

This is a quote from James Surowiecki's introduction that in my opinion has some serious errors.

"Galton was interested in figuring out what the "average voter" was capable of because he wanted to prove that the average voter was capable of very little. So he turned the competition into an impromptu experiment. When the contest was over and the prizes had been awarded, Galton borrowed the tickets from the organizers and ran a series of statistical tests on them. Galton arranged the guesses (which totaled 787 in all after he had to discard thirteen because they were illegible) in order from highest to lowest and graphed them to see if they would form a bell curve. Then among other things, he added all of the contestants' estimates, and calculated the mean of the group's guesses. That number represented, you could say, the collective wisdom of the Plymouth crowd. If the crowd were a single person, that was how much it would have guessed the ox weighed.

Galton undoubtedly thought that the average guess of the group would be way off the mark. After all, mix a few very smart people with some mediocre people and a lot of dumb people, and it seems likely you'd end up with a dumb answer. But Galton was wrong. The crowd had guessed that the ox, after it had been slaughtered and dressed, would weigh 1,197 pounds. After it had been slaughtered and dressed, the ox weighed 1,198 pounds. In other words, the crowd's judgment was essentially perfect. Perhaps breeding did not mean so much after all. Galton wrote later: "The result seems more creditable to the trustworthiness of a democratic judgment than might have been expected."

Bugs

Bug #1: The book's introduction says: "Then among other things, he added all of the contestants' estimates, and calculated the mean of the group's guesses."

However, Galton did not calculate the mean of the group's guesses, he calculated the median of the group's guesses.

Galton: "According to the democratic principle of "one vote one value," the middlemost estimate expresses the vox populi, every other estimate being condemned as too low or too high by a majority of the voters (for fuller explanation see, One Vote, One Value," Nature, February 28, p. 414).

In fact, in a different earlier article by Galton, he condemns using the average (the mean):

http://galton.org/essays/1900-1911/galton-1907-vote-value.pdf

"How can the right conclusion be reached, considering that there may be as many different estimates as there may are members? The conclusion is clearly not the average of all estimates which would give a voting poser to "cranks" in proportion to their crankiness. ... I wish to point out that the estimate to which is least objection can be raised is the middlemost estimate."

Bug #2: The book's introduction says: "Then among other things, he added all of the contestants' estimates, and calculated the mean of the group's guesses." And then the intro says, "The crowd had guessed that the ox, after it had been slaughtered and dressed, would weigh 1,197 pounds."

The 1,197 pounds number given in the book as the crowd's average ox weight guess seems to be clearly incorrect.

Take a look at the data in the chart above then add up the second column marked "Estimates in lbs." (22,747 total) then divide that total by the number of entries (19) in the column. The result is 1197.21 pounds. However, 1197.21 is clearly not the average guess of the 787 people in the crowd.

The 19 entries in Galton's data represent the various ox-weight estimates. Galton does not appear to have provided information on how many people guessed each of the 19 estimate values. Therefore, the 1,197 pounds average guess the book cites is not the average of the crowd's guesses - which we cannot calculate because of missing data.

To provide the crowd's average guess we would need to add the 787 ox-weight guesses then divide that sum by 787, which Galton did not report.

References

Another animal weight-guessing competition: Auckland Star, Volume XXX, Issue 279, 24 November 1899, Page 8.

An old study from 1899, before Galton's research uses a small data-set but it arrives at a different conclusion:

"Nearly all competitors over-estimated the weight of the sheep, and under-estimated the weight of the cattle."

How would Galton in 1907 have interpreted the sheep and cattle weight guessing differences?

Would Galton have formulated a different relationship between animal weight-guessing and political voting based on the differing results for cattle and sheep?