03: Common NPE Errors

"The first principle is that you must not fool yourself, and you are the easiest person to fool." - Feynman."Science is simply common-sense at its best; that is, rigidly accurate in observation and merciless to fallacy in logic." - T. H. Huxley."Wise men learn by other men’s mistakes, fools by their own." — H G. Wells.Lecture outline: which errors commonly negatively affect performance analysis studies thereby allowing inference of incorrect conclusions?

1. Logical fallacies

Fallacies of evidence: hasty generalization; questionable cause; Texas sharpshooter fallacy; appeal to ignorance.

Fallacies of relevance: personal attack; bandwagon argument; red herring; straw man.

2. Cognitive biases

Base rate neglect; ‘the law of small numbers’; stereotyping bias.

3. Systematic NPE errors

No goals or biased goals; inappropriate modeling; inappropriate workload/ metrics/ experimental design or analysis; not using scientific method of investigation.

Primary reference for this lecture:

1. Kevin B. Korb (1998) Research Writing in Computer Science. This explains some of what goes into good research writing, including argument analysis and an understanding of cognitive errors that people are prone to make. It also discusses research ethics.

2. “The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling” by Raj Jain; Chapter 2: “Common Mistakes and How to Avoid Them”

3. "Reasoning and Fallacies" by P. B. Stark (UC, Berkeley)

http://www.stat.berkeley.edu/~stark/SticiGui/Text/reasoning.htm