14: Empirical Methods and Measurements
"There cannot be a science without measurement" - Richard Hamming.
"Empirically observed covariation is a necessary but not sufficient condition for causality." - Edward Tufte.
Lecture outline: How to build the scientific method based on experiments to do performance evaluation of networks? How to measure anything? Studying measurement from a new perspective.
Preamble: empirical verification is the foundation of the scientific method.
Preamble: importance of measurement
1. What exactly is measurement?
Common misconceptions: science must be exact; all numbers must be exact.
All measurements are inexact. How to express the uncertainty of measurement?
Accuracy and Precision.
Statistically dependent measurements: the assumptions of IID and measurements.
Expressing measurement over-precisely (more precisely than the data, method of measurement allows)
Five ‘rules’ for measurement.
2. Two theories of measurement
Shannon’s information theory
Smith’s theory of measurement: various scales of measurement.
3. Simple (back of the envelope) measurements.
‘Rule of 5’
Eratosthenes and earth’s circumference
Fermi calculations
Example of Fermi question: In test cricket’s history, how many balls have been bowled?
Primary reference for this lecture:
“How to Measure Anything: Finding the Value of "Intangibles" in Business” by Douglas Hubbard; Chapter 3: “The Illusion of Intangibles, Why Immeasurables Aren‟t”
“Experimental Computer Science: The Need for a Cultural Change” by Dror Feitelson [link]
Secondary references for this lecture:
“Performance Modeling - Experimental Computer Science at its Best”, by Denning [link].