04: NPE Techniques and Metrics
"By three methods, we may learn wisdom: First, by reflection, which is noblest; Second by imitation, which is easiest; and third by experience which is the bitterest" – Confucius.
“When the only tool that you have is a hammer, every problem begins to resemble a nail.” - Maslow.
"All models are approximations. Essentially, all models are wrong, but some are useful. However, the approximate nature of the model must always be borne in mind… " - George Box.Lecture outline: what metrics and techniques can be used for evaluating and comparing the performance of networking systems/ algorithms?
1. Types of NPE techniques
Classification: Analytical modeling; Simulation; Measurement.
Comparison: Analytical modeling; Simulation; Measurement.
2. Types of NPE metrics: Time-based; Number-based; Rate-based.
Time-based metrics: Uptime; Mean time to failure (MTTF); Mean time between failures (MTBF); Mean time to repair (MTTR).
Rate-based metrics: Throughput.
Ratio-based metrics: Utilization; Availability; Fairness.
Selecting set of relevant metrics.
Primary reference for this lecture:
1. “The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling” by Raj Jain; Chapter 3: “Selection of Techniques and Metrics”.
2. "Analyzing Computer Systems Performance with Perl::PDQ" by Neil Gunther, Chapter 1: "Time---The Zeroth Performance Metric".
Secondary references for this lecture:
1. “Measuring Computer Performance: A Practitioner's Guide” by David Lilja; Chapter 2: “Metrics of Performance”
2. “Computer Systems Performance Evaluation and Prediction” by Fortier and Michel; Chapter 3: “Fundamental Concepts and Performance Metrics”