40: Verification and Validation
"A theory has only the alternative of being right or wrong. A model has a third possibility: it may be right, but irrelevant." - Eigen, Manfred“Verification is solving the equations right; Validation is solving the right equations.” - Unknown.
Lecture outline: How can simulation models be verified and validated?
1) Common simulation mistakes.
Checklist for simulations.
Simulation results validation (validation and output analysis).
2) Verification and Validation
Verification---are we building the model right?
Model verification techniques.
Validation---are we building the correct model?
All models are approximations.
Principles of simulation modeling (Alan Pritsker's advice).
Calibrating models is iterative.
Principles of V&V
Simulation results validation.
3) Credibility of simulations.
A cautionary note on simulations.
Repeatability of results.
Primary reference for this lecture:
“The Art of Computer Systems Performance Analysis: Techniques for Experimental Design, Measurement, Simulation, and Modeling” by Raj Jain; Chapter 25: “Analysis of Simulation Results”; Section 25.1 and 25.2.
Secondary references for this lecture:
1. “Fundamentals of Performance Evaluation of Computer and Telecommunication System”, by Obaidat and Boudriga; Chapter 11: “Analysis of Simulation Results”.
2. "Handbook of Simulation", edited by J Banks, Part 2: Chapter 10, Verification, Validation, and Testing.
3. "Why we don't know how to simulate the Internet", Paxson and Floyd, [link].
4. "Improving Confidence in Network Simulation", Weigle, [link]