Notes

Experimental observations form an important part of network performance evaluation. It is through experimentation that models are validated, simulations parametrized and systems tuned.

In the fields of statistics, a well-designed experiment produces valid results as efficiently as possible. Modern applied statistics is being used routinely in other areas to solve problems which are similar to those faced by experimenters in performance evaluation. Knowledge of statistics can provide assistance in both the design of experiments and the analysis of data.

Sports statistics are readily available from many sources and are frequently used in teaching statistical concepts. Assume that a particular college basketball player has a very poor free throw shooting percentage, and his performance is charted over a period of several games to see if there is any trend. This would constitute observational data—we have simply observed the numbers. Now assume that since the player’s performance is so poor, some action is taken to improve his performance. This action may consist of extra practice, visualization, and/or instruction from a professional specialist. If different combinations of these tasks were employed, this could be in the form of a designed experiment. In general, if improvement is to occur, there should be experimentation. Otherwise, any improvement that seems to occur might be only accidental and not be representative of any real change.

Designed experiments have many potential uses in science in general and in NPE in particular. Some of these uses are: [http://www.moresteam.com/toolbox/t408.cfm]

1) Comparing Alternatives. In the case of our cake-baking example, we might want to compare the results from two different types of flour. If it turned out that the flour from different vendors was not significant, we could select the lowest-cost vendor. If flour were significant, then we would select the best flour. The experiment(s) should allow us to make an informed decision that evaluates both quality and cost.

2) Identifying the Significant Inputs (Factors) Affecting an Output (Response) - separating the vital few from the trivial many. We might ask a question: "What are the significant factors beyond flour, eggs, sugar and baking?"

3) Achieving an Optimal Process Output (Response). "What are the necessary factors, and what are the levels of those factors, to achieve the exact taste and consistency of Mom's chocolate cake?

4) Reducing Variability. "Can the recipe be changed so it is more likely to always come out the same?"

5) Minimizing, Maximizing, or Targeting an Output (Response). "How can the cake be made as moist as possible without disintegrating?"

6) Improving process or product "Robustness" - fitness for use under varying conditions. "Can the factors and their levels (recipe) be modified so the cake will come out nearly the same no matter what type of oven is used?"

7) Balancing Tradeoffs when there are multiple Critical to Quality Characteristics (CTQC's) that require optimization. "How do you produce the best tasting cake with the simplest recipe (least number of ingredients) and shortest baking time?"

"If you're a politician, admitting you're wrong is a weakness, but if you're an engineer, you essentially want to be wrong half the time. If you do experiments and you're always right, then you aren't getting enough information out of those experiments. You want your experiment to be like the flip of a coin: You have no idea if it is going to come up heads or tails. You want to not know what the results are going to be.", Peter Norvig.