What do the experts have to say about checking model assumptions?
The notes below contain excerpts from many references on various statistical methods and topics, written by credible authors and recognized experts.
[Each model has] implicit assumptions based on properties inherent to the process modeling methods themselves. Successful use of these methods in any particular application hinges on the validity of the underlying assumptions, whether their existence is acknowledged or not.
R-squared is NOT enough!
Model validation is possibly the most important step in the model building sequence. It is also one of the most overlooked. [...] Use of a model that does not fit the data well cannot provide good answers to the underlying engineering or scientific questions under investigation.
Chapter 3 (page 100)
When a regression model is considered for an application, we can usually not be certain in advance that the model is appropriate for that application. Any one, or several, of the features of the model, such as linearity of the regression function or normality of the error terms, may not be appropriate for the particular data at hand. Hence, it is important to examine the aptness of the model for the data before inferences based on that model are undertaken.
Chapter 3 (page 76)
The decomposition of the variability in the observations through an analysis of variance identity is a purely algebraic relationship. However, the use of the partitioning to test formally for no differences in treatment means requires that certain assumptions be satisfied. Specifically, these assumptions are that the observations are adequately described by the model [ yij = mu + taui + eij ] and that the errors are normally and independently distributed with mean zero and constant but unknown variance. If these assumptions are valid, the analysis of variance procedure is an exact test of the hypothesis of no difference in treatment means. In practice, however, these assumptions will usually not hold exactly. Consequently, it is usually unwise to rely on the analysis of variance until the validity of these assumptions has been checked. Violations of the basic assumptions and model adequacy can be easily investigated by the examination of residuals. […] Examination of the residuals should be an automatic part of any analysis of variance.
Chapter 3 (page 83)
The ANOVA techniques considered […] are theoretically based on independence, random samples, normal distributions, and equal population variances. In practice, we do not expect the model assumptions to be satisfied exactly. For the procedures to yield reliable results, however, those assumptions must be reasonably satisfied.
Chapter 6 (page 182)
If we could rely on the exactness of the IID assumption, we could say that after these [analysis of variance] statistics had been calculated no further relevant information remained in the original data, and we could, therefore, ignore the residuals and the original observations and concentrate entirely on the interpretation of these statistics. In practice it would be very unwise to do this without further checks because data may contain valuable information not allowed for in the assumed mathematical model and therefore not revealed by the associated analysis of variance table. […] Discrepancies of many kinds between a tentative model and the data can be detected by studying residuals. […] When assumptions concerning the adequacy of the model are true, we expect to find that, apart from restrictions introduced by the analysis itself, the residuals vary randomly. If, however, we find that the residuals contain unexplained systematic tendencies, we shall be suspicious of the model. Therefore, as an automatic preliminary to further statistical analysis, a table of residuals should always be constructed and studied. [particular discrepancies] should be looked for as a matter of routine.
Chapter 4 (page 58)
The decomposition of the total variability in the observations through an ANOVA is purely algebraic. However, the use of ANOVA to construct confidence intervals requires that certain assumptions on the observations be satisfied. In general, ANOVA procedures are relatively robust to these underlying assumptions. In experiments where some or all of the factors are random (as is frequently the case in gauge R&R experiments), the assumptions are more critical. […] In practice these assumptions usually will not hold exactly. Consequently, it is usually unwise to rely on the ANOVA results without carefully checking the validity of these assumptions. Violations of the basic assumptions and model adequacy can be easily investigated by the examination of residuals.
Chapter 2 (page 29)
(Regarding SPC charts) The most critical assumption is that successive deviations are independent, so that variation within subgroups can tell us what the variation of the averages between subgroups should be.
Chapter 2 (page 51)
[…] ARLs are extremely sensitive to violations of assumptions.
Chapter 3 (page 61)
Difficulties frequently occur in the application of the control methods [discussed]. They relate to the inadequate approximations that underlie the use of these techniques and, in particular, the supposition that noise can be modeled as a stationary random variation about a fixed mean.
All models are approximations. Assumptions, whether implied or clearly stated, are never exactly true. All models are wrong, but some models are useful. So the question […] is not “Is the model true?” (it never is) but “Is the model good enough for this particular application?” The answer depends on two factors: (1) How close to the reality the assumptions are likely to be, and (2) How robust (insensitive) the derived statistical methods are to these likely departures from assumption.
Chapter 3 (page 63)
So in the practice of quality control charting or in the application of any other statistical method, you must ask: What are we assuming? How large can we expect deviations to be from these ideal assumptions? Will the conclusions be seriously affected by such deviations?
Chapter 3 (page 137)
The analysis of variance assumes that the model errors (and as a result, the observations) are normally and independently distributed with the same variance in each factor level. These assumptions can be checked by examining the residuals.
Chapter 1
(page 20-21)
Care should be used with [process capability] indices […]. Statistical properties and assumptions should be understood and met before using them, and index estimations should be qualified via confidence intervals.
(page 22)
Care should always be taken to ensure that the underlying assumptions are met […].
Reliability: Statistical Methods for Reliability Data (Meeker)
Chapter 17 (page 443)
An important part of any statistical analysis is diagnostic checking for departures from model assumptions. In conducting a failure-time regression analysis we recommend the use of graphical methods, using generalizations of usual regression diagnostics (including residual analysis). These diagnostic methods can be used in a manner that is similar to their use in ordinary regression analysis, except that interpretation is often complicated by the censoring. The analysis can also be complicated when fitting underlying non-normal distributions.
Chapter 1 (page 14)
Of course, any quantitative analysis relies upon a series of assumptions. When the conditions behind the assumptions no longer apply, the analyses should no longer be employed.
See the paper “Statistical Methods and Regulatory Compliance - Sampling Plans.doc”.
(Role of Statistics in QMS - paper is attached at the bottom of this page)
(Chapter 1, page 3)
The sources of error in applying statistical procedures are legion and include all of the following: […] Failing to validate models.
(Chapter 1, page 4)
Here is a partial prescription for the error-free application of statistics.
(7) Know the assumptions that underlie the tests that you use. Use those tests that require the minimum of assumptions and are most powerful against the alternatives of interest.
(Chapter 5, page 51)
Every statistical procedure relies on certain assumptions for correctness. Errors in testing hypotheses come about either because the assumptions underlying the chosen test are not satisfied or because the chosen test is less powerful than other competing procedures.
[… page 52 …]
[B]efore we choose a test statistic, we check which assumptions are satisfied, which procedures are most robust to violation of these assumptions, and which are most powerful for a given significance level and sample size.
(page 78)
An experienced data handler develops a nose for strange features of data and statistical summaries. An odd pattern, irregularity, or coincidence is often a clue that something fishy may be going on – a mistake in recording the data, a statistical miscalculation, a departure from the assumptions and conventions of a particular type of analysis, or -- in unusual cases -- even downright fraud.
(page 14)
A traditional assumption made in financial study is that the simple returns are independently and identically distributed as normal with fixed mean and variance. This assumption makes statistical properties of asset returns tractable. But it encounters several difficulties.
(page 44)
A fitted model must be examined carefully to check for possible model inadequacy. If the [time series] model is adequate, then the residual series should behave as a white noise. If a fitted model is found to be inadequate, it must be refined.
(page 193)
Many things can, and often do, go wrong when data are analyzed. There may be data that were entered incorrectly, one might not be analyzing the data set one thinks, the variables may have been mislabeled, and so forth. […] Besides problems with the data, the assumed model may not be a good approximation to reality. [… most] practitioners use regression diagnostics to detect problems and then attempt to remedy any problems that were detected. [… the] residuals can be used to check the assumptions behind regression.
(page 51)
Assumption [...] must be verified, not merely stated, in practice to ensure good results.