K. J. Halliwell

The Quality Measurement Plan: an overview and comments on its application.

Introduction:

The Quality Measurement Plan (QMP) is a statistical technique for evaluating non-conforming attributes (defects) found during process output sampling. A QMP evaluation assesses process quality performance as compared to a given standard or expectation. QMP is a relatively new technique, and it is often misunderstood. The following gives a brief history and overview of the QMP technique, and comments on its application as a statistical process quality monitoring and improvement tool.

The QMP technique is based on "Empirical Bayesian Analysis" theory (see addendum). During the first half of the 18^th century, Thomas Bayes developed statistical analysis ideas that were later refined and became known as Bayesian analysis. One aspect of Bayesian analysis is that the past (empirical or subjective history) influences the present; and, if history is taken into account, it provides a better estimate of the "truth" about sampled populations than methods that do not consider history. This history effect is known as Bayesian statistical inference.

Around 1980, Dr. A. B. Hoadley, of Bell Labs, studied and applied Empirical Bayesian Analysis theory to develop the QMP technique. Shortly thereafter, QMP replaced the "T" rate plan once used by Western Electric (now Lucent Technologies).

 

Technique:

The QMP technique uses data from the current and last five sampling periods of a given process, or process group, to estimate true process quality performance level and its variability. Input data consists of sample size, actual defects (demerits) and the normally expected or "standard" defects (demerits) for each period. The results are: Current Index (CI), Long Run Average (LRA), Best Measure (BM) and Posterior distribution confidence percentiles.

A Posterior distribution is the probability distribution associated with Estimated Bayesian Analysis. For QMP, it contains the range of possible "Quality Index" (QI) values wherein the true current QI value may exist. A QMP chart may show this QI probability range as a "box plot" bounded by its 99, 95, 5 and 1 percentile points, or it may show only the 95 and 5 percentile points.

The Current Index (CI) is simply the ratio of actual defects (found during sampling) to expected defects. In other words, the CI is a Quality Index (QI) value. Typically, all possible values (Quality Index range) for the CI form the "Y" axis of a "QMP chart" and the sample periods (time interval) form the "X" axis. A CI of one (1) indicates actual defects equal expected defects, less than one (1) indicates actual defects are less than expected defects, and greater than one (1) indicates actual defects are greater than expected defects. (Note: defects are viewed by the QMP statistical technique as demerits. A demerit may represent any observed count; i.e., defects may be demerits, but demerits are not necessarily defects. Depending on the application, a demerit may represent a defective unit, dropped calls, missed shipments or any other observed attribute oriented value.)

The Long Run Average (LRA) is similar to a six period running average of the CI, but it is not a simple arithmetic average. It takes period-to-period sample size variation and past CI values into account. The LRA is also known as the "estimated process average."

The Best Measure (BM) is the mean (average) of a given period’s Posterior distribution. It is not the distribution’s physical midpoint or median; i.e., a Posterior distribution is not a "Normal" or any other type of symmetrical distribution. The BM represents the current period’s most likely true Quality Index level. It results from a calculation involving the LRA, CI and several "weighting factors" derived from the current and historical Posterior distributions’ variability. When process performance is not stable, more weight comes from the current period; otherwise, the past periods’ distributions contribute more weight.

The Posterior distribution’s 99, 95, 5 and 1 percentile points are calculated using the "method-of-moments" along the Incomplete Gamma (IG) distribution’s characteristic curve family. In other words, the Posterior distribution (like all statistical distributions) can be represented as a function or transform of the IG distribution. To map from the IG to Posterior distribution, one uses BM squared, divided by its estimated variance as the IG "shape" factor; and the BM divided by estimated variability as the IG "range" value. The IG shape factor represents a specific IG distribution characteristic curve bounded by 0 and 1 on the Y axis, and the IG range bounded by 0 and infinity on the X axis. Where the given range value (X) intersects with the given shape (curve) yields the probability value (Y) of the Posterior distribution.

The QMP technique is somewhat complex; but it is based on widely accepted statistical theory and practice. It has all the elements of classical (non-Bayesian) statistical techniques; but it greatly improves upon them by utilizing history (past sample size and process quality performance variation) as an important influence on estimating current population mean (i.e., BM) and long term quality performance (i.e., LRA) from sample data. Thus, the QMP technique is robust; and the BM and LRA are excellent estimators and indicators of true process quality performance.

 

Application:

The QMP statistical technique with its Quality Index chart is a process quality performance monitoring tool. Additionally, it gives indications about process stability via the degree of inter-period BM variability (shown by the box plots) and the variability in CI and LRA. Thus, to a very limited extent, it may be used like a process stability control chart; although, it is not a process stability control chart, and it is technically incorrect to use it as a process stability control chart. But, as a process stability control chart substitute, one may view a QMP chart as a classical, one-sided process stability control chart with its upper limit equal to a Quality Index of one. The risks of this unorthodox usage are that a true unstable (out-of-control) state may go unnoticed even though a process is performing within its expected limit, or a process may be mistakenly viewed as unstable (out-of-control) simply because it operates outside its expected performance limit. In some cases, process stability control may not be a significant concern relative to performance control, and, thus, out-of-control is taken to mean operating outside of expected performance limit. For these cases, a QMP chart will adequately serve as a "control chart."

Also, one may use a QMP chart to monitor process improvement. Regardless of process stability or control level, BM and LRA values will show trends, over time, to indicate whether or not a process is improving, and reflect its relative stability. Unlike control chart substitution, a QMP chart is extremely reliable and technically correct for monitoring process improvement.

As a process performance monitoring tool, one can easily establish decision rules for QMP output parameters that determine when a target process is meeting or exceeding expectations (capable), borderline (approaching not capable), or failing (not capable). These three conditions may be displayed as green, yellow and red colors on a signaling device linked to a QMP data processing system. Thus, a complex data analysis technique and its somewhat cryptic chart can be converted into a simple and easy to understand process quality performance indicator.

In any sampling based statistical method (including QMP), confidence in estimates of the true mean has an inverse relationship to sample size. In other words, the larger the sample, the higher the confidence that the sample mean truly represents the total population mean value. Thus, one must take relatively large samples to confidently detect small expected values, and relatively small samples to confidently detect large expected values. Usually, relatively large expected values and their small sample do not cause economic problems; i.e., small samples have minimal economic impact. But economic problems may occur with taking large samples to confidently detect small expected values. There is no easy solution to this classic sampling problem. Although, one can apply techniques such as small samples over longer periods; and taking into account that a stable process, with little or no history of control problems, may not need frequent monitoring, but, rather, occasional monitoring with relatively large samples. Obviously, these approaches are not without risk; but this is an inherent trade-off with sampling.

For the QMP technique, a rule-of-thumb is that sample size should not fall below a level that will yield a Quality Index of one (1) for sample size multiplied by expected defects percentage. Intuitively, this is sensible because it means the sample size provides a chance to observe at least one (1) defect at the expected level; although, this is not guaranteed. On the high-side, taking samples greater than ten times sample size times expected defect percentage yields an extremely high confidence level that approaches 100% inspection, and, thus, cannot be economically justified. The current practice is to take sample sizes that yield an expected defect index of two or three times sample size times expected defect percentage.

Generally, the QMP technique performs very well with relatively small sample sizes. But a small sample size may not provide as high of a quality performance level indication, for a given BM, as a larger size; although, the QMP results continue to provide a good indication of process quality performance control and improvement.

The QMP technique is a special application of Empirical Bayesian Analysis. It is a modern and robust statistical technique primarily intended for non-conforming attribute (defect) performance monitoring. It works well in environments with high variability in process stability and sample size, as well as ones with low variability. To a very limited extent, it may be used like a one sided, stability control chart where the expected or desired quality level appears like an upper stability control limit; although, a QMP chart is not a process stability control chart. The QMP technique is an excellent statistical tool for showing process performance control, and monitoring improvement trends.

Quality Control Charts and Empirical Bayesian Analysis

The following is a synopsis of information found in the "Handbook of Statistical Methods for Engineers and Scientist", chapter 10.4, by Thomas W. Calvin, published by McGraw-Hill, ISBN 0-07-067674-7. When reading, please keep in mind that Empirical Bayesian Analysis is fundamental to the QMP technique, but it is not exactly the QMP technique. Therefore, some concepts herein are slightly different than those of the QMP technique; e.g., the Bayesian "control chart."

In the ideal world, a manufacturing process produces units in well-defined lots, batches, runs or some other subgroup. When a process is stable and "in control," each unit within a subgroup and each subgroup is essentially the same, with slight unit-to-unit and subgroup differences. Additionally, there may be slight difference within each unit. Generally, differences within a unit are viewed as due to random process variation; whereas, differences between units and subgroups may be due to "assignable" causes; i.e., not due to random variation.

A histogram can show the total random variation found between units and subgroups. A probability distribution can be applied to the histogram, and its average and range may be computed. In turn, a probability distribution based on the idea of sampling units from subgroups may be used for estimating the process average and range. Sampling offers an economical technique for monitoring a process, but it may or may not give a good estimate of the true process average and range. Thus, with sampling, another variation is introduced – sampling variation (represented by the sampling probability distribution).

Sampling is often used in conjunction with a statistical control chart. Ideally, a control chart should be applied only when a process has achieved stability and all assignable causes have been identified and removed. In other words, a statistical control charts’ purpose is to detect an occasional out-of-control condition based on sampled units from subgroups; it cannot do this reliability when the target process is inherently unstable or out-of-control.

In the real world, it is often impossible to identify and eliminate all assignable causes. This may be due to a new or complex technical processes, or built-in process or economic limitations. Of course, these residual assignable causes may vary within and between themselves, and they contribute to unit-to-unit and subgroup-to-subgroup variation. Thus, one is confronted with the paradoxical problem of providing a statistical means to monitor and evaluate the control state of processes containing assignable, random and sampling variation (a situation often found in chemical and electronics manufacturing industries) that denote inherent instability. If the unavoidable assignable causes are well understood and not highly variable, then classical control charts (e.g., X bar, R, P, C and U charts) may suffice, but this practice is questionable. Bayesian analysis offers an alternative statistical method that can deal with these "real world" processes. It takes into account past process experience (including all variation) to estimate current process performance and, to a limited degree, stability. This past experience may come from prior process sample, technical and subjective data. When available, prior process sample data is preferred; otherwise, technical or subjective data that approximates the past is usually better than nothing. The use of prior sample data with Bayesian analysis is known as Empirical Bayesian Analysis. (See the above cited reference for a detailed presentation of Bayesian analysis theory.)

Unlike a classical control charts, a Bayesian "control chart" uses a target value to represent process performance expectation. Essentially, the target value serves the function of the average value found in a classical control chart. This is necessary because the Bayesian best estimate (average) of process performance is not fixed from period-to-period like the classical average value. The best estimate’s value, relative to its target, indicates the degree to which a process is meeting (or not meeting) targeted performance or capability; and, when compared to previous best estimates, its stability. Further, by placing a standard "box plot" (each box end marks the 25 percentile points on each side of the best estimate) over the best estimate to show sampling and process variability, one gains further insight into process control state. To an extent, the box plot functions like control limits on a classical control chart; although, they are not fully identical concepts. When the Bayesian target value falls within the box, process capability is probably in control (meeting expectations); otherwise, it is probably out-of-control (not meeting expectations).

Empirical Bayesian analysis with targeted performance levels overcomes a weakness of classical control charts. Namely, a classical control chart may report process stability control state, for a given process performance level, but not process performance control state relative to an expected level. In other words, process stability is known, but capability is not known. Bayesian analysis and its "control chart" will report process performance control state and exhibit the relative process stability control state while accounting for variability introduced by any unresolved assignable causes and sampling. Its weakness is that it cannot report process stability control state, but only exhibit its variability as periodic changes in process performance best estimate value. Depending on the process control problem at hand (stability v/s performance), and the uncertainty of residual unassignable causes, Empirical Bayesian analysis may offer a better tool than a classical control chart.