Software Effort Estimation Terminology: The Tower of Babel

Introduction:

Software development effort estimates are the basis for project bidding, budgeting and planning. These are critical practices in the software industry, because poor budgeting and planning often has dramatic consequences. When budgets and plans are too pessimistic, business opportunities can be lost, while over-optimism may be followed by significant losses. The importance of accurate estimates is documented in a wide range of studies. For instance, the Standish Group [24] concludes that reliable estimation is among the top ten most important success factors in software projects. It is therefore unfortunate that, as indicated in a recent review of estimation surveys [41], there has been little improvement in software cost estimation accuracy over the last 20 years. We believe that one reason for this lack of improvement is the imprecise use of terminology for effort estimation. The following two case stories indicate that proper communication, interpretation and improvement of estimation accuracy measurements may be a problem when there is no precise use of terms related to estimation. This problem motivates the review and guidelines provided in this paper.

Guidelines for Estimation Terminology:

• Guideline 1: Do not mix estimation of most likely effort with planning, budgeting or pricing.
  • Different terms should be used for different concepts. In particular, a distinction should be made between estimated ‘most likely effort’, ‘planned effort’ and ‘budgeted effort’.
  • When conducting surveys or logging estimation information, it must not be assumed that the terminology used is understood, even if it is defined precisely. In-depth studies and triangulation may be needed to ensure that all the data are based on the same understanding of the estimation terminology used.
  • Different terms should be used for different concepts. In particular, a distinction should be made between estimated ‘most likely effort’, ‘planned effort’ and ‘budgeted effort’.
  • The estimation of most likely effort should be performed as an independent activity and separated from planning, budgeting and pricing. People in charge of bidding should, for example, not be in charge of the estimation of most likely effort, to ensure that pricing and realism are not mixed. Planning tools should not be used as estimation tools, or, at least, used with great care to avoid a mixing of concerns.
• Guideline 2: When assessing estimation accuracy, make sure that the estimated and the actual effort are comparable.
  • The actual efforts should be adjusted so that they are comparable to the estimated effort with respect to technical and functional parameters before the estimation accuracy is calculated. If functional and quality requirements are not available, the project plan should be investigated and interviews should be used to identify changes in scope and/or quality. If estimates are of types other than most likely effort estimates, they should be transformed to most likely estimates before the accuracy is calculated.
  • When estimates cannot be reliably transformed to values that are comparable to the actual result, great care should be taken when using these results, or the projects for which such transformation cannot be performed should be removed from the data set.
  • The scope and other assumptions of the estimate of most likely effort should be recorded. The version of the requirement specification, and other documents that the estimate of most likely effort is based on, should be specified.
  • Deviation from estimated scope, quality, and development process should be recorded.

Summary:

Effort and schedule overruns are serious problems in the software industry. In this paper we argue that the lack of a precise software effort estimation terminology is an important obstacle for the improvement of estimation accuracy. We reviewed the currently most popular software textbooks and a representative set of software estimation research papers and found systematic shortcomings in use of estimation terminology. For example, estimates of most likely effort are frequently mixed with planned effort, budgets and price. In addition, effort estimation accuracy is frequently measured without adjustments being made for differences in the scope and/or quality assumed when estimating the effort and the system actually implemented.

In order to improve effort estimation accuracy, a more precise terminology for software effort estimation is needed. We provide two simple guidelines for this purpose: (1) Do not mix estimation of most likely effort with planning, budgeting or pricing, and (2) When assessing estimation accuracy, ensure that the estimate and the actual effort are comparable. Although these guidelines are not innovative and might seem obvious, they are nevertheless worth stressing. As this review points out, they are frequently violated.