Methodologies for Information Systems Investment Evaluation at the Proposal Stage: a Comparative Review

Abstract:

The evaluation of information systems (IS) investments has been a recognized problem area for the last three decades, but has recently gained renewed interest of both management and academics. IS investments constitute a large and increasing portion of the capital expenditures of many organizations. However, it is difficult to evaluate the contribution of an IS investment to the goals pursued. Consequently, there is a great call for methods and techniques that can be of help in evaluating IS investment at the proposal stage. The contribution of the paper to the problem area is twofold. First, the different concepts which are used in evaluation are discussed and more narrowly defined. When speaking about IS investments, concepts are used that originate from different disciplines. In many cases there is not much agreement on the precise meaning of the different concepts used. However, a common language is a prerequisite for the successful communication between the different organizational stakeholders in evaluation. In addition to this, the paper reviews the current methods and puts them into a frame of reference. All too often new methods and guidelines for investment evaluation are introduced, without building on the extensive body of knowledge that is already incorporated in the available methods. Four basic approaches are discerned: the financial approach, the multi-criteria approach, the ratio approach and the portfolio approach. These approaches are subsequently compared on a number of characteristics on the basis of methods that serve as examples for the different approaches. The paper concludes with suggestions on how to improve evaluation practice and recommendations for future research.

Introduction:

Investments in information systems (IS) are large and increasing. They constitute up to 50% of the capital expenditures of large organizations [l-3]. Information systems are not only used in administrative and decision making tasks but are changing the shape of production processes and enable the development of new products and services. Recent empirical studies show that organizations have several problems with the evaluation of proposals for IS investments [4-8]. A number of causes can be identified. Because information systems are often for a great extent integrated in the organization, it is difficult to establish the boundaries of the system. For instance, which user costs of a new electronic mail system should be considered in an investment proposal?

Another possible cause is the ongoing dispute on the relevant decision criteria. How should, for instance, long-term consequences of an IS investment be incorporated? An example of this is the contribution of a database management system to the realization of data infrastructure in an organization.

A plethora of methods and techniques has been proposed to assist in the evaluation of IS investment proposals. Different Dutch researchers in the universities of Delft, Eindhoven and Amsterdam [9,10] identified over 65 methods that all aim to be of help in the evaluation of IS investment proposals (see Appendix). Already in 1961 the International Federation of Information Processing devoted its first conference to evaluation issues [11] and in 1968 Joslin wrote his book on computer selection [12].

However, all too often new methods and guidelines for IS investment evaluation are proposed, without building on the extensive body of knowledge that is already incorporated in the available methods. The purpose of this paper is to improve insight into the current methods for the evaluation of IS investment proposals, For the moment this is the maximum that can be strived for, as research that has validated evaluation methods is hardly available. General prescriptions about the use of which method in which circumstances cannot be given. Current research is still focusing on finding the essential evaluation criteria, the circumstances in which these should be used and the inclusion of the criteria in the evaluation process.

Section 3 reviews the current methods. Subsequently, in Section 4 an assessment of the methods on different characteristics is presented. In the preceding Section 2 the different concepts used in evaluation are discussed and defined. This is prerequisite for the comparison of the methods in the following sections. Finally, Section 5 concludes with suggestions on how to improve evaluation practice and some recommendations for future research.

Evaluation Method Comparison:

Conclusions:

Evaluation of IS investment proposals currently is a major issue for both management and academics. This paper has first introduced and defined the concepts used in evaluation. These definitions can be used to improve communication between the different stakeholders in the evaluation process and to gain insight in the differences between the several evaluation methods. The review and comparison of evaluation methods showed that the available methods differ in many respects and that conclusions regarding the overall quality cannot be drawn. However, some general observations can be made:

• • The available non-financial evaluation methods are hardly underpinned by theory: they are usually based on single case studies and lack theoretical basis. Consequently, the choice of criteria seems rather arbitrary.

  • The available methods focus on the evaluation criteria, less attention is paid to the evaluation process. The experimental study of Klompe and Berghout [41] shows that altering the decision-making process significantly influences decision-making.

  • There seems to be trade-off between the inclusion of non-financial criteria and the ease of use of a method. Graphical tools as used in portfolio methods can be of help.

  • The differences between the methods can partly be overcome by combining features of the different approaches. For instance, a financial or ratio assessment combined with the non-financial consequences (‘contribution’) represented in a portfolio.