Changing the Paradigm of Software Engineering

Introduction:

There has been much discussion as to what degree iterative and agile processes and software evolution are different from previously promoted practices based on the waterfall life cycle model. Here, I argue that the departure is fundamental and, in fact, it represents a new paradigm. The notion “paradigm” was used by Kuhn [8] to describe a “coherent tradition of scientific research” that includes knowledge, techniques, research agenda, and so forth. Kuhn collected extensive historical data and proved that change from an old paradigm to a new one takes place when the old paradigm is in crisis and cannot explain compelling new facts. It is a genuine revolution in a particular branch of science and it represents a discontinuity in the development of that branch.

Kuhn argues that there is always a substantial investment in the old paradigm that causes a resistance to the paradigm change. This resistance is fueled by the fact that as a result of the paradigm change, knowledge accumulated up to that point may lose its significance. This makes the advantages of the new paradigm disputable, especially among those with the greatest knowledge of the old paradigm. Those who have a vested interest in the old paradigm will always attempt to extend that paradigm to accommodate new facts. Thus, the final victory of the new paradigm can only be guaranteed by a generation change.

With just a slight shift in the meaning, I use the notion of “paradigm” as “coherent tradition of software development,” and argue the current movement toward software evolution and agile and iterative processes of software development represent nothing less than a paradigm change. In order to describe the scene in which this change is taking place, it helps to review the history of software technology.

Software separated in the 1950s from the underlying hardware and emerged as a distinct technology, turning out independent products and requiring a specialized set of skills. Original programmers were recruited mostly from the ranks of hardware engineers and mathematicians who used ad-hoc techniques they carried from their former fields. The first paradigm change occurred in the late 1960s. It was precipitated by the fact that the ad-hoc techniques did not scale up to the large systems. The situation is vividly described by Brooks in [2] where the demands of the new operating system OS/360 taxed the limits of the programmers, project managers, and the resources of IBM Corp. The result of the crisis was a paradigm change that established a new academic software engineering discipline and introduced the waterfall metaphor into the production of software.

The waterfall metaphor is widely used in the construction industry and product manufacturing. It requires the developer first to collect the requirements that describe the functionality of the future product, and then to create a design that will be followed during the entire construction. When the product is finished, it is transferred to the user and any residual problems that may surface afterward are resolved through maintenance. The waterfall metaphor is an intuitively appealing metaphor. Common sense dictates to try to avoid the expensive late rework; requirements elicitation up front and good design based on these requirements lessens the need for the expensive rework. Waterfall became the dominant paradigm through which the software development was viewed.

However, in the context of software development, waterfall is beset by a serious problem of requirements volatility. Requirements are not fully known in advance and are often added during the course of the project. For example, Cusumano and Selby [3] found that 30% of the requirements for Microsoft projects were not elicited in advance, but emerged only when the development was already under way, possibly as a result of the developer’s learning. Any design based on volatile requirements must of necessity be only temporary and cannot guide developers through the entire development process. Additional data indicated that in spite of widespread use of waterfall, there were very few successful projects [7]. In an oft-quoted report, the Standish Group revealed that in 1995, 31.1% of all software projects were canceled, 52.7% were “challenged” (completed only with great difficulty, and with large cost or time overruns), and only 16.2% could be called successful. Obviously, the waterfall metaphor did not solve the problems of software development.

Incremental Change Design:

Conclusion:

The old waterfall paradigm tried to freeze requirements for the duration of software development and created a situation where requirements volatility caused too many project failures, while the new paradigm addresses this shortcoming by its emphasis on software evolution. The new paradigm brings a host of new topics into the forefront of software engineering research. These topics have been neglected in the past by the researchers inspired by the old paradigm, and therefore there is a backlog of research problems to be solved.