Introduction

A happy new year to you all!

If you ask a C++ Report subscriber (or most any software practitioner) what a pattern is, most responses will contain the word "object" and "design," or some other software cue. Previous installments of the "Column Without a Name" have made it clear that patterns needn't be object-oriented. Here, we go one step further and completely divorce patterns from software architecture. This month, we introduce patterns for organization and process.

If you really appreciate object-oriented design (or just plain design), you appreciate why it's important to study organization and process. Good design attacks complexity with abstraction, but must balance abstraction with good module de-coupling while meeting business needs. Why are abstraction and de-coupling important? The software itself doesn't care about how abstract or de-coupled it is; in fact, abstraction and modularity often come at some performance expense. We care about these properties for their human implications. Good de-coupling helps developers work independently, unencumbered by excessive meetings and negotiations.

Process and organizational patterns have become surprisingly popular, given that software architecture concerns dominate the pattern literature. The Generative Development-Process Pattern Language of PLoP/94 [Coplien95a] raised many eyebrows and defied the prevailing conventions of the time. That process pattern language took the pattern concept beyond the structure of software itself, and conveyed structures of software development organizations. Process patterns are now finding a home in the sector I reluctantly refer to as "business process re-engineering": these patterns have started to draw critical acclaim, and many major corporations are applying such patterns.

New dimensions of process and organization

Software engineers[1] have been studying development process in earnest for many years, but usually have cast process in computer science terms. For example, Osterweil believes that the abstractions and techniques of program analysis and development are well-suited to process development and enactment: that, in short, organizing people and orchestrating their actions is programming [Osterweil87].

But organizations build on quirky, uniquely intelligent building blocks called human beings, whose dynamics are usually a lot more interesting than a sequence of time-ordered tasks. In the studies of development processes and organizations at AT&T and elsewhere, we usually find wide variation between successive enactments of the same process even in organizations with ISO process certification or healthy capability maturity model ratings [Cain+93]. What's worse, we find only superficial correspondence between actual practice and the written process description (the "process program") in these organizations. The temporal ordering of development tasks is unstable over time, as is the constitution of some of the tasks themselves. Industry interest in the literal process programming vision of Osterweil seems to be diminishing, though vestiges of it remain in the work flow community, and live on in other areas of development process automation.

One must adopt a different perspective to see organizational continuity and stable process structure, and patterns offer a perspective and vocabulary to capture this structure. Patterns offer a refreshing new perspective to complement techniques already in use.

Organizational patterns are nonetheless an old idea. One definition of cultural anthropology is the study of patterns of relationship in cultural units. As a science, anthropology has a history that goes back at least 60 years. With tongue only slightly in cheek, we can stretch the history back to Lao Tsu's studies of military and government organizations in the Tao te Ching. This discipline holds a much richer cache of abstractions and techniques than the much less mature domain of software engineering. While traditional cultural anthropology focuses on social and family units, modern anthropology also admits industrial and professional organizations as social units worthy of study (see [Brajkovich94] ).

Branching into cultural anthropology not only has utilitarian value, but serves a widely held pattern community vision. Part of the pattern vision, all the way back to Christopher Alexander himself, is that patterns should serve people by making them feel "fully alive." Alexander strove to serve human needs through architecture, the structural artifacts that shape daily interactions between people. Organizational patterns are the contemporary workplace extension of these structures: organization structures and processes portend as much for human interaction and comfort as do office building architectures. Even Alexander noted the relationship between the structure of a building and the structure of the natural social groups in the building, as we find in his patterns Wings of Light, Structure Follows Social Spaces ( [Alexander77], pp. 529 and 940), and others. In this sense, patterns of process and organization are closer to the Alexandrian ideals than are patterns for C++, for object-oriented design, or for any software structure. A software design pattern might describe how it makes life easier for the software designer or how it delights a customer (see the July/August "Column Without a Name" [Coplien95b]); organizational patterns make human concerns a primary agenda. Other patterns for requirements acquisition and training also fit these categories. These are among the most human of patterns and pattern languages.

Organizational patterns serve other facets of the pattern community vision by engaging disciplines outside computer science. When the Hillside Group first met to refine the relationship between software and Alexander's patterns, its members were conscious of their own efforts to step outside the accepted norms of their own discipline to seek analogies in building architecture. Organizational patterns borrow from the social sciences and from the business management world. Such interdisciplinary influence reflects a form of lateral thinking, and may be an important component of truly generative patterns.

This generativity--the use of indirect solutions that elicit emergent behavior in a system, instead of tackling symptoms head on--is a major feature of patterns that distinguishes them from other technical solutions (e.g., design methods, process programming and many other traditional development process analyses and formulations). I won't explore generativity in depth this month, but those who are interested can explore the discussion in the October 1995 "Column Without a Name" [Coplien95c].

Organizational Pattern Mining

Where do organizational patterns come from? We use a form of pattern mining [Coplien95c] that extracts organizational patterns from numerous successful organizations in AT&T and elsewhere in telecommunications, in aerospace, in consumer software, in the government sector--just about everywhere. A typical study (albeit of a very atypical organization) was the analysis of Borland's QPW project [Coplien94a].

Figure 1. A developer-centric organization on the left, and a manager-centric organization on the right. From [Coplien95a].

Our technique uses a CRC card session to capture the responsibilities of and interactions between organization roles. We analyze these data using sociometric analysis tools. Not only do these tools produce standard sociometric data, but they produce visualizations of the social network data such as social network diagrams and interaction grids (Figure 1). These visualizations provide much of the raw material for pattern mining. We look for patterns in interactions between organizational roles much in the same way that DePauw et al. looked for patterns of interaction in object-oriented programs [DePauw+93]. Perhaps this is Conway's Law at its best: the structure of the software may reflect the structure of the organization more literally than any of us ever imagined. By comparing social network diagrams with DePauw's pictures, we may just be able to prove that. But more about Conway's law later.

Gamma patterns to generative patterns

Search carefully for the characteristic patterns in the interaction grids of Figure 1. Can you find them? Notice that the points in the left interaction grid tend to form a horizontal rectangle, and the points in the right grid suggest a verticle rectangle. What do these patterns mean?

To understand these patterns, you must first understand the graphs' semantics. Each point on the X-axis corresponds to a role. Roles are ordered along the X-axis according to how tightly coupled they are to the organization as a whole, with the most tightly coupled roles on the left. The Y-axis is a clone of the X-axis. There is a "point" on the graph for each directed interaction from a role on the X-axis to a role on the Y-axis. The color intensity of the "points" encodes the intensity of the interaction.

What does it mean if such a graph exhibits a vertical rectangle near the Y-axis? It means that the most intense roles (those nearest the origin) are generating many requests to roles all across the process. On the other hand, if the rectangle is horizontal and lays on the X-axis, it means that roles all across the organization are generating requests to the core of the organization. That is, the orientation of the rectangle tells us whether work flows inward in the organization, towards core roles, or whether it flows outward from the core roles to the peripheral roles.

Now we can look at more detail. If the central roles (those near the origin) are managers and if work flows from the center to the periphery, this is an organization where managers wield control. If the central roles are developers, and if work flows from the periphery to the center, then the developer is in control but is responding to inputs that come from customers and other process roles. These two pictures point to two very different management styles.

We also make a subjective assessment of organizations' health, productivity (using coarse characterizations like log10(KNCSL) / staff-month)[2], and success using interviews and outside information (e.g., market reviews). We search for patterns that separate particularly successful organizations from the rest. We can usually trace these patterns to root causes either by common sense or by appealing to the large body of organizational literature. These assessments lead to patterns such as the following, derived from diagrams like those in Figure 1 (adopted from [Coplien95a]):

Work Flows Inward

Problem: Work that adds value directly to the product should be done by authoritarian roles.

Context: An existing organization where the flow of information can be analyzed. The organization exhibits a management pecking order.

Forces:

Some centralized control and direction are necessary.

During software production, the work bottleneck of a system should be at its center (in the adjacency diagram sense). If the communication center of the organization generates work more than it does work, then organization performance can become unpredictable and sporadic.

The developer is already sensitized to market needs through Fire walls and Gatekeeper (no centralized role need fill this function).

Solution:

Work should be generated by customers, filter through supporting roles, and be carried out by implementation experts at the center. You should not put managers at the center of the communication grid: they will become overloaded and make decisions that are less well-considered, and they will make decisions that don't take day-to-day dynamics into account.

Resulting Context:

An organization whose communication grid has more points below the diagonal than above it (see Figure 1).

Design Rationale:

Katz & Kahn's analysis of organizations shows that the exercise of control is not a zero-sum game ([KatzKahn78], p. 314).

The work should focus at the center of the process; the center of the process should focus on value-added activities (Developer Controls Process).

Organizations run by professional managers tend to have repeatable business processes, but don't seem to reach the same productivity plateaus of organizations run by engineers. In programmer-centric organizations, the value-added roles are at the center of the process (Developer Controls Process; Architect Also Implements). The manager should facilitate and support these roles and their work (Patron; Fire walls).

Mackenzie characterizes this pattern using M-curves, that model the percentage of task processes of each task process law level (planning, directing, and execution) as a function of the classification [MacKenzie86].

The rationale is supported with empirical observations from existing projects.

Managers should still make day-to-day decisions for the business process, and pursuant to their responsibility to "keep the pests away" (Fire walls).

(The patterns referred to by this pattern can also be found in [Coplien95a]).

Where technology and people meet

So if process pattern languages are so powerful, so attuned to human comfort, and if we believe Conway's Law that suggests that software architecture is a by-product of organization anyhow, why do we need software architecture and design patterns?

In practice, the solutions to difficult software problems have both a design component and an organizational component. This dualism can emerge in several different ways. The July/August "Column Without a Name" [Coplien95b] introduced simple code formatting patterns that resolved human forces. Those patterns present a solution in the technical domain to solve a problem in the human domain. This kind of "back door" generativity is often the sign of a powerful pattern--powerful because straightforward technical tweaks generate eagerly sought human results. That is the essence of all Alexander's patterns [Alexander77], whose architectural solutions help humans be more "fully alive."

Alexander sprinkles human concerns through the structure of technical patterns. Another approach, pioneered by Steve Berczuk at MIT, weaves organizational patterns and technical patterns together [Berczuk95b]. The technical patterns come from Steve's contribution to PLoP/94 [Berczuk95a]; the organizational patterns come from [Coplien95a] and elsewhere. Steve "includes social context as a motivating context for a pattern." He observes that both organizational problems and technical problems figure heavily in satellite telemetry software development, largely because such projects employ geographically distributed development teams. These projects lack a single architectural authority, making it difficult to apply the pattern Architect Controls Product. Because groups are geographically isolated and have widely divergent interests, it's important to localize changes to the software owned by a team (Code Ownership, Organization Follows Location). The solution is to de-couple teams through flexible architectural interfaces, employing patterns like Callback, Parser Builder, and Hierarchy of Factories. The resulting architecture and organization help reinforce patterns like Developer Controls Process and Code Ownership.

This is a delightful pattern language to read. If you believe that an OO methodology will solve your development problems, or that design patterns alone will solve your development problems, or that organizational patterns alone will solve your development problems, you won't go away from this pattern language unchanged!

As we write software design patterns, it's important to remember that the software doesn't care about its coupling and cohesion--we worry about them because of the ramifications for interaction between people. The software may not care, but the people do. Always remember that good patterns solve problems for people, and that good patterns seek to relate their human component to the reader.

Of course, technical and organizational issues are just two of many dimensions of a complex software problem. Most solutions have political, technological, psychological, and economic ramifications, to name a few. For the time being, we fold many of these "soft" perspectives into the organizational perspective, but some day we should treat these dimensions more rigorously.

Incorporating the organizational perspective in our design patterns is a quantum leap in its own right. At PLoP, I was discussing pattern categories with Brandon Goldfedder of the Dalmatian group. He recognized the need for architecture patterns, design patterns, and idioms but felt that a category called "analysis patterns" was needed as well. What he thinks of as analysis patterns, I think of as organizational and process patterns. Organizational concerns aren't a higher level concern than architectural and framework patterns: they are in a different dimension. The organizational dimension and technical dimensions overlap in complex and interesting ways: Conway's law runs rampant, as we see in Berczuk's telemetry pattern language.

When one discovers this relationship between design patterns and organizational patterns, one can see beyond rule systems of direct symptom/solution pairs to the underlying principles that drive the solutions. Alexander's great patterns are like that: they balance psychological and social forces, not just the force of gravity. There is an even deeper level of forces at work in Alexandrian patterns that goes beyond principles to values. Great organizational patterns work at this level, too: they help the organization answer the question: "who are we, and how do we want our customers to perceive us?", which is more profound than principles like: "how do we do business with customers?", which is higher than the mundane: "how many attendants should staff our complaint department telephones?"

How do organizational patterns work?

I run into many people who are secretly disenchanted with the process standards of the past few years ("secretly" because such standards wield considerable political clout). ISO process standards and the quest for an ever-higher CMM rating yield occasional improvements, but they often take a large human toll in morale, motivation, and innovation. Why should anyone embrace process patterns any more than they embrace ISO or the CMM?

Do they work?

To provide a setting for why I think organizational patterns work, it's probably a good idea to establish that they do work. Let me tell you a story about ParcPlace Systems, a company that went through quite a few changes a couple of years ago. They were in the middle of deploying a new product, and were going public at the same time. The new product moved them into new markets, markets where the old-timers had little expertise. Changing from a private company to a public company radically changed the culture: the old ways of working wouldn't scale to meet the new demands. Reward mechanisms changed: stock options that used to go to employees were now headed to Wall Street. Growth pressures had tangible results: project members were scheduled to move to a new building in the middle of their final shipping deadlines. In the old days, the organization had been a well-oiled machine. The old approach wouldn't work with the new organization and new markets, and the organization was starting to see signs that old ways were breaking down.

In the middle of this (December 1993), ParcPlace brought on Richard Gabriel as a Distinguished Scientist. Shortly after that, Dick and I discovered our mutual interest in organization analysis, and started trading notes. We discovered many common perspectives on what makes software development organizations great (see [Gabriel94] and [Coplien94a]). He worked on side projects at ParcPlace until mid-1994, when he became Vice-President of Engineering. Dick worked with the engineering organization to help it reform around the new corporate priorities.

At PLoP/94 in early August, Dick learned more about the organizational pattern language, and invited me to visit ParcPlace to talk to his engineering organization. The visit would provide me with an opportunity to collect data to support my research, and provided ParcPlace with an opportunity to introspect about their organization from a fresh perspective.

Anticipating my visit to ParcPlace, Dick left copies of the pattern language by the mail boxes at ParcPlace so people could learn a bit more "what this organizational pattern stuff is, and how it might help us." I met the ParcPlace engineering folks that October. We spent much of the day in the organizational role-play, replaying the events of a recent project development. Project members tracked the responsibilities and interactions of roles using CRC cards. I gave the organization some preliminary feedback and later made a more thorough report.

Six months later, Dick told a story at our spring Hillside Group meeting of how ParcPlace engineering had turned around. They had started using organizational pattern names in their day-to-day vocabulary. Seeing what was possible, and knowing that the patterns came from real-life successful organizations, they aspired to renewed success in their new business setting. At the time of his report, the ParcPlace engineering had rounded the corner and was on the mend: morale was up, the project was starting to serving their new markets well, and the organization was becoming productive.

Introspection, not prescription

Of course, process patterns themselves are no more a panacea than process re-engineering programs designed around ISO or the CMM. The Quality Without a Name comes from the people and the organization itself, in whom the patterns take root and become alive. There was no systematic program at ParcPlace to "install" or "implement" the development process pattern language. The change came from within the organization. When employees participated in the role play, they gained a bird's-eye view of their organization that was difficult to see from in front of their computer screen. Having the "big picture", they could start to see recurring problems--patterns--of the development process. For example, no individual role felt culpable when asking the Developer role to do an urgent task on their behalf. The role-play setting put these interruptions in perspective so the team as a whole could discover what the Developer already knew: that more than eight roles wielded enough power to unilaterally redirect the Developer's work. The Developer accomplished little beyond servicing interrupts! The organization as a whole learns, a much different dynamic than if all organizational individuals learn.

The process pattern language showed them not only that it was possible to overcome these problems, but how to overcome the problems. That the pattern names became part of their vocabulary implies that the patterns became part of their mind-set. Project members could tailor the patterns to their own context, and even bring their own patterns to bear. When they did, the organization righted itself.

I think the big picture perspective and the patterns themselves helped the project think of itself in terms of principles, and perhaps values, instead of dwelling on surface symptoms. In this sense, the change was truly generative: changes in values and principles lead to lasting changes in behavior. Changing behavior for its own sake often leaves principles and values untouched. In the long term, the values and principles win out--even if the organization gives lip service to the changes dictated by a process program. This is probably why so many organizations exhibit process behavior that departs from their formal process documents in major ways.

Signposts and such

Many thanks to Steve Berczuk of MIT and to Dave Smith of ParcPlace/Digitalk for their comments on this month's column. Many thanks to Richard Gabriel for many past exchanges about this topic.

For more details on organizational patterns, look forward to the Annals of Software Engineering, to be published December, 1996.

PLoP/96 will again be at Allerton Park in Illinois, September 4-6, 1996. Doug Schmidt (schmidt@cs.wustl.edu) is program chair, and Brian Foote (foote@uiuc.edu) is conference chair. And for our European readers, the first EuroPLoP will be held in Aarhus, Denmark, August 7-11 this year; for more information, contact Frank Buschmann (Frank.Buschmann@zfe.siemens.de) or Bruce Anderson (bruce@essex.ac.uk).

References

[Alexander77] Alexander, Christopher, et al. A Pattern Language. New York. Oxford University Press, c1977.

[Berczuk95a] Berczuk, S. P. "Handlers for Separating Assembly and Processing." In Copolien, J. O., and D. Schmidt, eds., Pattern Languagaes of Program Design. Reading, MA: Addison-Wesley, 1995, 521-538.

[Berczuk95b] Berczuk, S. P. "A Pattern Language for Ground Processing of Science Satellite Telemetry." Draft Proceedings of PLoP/95, Allerton Park, Illinois, 6 September, 1995.

[Brajkovich94] Brajkovich, L. F. "Sources of social structures in a start-up organization: work networks, work activities, and job status." Netherlands: North-Holland Elsevier Press, c1994, SSCI 0378-8733(93)00237-Q.

[Cain+93] Cain, B. G., and J. O. Coplien. "A Role-based process modeling environment." Proc. of the Seconod International Conference on the Software Process, Los Alamitos, California, February 1993, IEEE Computer Press, 125-133.

[Coplien94a] Coplien, J. O. "Evaluating the Software Development Process." Dr. Dobb's Journal, Vol. 19, No. 11, October 1994.

[Coplien95a] Coplien, J. A Development Process Generative Pattern Language. In Coplien, J. O., and D. Schmidt, eds., Pattern Languages of Program Design. Reading, MA: Addison-Wesley, 1995.

[Coplien95b] Coplien, J. "The Column Without a Name: Software Development as Science, Art, and Engineering." C++ Report 7(6), July/August, 1995, ff. 14.

[Coplien95c] Coplien, J. "The Column Without a Name: Pattern Mining." C++ Report 7(8), October, 1995.

[Gabriel94] Gabriel, R. P. "Productivity: Is There a Silver Bullet?" Journal of Object-Oriented Programming 7(1), March/April 1994, 89-92.

[DePauw+93] DePauw, W., R. Helm, D. Kimelman et al. Visualizing the Behavior of Object Oriented Systems." SIGPLAN Notices 28(10), 1993, 326-337.

[KatzKahn78] Katz, Daniel, and Robert L. Kahn. The Social Psychology of Organizationos, 2d ed. c1978, John Wiley and Sons.

[MacKenzie86] Mackenzie, K. D. "Organizing High Technology Operations for Success." Managing High Technology Decisions for Success, J. R. Callahan and G. H. Haines, eds., 1986.

[Osterweil87] Osterweil, Leon. "Software processes are software, too". Proceedings of the 9th International Conference on Software Engineering, Monterey, CA, USA, 1987, pp. 2--13.