Preface

We are satisfied by doing real work. Software is like a plant that grows: You can't predict its exact shape, or how big it will grow; you can control its growth only to a limited degree. There are no rules for this kind of thing--it's never been done before.- Charlie Anderson, Architect, Borland Quattro Pro for Windows.

You will find no books on the bookshelf here that tell you how to start up a new discipline. Software has been seeking its own way as a relatively young discipline for the past 40 years. Every new discipline struggles to find practices suitable to its survival and growth. Sometimes this struggle is incremental. Sometimes disciplines undergo more substantial shifts in process, structure and values that break more with the past to explore new ground. What Charlie Anderson said above about the Borland Quattro Pro for Windows effort in particular applies to the rhythms of software development in general. Get ready for change, for it will come tomorrow.

The most exciting advances in science go hand in hand with radical social change. The move from classic physics to quantum physics precipitated from a crisis in physics. We talk about the software crisis, yet no individual crisis in software -- let alone the Software Crisis, whatever that might be -- has precipitated the same kinds of change that we associate with great advances in science. Software development has perhaps yet to face its first true crisis that leads to the first true industry-wide systemic change.

But that doesn't mean that software is static. We can identify different faces of change in software development over the past five decades. Our interest in this book is what software development has learned about itself from an organizational and social perspective. Software development is perhaps working in its fourth social style of system development. Yet what is really interesting about these social styles is their ties to technical advances in the art. The first style of software development goes back to the first computers that were programmed manually with console switches. The second style came with the advent of programming languages that allowed scientists to work individually or in small teams, interacting with the machine through a language. In the third style, what we learned from hardware design and manufacturing carried over into software. Formal processes drove development, management was visible and explicit, and both the system and the organizations that worked on the system were highly hierarchical. Now we are in the fourth style: one that breaks down hierarchy, that features dynamic social structures and communication paths, and that values immediacy. This fourth style often bears the label "agile," but that is just one of many characterizations of a broad new way of developing software that has emerged over the past decade.

Yet human endeavors tend to take the same shape century after century, and the fact that human endeavors all have the common element of human nature bounds the range of human undertakings. For example, most organizations have leaders, and cliques, and their own little rituals, their own nuances of meaning for terms of the trade, a correspondence between physical space and organizational structure, and hundreds more. The organization of any major human endeavor follows basic laws of efficiency of communication, span of control, xenophobia, specialization, and other sociological forces that drive most large-scale projects to similar practices and structures. Much has been made of the similarity between vernacular housing architecture and the construction of software systems. [BibRef-CoplienDevos2000]. The same may be true for the organization of any social animals, but we leave verification of that claim to readers better tooled in those disciplines than we are.

Going deeper yet, the systems of nature have common rhythms and trends that underlie their emergent complexity. These properties come from the structure of the organization: the deeply held relationships that define the organization as a social entity. Senge [BibRef-Senge1990] popularized this aspect of social behavior in a discipline he called systems thinking, a discipline that goes beyond our everyday disciplines that are based on a simplistic relationship between cause and effect. Many early attempts at software process improvement relied on this simplistic cause-and-effect relationship, particularly as the third paradigm of software development started to take hold in the industry. Most ISO 9000 process improvement efforts were run this way: find what we're doing wrong, find the place in the process that's wrong, and make it right. There was rarely any notion that the process as a whole might be wrong and that, for example, a step-by-step process should be replaced by a more reactive, agile process. And it was heresy to conjecture that process itself might be the wrong formalism to capture the crucial properties of efficient, effective systems.

When we started this work in the early 1990s, we started with documented research that showed serious shortcomings in process-based approaches such as ISO 9000. We asked: if process isn't the answer, then what is? We chose organizational structure--and, in particular, the structure of relationships between roles--as the basis for system understanding. All systems are about relationships, and most disciplines that study systems study the relationships in those systems. The idea worked. What's better, the technique didn't displace the process-based approaches in existing organizations (it's always hard to tell an organization to stop doing something they think is helpful, anyhow), but complemented them by adding insight into deep structure that explained behavior at the process level. So if you already have a process improvement program in place, this book can add an enriching dimension that builds on your own culture and helps develop your peoples' insights into that culture.

Patterns provide a way to capture both the broad, invariant practices of socially built artifacts as well as the specialized practices of individual disciplines, along with an understanding of how those practices build on each other. Long before Alexander started using patterns for the field of architecture, anthropologists were using them to describe human social structures [BibRef-Kroeber1948]. The pattern languages in this book combine the timeless human structures that transcend disciplines with the best practices of contemporary software development. These patterns are all empirical: they capture the major rhythms and structures of successful software-intensive organizations today. Many of the patterns come from our own research, but we also incorporated patterns from other authors working in the same field. This is a collected work and in many ways reflects a community-wide effort.

There are two equally valid views of this book: as a guide to organizational improvement, and as a record of the "best typical" software development structures of the fourth social paradigm of software development. Most readers will use the book in the first sense. However, it is our hope that this book reflects a well-enough grounded view of contemporary software development to serve as a touchstone that records what life was like in software development organizations in the late twentieth and early twenty-first centuries. Fifty years from now, will this book be a sobering admonishment to the industry? Will we just chuckle at how things used to be? Or will time leave this fourth-generation culmination of software progress largely untouched? In that spirit, we greet the rare reader who finds an archival copy of this tome on a dusty bookshelf in the middle of the twenty-first century, and we salute your efforts to use this understanding to further improve the lot of the organizations of your time and place.

Our sense of history extends in the other direction as well. Christopher Alexander's book includes a picture at the beginning of every pattern [BibRef-Alexander1977]. Each picture sets a broad tone for the pattern that follows. We wanted that same feeling for this book, and we strove to include a picture for each pattern. We initially felt that the social network diagrams would suffice as pictures, but that gave the book an "academic" feel that left a bad taste in our mouths. Paul Bramble turned us on to the prints and photographs division of the Library of Congress (http://lcweb.loc.gov/rr/print/catalog.html), which provided a wealth of vintage photos. Most come from a collection of depression-era photographs sponsored by the Farm Services Administration. Some of the photos are strikingly poignant or relevant to the patterns; that was a surprise, since they come from an era that predates the software culture that is such a large part of this book. But we feel that the age and human element of the photos lends an overall charm to the book that totally would be lacking in the social network diagrams. They also give us a feel of the timelessness of the basic human issues facing organizations of any culture, era, or ideology. This is not a book about ideology, but about human nature. Last, the pictures might help make the patterns more memorable for you: pictures are powerful association tools.

Many of the pictures have a military theme. Please remember that the purpose of patterns is human quality of life and comfort, and that patterns help us capture as much learning from history's tragedies as from its moments of peak culture. Also remember that the earliest patterns of human organizations (such as [BibRef-Clavell1989]) have roots in military organizational structure.

Acknowledgments

In doing this book we view ourselves as editors and chroniclers of others' work and ideas. There are literally thousands of people who contributed to this book through their participation in the empirical studies from which we mined the patterns. We don't have all of their names here, but we appreciate all of them for their time and energy. Especially noteworthy were the Borland QPW Group, coordinated by David Intersimone, a remarkably productive project at AT&T led by Judy Tschirgi, highly effective projects at Schlumberger in Oslo, where we were hosted by Lise Hvatum, and the group managed by Richard Gabriel at ParcPlace Systems that was undergoing a sobering restructuring at the time we visited Richard.

There are other people who put even more of their own energy into this book by building things for us and doing things with us. Brendan Cain was one of the original members of the Pasteur research effort at Bell Laboratories, and he wrote many of the original analysis tools. Anthropologist Peter Bürgi was another early member of the research team; he contributed many of the insights on schismogenesis and on other direct parallels between the corporate world and the more "traditional" world of cultural anthropology. Tom Burrows' early work on the GIL and Romana environments at Bell Labs provided a platform for many of the early tools. We are grateful to Steve North for the dot tool that not only provides good supporting visuals that map out the pattern languages, but which was a key research tool in its own right.

We are particularly indebted to authors who let us reproduce their patterns here. Pieces of the patterns of Alistair Cockburn, Ward Cunningham, Bruce Whitenack and Steve Berczuk have all made their way into this collection. Thanks for sharing, folks. Gerard Meszaros wrote several patterns, including ArtifactOwnership_, ArchitectureDefinitionTeam_, and ArchitectureOrganization_, whose contents filtered into many patterns in these pattern languages.

Other people steered us in the right direction. Diane Grinnell pointed us to the references on organizational incest and its parallels to dysfunctional constellations in family therapy. Tom Stone, then at Addison-Wesley, pointed us to some dynamite references on organizational learning, and in particular the studies that came out of Royal Dutch Shell. Bindu Rama Rao of Lucent pointed us to the work by Kroeber, which gave us strong ties from patterns back to the world of anthropology. Urvashi Kaul of Allstate developed pattern taxonomies that shaped how we organized these patterns into pattern languages. But most of all, we're grateful to Moody Ahmad, who first gave us a hint back in the mid-1980s that software development research should be investigating not just the technical issues, but the human issues as well.

Dozens of people reviewed these patterns in writers workshops at pattern conferences and local pattern groups. Additionally, we enjoyed the feedback of a team of focused and thorough reviewers who weren't afraid to give us the benefit of their opinion in places where they felt we were misguided, and they were usually right. Gerhard Ackermann at Siemens in Vienna offered a wealth of first-hand insights in organizational growth and repair. Paul Bramble and Ian Graham were our two main manuscript reviewers for the late versions of the manuscript, but they offered a lot of useful advice on the organizational of the book. Joshua Kerievsky pioneered the conversion of these patterns to Alexandrian form with his outstanding editorial efforts on SoloVirtuoso_, SizeTheOrganization_, and SelfSelectingTeam_. And many thanks to our favorite Mercenary Analyst, Betsy Hanes Perry, for her contributions to PublicCharacter_. Other key early reviewers included Jay Stagnone, ...

There are others who worked with us as partners along the way, and whose editorial feedback and suggestions were fundamental to the shape of the book. Martine Devos (then of Argo in Belgium, and currently of Avaya Labs) and Steve Berczuk invested much of themselves in this work, and we are grateful for their energy and dedication.

We enjoyed a good technical and organizational infrastructure to support out work. At Bell Labs, the research organizations managed by Eric Sumner, Mary Zajac, and David Weiss actively supported this work over a decade. That is a long time not only in Internet years but even by research project standards. We honor their vision, patience, and forbearance. David Weiss continued to support this work in a similar capacity at Avaya Labs. Many thanks to Universität Karlsruhe for the Wiki that we used to develop the book manuscript, and in particular to Dr. Helmut Goos. The support came in part under the auspices of the IST 1999-14191 EasyComp joint research project of the European Community, and we are grateful for that support. Research Chair John Roddick and fellow professor Paul Calder sponsored the infrastructure and time for this work while Jim Coplien spent a summer (or was it winter?) at Flinders University in Adelaide, Australia.

And of course, where would we be without our editorial support? Alan Apt has always been there in the wings, but not bugging us too much, supporting us in mighty ways. It's been a joy working with him.

This book was generated automatically from a Wiki web site using tools that generated a MIF file for transmission to the publisher.