BeyondProcessToStructureAndValues
A good organization isn't just about process, at least not in the sense that the term is used in ISO 9000 series organizational work. Process emerges from structure, and structure from values.
Swieringa and Wierdsma are systems thinkers who describe several types of organizations [BibRef-SwieringaWierdsma1992]. They describe organizations as organisms that exhibit certain behaviors, driven by principles, insights, and rules, with the goal of producing certain results. Organizational learning shows up as a change in behavior. There are three kinds of learning: single-loop, double-loop, and triple-loop learning.
In single loop learning, collective learning causes the rules to change ([BibRef-SwieringaWierdsma1992], p. 37). They note that "[m]any of the measures applied in industry to improve quality, service and customer relationships take place at the level of single loop learning." But they note that it is a surface effect: "No significant changes take place in the strategy, the structure, the culture or the systems of the organization." It is a question of changing the how, but hardly ever the why. There is a hope to improve, largely by doing more of the same, but doing it better.
In double-loop learning, the focus is on learning at the level of insight. Now the focus moves to why, on knowledge and understanding rather than just "improving." They also call this renewal learning since it relates to a renewal of insights in the organization.
Last is triple-loop learning, which is about the organization's identity, which they call organizational development. It asks: what kind of business do we want to be? It asks: what are our values and principles?
All of these modes can be beneficial. The deeper one goes into learning, the longer it takes (single loop can take place over days, double loop over months, triple loop over years.) Whereas single loop is about process and reaction, the other levels deal with the structure of the organization and with learning how to learn.
Learning is a process, and one must learn how to learn. The patterns in this book are learning tools, and moving from pattern to pattern is a learning experience. Most of the patterns tend to be double-loop. At the core of this learning process is the Fundamental Process of PiecemealGrowth. This focus on organizational learning distinguishes organizational patterns from other organizations such as Extreme Programming [BibRef-Beck1999] which, though rooted in principles, imposes the principles from without instead of from within the organization. And it offers no process for "learning" one's way to success, supposing instead that the incorporation of all of the principles and practices (in some unspecified order over some unspecified amount of time) can lead to success. Patterns put the organization and its members, and their collective talents, insights and intelligence, at the center of the learning process.
The Shortcomings of Process
Many traditional approaches to software productivity have focused on processes; what steps are taken, and how they are executed. Indeed, the implication is that if the process is followed, then the software will be high quality, and will be developed efficiently. Let's examine this premise in more detail.
If following the process produces high quality software, then what is the cause of failures in the software? It must be either that the process is not being followed, or that the process is deficient somewhere; the process missed something. If you think this is far-fetched, think again: one of the authors once attended a briefing session for an upcoming ISO 9001 audit. The leader stated that everyone should remember that the person was not being audited, but rather the process and compliance to it. This statement was given to reassure people, and it probably did provide comfort--temporarily. For after all, if processes can be tweaked to handle every eventuality, then developers stand in grave danger of being replaced by programming robots.
But we all know that software developers are in no danger of being replaced by robots, because software design is such a highly creative activity, done by highly intelligent people. And this makes process much less relevant. In fact, highly intelligent people have been known to ignore or subvert the official process when it does not apply to them. Every organization has two processes: the official process, and the one actually followed. One of the authors once interviewed a group of key developers from several different projects. Although used processes that called for design documents and design reviews to precede coding, each one admitted that they completed the code first, and then wrote the design document and reviewed it — "so we can check off the design review box." They had found that for them, design documents were not relevant, and it was easier to write them once the design was instantiated in the code.
Structure
So if process has little impact on organizational effectiveness, what does? It turns out that the structure of the organization is more stable over time, and is a better indicator of effectiveness. This is the break between single-loop learning and higher order feedback loops. One reason for this is that the structure of an organization reflects its values, or principles, and values drive the organization. Some examples illustrate this.
Prior to the breakup of AT&T in 1984, the Bell System was a monopoly, and one of its core values was to provide telephone service to everyone, all the time, at a reasonable cost. This philosophy led to groups dedicated to extremely high availability hardware and software. On the other hand, because cost and speed were non-issues in a monopoly, organizations became bloated, and were not structured for speed.
In 1993, Borland was in the midst of developing QuattroPro for Windows (QPW). Risk taking, hard work, and individual closeness were important, even to the point that employees played jazz with Philippe Kahn, the president. Not surprisingly, the QPW team showed some of the tightest communication coupling we have ever seen.
Organizations that have a strong commitment to satisfaction of customers often have customer, surrogate customer, product support, and customer service roles that are tightly coupled to the rest of the organization. A thread that runs through all our patterns, and which is fundamental to the principles of the pattern languages presented here, is that the focus is on product.
Neither the development processes, nor the internal documentation are delivered to the customer. The customer does not pay for elaborate project plans or architecture documents or, if they do, it is probably a sign that something is more deeply wrong than such measures can ever address. These patterns don't focus on the development of a process, nor do they advocate having a process organization. The patterns strive to manage the resources that go into internal documentation. The focus is always on product. A product has strong structural elements, and that structure reflects itself in the organization. Aligning these structural elements through good communication practices is what we believe offers the key to effective development.
Values: The Human Element
Heeding the communication problem implies attentiveness to human issues. While individuals sometimes stand out in history for their accomplishments, most great things are done in groups, teams, and societies, guided by cultural or societal norms. To the degree the group has a common goal, a common vision — a UnityOfPurpose -- the organization can come together to do great things. This is perhaps the deepest hallmark of a great organization: it works as one mind, to some purpose: triple-loop learning. Perhaps the second deepest hallmark is the dedication and care with which they pursue this vision, including a dedication to learning and improvement.
Once in a great while, great minds think alike. But for the rest of us, it takes communication to align minds, motives, and methods to build an effective team. Communication is a complex human activity, filled with social context, psychological complexity, and emotion. So building a communication structure that supports the building of a product is also a complex activity. It requires a cultural setting conducive to effective communication. There is no guarantee that any set of guidelines can produce such an environment. However, these patterns can provide one foundation for an effective communication environment by defining a development culture suited to the needs of software development, taken from projects attentive to human needs. They also provide structures that can contribute to reduction in time to market, in solving the right problem, and in meeting other customer expectations. The patterns can contribute to a high(er) context culture where there are shared patterns, and therefore a shared vocabulary and shared culture, rather than a mechanical or bureaucratic environment full of rules that are either inhuman or arbitrary.
The focus explicitly is not on process. Process is a good tool in mature domains with predictable steps. In other domains, it's not clear what good process portends for product quality. A flexible process can reduce interval and contribute to good communication in a domain that must deal with change. But one can't just install a process; a process must emerge from the structures of communication and production beneath it. And the structures, in turn, are held in place by the values of the organization. Thus the values of an organization are the foundation of not only what is done, but how it is done.
Some organizations' values are rooted in making money. If such values are pervasive, the organization may have strong links to marketing or sales roles. Customer roles are likely to be present, but less central than customer satisfaction-oriented organizations.
Many organizations we see value management highly. Manager roles show up in the middle of organizational diagrams, sometimes even trumping the critical producer roles such as developer. When we probe, we often find that people are rewarded more for management than for development.
It shouldn't be surprising that most of our patterns have a strong, although not obvious impact on the values of the organization. They go beyond the superficial processes to get at the heart of an organization. Yet changes at this level do not come easily. It may take years, or perhaps a crisis, to shake the foundation of the organization — its values.