Publications‎ > ‎Patterns‎ > ‎

TopTenPatterns


Organizational Patterns: Building on the Agile Pattern Foundations

On 1 June 2005 Cutter offered the paper "Organizational Patterns: Building on the Agile Pattern Foundations" for free on their web site, provided that you left your credentials. The publication agreement, which stipulated such free access, had a duration of one year. That year is up, and the publication is still available from Cutter at $US 150.00. As Cutter holds the copyright they are the source for this work. The abstract of the work is:

What really works in software development management? Practices and deep organizational structures -- what the authors refer to as organizational patterns -- that result in customer satisfaction and lay the foundation for organizational adaptability and agility. The following Executive Report by James O. Coplien, Neil B. Harrison, and Gertrud Bjørnvig discusses the top 10 patterns that successful organizations and teams have used and provides a framework within which you can customize these practices for your own enterprise.

The executive summary starts:

You're a project manager, an executive, a line manager, or maybe even a developer, and you have more than a hunch that you could deliver high-quality software faster. You believe that you really could meet those schedules and that you really could meet requirements regularly, if only ...

The work describes what Organizational Patterns are and presents the ten patterns that research has found most strongly correlate to business success. The patterns are drawn from the popular Organizational Patterns of Agile Software Development. The top ten list itself has been a frequent topic of many public talks, and is as follows:

  1. UNITY OF PURPOSE.
  2. ENGAGE CUSTOMERS.
  3. DOMAIN EXPERTISE IN ROLES.
  4. ARCHITECT CONTROLS PRODUCT.
  5. DISTRIBUTE WORK EVENLY.
  6. FUNCTION OWNER AND COMPONENT OWNER.
  7. MERCENARY ANALYST.
  8. ARCHITECT ALSO IMPLEMENTS.
  9. FIREWALLS.
  10. DEVELOPER CONTOLS PPROCESS.

Follow the links to the patterns themselves below. These patterns are a local cached copy of the content at the longstanding EasyComp web site whose existence long predates the Cutter article. Or see all the patterns in Organizational Patterns of Agile Software Development, available at Amazon.com for $37.35.


Unity of Purpose


...the team is beginning to come together. Team members may come from different backgrounds and may bring many different experiences.

* * *

Many projects have rocky beginnings as people struggle to work together.

Often, the people have different ideas about what the final product should be. In fact, the final product may well be a pretty fuzzy concept. Yet the people must have a consistent view of the product if there is any hope of it getting done.

Each person is different and has different views and opinions. They come with different backgrounds and experiences. They must learn to work together.

It is important to get off to a good start -- initial impressions, good or bad, tend to be lasting.

Therefore,

The leader of the project must instill a common vision and purpose in all the members of the team. This "leader" can be a manager, or the PatronRole, or a customer advocate, but should be someone who holds the team's respect and who has influence over the team's thinking. This is an overt action; you can't count on it happening automatically. The leader should make sure everyone agrees on the following: What is the product supposed to do? Who are the customers, and how will it help them? What is the schedule, and everyone must feel personally committed to the schedule. Who is the competition?

An important component of this action is to identify strengths of the team, and use them as rallying points as well. This is related to identifying the challenges and competition, and uniting to overcome and surpass them.

As time goes on, the Unity Of Purpose continues to emerge from ongoing dialogue within the team and with customers and other stakeholders. While the team leader primes the pump, team dynamics take over and keep things going.

* * *

The obvious result is that the team is on the same page, and is working together, rather than at cross purposes. But a more subtle, but probably more powerful effect is what it does for the morale of the team. The best teams tend to feel that they are somehow better than others -- and they work to prove it!

This pattern relates to some deep-seated principles and values of organizational health. There may be no more important single property of an organization than that its members have a shared vision they are motivated to achieve. Communication--which receives the bulk of the attention in this book--is just a means to achieving that shared vision. Unity Of Purpose is a deeper principle even than effective communication; communications are just a means to Unity Of Purpose.

Related Patterns:

SharedClearVision ([BibRef-Bramble2002], p. 80) notes the importance of a clear vision in creating unity, from the point of view of writing use cases. SelfSelectingTeam outlines how a team should come together, but that alone is insufficient to achieve Unity Of Purpose. LockEmUpTogether helps achieve unity, particularly of architecture. A GateKeeper can help. It can help the team be more unified on what requirements to EngageCustomers. This pattern sets up CompensateSuccess: it's much easier to compensate success when everyone knows what success means. And while Unity Of Purpose is important to galvanize the team, effective team dynamics can come only if every team member is also valued as an individual: HolisticDiversity comes to play here.


Engage Customers




Clerk measuring customer for a suit of clothes, San Antonio, Texas.

A friend of one of the authors once designed and implemented the user interface for a large system. He got input from customers on how to make it useful for them. Unfortunately, the requirements writers had a different idea, and made him remove the features the customers liked. But then the customers asked for the missing features, and the requirements writers were forced to relent. I guess it didn't help relations between my friend and the requirements writers.

...an organization is in place, and its Quality Assurance function has been generally shaped and chartered. The Quality Assurance (QA) function needs input to drive its work. Many people in the enterprise are concerned about quality.

* * *

It's important that the development organization ensures and maintains customer satisfaction by encouraging communication between customers and key development organization roles. This isn't the responsibility of any single "customer satisfaction" organ, but the need pervades the entire organization structure. Most organizations are averse to direct contact between developers and customers, fearing that the developers are "loose cannons on deck" who will promise to deliver things that go beyond the scope of a job.

Yet you can't know all the requirements up front, so developers need to keep going back to customers for more information--and customers need to keep coming back to developers with their insights, particularly when developers BuildPrototypes. Requirements changes occur even after design reviews are complete and coding has started.

Many organizations depend on their marketing organization to provide requirements and needs. But marketing doesn't provide design data (BibRef-BeyerHoltzblatt1998], p. 30). The best that marketing can do (or should do) is to understand what will sell and why people will buy what you want to sell. Designers in turn must understand how people will use the product in a way that creates value for them. Good value sometimes leads to good market potential, but marketing usually looks at other factors (brand name recognition, product name and posturing n the market) about which designers care little.

Missing customer requirements are a serious problem: most problems in software systems can be traced to requirements problems ([BibRef-Daley1977]; [BibRef-Boehm1976]). Yet it seems like so much effort to elicit them -- which is work that is not directly producing a marketable artifact. It seems like makework and overhead.

Customers are traditionally not part of the mainstream development, which makes it difficult to discover and incorporate their insights. Yet customer contact correlates with project success [BibRef-KeilCarmel1995].

Trust relationship between managers and coders are often strained, so you don't want them to be the sole intermediary between developers and customers.

Therefore:

Closely couple the Customer role to the Developer and Architect, not just to QA or marketing. In short, developers and architects must talk freely and often with customers. When possible, engage customers in their environment rather than brining them into your environment.

Two things are necessary for this to happen: opportunity and culture. Developers must have the opportunity (and the means) to communicate with customers. They should meet customers personally to establish trust and free flow of communication.

But these visits will be superficial if the organization culture builds walls between customers and developers. In particular, if system requirements must go through a lengthy formal process to be approved, the developer will be hamstrung -- unable to respond to customer requests. Therefore, the organization must develop a culture where developers have some latitude to respond to customers. This is not saying, however, that all control of requirements should be relegated to the developer. Order is necessary.

Beyer and Holtzblatt note that "many common ways of working with customers remove them from their work." ([BibRef-BeyerHoltzblatt1998], pp. 36-7). One way to help this is by "putting designers and engineers directly in the customer's work context" ([BibRef-BeyerHoltzblatt1998], p. 20). This is particularly important if you are using customer engagement to create wholly new market directions for the enterprise, rather than refining existing work. Putting developers in the customer work environment also trains developers' intuition about good design and good human interfaces, and this intuition can fill in when specific detailed requirements are unavailable [BibRef-BeyerHoltzblatt1998], p. 35).

Language is a key element of culture that can smooth customer engagement if treated properly, and smother it if treated badly. Don't make your customers learn UML or other technical notations; do your best to learn their language and to communicate with them in the terms of their culture.

QA can monitor the relationship to keep the direction within contractual business limits, while allowing a free flow of insights back and forth between developers and customers. Such communication can often flow unimpeded; however, see the pattern GateKeeper.

Note that this pattern is all about relationships and culture. It is the culture of respect for and communication with customers that makes the communication effective, for example, during the writing of use cases, as described in ParticipatingAudience ([BibRef-Bramble2002], p. 35).

* * *

This pattern supports requirements discovery from the customer, as required by ScenariosDefineProblem and BuildPrototypes. Other patterns like FireWalls also build on this pattern. The pattern RecommitmentMeeting is a more formal derivative of this pattern in a different context.

A good understanding of customer needs can avoid rework after implementation is done. While it is also important to continuously engage customers through each development episode of iteration, early understanding helps launch the effort in the right direction. A Navision project in Copenhagen felt that improvements in customer engagement helped save time on their development schedule (from a draft pattern "Scandinavian System Development" by Flemming Pedersen, 24 January 2002).

This was a strong pattern in the Borland Quattro Pro for Windows case study. Also, see [BibRef-Floyd1992] and in particular the works of Reisin and Floyd therein.

Some processes and methods are founded on customer engagement, such as IBM's Joint Application Development. Other methods are conducive to customer engagement, such as Cunningham and Beck's CRC design technique. Other methods, and especially most CASE-based methods, are indifferent or harmful to customer engagement.

Even some of the best customer engagement techniques tend to stop once they achieve some level of contractual agreement about what is to be delivered. Customer engagement in agile processes goes far beyond that. Developers need to assimilate the context in which their product will be used: this is called contextual design. Contextual design means gathering data on customers' models of how they do their work rather than creating models of how the program will solve the problem. Use Cases are about the latter; contextual design is about the former. See [BibRef-BeyerHoltzblatt1998].

The pattern is "Engage Customers", in the plural, to support a domain view and to avoid being blind-sided by a single customer.

The project must be careful to temper interactions between Customer and Developer, using FireWalls, GateKeeper, and the QA organizational presence as in EngageQualityAssurance. A big part of interacting with the customer is to learn how they want to interact with the project as the unfolding software uncovers problems in requirements and systems engineering (see ApplicationDesignIsBoundedByTestDesign).

Note that "maintaining product quality" is not the problem being solved here. Product quality is only one component of customer satisfaction. Studies have shown that customers leave one company for another when they feel they are being ignored (20% of the time), or because the attention they receive was rude or unhelpful (50% of the time). For customers having problems that cost over $100 to fix, and the company does not fix it, only 9% would buy again. 82% would do business with the company again if the problem was quickly resolved after they complained. (The source for the former pair is The Forum Corporation; for the latter pair, Traveler's Insurance Company [BibRef-ZuckermanAndHatala1992].)

Joe Maranzano [BibRef-Maranzano1992] notes that this pattern probably should come earlier in the language. However, it is important that the project roles be defined first--particularly those that interact with the customer, and those that are driven by customer input (such as Quality Assurance). Said in another way, the organization exists to serve the customer, so the organization should be in place before the customer is fully engaged.

This pattern works only if customers are directly accessible to the development team. If that is impossible for business reasons or because of geographic separation, consider SurrogateCustomer.


Domain Expertise in Roles



Naval air base, Corpus Christi, Texas. A top notch mechanic, Mary Josephine Farley expertly rebuilds airplane engines. Although she's only twenty years old she has a private pilot's license and has made several cross country flights.

...you know the key atomic process roles (FormFollowsFunction) including a characterization of the Developer role.

* * *

Matching staff with roles is one of the hardest challenges of a growing and dynamic organization. All roles must be staffed, and, all roles must be staffed with qualified individuals. Just as in a play, several actors may be assigned to a single role, and any given actor may play several roles.

You'd like to use domain-inspecific qualification criteria like college grades or years of experience to qualify people for jobs. Such an approach gives the project flexibility in staff allocation; it helps it avoid being overly dependent on individual skill sets and experience. In short, the hope that such criteria might work provides project managers a basis for keeping the project from becoming overly dependent on certain individuals; such individuals may leave or may hold the organization hostage for higher salaries or to see their own policies implemented unilaterally. Yet successful projects tend to be staffed with people who have already worked on successful projects.

Spreading expertise across roles complicates communication patterns. It makes it difficult for a developer or other project member to know who to turn to for answers to domain-specific requirements and design questions.

Therefore:

Hire domain experts with proven track records, and staff the project around the expertise emodied in the roles. Teams and groups will tend to form around areas of common domain interest and focus. Any given actor may fill several roles. In many cases, multiple actors can fill a given role.

Domain training is more important than process training.

Local gurus are good, in all areas from application expertise to expertise in methods and language.

* * *

This is a tool that helps assure that roles can be successfully carried out. It also helps make roles autonomous. Empirically, highly productive projects (e.g., QPW) hire deeply specialized experts. OldPeopleEverywhere? ([BibRef-Alexander1977], ff. 215), talks about the need of the young to interact with the old. The same deep rationale and many of the same forces of Alexander's pattern also apply here.

This is also a systems principle that one finds in software development; see http://gee.cs.oswego.edu/dl/rp/roles.html.

A seasoned manager writes, "The most poorly staffed roles are System Engineering and System Test. We hire rookies and make them System Engineers. (In Japan, only the most experienced person interacts with customers.) We staff System Test with `leftovers'; after we have staffed the important jobs of architecture, design, and developer."

Other roles (Architect Controls Product, Developer Controls Process, Mercenary Analyst, and others) are prescribed by subsequent patterns.

If expertise becomes too narrow, the organization is at risk of losing key expertise if a single person leaves, is promoted, etc. Temper this pattern with ModerateTruckNumber.

Domain experts can naturally come together in ProgrammingEpisodes. The pattern ApprenticeShip helps maintain this pattern in the long term. DiverseGroups is, in some sense, a more general version of this pattern.

See also SubsystemBySkill and UpsideDownMatrixManagement.


Architect Controls ProductA


...an organization of developers exists and needs strategic technical direction.

* * *

Even though a product is designed by many individuals, a project must strive to give the product elegance and cohesiveness. One might achieve this by centralizing control, but totalitarian control is viewed by most development teams as a draconian measure. One person can't do everything, and no single person has perfect foresight. However, the right information must flow through the right roles; individual areas of competency and expertise must still be engaged.

Furthermore, there needs to be some level of architectural vision. While some domain expertise is distributed through the ranks of the development team (DomainExpertiseInRoles), the system view -- and in particular, the design principles that create a common culture for dialogue and construction -- usually benefit from the conceptual integrity we associate with a single mind or small group.

Therefore:

Create an Architect role as an embodiment of the architectural principles that define an architectural style for the project, and of the broad domain expertise that legitimizes such a style. The Architect role should advise and influence Developer roles, and should communicate closely with them. The Architect doesn't dictate interfaces (except in cases where arbitration is necessary). Instead, the Architect builds consensus with individual developers, developer sub-teams, and if necessary with the entire development staff, commensurate with the architectural style. The Architect is the principal bridge-builder between development team members.

The Architect should also be in close touch with Customers so the domain expertise is current, detailed, and relevant.

* * *

This does for the architecture what the PatronRole pattern does for the organization: it provides technical focus, and a rallying point for technical work as well as market-related work.

The architect doesn't control the product in any dictatorial sense; it is more inspirational guiding and leadership. We could have called this "Architect Leads Product" or "Architect Guides Product" but all these words have their own problems.

Resentment can build against a totalitarian Architect; use patterns like StandUpMeeting to temper this one.

Intellectually large projects can build an ArchitectureTeam.

We have no role called Designer because design is really the whole task. Managers fill a supporting role; empirically, they are rarely seen to control a process except during crises.

While the Architect controls the architectural direction, the Developer Controls Process, and there is still an OwnerPerDeliverable. The Architect is a "chief Developer" (see pattern Architect Also Implements), or as Alexander thinks of himself, a "master builder." Their responsibilities include understanding requirements, framing the major system structure, and guiding the long-term evolution of that structure. The Architect controls the product in the visualization accompanying the pattern EngageQualityAssurance.

Because OrganizationFollowsLocation and ConwaysLaw, there should probably be an architect at each location. Architects can be the focus of local allegiance, which is one of the most powerful of cultural forces in geographically distributed development.

A more passive way of implementing this is to have the architect review everything. We have seen this work in several projects. However, in most of these projects, we fear that it put the "truck number" in danger (see ModerateTruckNumber). Also, if there is a conscious plan for the architect to review everything, the architect -- in capacity as a developer (see ArchitectAlsoImplements) may "swoop" and fix things that are the responsibility of others (see CodeOwnership). Such "swooping" can be demoralizing to the original code author. The architect can review everything if that role still defers to the implementor for execution and even for the decision about making the change. See, of course, StandUpMeeting.

Architectural control must balance developer authority, and this role of being "keeper of the flame" and of the principles should tread neither on developers' feelings of ownership of their code, nor on their ownership of the code development processes. Architects intervene in processes largely at the business level, and should meddle in implementation processes only in exceptional circumstances.

"Les oeuvres d'un seul architect sont plus belles...que ceux d'ont plusiers ont taché de faire." ("The works of a single architect are more beautiful than those that several have tried to achieve") -- Pascal, Pensées.


Distribute Work Evenly


A twenty-mule team distributes work and communication evenly.

...an organization is working to organize in a way that makes the environment as enjoyable as possible and which makes the most effective use of human resources.

* * *

It is easy to depend on just a few people to carry most of the organization's burdens. Managers like this because it minimizes the number of interfaces they need to manage. And some employees strive to do all they can out of a misplaced feeling of monumental responsibility. In fact, we find that ProducerRoles tend to have stronger communication networks than other support roles.

But if this unevenness continues, it is difficult for a heavily loaded role to sustain the communication networks necessary to healthy functioning of the enterprise as a whole. Resentment might build between employees who don't feel like they are central to the action. And the central people may easily burn out.

Define the communication intensity ratio as the ratio of the number of communication paths of the busiest role to the average number of communication paths per role. Empirically, one finds that the organization has a problem--some unhealthiness--if this ratio becomes too large.

Therefore:

Try to keep the communication intensity ratio to two or less. (We have found that it isn't easy to get much below two.) The easiest way to do it is to have FewRoles. It also helps to identify the ProducerRoles and eliminate any deadbeat roles. You can also identify all the communication to the most central role and see which are really necessary.

Some of this communication overhead isn't very subtle, and these cases are easy to identify. You can eliminate redundant or misdirected communication using simple and direct methods, without going to the level of deep structure or principles of the organization, in these cases.

Other situations take more finesse and generativity, building on other patterns in this pattern language.

* * *

If an organization becomes so out of balance that the work is concentrated in a few people, the organization is more likely to have spots of burnout. Such unevenness might also point to deeper problems in the organization. For example, the more lightly loaded people may not have the technical skills or the human interaction skills to be able to integrate into the larger team or organization. Personality differences can be compensated for with human effectiveness training programs that help communication from the level of appreciating differences to the level of effective presentation. Skill mismatches can be dealt with by re-assigning people or by training.

Unbalance may also point to insecurity in the person or clique that tries to take on all the work. Such insecurity may manifest itself as lack of trust of others. Encounters between the insecure parties and the rest of the project polarizes the positions of each, and a form of schismogenesis may set in--the rise of factions in the organization (see TheOpenClosedPrincipleOfTeams). It may show up either as the insecure subgroup withdrawing, or as in the insecure subgroup trying to hijack the project by strong-arming people into doing their bidding. This may be accompanied by some of the dynamics of burnout; e.g., shutting down communication with "outsiders." Patterns like GateKeeper, JesterRole, and Patron can help avoid this.

In any of these dysfunctions, it is the job of the ManagerRole to counsel the insecure or dysfunctional parties and to take strong intervention. The fix is often intricate and time-consuming.

This pattern follows ProducerRoles and ProducersInTheMiddle, which are prerequisite to ShapingCirculationRealms. This pattern itself is a refinement of ShapingCirculationRealms. FewRoles makes this pattern happen.

This pattern can be implemented and elaborated by using ThreeToSevenHelpersPerRole and ResponsibilitiesEngage.

Here are data on communication intensity ratio for some of our early research subjects. We find that the successful organizations tend to be near the origin of the graph.


Function Owner and Component Owner



...the functions in your project tend to cut across the components of your project. You recognize the importance of organizing according to the architecture of the project (ConwaysLaw), but things aren't simple.

* * *

If you organize teams by components, functions suffer, and vice versa.

You may be organized by function or use case, with no component ownership. On the other hand, you may be organized by class or component, with no function or use case ownership. In either case you the the same question: "What happens when two people need to program the same function?"

You want ownership and consistence in the functions, but also in the components. And the components must be shared across the teams.

If you just assign function owners, the components become shared, and lose their identity and integrity. But if you have only component owners, the functions become orphans. Who makes sure they get done?

Therefore:

Make sure every function has an owner and that every component has an owner.

Make sure every component has a responsible owner and make sure every delivered function has a responsible owner. The component owner answers for the integrity and quality of the component. The function owner ensures that the function gets delivered. If the component owners all refuse to incorporate something needed to deliver end functionality, the function owner sees that the missing code is put in a function-specific place.

* * *
 
There may be friction between component and function owners. You may find the need for the Envy/Developer model of ownership, in which there is a common part to each component or class, and an application-specific part.

No conflict resolution is needed in this strategy, since the component owner has right of refusal to any request, and the function owner has recourse to another to get the job done.

Related patterns:


OwnerPerDeliverable - the general form of this pattern.

TeamPerTask - where a task is the unit of ownership and conflict.

Sample situations:

A. Project Reginald, among others, started out with teams centered around classes or components. At delivery time, the end function did not work. Each team said, "I thought you were taking care of that. It doesn't belong in my class."

On Project Reginald, each function ended up with one responsible owner. That person had to negotiate between the teams to see if some team was willing to pick up the gap, or to see that some extra, special code was created to cover the gaps between the classes.

B. Project Winifred, among others, started out with teams centered around use cases. They soon found that with nobody responsible for cleaning up a class, it develops into an arbitrary collection of state variables and functions. At that point, nobody can clean it up. In one team of three, the entire team sat around one workstation one day to clean up their classes. All three people were needed to understand the code.

C. Brooklyn Union Gas, Project Othello, and eventually Project Winifred were organized with both class and function owners. This changed people's responsibilities, so that a class owner would refuse to put a requested function into a class. They would say it was not part of the class's responsibility, only a special need for an isolated function. The function might end up in the class for the use case.

Fortunately, there are tools such as Envy/Developer that allow a method/functions to be attached to a class just for a specific application. They support creating ownership around common parts of a class, and application-specific parts of a class.

D. Envy/Developer. This model is straightforward to apply to Smalltalk code since Envy/Developer directly supports it. The presence of that tool and this pattern changes the economics of the argument that view classes are needed separately from model objects. While view classes are still valid, Envy/Developer makes it straightforward, safe and local for one application team to add their function-specific methods to the class.

A version of this pattern first appeared in [BibRef-Cockburn1998].


Mercenary Analyst


On one of his many journeys in the Appalachian Mountains, the itinerant folk song collector John Jacob Niles heard a woman singing a particularly beautiful song. He persuaded her to repeat the now-famous Christmas song, "I Wonder as I Wander," until he had learned it himself. He later said, "I never saw her again."

... you are assembling the roles for the organization. The organization exists in a context where external reviewers, customers, and internal developers expect to use project documentation to understand the system architecture and its internal workings. (User documentation is considered separately). Supporting a design notation, and the related project documentation, is too tedious a job for people directly contributing to product artifacts.

* * *

Technical documentation is the dirty work every project must do. It's important to create--and, more so, to maintain--good documentation for subsequent use by the project itself. Who writes these documents?

If developers do their own documentation, it hampers "real" work. Meeting software deadlines means money to the organization; technical documentation is one of those things we tell ourself can be deferred until there is time to do it. But "the time to do it" often never comes, and an organization without good internal technical documentation of its system has a serious handicap.

Documentation is often write-only.

Engineers often don't have good communication skills.

Many projects use tools like Rose to do design, that produce pretty pictures. A good picture is not necessarily a good design, and architects can become victims of the elegance of their own drawings (see the rationale below).

Therefore:

Hire a technical writer, proficient in the necessary domains, but without a stake in the design itself.

This person will capture the design using a suitable notation, and will format and publish the design for reviews and for consumption by the organization itself.

* * *

The documentation itself should be maintained on-line where ever possible. It must be kept up-to-date (therefore, MercenaryAnalyst_ is a full-time job), and should relate to customer scenarios (ScenariosDefineProblem). Note, though, that all team members need to provide input to keep the documentation up to date. The Ad-HocCorrections pattern [BibRef-Weir1998] suggests that a master copy of the documentation be kept, and that team members write corrections in the margin. One team member is assigned to periodically update the document.

The success of this pattern depends on finding a suitably skilled agent to fill the role of mercenary analyst. If the pattern succeeds, the new context defines a project whose progress can be reviewed (the pattern StandUpMeeting) and monitored by community experts outside the project.

If the MercenaryAnalyst_ really is a "mercenary" who "rides into town, gets the early stuff documented, kisses his horse, saddles up his girl, and rides off into the sunset" (Paul Chisholm), then it's good to keep some of the expertise behind by combining MercenaryAnalyst_ with DevelopingInPairs.

This pattern is uncommon but empirically grounded and effective, found in Borland's Quattro Pro for Windows and many AT&T projects (a joint venture based in New Jersey, a formative organization in switching support, and others). It is difficult to find people with the skills to fill this role.

Rybczynski writes:


Here is another liability: beautiful drawings can become ends in themselves. Often, if the drawing deceives, it is not only the viewer who is enchanted but also the maker, who is the victim of his own artifice. Alberti understood this danger and pointed out that architects should not try to imitate painters and produce lifelike drawings. The purpose of architectural drawings, according to him, was merely to illustrate the relationship of the various parts... Alberti understood, as many architects of today do not, that the rules of drawing and the rules of building are not one and the same, and mastery of the former does not ensure success in the latter. -- [BibRef-Rybczynski1989, p. 121].

A passage from Manzoni's I Promessi Sposi (The Betrothed [BibRef-Manzoni1984]) might amuse the MercenaryAnalyst_.


The peasant who knows not how to write, and who needs to write, applies to one who knows that art, choosing as far as he can one of his own station, for with others he is hesitant, or a little untrusting. He informs him, with more or less clarity and orderliness, of who his ancestors were, and in the same manner tells him what to set down on paper. The literate person understands part and guesses at the rest, gives a few pieces of advice, suggests a few changes, and says "Leave it to me."


He picks up his pen, puts the other's thoughts as well as he can in literary form, corrects them, improves them, embellishes them, tones them down, or even omits them, according to how he thinks best, because--and there's nothing to be done about it--someone who knows better than others has no wish to be a mere tool in their hands, and when he is concerned with the business of others he wants it to go a little in his own way.

Richard Gabriel [BibRef-Gabriel1995] notes the following are important traits of this role:

  • good meeting facilitator
  • likes things organized
  • good attention to details
  • has written instructional material (for software)
  • has no ego to invest in the material being documented
  • very smart, highly educated (Ph.D. in literature from Cornell in my case)

In exceptional cases, the MercenaryAnalyst_ can actually take a stake in the design. Betsy Hanes Perry writes:


When I fill this role, I most definitely have a stake in the design: I want to make sure it's elegant, consistent, and clean. The architect has primary responsibility, of course, but I also suggest places in which the design conflicts with itself or may lead to future misunderstandings. As I see it, a software architecture is an idea. The designer/implementors are responsible for expressing that idea (or those ideas) as code; I express it/them as prose. Both are projections of the idea into a particular plane. When there's a conflict, the code is probably correct.

Many projects put faith in tools and notations such as UML to improve quality. But, as Betsy points out, the tool largely provides the forum and opportunity for a human being to engage in the processes and convey the insights that contribute to quality. For documentation to have added value as a quality tool, the documentation process must proceed in the spirit of this admonition.

Paul Chisholm offers the following about the history and rationale of MercenaryAnalyst_:


MercenaryAnalyst_ came from two sources:


(1) Borland's Quattro Pro for Windows, which Cope's identified as the most productive software development organization he's ever seen (average 1000 delivered non-commentary source lines of C++ per staff week), in large part due to the fact that developers had people to write the development documentation for them).


Designer/coders have responsibilities that cannot be delegated. Some responsibilities, such as documentation, can be delegated. Besides, many excellent programmers and most average ones are less than stellar writers. (Richard [Gabriel] may disagree that this *is* the case, and will certainly disagree that this *should* be the case...


(2) A combination of two patterns. One, from Tony Hansen's group, is DisposableAnalysis?: do analysis once, translate to design, throw away the analysis, keep only the design up to date with the code. The other is my observation that most CASE tools require significant experience in the method and the tool itself. If you have DisposableAnalysis? (which few projects plan to do but many follow unintentionally), you should not develop local expertise in CASE tool operation.


It's bad enough learning Framemaker. CASE tools tend to have lousy user interfaces; it's a real pain to use them, or learn how to use them.


The "mercenary" in Mercenary Analyst. comes from the "hired gun" quality a MercenaryAnalyst_ might have; rides into town, gets the early stuff documented, kisses his horse, saddles up his girl, and rides off into the sunset. That's the DisposableAnalysis? model, not the Borland Quattro Pro for Windows model!


Mercenary Analyst plays well with DevelopingInPairs.


Someone quoted by Jim Coplien wrote that "Mercenary Analyst is the professional technical writer who takes care of all the project diagrams and documentation so it doesn't get in the way of the architects."


Maybe not a "tech writer", and not "all the diagrams and documentation," but, yes, that's the idea.


What should be a MercenaryAnalyst_'s education? Mastery of his or her tools (e.g., word processor, CASE tool) beyond that of most users. Experience (perhaps expertise) in the "method" behind the documentation (e.g., an ObjecTime? Mercenary Analyst would have to know ROOM well, someone writing requirements would need systems engineering and/or software development experience).


What is the Mercenary Analyst's motivation? To get the software (not the documentation) out faster!


How can one paint CASE diagrams without knowledge of software? I had some naive hope that a CASE tool Mercenary Analyst could be a highly skilled clerk. I've given up on that. There may be some way of combining Mercenary Analyst with DevelopingInPairs (or a variant for triples) to make Mercenary Analyst some sort of entry-level or apprentice position.


Domain Knowledge. While knowledge of the domain is important for a project (DomainExpertiseInRoles) I don't think the MercenaryAnalyst needs it. (I hope not!)


Knowledge of software is important. Would you trust a driving instruction manual written by someone who'd never driven?

Architect Also Implements


Architects of a housing development working on-site, 1942.

...an organization is being built to serve an identified market (OrganizationFollowsMarket) or markets. Going forward, the project needs the necessary architectural breadth to cover its markets and to ensure smooth evolution, but it can't be blindsided by pragmatic engineering and implementation concerns. Furthermore, the project needs to carry through a singular architectural vision from conception to implementation if it is to have conceptual integrity.

* * *

A software project must broaden the scope of leadership without sacrificing depth and attention to pragmatics. Though developers are good at making individual design and implementation decisions, a project needs an overall guiding strategic technical direction. This usually comes from the architect. However, too many software architects limit their thinking and direction to abstractions, and abstraction is a disciplined form of ignorance. Too many projects fail on "details" of performance, subtleties of APIs, and interworking of components--or, at best, they discover such problems late.

If an omniscient plan were possible, one could solve this with totalitarian control. But even if that were possible, totalitarian control is viewed by most development teams as a draconian measure.

The right information must flow through the right roles; in particular, the developers must latch onto the strategic vision and carry responsibility for implementation. The architect, and to some degree the developers, must also understand the application needs and what they portend for the long-term structure of the system. But there should be a more centralized locus of strategic direction that keeps the project from floundering, makes sure the necessary details are attended to, and keeps track of the emerging "fit" of all the pieces into a whole. Sometimes this "fit" requires understanding low level details of component interaction, protocols, APIs, performance, or reliability concerns.

Therefore:

Beyond advising and communicating with Developers, Architects should also participate in implementation.

The Architect should be organizationally engaged with Developers and should himself or herself write code. The Architect may implement along with a developer using DevelopingInPairs.

* * *

If the architect implements, the development organization perceives buy-in from the guiding architects, and that can directly avail itself of architectural expertise. The architects also learn by seeing, first-hand, the results of their decisions and designs: an important place for feedback in the development process.

The importance of making this pattern explicit arose recently in a project I work with. The architecture team was being assembled across wide geographic boundaries with narrow communication bandwidth between them. Though general architectural responsibilities were identified and the roles were staffed, one group had expectations that architects would also implement code; the other did not.

One manager suggests that, on some projects, architects should focus only on the implementation of a common infrastructure, and that the implementation of non-core code should be left solely to the Developer role. This may work in some projects; however, it is crucial that the architect have a strong feel for the application needs. It is by understanding recurring application needs that the architect can build long-term robust frameworks. If architects work only on infrastructure without an engaged appreciation of application needs, there will be a disconnect between the infrastructure (framework, middleware) and the application.


"It would be convenient if architecture could be defined as any building designed by an architect. But who is an architect? Although the Academie Royale d'Architecture in Paris was founded in 1671, formal architectural schooling did not appear until the nineteenth century. The famous Ecole des Beaux-Arts? was founded in 1816; the first English-language school, in London, in 1847; and the first North American university program, at MIT, was established in 1868. Despite the existence of professional schools, for a long time the relationship between schooling and practice remained ambiguous. It is still possible to become an architect without a university degree, and in some countries, such as Switzerland, trained architects have no legal monopoly over construction. This is hardly surprising. For centuries, the difference between master masons, journeymen builders, joiners, dilettantes, gifted amateurs, and architects has been ill defined. The great Renaissance buildings, for example, were designed by a variety of non-architects. Brunelleschi was trained as a goldsmith; Michelango as a sculptor, Leonardo da Vinci as a painter, and Alberti as a lawyer; only Bramante, who was also a painter, had formally studied building. These men are termed architects because, among other things, they created architecture--a tautology that explains nothing." -- [BibRef-Rybczynski1989, p. 9].


[BibRef-Vitruvius1960] notes: "...[A]rchitects who have aimed at acquiring manual skill without scholarship have never been able to reach a position of authority to correspond to their pains, while those who relied only upon theories and scholarship were obviously hunting the shadow, not the substance. But those who have a thorough knowledge of both, like men armed at all points, have the sooner attained their object and carried authority with them."


John Thomas [mail of 18 Mar 1997] writes: "C. E. Walston and C. P. Felix did an extensive multiple regression study of software productivity -- reported in the IBM Systems Journal, vol.16, 1977, pp. 54-73 'A method of programming measurement and estimation.' [BibRef-WalstonFelix1977]


John continues, "As I recall, the proportion of architects who were also on the implementation team had a very large coefficient. It was a much more powerful variable, e.g., than use of a high level language or use of structured programming."

Though the architect should be able to understand the minutiae of development, it is not necessarily the architect's business to deal with detail day in and day out. Much of what the architect does is to be the keeper of the flame, the owner of the principles that the project follows: principles that in turn shape structure. Much of the structure can come out of a consensus process guided by the architect; in fact, in practice, that's much of what architects do. [BibRef-CoplienDevos2000]

A related pattern is GuruDoesAll from the collection of Don Olson [BibRef-Olson1998a, 153-154], which states [BibRef-Rising2000, 130]:


A newly formed team is given a project with a tight schedule, uncertain requirements, uneven distribution of skills, and new technologies. Let the most skilled and knowledgeable developer drive the design and implement the critical pieces. This can be an antipattern.

The important element of this pattern is to give the critical pieces to the most skilled and knowledgeable practitioners (DomainExpertiseInRoles and, of course, ArchitectAlsoImplements_). But it also has elements of SoloVirtuoso, and can be thought of as an interim application of SoloVirtuoso_ in the context of a project that will mature out of the need for such a pattern. This pattern should be tempered with ApprenticeShip, PhasingItIn, DayCare and others to move toward more of a peer team over time. (DayCare_, in fact, talks explicitly about problem of people starting to say that "a few experts could get the project done faster"). Putting too much burden on one developer can lead to early burnout (see TheOpenClosedPrincipleOfTeams). It is sometimes difficult for a lead developer to give up the SoloVirtuoso_ behavior on a given project once having filled that role, so this pattern should be applied with care.


Firewalls


Nobody gets past this point without my permission!

"A manager should be like the sweeper in curling: The sweeper runs ahead of the stone and sweeps away debris from the path of the stone so that the progress of the stone will be smooth and undisturbed -- does this sound like your manager?" [BibRef-Gabriel1996]

Unfortunately, heavy human use in this same area could lead to bear/human interactions which could injure humans and cause management actions against the bear. -- Sign at an entrance to Boulder Mountain Parks, Boulder, Colorado

...an organization of developers has formed in a corporate or social context where they are scrutinized by peers, funders, customers, and other "outsiders." Project implementors are often distracted by outsiders who feel a need to offer input and criticism.

* * *

It's important to placate stakeholders who feel a need to "help" by having access to low levels of the project, without distracting developers and others who are moving towards project completion

Isolationism doesn't work: information flow is important. But communication overhead goes up non-linearly with the number of external collaborators.

Many interruptions are noise.

Maturity and progress are more highly correlated with being in control than being effectively controlled.

Therefore:

Create a ManagerRole, who shields other development personnel from interaction with external roles. The responsibility of this role is "to keep the pests away."

* * *

The new organization isolates developers from extraneous external interrupts. To avoid isolationism, this pattern must be tempered with others, such as EngageCustomers and GateKeeper.

This pattern was present in both BorlandQuattroProForWindows and in AHyperproductiveTelecommunicationsDevelopmentTeam. See also the pattern EngageCustomers, which complements this pattern.

GateKeeper is a pattern that facilitates effective flow of useful information; Firewalls restricts detracting flow of (even potentially useful) information. You need a balance between them. In the park in Boulder, people (customers) come to see nature, and bears are a part of that nature. But if the customers interact too closely with the core contributors--to the point where it is a distraction--things can get out of control. Developers need information, and they can take advantage of customer contacts and GateKeepers_ to get the information they need. But they can also use managers as a shield. Furthermore, managers may need to step in to "help" developers who may be afraid to ask not to be bothered by customer contacts, or who are at risk of not fulfilling their own responsibilities if they are embroiled in customer matters.

Be warned that if the organization fills this role with someone motivated largely by personal power, the potential damage to the organization can be large. If other roles like GateKeeper_ maintain good contact with other organizations, communications are more likely to remain open and Firewalls will more likely be called to account for self-serving actions.

Sun Tzu notes: "He will win who has military capacity and is not interfered with by the sovereign." [BibRef-SunTzu1989]


Developer Controls Process

A journeyman devises effective and efficient processes for manufacture of self-sealing fuel tanks, WW II.

...an organization has come together to build software for a new market in an immature domain, or in a domain which is unfamiliar to the development team. Progress will be marked by an InformalLaborPlan. The necessary roles have been defined and initially staffed.

* * *

A development culture, like any culture, can benefit from recognizing a focal point of project direction and communication. Successful organizations work in an organic way with a minimum of centralized control. Yet there are important points of focus, embodied in roles, that tie together the ideas, requirements, and constraints into an artifact ready for testing, packaging, marketing, and delivery.

Totalitarian control is viewed by most development teams as a draconian measure. The right information must flow through the right roles. You need to support information flow across analysis, design, and implementation.

Because developers contribute directly to the end-user-visible artifact, they are in the best position to take accountability for the product. Of all roles, they have the largest stake in the largest number of phases of product development. And there should be no accountability without control. The ManagerRole has some accountability as well, to the extent that it indirectly supports delivery of the user-visible artifacts. These are process issues.

Therefore:

Make the Developer the focal point of process information. Place the Developer role at a hub of the process for a given feature, in the spirit of OrganizationFollowsMarket. A feature is a unit of system functionality (implemented largely in software) that can be separately marketed, and for which customers are willing to pay. Responsibilities of Developers include understanding requirements, reviewing the solution structure and algorithm with peers, building the implementation, and unit testing.

The developer is central to all activities of this end-to-end software development process.

Note that other hubs, such as the ManagerRole, may exist as well, though they are less central than the Developer.

* * *

The Developer who is at the hub of a particular feature may be accorded that position according to FunctionOwnerAndComponentOwner but, more generally, the developer should be at the communication hub of whatever process engages them in writing code for the customer. This pattern encourages a structure that supports its prime information consumer. The Developer can be moved toward the center of the process using patterns WorkFlowsInward and MoveResponsibilities. Though Developer should be a key role, care must be taken not to overburden it. This pattern should be balanced with MercenaryAnalyst, Firewalls, GateKeeper, and more general load-balancing patterns like HallwayChatter, ResponsibilitiesEngage, and MoveResponsibilities. Conflicts can be escalated to the PatronRole when consensus breaks down. and the Developer should enjoy particularly strong support from the PatronRole.

If the Developer controls the process, then it's possible to have WorkFlowsInward.

Developers of course don't "control" the process unilaterally, but as a collective group, starting with DevelopingInPairs.

We have no role called Designer because design is really the whole task. Managers fill a supporting role; empirically, they are rarely seen to control a process except during crises. While the Developer controls the process, the Architect controls the product. (In the figure, the Architect role is split across Framework Owner and ArchitectureTeam.) This communication is particularly important in domains that are not well understood, so that iteration can take place to explore the domain with the customer.

In a mature domain, consider HubSpokeAndRim as an alternative.

You can still write down your process as part of a process improvement program. But keep the documentation light; many organizations have found that one page per process is good enough. And make sure each process step meets a need that you can tie to your organization's value proposition. Most often, this value is or should be tied to the product you are producing for a paying customer. If it isn't obvious how the process step helps achieve what you know the customer wants, the do the right thing instead.


Last edited Sun Jul 9 17:56:50 CEST 2006

mailbox JOCoplien@cs.com
Comments