Agile Transformation in Organizations that Suck

First published in The Agile Zone, 4 October 2013

“The fastest way to succeed is to look as if you’re playing by somebody else’s rules, while quietly playing by your own”

- Michael Korda

When Transformation Goes Wrong

The failure of the UK Government’s flagship agile project has raised questions about just how “agile” it really was. I've blogged some thoughts on this matter and, along with others, I've indicated that the main contributing factor to the Universal Credit debacle may have been inappropriate procurement. This would make it one of several recurring issues to have affected public bodies that are supposedly in transition to agile ways of working. In fact, Universal Credit is just one example of how large organizations are bounced into paying lip service to the paean of agility, when they lack the skills and cultural grit to effect the necessary components of change. In my experience the rate of genuine agile transformation in such places can be 10 times slower than elsewhere. Transitions that might take 2 to 3 years may, I suppose, end up taking 20 to 30 years… if indeed they ever really happen at all. In this article we’ll look at how agile adoption can be nurtured in these environments and brought to fruition.

The public sector has a reputation for being the graveyard of many an agile coach. Yet similar problems can afflict large commercial enterprises too. The cultural barriers to agile adoption are common ones and are essentially a function of organizational culture and scale. In truth we should define the transformation problem in a more general sense of bureaucratic impairment. Bureaucracies militate against reform and intimidate those who would bring about change, something which is true regardless of the sector involved. This is a well-known issue and it has made Dilbert the alter-ego of us all.

Yet as agile coaches and change agents, we have to take the rough with the smooth when it comes to agile adoption. Some clients make cheering progress while others, particularly those at scale, advance at a glacial pace and with many false starts. We cannot sit at our desks, discouraged and reluctant to engage. They are clients after all and they need our help. In short, we must be prepared to facilitate agile transformation in organizations that suck.

Understanding the Pathologies

The typical scenario in a large corporation is to have agile buy-in at a very high level. Every C-level executive wants to think that he and his people are "agile." This is the realm of strategic vision and policy, where portfolios of strategic assets are controlled and programs of work are defined. For example, the driver behind the Universal Credit attempt at agile adoption was a policy decision by the UK Government. We can also expect substantial (though not universal) buy-in at the other end of the food chain. Developers often want to learn new agile practices, even if it is only to improve their career prospects and future employability. Contract staff expect the sort of agile values they have experienced elsewhere. There can be die-hards in the permanent cadre, people who regard change as a threat rather than an opportunity. They may even try to sabotage an agile transformation, but they can usually be dealt with by exposing them through transparency.

The sticking point in agile adoption usually lies in middle management. This is where a transformation can really founder. In fact I hesitate to call this obstruction “management” at all. It is more a series of layers of vested interests in the status quo. These layers include other die-hards… those without operational skills, but who, as managers, are influencers nonetheless. Their lack of co-operation often means that there is no coherent strategy in place for the agile rollout the higher-ups supposedly desire. This is a problem. It means that nothing connects organizational policy to the teams' operational demand for a more joined-up agile environment. The resulting pathologies include the undervaluation of transparency, indifferent product ownership, weak pull, reactive management styles, severe constraints placed upon the ability of teams to inspect and adapt, and many other dysfunctions.

What can a Scrum Master do about the situation?

Bureaucracies -- including those troublesome middle managers -- often hire “Scrum Masters” without fully appreciating what the role entails. To them, a Scrum Master may be nothing more than a rebadged team or workstream lead. Agile transformation often degenerates into a rebranding exercise when the journey is seen to threaten established terms of reference and cultural norms. The concept of servant leadership can seem alien and dangerous to bureaucrats. The removal of impediments by a Scrum Master, such as wasteful meetings, might be seen as an act of insubordination, while organizational agile coaching may be regarded as incitement to rebel. It’s a tricky situation to deal with. Here are three things a notional “Scrum Master” in this situation can actually do:

1. Apply Scrum at a team level in order to provide some transparency. Unplanned work coming in mid-sprint needs to be made visible, as does a lack of planned work if product ownership is impeded. It needs to be made obvious how the lack of appropriate and timely pull impacts burn and velocity. Also, the dependencies and blockages that affect a team need to be exposed. You might not be able to fix all of the problems within the limited and poorly framed remit you have been given, but you can always expose them. Transparency is the first step in any agile transformation. That's why methods like Scrum are worth doing even in this disheartening situation. This bottom-up method of agile adoption is sometimes called the “guerrilla agile” approach.

2. Apart from providing transparency, the second most important single thing a Scrum Master can do is to keep a team together so that comparable metrics can be elicited. The nightmare scenario is when team members are pulled out by managers to help them firefight. How can a team inspect and adapt itself reliably, without reliable data based on reliable membership? It's much better to get unplanned work put on the backlog or fast tracked by the team. In an agile way of working, scope change is easier to deal with than uncertainties in team composition.

3. Thirdly, keep sprinting on a project, even when product ownership is weak. It can be very tempting to say "the organization can't support effective sprint planning, therefore we'll move away from sprints altogether." This might be fine and dandy for BAU work, where there are only small and repeatable changes to be done, continuous integration and deployment is in place, and there is no project risk. However it's not appropriate as a solution when there is project work to do. Sprints de-risk projects by putting the emphasis on clear sprint goals and on the building of a potentially releasable increment. When an organization can't do this, that's exactly what needs to be exposed to scrutiny. When sprint value can't be established, that needs to be made clear so product ownership can be challenged and improved. Abandoning sprints will mask these problems, not solve them.

Seven Steps to Transformation

Now let’s consider the transformation problem in a broader sense. There’s only so much you can do by growing an agile transition from "the bottom up." It’s just the first of seven things that I would suggest are necessary. Here’s the list:

1. Go Guerrilla... but realize the limitations

Starting an agile transformation with development teams makes a certain amount of sense. After all, if Scrum Masters have been assigned it implies that these teams at least are expected to function in an “agile” way. Furthermore an enterprise transition must start somewhere, and it is clearly important that development teams should have their house in order.

Now this is all very well, but the limitations will soon become apparent once transparency is in place. For example, the development teams may not actually be in a position to deliver on a sprint-by-sprint basis. It is often the case that they have dependencies upon other teams before their work can be brought to completion. This can be made visible by marking such items as “blocked” on an information radiator such as a Scrum or Kanban board. Transparency comes first. In addition, the team should be coached not to accept work that they are unable to wholly complete. Another issue that can be made clear through transparency is weak product ownership and pull. If there is no demand for work to be done, backlogs may be left unpopulated and work may accumulate pending its approval.

Transparency is great for exposing these issues. What it doesn’t do is to solve them. For that to happen, genuine organizational change will needed, and the Scrum Master must start looking beyond his or her team.

2. Understand the Operating Model

At scale an organization can lose sight of how it intends to keep its promises to stakeholders. This needn’t be much of an issue for small companies that provide and service only one or two products. Unfortunately, once the portfolio of assets grows beyond a certain level, this model can become unclear. For example, if a new middleware component is needed to glue several existing products together:

• Will this be developed by an existing team or a new one?

• How will the journey into service for this component be managed?

• Have support personnel been resourced and trained, or has a budget for doing so at least been provided?

• Has operational expenditure been distinguished from capitalized resources?

• What will the value stream look like?

In short, what is the organization’s strategy for turning portfolio inventory into working product? This strategy is known as the operating model… the model under which the organization operates in order to provide value.

Try to find out what the operating model is, assuming that there is one at all. A model will reveal what the team dependencies are assumed to be. Does this match the reality though? If it does, then great. You have a shared frame of reference with others in the organization, and a way to develop a shared understanding of the dependencies you identified earlier. The operating model and its dependencies can then be challenged. On the other hand if the operating model doesn’t match the reality of where dependencies lie, then the organization is just making it up as it goes along. In either case the lack of an agile operating model will cause organizational pain. The next step is to gauge the extent of it.

3. Diagnose the Pain Points

Once the portfolio of assets grows beyond a certain point, the operating model can become confused. In fact the attenuation of the operating model is often what leads senior managers towards agile adoption in the first place. It is a pain point for them. It is observed by these managers that their organization cannot deliver sufficient value in a timely manner. The need to “become more agile” is latched onto even though the premises and consequences of this are not understood.

As a Scrum Master, you have already identified the pain points of your own team. You elicited them in Step 1 and made them transparent. Now you should encourage other teams to do the same thing. You know where to look for them though… follow the chain of dependencies out from your own team. Find out where each team is on the path to an agile way of working:

• Is their membership reasonably stable, such that they can inspect and adapt?

• Do they share a common terminology, and present themselves along with other teams in a consistent way to stakeholders?

• How is product ownership represented?

• Where does prioritization and pull come from?

• What metrics are valued?

• How much “work in progress” do the teams have?

• Where do their own dependencies lie?

Remember that the pain points that stop a work item from being completed can actually involve multiple teams. Your team might be waiting on others, but those others may well have dependencies of their own. There’s also a chance that certain teams will be dependent upon work being done by yours. Moreover, if one team doesn’t deliver assets to another team until a Sprint finishes, then clearly the combined work will not be completed in one Sprint cycle. This will mean that an incremental release cannot be made. Release management is a common problem, so now it is time to review the strategy behind it.

4. Review the Release Strategy

Small organizations with a simple product can expect to have a comparably simple operating model. Perhaps only one team will be involved in development and delivery. That’s comparatively simple, because only the products of one team need to be considered when making an incremental release.

This situation changes once multiple teams are involved. A more complex product will bring in different workstreams, potentially at different times, and their deliverables must align. We briefly considered an example of this in Step 2: a piece of middleware that links the outputs of multiple development teams, and which will need to be delivered into service at some point. Things to look for include:

• A release schedule, and whether or not it is iterative

• The extent to which continuous integration and deployment is in effect and release-on-demand is supported

• Whether all teams contribute towards the same release or if disparate releases versions are targeted simultaneously

• Whether or not there is a release manager who co-ordinates it all.

If the organization’s operating model is well defined then the mechanics of release management should also be clear. If they aren’t, you can expect an ad-hoc approach to team collaboration and a hit-and-miss record on the timely delivery of releases of value. The defining characteristic of an agile transformation at scale is the need to co-ordinate multiple teams. When this is done well, incremental value is still delivered and is of sufficient quality to be potentially releasable.

When working as a Scrum Master with a lone voice, I’ve usually been able to get this far. I’ve been able to gauge the pain points between teams, relate them to an established operating model should one exist, and get a handle on what the release strategy is. All of this can be ascertained by casual meetings with stakeholders such as a release manager or old development hands. Yet if the transition is to move further then stronger relationships must be cultivated.

5. Set up a Scrum of Scrums

By this point you should know which teams have dependencies upon each other, including your own team. You’ll also know the release strategy – if there is one – to which each of these teams must align. The next step is to get the teams to collaborate more effectively so timely releases can be made.

A Scrum of Scrums is a timeboxed event, held at least once per week and ideally daily, in which representatives from each team collaborate towards making a release. A 15 minute standup is the typical format. A representative from each team describes:

What has been done since the last session

What they plan to do next

What impediments or dependencies they have

What impediments they are likely to be putting in another team’s way.

The session is an inspect-and-adapt opportunity for all of the teams to smooth their progress towards a release goal. It doesn’t matter how agile the teams are before they attend. The goal is to improve collaboration so timely releases happen and waste is reduced. Be prepared to coach them as needed.

Note that a Scrum of Scrums will require a common Definition of Done so the quality of a release can be vouched for. Each team contributing to an incremental release should include it in their own Definition. A Scrum of Scrums may also benefit from a shared glossary or guide that keeps their terms of reference consistent.

6. Inventorize the system assets

Thus far, we’ve looked at how to extend an agile transformation horizontally across the various teams that contribute to releases. The next step is to extend the transformation upwards to senior management. We want to connect the activities conducted by Scrum and Kanban teams to the organization’s agile policy. That means inventorizing the assets the various teams work on, and mapping them to the organization’s portfolio of work.

Inventorizing system assets for correlation to a portfolio can become a problem for a Scrum Master. In a bureaucratic organization, where a Scrum Master is assumed by middle management to be a spiffed-up team lead, it may be seen as the exceeding of one’s authority. Framing enquiries in a Scrum of Scrums context can mitigate this risk somewhat. After all, you don’t want or need a complete list of every system the company has or is working on. You only need to enumerate those assets which the various teams are responsible for producing and maintaining – that is to say, the teams represented in the Scrum of Scrums. An understanding of the assets that these teams are responsible for would clearly not be unreasonable. In fact, it might be possible to draft a list by pooling the knowledge of Scrum of Scrums team members. Even so, you should try to validate this information. The obvious person to ask would be a release manager.

Release managers generally know what the touchpoints to systems are, and this information can be correlated to that provided by Scrum of Scrums members. A simple spreadsheet which maps these systems to the Scrum teams, or at least to project managers, is a good start. Release managers may already have a mapping like this as a tactical tool to help them do their jobs.

7. Build Product Ownership

In this approach, we’ve done our best to link the operational needs of Scrum and Kanban teams with agile sponsorship at a senior policy level. We’ve cut through bureaucracy and correlated at least some of the organization’s portfolio of assets to the means of production. Yet none of this should have been necessary if adequate product ownership was in place. A Product Owner should represent an asset to development teams, and will be responsible for the return on investment to senior stakeholders. Agile teams need these Product Owners because they actually have skin in the game, and can explain requirements and priorities and value. Teams with delivery responsibilities have far less use for middle managers.

So now we need to go through the inventory and see what each item has in the way of Product Owner representation. It’s quite likely that the portfolio won’t reveal coherent ownership at all, but rather that the assets are correlated to a mix of stakeholders. These can include project managers - who report on progress towards stage-gated milestones - and departmental heads who have some sort of notional accountability for it all. As in the case of Universal Credit, the stakeholders may include external suppliers. Sometimes attempts are made to outsource product ownership when it cannot be identified within… a patently ludicrous strategy.

Of course, in an agile organization, none of this is wanted. Instead it demands an environment in which frequent releases are used to learn about product value, and to inspect and adapt the innovation process behind delivery.

For this to happen senior managers will need to sponsor robust product ownership. They’ll need the courage of their supposed agile convictions. This can mean, for example, vetting Product Owners for a project before it is authorized, and before the problems of weak ownership impact teams at an operational level. As Eric Ries has put it in his book The Lean Startup: “…for established companies looking to accelerate their innovation teams, building this platform for experimentation is the responsibility of senior management”. A Scrum Master cannot pull a Product Owner out of a hat. All he or she can do is to make weak product ownership transparent, and coach the organization’s stakeholders to act on that information accordingly. Unfortunately there is no shortcut to agile adoption. It cannot be created magically or bought. It must be committed to and earned.

Conclusion

Agile transformation will reveal dependencies that inhibit any given team from bringing a Sprint Backlog through to completion. Most "small" teams in an enterprise at scale don't truly own their process. Raise a Scrum or Kanban board for such a team and it soon fills with work that is "blocked". Moreover, getting it "unblocked" requires authorizations and organizational culture changes that no one Scrum Master or Agile Coach can bring into effect without top-level support. Guerrilla Agile, although laudable and useful in some cases, can take an enterprise-level transformation only so far.

This is something that takes many clients by surprise, and some are not prepared to deal with the harsh realities that are exposed. Clients should be fully appraised of the need to address home truths, and for the need to build product ownership around a meaningful release strategy that promotes further learning. Agile adoption is a harsh mirror. If you don’t want to know how ugly you are, don’t look in.