Scaling between Lean and Agile Practice
First published in The Agile Journal, 24 August 2011
Introduction
Agility can be reduced to a few rules. Scrum is a case in point; its rules are well documented and have been popularised for many years. These rules encompass simple things like having a product owner, prioritisation sessions, daily standups, and retrospectives. Of course this level of distillation doesn't provide for much slack, and so the rules must be enforced scrupulously if chaos is to be avoided. Other methods such as XP, DSDM, and those at the harder end of the Crystal series are comparatively more prescriptive. They offer more guidance about the things that need to be done - things like allocating more varied and focused roles, pair programming, test driven development, the structuring of deliverables, continuous deployment, or a greater focus upon project management or upon the business case.
These more prescriptive methods can seem overly complex when compared to Scrum, but their "bells and whistles" do provide added value. Each represents a roadmap to solving a particular kind of Agile problem. Do you have a small project that would benefit from the rapid delivery of prototypes? Consider XP. Do you need to integrate your project with enterprise processes? Consider DSDM. Not sure? Look at Crystal, or perhaps you can just start with the few rules of Scrum and scale them up accordingly.
Scaling up, scaling down
This idea of "scaling up" to more prescriptive Agile approaches is only one side of the coin. It's clearly an important one, because a successful project is likely to grow, and the demands placed upon its management will increase. But the fact is you might also want to scale a successful project down. There are two circumstances in which this is likely to happen:
You're moving from a broadly functional enterprise process towards a more optimised Agile one. For example, if you're working on a project within the context of an ITIL or PRINCE2 environment, you might encapsulate a lightweight Agile approach (such as Scrum) within work package delivery, and/or to adopt a "whole project" Agile method such as DSDM.
You've successfully delivered and transitioned a large system and are entering maintenance mode, such as with a long-tail Agile project. But you're finding that the Agile method you used to build the system is no longer appropriate, since the demands of service support and delivery are not the same as those of development.
The first of these circumstances is the most widely encountered, and perhaps the best understood. I've written about it previously in the Agile Journal. Others have authored books on DSDM and its incorporation into the PRINCE2 framework* and ITIL**. It is the second of these circumstances that is the more controversial. This is because it implies moving away from a proven Agile method, with the hope of finding something better suited to a post-delivery context of small changes and reduced risk.
Enter Lean, pursued by a Bear
Whereas Agile methods use iterations and incremental releases to mark time and progress, Lean uses another measure - the throughput of delivered items, also known as velocity, and the conformance of products to specified tolerances. If the Kanban is supplied with a diet of small and repeatable changes then the velocity can be calculated and the products can be compared to a standard by using appropriate metrics. This provides a mechanism for optimising the entire value stream. By improving team response times throughput can potentially be increased, and by improving development practices the quality of deliverables can potentially be improved. Now all of this might sound perilously Victorian, with Lean developers reduced to the status of Dickensian mill workers. It could have the makings of a good costume drama. Perhaps the Scrummaster can be replaced by an overseer with a pocket watch, glaring at the Kanban board and stamping up and down, growling and scowling if he sees production or quality deviate from projections. A ruddy complexion and mutton-chop whiskers would complete the part.
A conscientiously implemented Lean process will avoid these problems altogether. The primary mechanism for this is the Inversion of Control. This is a well-known design pattern in software engineering and it is certainly interesting to see it used in social practice. The role of the manager is not one of didactic control. There is no "overseer" as such; rather, the organising and actioning of tasks is delegated by the team to a vested authority in those situations where it adds value to do so. In short, the team will direct and update a manager regarding the state of the project. That manager will effectively become a facilitator...much like a Scrummaster in fact. The principle is one of a self-organising team in which members can no longer be adequately considered as resources but as agents of the process itself.
So attractive is this proposition that many Lean principles have been adopted and adapted by the Agile community at large. These include self-organisation and the almost mystical concept of pull. If team members are fully empowered and in charge of their own work, then they are the ones who will generate the driving force for throughput. They'll be constantly removing tickets from the backlog and actioning them, exerting a demand upon the Product Owner for further requirements. When this works it is a beautiful and awe-inspiring thing...you can almost hear the engine humming.
Big Ticket Items: the pressure to upscale again
You'll have noticed that when I was talking about the small and repeatable changes which represent the correct diet for a Lean Kanban, I didn't sound as though I was completely sold on the matter. It isn't that the prescription is wrong. On the contrary, if we are to have a consistent and steady throughput, the team can't spend its time looking at a requirement, turning it over, and cutting it into smaller pieces. It's just that these larger requirements can and do emerge, even in the context of service delivery, and at the very least they'll need to be triaged. There's no guarantee that requirements changes occurring post-release will be of uniform size.
What can we do then? It's best to think of it this way. If it's too big to be picked up by chopsticks then don't put it into a Lean backlog. If it's more of a knife and fork job then you should really triage it accordingly and put it into an Agile one.
Agile methods such as DSDM and Scrum handle big tickets well because they are expressly iterative. They allow requirements uncertainty to be tackled by means of empirical process control. The resolution of scope isn't achieved so easily with Lean since it does not put the same degree of emphasis on timeboxed checkpoints...that is to say, the process gates in which scope is baselined and reviewed. Instead, Lean puts its emphasis on a different kind of empirical process control - checking how well the deliverables meet certain tolerances. Apart from that, all you really get is a "heartbeat" marking the periods of time over which velocity can be measured. If there's any sense of retrospective, it's one that's looking at throughput, and at measuring conformance using like-for-like metrics. But as long as the tickets are kept small enough to flow and compare well, it's all fine and dandy.
Upscaling from Lean to Agile: the nitty- gritty
Of course, what this means is that when you do get the inevitable big tickets items (BTI's) appearing on a Lean team's Kanban board, there are some choices that you will have to make.
Adopt a hybrid approach, such as the Lean development method advocated by the Poppendiecks; it is iterative but retains the Lean emphasis on optimised value streams. Alternatively you could adopt a "Scrumban" variant. All tickets will be subject to estimation and the measured velocity will be adjusted to reflect each ticket's size or complexity. Use timeboxed iterations of 2, 3, or 4 weeks in order to narrow scope, and develop a Sprint backlog as well as a Product one. These hybrids are essentially fast-iteration Agile processes, but there can be some flexibility for modifying the rules so they work in a "leaner" fashion. For example, you can decide whether or not you will allow tickets to be introduced or reprioritised mid-iteration. Business owners often expect the right to introduce and reprioritise tickets at the drop of a hat, and a Lean process should be able to support this. However, Scrum does not permit such interference because it interrupts flow, shifts the posts for tracking burndown, and carries the burden of re-estimation. These consequences must be explained very carefully to all stakeholders.
Stick with Lean, but use a ticket-cascading method to achieve empirical process control over scope. With this technique you raise a ticket specifically for analysis of the BTI's requirements. I favour using the BTI itself as a holder; there is no point in abandoning the ticket just to raise a new one. The deliverables will be a broken-down set of requirements, costed if appropriate, and an architectural impact assessment with an appraisal of risk. New tickets can then be raised for each of the derived requirements, and linked to the original BTI which may then be retired. The linking gives you a narrative thread for tracking progress of multiple fine-grained tickets over time, which is certainly much better than allowing a behemoth of a ticket to choke the Kanban. Ticket-cascading might seem like a bit of a cheat, but I have used it successfully and the process overheads are low.
Set up a separate temporary Scrum just for the BTI. You will generate estimates for all tickets, and the BTI's which are identified will be taken outside of the Kanban for impact assessment and triage. For seriously large BTI's a dedicated Scrum team may be warranted. The Scrum developers that are needed to progress the requirement will be identified at that point, and it should be broken down into more detailed requirements for population in the Scrum backlog. Note that the Scrum developers should be sourced separately from the Lean team. A developer can't be on a Lean team and a Scrum team simultaneously, unless a ticket is raised on one of the Kanbans to account for that developer's time. Even then it is only viable as a short-term measure.
Set up a separate permanent Scrum. Not all BTI's are large enough to justify a dedicated Scrum team. Instead, you can set up a permanent Scrum with the remit of progressing the smaller BTI's. Note that by "small" I still mean bigger than chopstick-sized. They're not huge BTI's, but they've still been assessed as too large for progression through a Lean operation. The singular advantage of a permanent Scrum is that you save on project setup and teardown costs, since you have one Scrum progressing multiple BTI's. The downside is that you end up with fairly large tickets on the Kanban which may represent capitalised projects (or portions thereof) from different business areas, and so you may need to handle cross-billing within the Scrum. It will also be necessary to identify a very senior Product Owner who can arbitrate during the prioritisation bun-fights that will inevitably follow.
Transitioning the Kanban
Aglie and Lean processes are not the same thing, and we've had a look at their relative positioning. We've also seen that you can move from one to the other when scaling a project up or down in order to keep pace with changing project or programme demands.
Now that's all very well, but it still leaves us with a problem - the tickets on the board. When a decision is taken to switch from Agile to Lean or vice-versa, we can't expect the Kanban to be wiped clean. The tickets will need to be transitioned "on-the-fly" and while many of them are in progress. This is potentially problematic, because we can't always expect the rules under which the Kanban operates to remain the same when transitioning between Agile and Lean. Here's why.
There is potentially an impedence mismatch between heartbeats and Sprints. For example, if you move from a Lean Kanban with a weekly heartbeat to Scrum, you'll be hard pushed to translate that heartbeat into a weekly Sprint. Too much time will be eaten away by housekeeping activities such as requirements analysis, estimation, demos, and retrospectives. That said, I have known it to be done successfully, but only after a bedding-in period of longer Sprints which allowed the team and its practices to mature and gel thoroughly.
A Lean ticket, once removed from the Backlog, should not be returned to it. I know that some people allow it, but in my view this is definitely a Bad Idea. Once the ticket is returned then the measure of velocity will be compromised - it might as well never have been removed in the first place. With Lean, there should be a strict flow from one bucket in the Kanban to the next unless the ticket is Blocked, in which case it should be moved to the Blocked bucket. It is then possible to tell at a glance which tickets are having their progress impeded and the appropriate action can be taken. This rule does not need to be enforced so strictly with Agile because the successful delivery of an increment is more important than the velocity achieved within the Sprint.
With Lean, a ticket that is Blocked may be subject to constraints that do not apply on Agile projects. For example, it is reasonable on a Lean project to enforce a rule whereby any tickets that are blocked by Business for a period of time (e.g. more than one week) will be discarded. A fresh ticket will need to be raised once Business is in a position to progress it. This rule assists in the maintaining of velocity and encourages discipline on the part of stakeholders. However, no such rule is needed on an Agile project, since unresolved blockages should be tracked as risks against the delivery of an incremental release, and actioned accordingly.
There are potentially other differences that will affect the transitioning of the Kanban, such as different measures of ticket estimation. However the issues I have outlined above are the main ones. The way to handle these mismatches is tool-dependent. If all you're using for your Kanban is a whiteboard and Post-It notes then you don't have much of a problem...any rule differences can be carried in team member's heads. But if you're using a Kanban application of some sort, you may have rules configured for each board. You'll then need to figure out how to move a ticket between boards that exhibit potentially different rules, or at least you'll need to figure out how to change the rules of the existing board. Jira, for example, identifies any field mismatches when refactoring tickets across projects, and will invite manual mapping at that point.
Conclusion
A project will evolve continually, and the methodology that is followed needs to evolve with it. The use of an Agile Method does not provide any sort of exemption. Agility is positioned as being adaptive in nature, but we should remember that this quality does not relate exclusively to the evolution of a product. The process that is being followed also needs to adapt so it can meet changing project conditions. The ability to transition between Lean and Agile presents certain unique challenges in this regard, and it's a skill that team leads and project managers should certainly be able to apply.
* Richards, Keith. “Agile Project Management: Running PRINCE2 projects with DSDM Atern”. The Stationery Office, London, 2007.
** Tudor, Dorothy. "Agile Project and Service Management: delivering IT services using ITIL, PRINCE2 and DSDM Atern". The Stationery Office, London, 2010.
