Running Scrum Sprints in DevOps

Everyone knows a partnership where neither partner is ever impressed by what the other one does. Whatever was accomplished could have been done better, or sooner, or more elegantly, or more efficiently. If only the other one listened more, they'd see sense and that the other's way is wiser. Yet at the same time, each partner takes what the other does for granted and without openly valuing it. The result is a strange love-hate sort of relationship, always seemingly on the point of combustion, but it somehow manages to work. Each partner appears to have something the other partner needs. Perhaps reluctantly and unadmittedly, they have reached a sort of mutual accommodation.Scrum and DevOps have been in that kind of oddball, finger-pointing relationship for several years now. Each partner has its advocates and the backing of strong belief and opinion:"DevOps bridges the gap between Development and Operations."

"Whoopy-doo. Scrum did that years ago. Ain't no separate ops role in a Dev Team."

"Look, DevOps work is put into production with minimal cost of delay."

"You're telling me about production? In Scrum, each increment must be of release quality!"

"Ah, but DevOps is more efficient because it leverages continuous deployment!"

"There ain't nothing to stop continuous deployment during a Sprint, you nut!"

"But batching work into a Sprint Backlog is wasteful, DevOps doesn't do that!"

"Look, a Sprint Backlog ain't no batch, it's a forecast to meet a Sprint Goal!"

"Who the heck needs a Sprint Goal? Who needs your damned Sprints anyway?"

"Ignoramus. Come back when you have a complex product to de-risk for grown-ups!"

Clearly, an understanding must be reached when this odd couple finds themselves in the same department. Let's look at the habits of DevOps and see if they can be reconciled with those of Scrum, who, as you may recall, is trying to meet a Sprint goal.

The "New Work" Episode: Continually Delivering Into Production

DevOps teams really value a lean flow of work into production. They want it to be integrated, tested, and of release quality. If they are using Scrum, they can be quite reluctant to bring new work forward into a Sprint if they doubt their ability to finish it within the time left. However, in Scrum, it is the achievement of a Sprint goal that is considered to be important, rather than the completion of individual discrete items of backlog inventory. The work that should be completed within a Sprint is that which is needed to achieve the Sprint Goal. That is the team's commitment.

Other work may or may not be done, though it is reasonable to expect a team to try and further maximize the value provided to stakeholders within the remainder of the Sprint time-box. Hence, a team using Scrum can take on further work, and only partially complete it, as long as doing so does not put the Sprint Goal at risk. If the goal has already been achieved then any new work must not compromise the release of the increment which delivers that value, nor should it fog the ability of the team to assess whether or not the increment meets the definition of done.

In other words, as long as the empirical achievement of the Sprint goal is not put in jeopardy, then it is possible for a DevOps team to pull in new work with no expectation that it be completed by the end of a Scrum Sprint. It was not part of the team's commitment and there ought to be no expectation that it will feature in the increment they provide.

The team should bear in mind that there is no partial credit for undone work in agile practice. It is reasonable, however, for the size of a backlog to be revised based on the latest estimates of the work remaining. Any effort expended by a team on a piece of work may affect the corresponding estimates. Items that have not been completed by the end of a Sprint should be re-estimated. Partially done items may be expected to result in lower estimates as there is likely to be less work remaining.

The "Debt" Episode: Worrying About Undone Work and Technical Debt

The key objection to leaving work partially completed is one of technical debt. The value of that work will degrade the longer it is left only half-done, and as such, it can represent a technical debt liability. This is the most important thing that a DevOps team must consider before bringing new work forward: Any work they start but do not finish will be subject to depreciation before it even gets into the hands of stakeholders. They may simply and reasonably decide not to incur that debt. If they go ahead and are not able to complete it by the end of the Sprint, the Product Backlog should reflect the work that they believe to remain, including any depreciation of the work which is partly done.

Note that since any work which is not yet done should be re-estimated on the Product Backlog, there is never any "carry-over" of work from one sprint to the next. Undone work is simply accounted for on the backlog for possible actioning in the future. The choice of which Sprint that work is subsequently planned into, if any, is a decision the Product Owner must make. That decision may be partly based on the depreciation of accumulated value and the advice given by the development team.

It's also important to remember that there is no intrinsic value in things like story points. The purpose of estimates is to help a team figure out how much they can complete in an iteration. This is not an exact science. The team should certainly plan to deliver the increment they forecast for completion in that time-box, but they may engage in further activities should the time prove to be available. If any additional items are not completed then the size of the product backlog should be re-estimated to reflect the work that is believed to remain.

The "In Progress" Episode: Wanting to Complete Individual Pieces of Work

At the end of each Sprint, no work should remain in progress. It should either form part of the increment or be accounted for on the product backlog, so it can potentially be completed in a future sprint. What matters in Scrum is that an increment meets the definition of done, that the Sprint goal has been met, and that it is not subsequently compromised by any further work that might be performed. Whether any individual backlog item is completed or not is irrelevant and essentially a matter of technical debt appetite. It is possible that none may have been completed at all in a sprint.

This last point may sound very strange to a DevOps team which is trying to leverage a continuous flow capability. It can raise the question of how a team can allow undone work at the end of the Sprint if those items fail to meet the Definition of Done. The critical thing to understand, however, is that in Scrum a definition of done applies to the increment and not to product backlog items (PBIs) such as user stories. Also, the PBI's that are planned into a Sprint Backlog are merely a forecast of the scope that will be actioned to meet the Sprint goal. Furthermore, a team must deliver a "done" increment that actually meets the goal and not the forecast.

Suppose, for example, that a DevOps team has the Sprint goal of delivering a basic online shopping cart function. The acceptance criteria for each item mandate a certain corporate look-and-feel. Unfortunately, there is a problem during the Sprint; the team is not provided with the new artwork. None of the items can now fully meet their acceptance criteria. The planned tasks for look-and-feel are therefore not completed.

However the PO, and the stakeholders represented, may not care about branding in this sprint. The MVP may simply be for a basic working shopping cart, as per the agreed Sprint Goal and which is fit for release to partners in their early adopter program. In this case, the agreed Goal has been met and the increment is fit for purpose. At the end of the Sprint, the work for each PBI is simply re-estimated on the Product Backlog to reflect what is still to be done with branding in order to satisfy the A/Cs. Note that not one story was completed by the DevOps team, even though the Goal for the Sprint was in fact accomplished.

Of course, a team should try to make a good forecast (i.e. to plan a good Sprint Backlog) in order to have transparency over the work remaining in a Sprint. Yet that forecast could turn out to be wrong, perhaps with no story being completed at all by the end of the timebox, and the Sprint Goal could still be met by a competent team with a useful increment delivered. Valuable work will have been performed which meets the Definition of Done and the Sprint Goal, and which is fit for potential release. That work will have been "done" and invested in the increment.

Remember that any undone work must be accounted for on the Product Backlog. Any PBI's that are not completed by the end of the Sprint must be re-estimated, revised if necessary to better indicate the scope remaining, and ordered in the Product Backlog by the Product Owner.

The "Done" Episode: Thinking Multiple Levels of Done Are Strange

The Scrum Guide acknowledges that the term "done" is sometimes applied to individual items of work, and thus warns of the need for clarity when the term is used. However, in Scrum, it is essential to have a definition of done that applies to an increment. The Guide explicitly says:

"The purpose of each Sprint is to deliver Increments of potentially releasable functionality that adhere to the Scrum Team’s current definition of 'Done.'"

This is at odds with certain norms and conventions found elsewhere in agile practice. A DevOps team must be aware of these differences. In Lean-Kanban operations, for example, it might be reasonable to expect individual backlog items to be completed as per a "done" assertion of quality. Scrum, on the other hand, puts the emphasis on delivering a "done" increment that meets a Sprint Goal. The purpose of backlog items is to provide transparency over work in progress, but their actual completion is, strictly speaking, quite irrelevant. The completion of individual items will only matter if the team decides that a continuous flow of such items is the best way for them to achieve the Sprint goal that has been agreed.

Note that it can be best to avoid language in which a “separate" DoD is applied to backlog items, as that is likely to cause confusion. It is possible to say that there may be a certain "level of done,” although in the case of user stories we might simply use the term "acceptance criteria." A definition of done may prescribe many "levels of done," and having those levels can be good practice. They can be a good waste limitation strategy for catching and addressing problems prior to integration in a DevOps value stream. Yet, the DoD itself applies to the increment, no matter how many levels of “done” it encompasses.

The Last Episode: Some House Rules Are Agreed Upon

If they are using Scrum, then a DevOps Team must deliver the best possible value to the Product Owner during a Sprint. If they have the opportunity to provide not only an increment but also to further reduce the estimated work remaining in the Product Backlog, then they may do so if the Product Owner agrees. The team is under no obligation to do so or to give any assurances regarding completion of that work. What they need to bear in mind is that any work left undone will be subject to depreciation before it even gets into the hands of stakeholders. At the end of a Sprint, items on the product backlog should accurately reflect the work that remains for their completion, including any depreciation of such work which has been started but not yet delivered.