The Empirical Product Owner: Innovation Accounting in Scrum

First published on Scrum.org, 11 April 2017 (syndicated)

"Few ideas work on the first try. Iteration is key to innovation” - Sebastian Thrun

The Agony and the Ecstasy

Have you ever seen the 1965 film The Agony and the Ecstasy, where Charlton Heston plays Michelangelo painting the Sistine Chapel? Each day the Pope looks up and asks him “When will you make an end?” The artist always mouths back the same answer: “When I am finished!”

“When will the project be done?” is similarly a common refrain. If you work on a software development team you might be able to put a number of faces to that voice, although none are likely to be the Pope's. Like Michelangelo, you may have heard this question many times before, and perhaps you sigh inwardly on each occasion. Yet unlike Michelangelo you are never entirely sure of how to respond. “When I am finished” may sound trite coming from your lips instead of those of a renaissance maestro. You aren’t painting the Sistine Chapel after all, adding further and further detail which only you are in a position to appreciate, with a trained eye from your own vantage point. It’s a software initiative, and people simply want to know when they will get the delivery they actually expect.

Yet the question brings with it some telling implications. Firstly it suggests that any value you may have delivered so far is inadequate for stakeholder purposes - hence the question being asked. Perhaps nothing has been seen by them at all. The scaffolding of work-in-progress may block their view, and as of yet any value might remain unevidenced and unappreciated. Secondly, and no less importantly, it suggests that the creative process lies outside of stakeholder control - hence they are asking that question of you.

These negative implications typically hold true in a linear, stage-gated software production context. Very little, if anything, may be delivered until very late in the initiative and the outputs may lie under the control of project managers. Stakeholders might have only very limited involvement once a mandate has been given and requirements documentation has been “signed off". With no obvious control from that point, and no clear evidence of value earned, as time goes on the plaintive wail of “when will the project be done” can sound increasingly shrill.

Agile Value

However in an incremental agile way-of-working, to hear such a question being asked is a different matter. Those implications ought not to hold true at all. Value should be adequate and timely, and control over its delivery should be indubitably evident. Hence if the question is asked, it tells you that something is fundamentally wrong in a number of ways:

• Firstly, that stakeholders are looking for a definitive end-point at which significant value will be released, instead of maximising value delivery on an incremental and ongoing basis 

• Secondly, that stakeholders have not yet obtained empirical evidence of a return on investment, and thus cannot forecast the likely value of continued funding 

• Thirdly, that stakeholders do not seem to recognise the "end-point” to the initiative as being value-driven at all, and hence not as something which lies under their Product Owner’s control

These are all contra-indications to Scrum practice, and they suggest that agile delivery is not in effect. How to fix this though? Well, value is a recurring theme and it must underpin the remedy. In Scrum, a clear role is elicited where the responsibility for “value" must incontrovertibly lie. This is the Product Owner, a role which is sometimes crystallised in the terse definition of “value maximiser”. That’s what a Product Owner really is. It’s someone who can liaise with stakeholders and arbitrate between them, so that the best product value can be articulated to teams and subsequently delivered. Hence, in Scrum, the question “when will the project be done” might not be a good one to ask a Development Team, since their focus ought to be on meeting Sprint Goals and delivering a release-quality increment. In effect, each Sprint might be a project to them. It is perhaps a better question to ask of a Product Owner, given that the Product Backlog ought to be at his or her disposal, along with empirical measures of ongoing value delivery. That is the person who will be able to account for the value obtained so far, and for how much remains to be provided, and whether the cost of doing so is likely to be justified. This is the person who will work with the team to ensure maximum value is released to stakeholders, incrementally every Sprint, and not at a deferred point in the future. The Product Owner is the individual who should decide, on behalf of the stakeholders represented, if and when Sprinting is no longer justified and an end-point to the initiative has been reached.

Sprinting to an answer

Scrum is agnostic as to how a Product Owner might actually calculate “value". Obviously the notion of what constitutes value is likely to differ between organisational contexts. A supermarket chain, a bank, a government office, and a charity may have very different understandings of what it means. The ways of assessing it are perhaps as varied as the number of complex products which can be delivered. All, however, must involve the Product Owner putting the Product Backlog in some sort of order which will allow “value” - whatever it is - to be maximised each and every sprint.

As such, there are a number of techniques which can be applied, including throwing the definition of “value" back at stakeholders and using brute arithmetic to calculate a return on investment (ROI). Here’s a dirt-simple method a Product Owner (PO) might be tempted to employ:

• Ask stakeholders to estimate the value of each item in the Product Backlog using value poker, or whatever other method they see fit

• Ask the Development Team to estimate the size of the Product Backlog using planning poker, or whatever other method they see fit

• Ask the Development Team to estimate their likely velocity each sprint, whether it be by experience, gut-feel, or some other method

• Divide size by velocity to calculate the projected number of sprints

• Divide the total estimated value of the Product Backlog by the projected number of sprints to calculate the projected value per sprint

• Estimate the cost for funding each sprint, taking into account team size and salaries, fixed costs etc.

• Divide the projected value per sprint by estimated cost per sprint to yield the projected ROI per sprint. 

In other words, if up-front estimation is done on a Product Backlog, it is possible to dial in certain numbers, turn a handle, and get other numbers out. A Product Owner can calculate the projected number of residual sprints and an average ROI per sprint. This can be done at any time during a project with respect to the work which is thought to remain. A “value-based” decision might thus be made on whether or not to continue.

Clearly though, what this block-headed approach doesn't do is to allow for a variation in worth across the increments themselves. Remaining value is calculated up-front, puréed and spread out evenly over each future iteration. Obviously things are unlikely to work out in such a way in reality. The outcome of one iteration can and should influence the value which is planned and delivered in the next. Each increment may be expected to achieve different things, perhaps with quite different Sprint Goals, and some iterations will therefore be worth more or less than others. That's the shortcoming of a predictive approach such as this. No allowance is made for adaptive change and validated learning. The emergent nature of value, which comes from inspection and adaptation, simply isn’t taken into account. No thought is given to innovative potential, and yet without this very little will be learned or improved. A predictive calculation makes a nonsense of agile delivery. After all, if you really wanted to be predictive about ROI, and didn't care about empiricism, learning, and innovation, why would you choose an iterative and incremental approach in the first place?

This is why we need to look beyond such coarse methods in agile practice, and towards a more useful way of assessing emergent value. One technique is known as Innovation Accounting. Let’s look at an illustrative example of how this might work: a public sector health-service initiative in a distant country, where you have been brought in as an agile consultant.

Welcome to Ruralania

There are few towns of any size in Ruralania. The capital city, Ruralpol, is the largest, and half of the country’s inhabitants live there. Ruralpol has the best equipped hospital in the land with eight hundred beds, five operating theatres, and outpatient services. Its diagnostic capabilities include an MRI scanner. The rest of the country is made up of remote and scattered villages. There are only a few landlines. However, mobile phone use has expanded greatly over the past 10 years. There is good network coverage from providers who recognised the market opportunity, and today it is normal for Ruralanians to have a smartphone of some kind.

The village hospitals

Each outlying village has a hospital or clinic averaging 20 beds, although many have only 4 or 5. There are 1 or 2 operating theatres in each facility. Difficult cases are referred to the capital. Due to the remoteness of each location, and a historical lack of telecommunications infrastructure, a comprehensive IT capability has never been developed for the health service. There is no network linking hospitals with each other, and there is no consistency in terms of IT estate. All that can be established with confidence is that each hospital has at least one working land-line telephone.

Your project

Funding has been obtained from international sources for technical uplift across the health service. The first initiative is to upgrade and centralise the hospital patient management system. A pet project of the Ruralanian health minister and its senior sponsor, he wants each hospital in the country to be able to record and update the beds it has available, in real-time. Patients can then be transferred as swiftly as possible to the next nearest clinic if their local one is full. He forecasts treatment efficiencies in the region of 28 million rurals per year. New PC’s will be installed in order to provide consistency of estate, while bandwidth demands will be kept low in order to support the existing analogue landline capability.

However, you are suspicious about the fundamental use-case itself, because it is predicated upon the manual entry of data. It does not make use of any sort of automated patient tagging. You point out that this may be a missed opportunity to have a less error-prone system, but the minister considers the idea of tagging to be far-fetched and radical.

A problem arises

Unfortunately, after the first couple of sprints, it becomes clear that hospital staff aren’t really engaging with the new system at all. Data entry is very patchy, and it can in no way be relied upon for accuracy when needed. The project looks like it will be a dismal failure, and the health minister is now afraid of losing face. You avoid saying “I told you so” and head up country to find out exactly what’s going on.

What you discover is that in most of the hospitals, the staff just track bed availability in their heads. If they ever have any doubt about capacity - which they hardly ever do - they just need to walk a few paces, look at their ward, and see how many beds are in fact available. Admissions and discharges are generally recorded in a physical ledger. The new system seems like a strange proposition to the staff. It means extra work and they see no clear incentive to use it.

An opportunity to pivot

Interestingly however, you notice that a completely unforeseen use case has emerged. The new system happens to include an elementary messaging feature of sorts. The intention behind it was to allow real-time qualitative observations and clarifications to be entered regarding hospital status. This is being used quite creatively, even though the fields concerning bed availability are largely ignored. Admittedly, sometimes the doctors just use this feature to update cricket scores, but they’ve also found that by tethering to a 4G connection they can use an attachment capability. This gives them a means of sending photographs and X-rays to each other, or to consult regarding MRI scans from patients who have been referred to the capital. Also, the nursing staff have discovered they can use this same messaging capability as a forum for exchanging medical supplies. They have developed an ad-hoc convention of prepending "SUPPLIES" to a comment. They find it to be easier than phoning around, which is what they usually do, and it’s much quicker than reordering material through the Ruralanian bureaucracy.

Your recommendation

You suggest to the health minister that he might pivot to these new use cases instead. A new purpose has emerged for the evolving system which will add immediate value. He sees potential savings of 35 million rurals per year in reduced treatment delays and 14 million rurals in supply chain efficiencies. He therefore agrees to your suggestion and authorises an appropriate scope change. Moving forward, the system will support the exchange of medical inventory and patient data, while making use of the new mobile infrastructure which is already in place. He also agrees to look again at the possibility of patient tagging, which he now accepts may be a more reliable indicator of hospital utilisation.

Conclusion

Trying to predict the value that will be obtained from an evolving and complex system often proves to be a fool’s errand. The hard financial forecasts of conventional accounting can look rock-solid, but they reassure us with false certainty, and represent pixie-dust rather than precision. It is usually best to formulate an hypothesis, test it as soon as possible, and inspect and adapt progress based on the empirical evidence obtained. New and better ideas can then be proposed and tested in turn. However a Product Owner defines “value”, it ought to be predicated on such empiricism. Once the core discipline of validated learning is brought to bear, a more appropriate means of assessing value becomes possible, such that even the most basic premise can be revised.

Innovation accounting ought to be appreciated in terms of the validated actions which can be taken, Sprint by Sprint, based on empirical evidence obtained. That is more likely to represent the best use of seed capital than persisting with a discredited value proposition. The most limiting constraint is generally the speed at which the agile organisation is prepared to learn.