What is a Product Backlog for, anyway?

First published on Scrum.org, 9 November 2017 (syndicated)

"Perfection has to do with the end product, but excellence has to do with the process." - Jerry MoranScrum requires a Product Backlog and a Product Owner to account for the value of product increments. For as long as a product exists, a Product Backlog will exist to describe the work which ought to be done.

I remember the raw emotion, the sheer thrill, when as a Scrum Master of some eleven years standing I first had a team where this arrangement was genuinely in place. The sense of luxury rang through me like a divine peal of bells. I was tickled into a state of euphoria commensurate with shock. To have not only a curated Product Backlog, but a clear owner engaged in the managing of it...well, it seemed hedonistic, immoderate, and extravagant. Other Scrum Masters looked at me with envy from their positions of austerity, as they struggled to establish priorities by themselves, and goals, and expressions of value; as they fought to engage even a reliable proxy for product ownership, or indeed to have a coherent product at all. Theirs was a position I knew well of course, for it was mine until but recently. And so I started to feel guilt, as if I was the holder of some undeserved privilege. I had suddenly received an embarrassment of riches I was unable to share.

Yes son, I knew a real Product Owner once. What can we do though in a more wretchedly familiar existence, where the very idea of a product can seem implausible? For that matter what is a Product Backlog really for anyway?

Our experience in such matters tends to be qualitative. We know when we have good product ownership, which is to say ownership as it is defined in the Scrum Guide, and we think of our reality as levels of degradation from that ideal. Hence we begin to abstract a sense of the "best" sort of product to have. For example:

1. Best: There is a clear Product Owner who cares about the product, engages reliably with the team and with stakeholders, and who is keen to see the iterative and incremental delivery of value.

2. Second best: We have a value stream of some sort which a team owns. It will meet a real need and will yield value to stakeholders, even if it isn't a "product" in the best sense. Where product ownership exists it will be weak and may be proxied.

3. Third best: We see the alignment of a product and its backlog to an organizational function, such as a business or marketing division. There may not be a clear expression of value but there is at least an alignment with organizational stakeholders.

4. Fourth best: We can see that a team owns a capability within an organization, but the value to stakeholders is unclear and unaccounted for. All that can be said is that something will be produced. Apart from that, the team will organize themselves as well as they can. They will try to do such things as minimizing waste around hand-offs, or perhaps try to leverage any benefits that might be found in cross-training.

5. Fifth best: We may discern a capability which stakeholders expect to be provided, such as middleware, but do not wish to own. Benefit is more likely to be general to the enterprise than specific to one value stream. It may actually be a service. Stakeholders wait for one another to move first and fund its provision, so they can avoid the worst of the costs while plotting to reap the rewards.

Categorizing things preferentially in terms of how good they seem to be is a natural response when faced with a complex environment. In this case we might be tempted to perceive the above series of ablations, or falling away, from an idealized product state. The Product Backlog can indeed all too easily fall into a state of lost remembrance. Perhaps Yuriy Zubarev was right when, in the Cynical Agile and Scrum Dictionary, he described a backlog as a Land of Forgotten Dreams.

Now let’s go back to basics and consider the essentials of what a Product Backlog actually is, and see if we can re-evaluate the patterns which might lead to their creation.

In Scrum, a Product Backlog is an artifact. It comes from the mind of a curious Product Owner. Curious, because while he or she wholly owns it, a good owner is keen to elicit the needs of stakeholders and Development Team experiences, views, and opinion. Refinement of the backlog will occur, where value is framed for delivery in order to meet tentative goals. Workshops may be held and team estimates may be solicited so that those goals are realistic. In fact goals are essential to Scrum and they will shape a Product Owner’s thinking. The Product Backlog will be organized not just in terms of discrete items, but in terms of wider features and epics and themes, and likely Sprint candidates which will yield increments of empirical value.

In the November 2017 update to the Scrum Guide, ownership was strengthened yet further to encompass what is known about the product, rather than the things which are thought might be needed. We can see that with ever-increasing clarity, the emphasis in Scrum is being placed upon diligent Product Backlog curation by an authoritative owner. The semantics of what actually constitutes a product recede, if not quite into irrelevance, then certainly into more of an implementation concern. It doesn’t matter a hoot what we think counts as a valid product, or what might make one product better or worse in terms of validity than another. If potential value can be identified and ordered - which is to say authoritatively curated - then we have a product and a Product Backlog. From a Scrum perspective this is the essential distillation of principle.

The contrast is stark with a Lean or Kanban system where there may not be a Product Owner at all. We may find that there is an authority, such as the Chief Engineer in the Toyota Production System, or we may find that a backlog is curated by the team itself. We may find that it is not really owned, but that its emergence is driven through teamwork. Corey Ladas, who has promoted the Scrumban approach, has gone so far as to dismiss the Product Owner role as an “egregious error”. Whatever a Product Backlog might be in such a context, it is not necessarily the artifact of an owner’s mind, and hence there may be no concept of a product at all which can find coherent expression. Instead, there may be qualities of service and service level agreements.

These essential considerations lay out the ground for identifying backlog patterns and their driving forces. So far we have attempted a sort of Product Backlog typology, where we might categorize four or five ablative product conditions. They may seem real enough when they chime with our qualitative experiences, but this approach could easily lead us off into the weeds. The essential risk is this. Even if we do identify a value stream, or an organizational alignment, or a capability notionally “owned” by some group, or of course one which nobody wishes to own at all, we clearly have the problem of no true Product Owner being found. Applying these categorizations therefore risks leaving us with a false sense of product control. We can determine that a single critical factor must instead be considered - the presence or absence of a Product Owner to give purpose to the backlog, whatever that purpose might be. Like the Holy Grail, the question is not so much what the Product Backlog is for, but whom does it serve.

So now let's try something different. Rather than thinking in terms of an ablative model, let's start to identify the actual patterns and forces which might be associated with a lack of product ownership:

• A backlog may be a buffer or pool queue, the force being a need to minimize lead times between work items serviced. In this case it is driven by the express needs of consumers. The work itself may relate to more than one product or value stream, or to a stream without clear value at all. Irrespective of value, a delay in lead times is seen to create a pain-point for consumers.

• A backlog may be an artifact exerting pull for local team optimizations and a leaner workflow. Again, the work itself may relate to more than one product or value stream, or to a stream without clear value at all. However, the driving force is one of improving internal process rather than the service enjoyed by a consumer. Care is needed, as expectations may go beyond gains in mere local efficiency. For example, is an enterprise transformation supposedly going on? If so, a backlog which supports local pipeline optimization could just result in the wrong things being done more quickly.

• A backlog may be a roadmap for a big-bang release. This is the classic project situation where a leap-of-faith is taken and there is little sponsorship for incrementalism and empirical control. Risk is offloaded from consumer to supplier rather than managed. Product ownership is either absent or faked.

• A backlog may be a reflex of some strange phenomenon, such as an artificially induced constraint. For example, if money is left in the pot at the end of a financial year, there may be an incentive to spend it on something or indeed anything, rather than see a reduction in budget allocation the following year.

• A backlog may be an artifact exerting pull for team formation. Instead of having a known product, value stream, organizational function, or owned capability around which teams are formed, we see the effect of a new force at work. There may be a cleverness about it. Innovation is one example. The need to test MVPs very rapidly can emerge in a disjointed, occasional, and unmanaged way from unofficial factions within an organization and its stakeholders. If a backlog of this work starts to crystallize, an innovation laboratory might be sponsored to deal with it.

These patterns give us a sense of what Product Backlogs might be for, even when the owner they would serve appears to be missing. There will be others we can identify, but it's important to realize that patterns are all they really are. No procedure can be described for dealing with the diverse modes of weak product ownership. Unfortunately though, just as it is natural for the human mind to think qualitatively in terms of first and second best, so it is in the nature of organizations to seek firm protocols about what to do in given situations. That isn’t really what we have here or can reasonably expect. All we can do is to identify these and other patterns which give rise to backlogs, and use them to inform our thinking about product and service development. If we know why backlogs are brought into existence and understand something of the forces behind them, we can hopefully make decisions which lead to more valuable and sustainable delivery outcomes.