Scrum Roles from a Lean Perspective

First published on Scrum.org, 14 September 2017 (syndicated)

"It's weird when you get roles that coincide with your life" - Lily James

Introduction

Impressive-sounding job titles are a recurring joke in large organizations. They often bear little relationship to the devil's brew of chaos and drudgery which is a daily reality for most. Cynics may hold that these fripperies are offered by way of compensation for low wages and a dysfunctional working environment. Certainly, any feel-good moment from seeing a mighty role in your email signature is destined to be short lived. The higher-ups will be quick to claim that the performance of an underling ought to be commensurate with their imagined role, even when the salary and authority provided are not. It seems that the most underpaid junior is now considered to be a "manager" of some kind. Perhaps there are almost as many managers now as there are people.

Although dot-coms and startups can make a show of rejecting corporatist bunk, they often replace these sappy monikers with their own form of "role-play". You will undoubtedly run into people described as "test ninjas" or "automation rockstars", if you haven't done so already. My personal favorite was an "agile samurai"...or so his business card claimed. Needless to say it soon turned out he wasn't.

Roles can be a cultural barrier

Genuine agile transformation must be deep and pervasive. Any or all of an organization's current roles may need to change, if the enterprise is to improve the way it delivers value. No roles, not even those of well-paid senior executives, are sacrosanct. Any pretence about what people are actually doing must give way to transparency, so power and accountability can be devolved upon those who are responsible for doing the work. The essential challenge is one of effecting the necessary cultural change which circumscribes the things people genuinely need to do, and how they see themselves in relation to that work and to fellow team members. Without that change, organizational gravity will erode any impetus for transformation, as people will revert to established behaviors and cultural norms. Faced with the risk and uncertainty of change, organizational dysfunctions can appeal to us like old friends. Those seasoned roles and practices have carved out a groove leading from yesterday to today, and can presumably still lead us safely on to tomorrow. Why upset the cart which trundles along? Its wheels fit neatly into the well-worn rut.

Agile transformation, which must indeed be deep and pervasive, is therefore hard. The Scrum Framework elicits a mere three roles around which organizations are expected to align, while the Nexus exoskeleton for Scaled Professional Scrum adds just one more. Imagine the difficulty in changing an organizational structure, which may run into hundreds of roles, so that increments of release-quality work may be delivered at least once per month. Yet that is precisely what these few Scrum roles are configured to do, while the bloated structure of the typical large enterprise militates strongly against it.

Stage-gated roles

The essential problem is that organizational work is generally scheduled by resource availability, such that work is passed as a batch from one type of resource to another, and value is added to the batch at each stage. In large companies "resource" is a euphemism for people. Employees are objectified to the point that their value is equated to their role. If an employee leaves for one reason or another, the "Human Resources" department will try to ensure that the role is backfilled. Hence analysts are expected do their particular bit on a project, as are designers. Programmers, testers, architects, database specialists, and project managers are supposed do theirs. The batch of work will be handed off from one specialism to another at stage gates while a project manager exercises oversight. The associated scheduling and resource leveling can be complex, as any manager who has tried to maintain a serious Gantt chart can attest. The time taken from concept to project completion may run into years. By that point, the originally mandated scope may no longer be relevant in a continually evolving business context.

The leap-of-faith taken in such "waterfall" stage-gated initiatives can be daunting, and it will often degenerate into a so-called "death march", or at best a last-minute kludge. At first blush, the process may look like a well-oiled machine. Requirements documents are refined into design specifications, which are then implemented as source code, compiled, reviewed and tested, subject to version control, compiled again, integrated, and perhaps reviewed and tested again. However, each artifact which adds to the batch of work at each stage is nothing more than a promissory note for the final value, and the tasks ascribed to each role prior to release are an exercise in false precision. The case for a more iterative, evolutionary, and incremental practice of small cumulative batches was of course made years ago. The leap-of-faith is minimized to the size of a small batch before it is released. Unfortunately however, the leap needed to implement those agile practices may seem greater to executives than the challenge presented by any one of their vexing, lurching projects.

Roles in agile and lean production

An agile approach will not schedule work by resource availability. There will be no large batch of work which is processed in stage gates. Instead, resource will become largely orthogonal to workflow. This means that people will organize themselves around the work that needs doing and at the time when it is best done. In Scrum for example, the "schedule" is replaced with Sprints, and a release-quality increment will be made available by the end of each one. The scope of an initiative will be prioritized on a backlog by the Product Owner, and then gradually implemented in successive Sprints by the Development Team in accordance with its rules and work-in-progress limits. The Scrum Master will facilitate the team's progress and coach them in the best application of agile practice. These three roles of Scrum Master, Product Owner, and Development Team almost correspond to archetypes: we can discern in them a servant leader, a value maximizer, and those with perhaps no greater calling beyond a simple wish to do good quality work and make the results available. Articulating responsibilities in these terms is enormously challenging for most organizations, and it is certainly difficult to imagine how a workable set of roles might be reduced any further.

When Lean-Kanban practices are examined, however, we find that no roles are mandated at all. This isn't because roles are dismissed as being injurious or detrimental in some way, even though it is acknowledged that an organization's ill-considered mishmash of roles might indeed hobble the delivery of value. It's more the case that the formulation of a particular set of roles is irrelevant to the application of lean practice. If you approached an organization with a set of roles you thought should apply - even just the three elementary Scrum ones - then that would be seen as a prescription too far. In his book on Scrumban, for example, Corey Ladas denounces the role of Product Owner as an "especially egregious error", on the grounds that it trivializes the varied and complex skills which are needed. Whether product ownership is indeed such a trivialization, or the vital focusing of product authority and accountability, is of course debatable. We can identify similar roles in lean manufacturing such as Chief Engineer in the Toyota Production System. The point we need to draw out here is this: the roles and responsibilities which are thought appropriate for a Lean-Kanban team are determined not by prescription but by situational context.

When I first started to coach agile practice, it quickly became clear to me that most companies find it hard to implement deep organizational change. The Scrum roles of Product Owner, Scrum Master, and a cross-functional Development Team are usually a very great leap from the established orthodoxy. They might not appear to fit the organizational context at all. "Things just don't work that way here", I would be told by managers and old company hands. The pressure would be on to change the Scrum Framework instead...to "configure" or "customize" the role structure, and the associated artefacts and events, to match the corporate reality. The result of acquiescing to this pressure is of course very little change at all, and the preservation of the status-quo. The supposed sponsors of the agile transformation then become disheartened when the benefits they hope for continue to escape them. The opportunity to demonstrate success with agile practice is also lost. Any political head of steam that may have built up soon dissipates, along with the goodwill of employees to attempt change again in the future.

Stepping over the quagmire

Agnosticism about an organization's roles can therefore be a sensible policy, at least in the short term. Rather than trying to cultivate transformation in a reactionary environment, a more prudent approach is to just place a window of transparency over current ways-of-working and the things people do. Inefficiencies may then be exposed and the organization can change its practices, including roles, at a measured pace based on whatever evidence is presented and the advice of the coach. In fact, a tenet of Lean-Kanban is that it is best for organizations to start with what they are doing now, and only then seek to improve matters. That's basically what I learned to do in organizations where the appetite for Scrum was weak. Raising a Kanban board to visualize current workflows does not solve anything in itself, but it introduces a lean and agile grammar, and exposes areas of concern which can then be acknowledged and dealt with.

Although a Scrum implementation may emerge during this journey, a series of local optimizations is unfortunately the more probable result. The desire for radical change might simply not be there. Middle management, for example, very often see a duty on their part to defend the organization's culture and its perceived standards, even when deep agile change has been established as a strategic goal. The dysfunctional antics of the "frozen middle" or "permafrost" are very common and are well attested to. Where an incumbent process is shown to be flawed, the contradictions within it are more likely to be optimized than rooted out. Doing the wrong thing more efficiently is often seen as the less painful option, in so far as it involves comparatively small and uncontroversial changes, appears to be cheap, and kicks the can of serious reform further down the road. Transparency may very well be the first leg of an agile transformation, but it does not inject backbone into a company's managers. Yet without robust sponsorship for deep and pervasive change from the executive downward, organizational gravity will pull and tug at a reformation attempt until the trajectory matches the status-quo. Subordinate to the constraint of that greater force, incremental advances will eventually peter out and choke off. No significant change will happen. There can be local optimizations which are well worth doing, but the launching of any number of small squibs is unlikely to achieve escape velocity and set the company on a strategic course for enterprise innovation. The "Kanban-as-a-window" approach is not generally one which brings eye-popping improvement.

What it leaves us with is a confined opportunity to improve things as far as possible at a localized team level. It is possible to view weak sponsorship for enterprise change as an enabling constraint, since each team can at least focus on getting its own house in order. For example, team members may choose to optimize their own way-of-working in terms of supporting better craft production, or perhaps by organizing themselves as a more efficient feature crew. They may even take the principle of the "enabling constraint" to heart by limiting their work-in-progress, so that throughput, quality, and teamwork might be enhanced.

Craft production and feature crews

A focus on craft production should make a competent generalist out of each developer. There ought to be no aspect of development which a good craftsman is unable or unwilling to do. If one has the capacity to take on a new piece of work, he or she will do so and will have all of the skills to take it through to completion. Although such a system of craftsmanship may sound like a highly productive and perhaps even an idealized system, there are a number of problems with it. Firstly, creating competent generalists is likely to prove an expensive exercise when only specialists, such as testers, architects, designers, and programmers might well be available. Secondly, the opportunity to harness the exceptional skills of a true expert in a particular area will be lost, as it is the generalist who is valued. Thirdly, little in the way of knowledge sharing about the work being done, or collaboration in order to expedite that work more quickly, is likely to happen. Perhaps in a craft system there can hardly be said to be a team at all. There is merely a workgroup of independent craftsmen who happen to have been brought together in time and possibly space.

Alternatively, they may decide to put the emphasis squarely on teamwork by enhancing their competencies as a feature crew. None will be generalists, but in aggregate, the team will have all of the skills needed to create a finished increment of quality and value. Work will be passed from one specialist to another at an agreed pace. They will collaborate closely in order to ensure that these handoffs are properly synchronized, and that the team collectively observes the most productive cadence. Again this may sound like an efficient system, but yet again there are problems. Firstly, the rate of flow will have to be constrained to the speed of the slowest expert, or else work will surely build up somewhere in the process and a bottleneck will occur. Secondly, if hiccups such as that arise, it may not be possible for a different specialist to help out. Even if one had the time to leave his station and do so, the skills demanded at the problematic station would be unique. Just as we find in a stage-gated system, each person does their bit according to their role. Thirdly, adding further experts to avoid the risk of a build-up may prove to be wasteful. Complex work is not all of a piece, and bottlenecks can occur unpredictably and at different stations. Additional specialists brought in to fix today's snafu may not be the ones needed tomorrow.

Things Scrum can teach us

Fortunately, even when Scrum is not being implemented, the framework can still teach us much about how to think about a problem. A Scrum Development Team, like a feature crew, is responsible for creating a usable finished increment of quality and value. Note that the role "Development Team" is a collective term, and that it isn't "Developer", since all Development Team members must share in that collective responsibility. Yet Scrum says nothing about the specialisms which might be needed for a feature crew to work. In fact those team members could all be generalists operating within the protocols of craft production. In short, it is left to each Development Team to self-organize in the way they consider most appropriate for the work at hand. A Scrum Master, as a servant-leader and agile coach, ought to be able to help them to do so. Perhaps Scrum is not so far from determining roles and responsibilities by situational context after all.

The most common approach taken in Scrum is to combine elements of craft production within a feature crew, or at least to aspire towards doing so. The ideal developer is sometimes referred to as having "T-shaped" skills or as a "generalizing specialist". Such a Development Team member will have a specialism in which he or she is expert, and will also have at least a minimal degree of competence in each of the disciplines needed for production. The essential principle is that each team member should be able to "go to where the work is" instead of waiting for the work to come to them. If there is a build-up or bottleneck, they will be able to pitch in and help to expedite that work-in-progress before anything new is taken on. In an agile way of working, remember, schedule ought to be orthogonal to workflow. Yet we still have the problem that generalists are rare and investing in them is expensive. However if a team member can provide competency in just one supplemental skill, then that may be adequate to deal with any risk that is likely to emerge. For example, a developer may reasonably be able to conduct at least some testing if work is seen to be backing up in QA, while a tester may be able to author at least some test scripts when they find that no work is yet available for inspection. Perhaps a business analyst may also be able to step in as a tester if needed or vice-versa.

Conclusion

When team members can move around the work, instead of waiting for it to come to them, the requirements become orthogonal to the system of production. The person who does the work is not necessarily determined by the nature of that work. Instead, team members can develop a joint focus on the activities at hand. The constraint is no longer the skills of the team but how quickly and effectively they can deliver value to a clear and accountable authority such as a Product Owner, and the extent to which they are supported in those endeavors without interruption. Here we can see the value in also having a Scrum Master, a servant-leader and coach who ensures work is not pushed onto the team, so work-in-progress is kept deliberately limited and the best focus and throughput is attained.