> agilepatterns.org

Agile Development in Practice: the official reference book

Upcoming classes

Professional Scrum Master (PSM). A series of classes are being held over 2021, most of which are being delivered online. This is the official Scrum.org course. Two attempts at the PSM I assessment are included along with a 40% discount on PSM II. Click here to register. You can also register on EventBrite.

Enterprise Change. Organizational change is largely cultural. In partnership with Debrett's, we offer training courses on developing the soft skills for agile enterprise transformation, in the context of a transformation framework. Please get in touch for more details.

Agile Transformation in the News

OCTOBER 13 2020: 18F, a US Government digital services agency that advocates lean and agile development techniques, has produced a guide outlining common reasons why agency IT projects fail...and how to do better. Federal News Network has provided some commentary.

AUGUST 20 2020: Banks are only "pretending" to embrace agile development...a commonly heard complaint, and one that is reflected upon by efinancialcareers. "The fact is that many technology teams in banks still do waterfall development even if lip service is paid to Agile - They may have incorporated some of the agile ceremonies, tooling and principles into their day to day work, but their product will only be used when it’s finished". Note: It would help in the adoption of agile practice if inspect-and-adapt events aren't seen as "ceremonies", but as opportunities to improve product and process.

JUNE 13 2020: COVID-19 has required enhanced organisational agility, but will it stick as the lockdown eases? This is the question posed by the Agile Business Consortium, who surveyed over 150 people including executives, heads of transformation and delivery teams. There are three main findings. Firstly, Covid-19 has forced many organisations to operate at unprecedented levels of pace in order to adapt. Secondly, organisations have been able to achieve this step change by radically changing the way they manage change using agile principles. Thirdly, there is a concern as to whether or not the learnings and practices gained from the crisis will be sustained as the world returns to some level of ‘normality'. Read more on the Consortium website.

APRIL 28 2020: Digital transformation is failing to gain traction in the US Government, according to this article in Big Think. Billions have been lost on cancelled projects. "The list of federal tech-based projects that failed to deliver is long & getting longer, a clear sign of the desperate need for federal digital transformation". Note: A lack of strategic leadership is identified. Instead, there is a "highly risk-averse culture opposed to the continuous innovation (CI) and agile fail fast, fail quickly mindset necessary for successful digital projects".

MARCH 23 2020: "Your Agile-built IT platform was 'terrible', Co-Op Insurance chief complained to High Court". The Register reports on a case of agile failure which eventually went to court. Note: This can be seen as a typical example of "agile" practice being applied as a magic salve, with no desire to inspect & adapt in a timely fashion or to engage in validated learning

JANUARY 9 2020: "Organisational change is hard everywhere – even more so in governments" reports GovInsider Asia. Three key lessons on Singapore's agile government initiative are explored in this article. Firstly the right conditions must be in place for people to work together, secondly processes must support and prioritize change, and thirdly open source technologies should be used as an innovation backbone. Note: In an agile way of working, managers will increasingly need to demonstrate servant leadership. As the article observes, "Managers instead play the role of coach, facilitator and resource finder, supporting the team as they self-organise and make their own decisions."

DECEMBER 12 2019: Federal Computer Week has interviewed Justin Fanelli, a government agile veteran with experience of about a dozen agile initiatives. In the interview, Fanelli claims that obtaining user involvement in frequent feedback is, if anything, harder than it was when he first started. Note: The report indicates that, due to the growth of agile projects, the demand on the time of users has grown while Product Ownership is "not where it needs to be yet".

NOVEMBER 18 2019: Agile working is becoming a strategic transformation priority, according to Consultancy Europe. They report a new study by KPMG in which respondents from 17 countries participated. It was revealed that more than 4 out of 5 mid-to-large organizations have either initiated or completed an Agile transformation in the last three years. Note: 59% of respondents mentioned culture and performance management as their key challenge in their shift towards agility.

NOVEMBER 15 2019: The CIO of Bankwest has claimed that agile incubators are “doomed to failure”, reports IT News. Too many organizations sponsor agile transformation half-heartedly, or cherry pick the concepts they think will be easiest to apply. They then find their efforts smothered. Instead, commitment is needed "right across the organisation from the top down to address natural resistance". Note: The need for clear executive sponsorship underpins a successful agile transformation, and it must be strong enough to overcome organizational gravity.

SEPTEMBER 22 2019: The adoption of "fake" agile practices across the US Department of Defense presents a national security risk, writes Steve Denning in Forbes magazine. Although DoD software development projects are very often declared to be "agile", in reality, they often fall well short of this standard. Note: The DoD has issued a guide for detecting such problems including the WaterScrumFall antipattern.

JUNE 20 2019: According to Breaking Defense, the Pentagon is pushing hard for the adoption of DevSecOps in the Department of Defense. The driver is the desire for "a new software development model that gets the bugs out early through constant testing and improvement". Note: As with the transformation of a European bank described last month by McKinsey, a fail-fast mentality is being encouraged for which automation is understood to be critical. The Pentagon's idea, Air Force Chief Software Officer Nicolas Chaillan has said, is to “fail fast, but don’t fail twice for the same thing.”

MAY 9 2019: McKinsey have described an agile transformation at an (unnamed) large European bank. "First, they put together three Scrum teams consisting of a total of 14 developers and testers plus five business people", reports Computing Magazine. Note: Scrum inspect-and-adapt events were observed, while the Scrum Master role facilitated collaboration. A fail-fast mentality was encouraged for which automation proved critical.

MARCH 13 2019: Companies that embrace agile practices enjoy a financial and operational advantage over their rivals, according to new research from PA Consulting. Two thirds of 500 executives surveyed believed that their business model was at risk of becoming outdated, and that enterprise agility would assist in future-proofing. Interestingly, 70% recognized that the "internal DNA" of their organizations would need to change.

JANUARY 9 2019: The UK Ministry of Justice has halted one of the agile workstreams of its £280m Common Platform Programme, according to The Register. Warning signs began in 2017 when one of the key agile consultants quit. It seems that due to successive delays, keeping the old system going has diminished the "immediate need" for a replacement.

JANUARY 4 2019: The Norwegian Labor and Welfare Directorate has sought to transform its IT department to better support enterprise agility. In this interview on InfoQ, the director and the team lead describe the journey. Note: The move towards cross-functional teamwork was identified as a huge cultural change.

DECEMBER 31 2018: Accenture has released its State of Federal IT 2018 Report, in which it claims to have found gaps between the adoption of agile practice by government agencies and stated objectives. Only 47 percent of IT decisionmakers believe they’re making sufficiently effective contributions to mission agility.

OCTOBER 22 2018: Fake agile implementations are now so common that the US Defense Innovation Board has produced a guide for recognizing them, according to this article by Government CIO Media. The guide identifies red flags that an agile claim may not be authentic. Problems include users being kept out of the loop, and development teams not spending time with actual users. Note: The WaterScrumFall antipattern describes this pathology.

SEPTEMBER 27 2018: 18F, the US General Services Administration office for digital innovation, has identified 3 antipatterns which are prevalent in agile contracting. These include turning a Product Backlog into a "laundry list" of user stories, having a separate contract for every release, and the precocious establishment of vendor pools.

AUGUST 14 2018: "The Best & The Worst Of Times for Agile". In this Forbes article, Steve Denning says that although 90% of senior executives give “high priority” to becoming agile, less than 10% actually see their own organization as being “highly agile” and consulting firms are exploiting the gap. “As ‘scaling Agile’ becomes the fashion", he observes, "rather than descaling complex problems into manageable pieces of work, scaling frameworks are being sold that are in some cases just another name for bureaucracy”.

JUNE 11 2018: "Industry, Agency Leaders Praise Government Move to Agile Development". In this article by MeriTalk, the CIO of the US Department of Defense finds reason to be positive, despite the department’s failing grade in incremental development as per the terms of the Federal IT Acquisition and Reform Act (FITARA). She intimated that agile transformation is now underway for many DoD projects. Note: Under FITARA, eight agencies which at first received F’s in incremental development now apparently account for six A’s and two B’s.

APRIL 30 2018: According to this news article in The Guardian, the warning signs for TSB's IT meltdown were clear a year ago. An insider spills the beans. Note: This can be seen as a case study in the problems caused by technical debt. However, "debt" may be too grand a term, considering that the losses incurred were not quantified and there was no plan for repayment.

APRIL 25 2018: Government organizations are increasingly turning to Agile methods for software development, but applying them to large IT projects can be a challenge. In this report for the Wall Street Journal, Deloitte point out that a few key strategies can help. Note: Strategies such as these can be implemented by applying transformational patterns, including Release Orchestration, Sponsored Vision, Teamwork and Done.

APRIL 11 2018: The US Department of Defense could take a page from SpaceX on software development, an advisory panel to the Pentagon has reported. SpaceX is held up as an exemplar of an agile and DevOps capability which satisfies mission critical standards. For nearly a decade, SpaceX has used Scrum for enterprise resource planning, space operations, and apparently even for finance and human resources. There is also a continuous deployment pipeline for updating critical internal information multiple times per day.

MARCH 2 2018: Can agile techniques be applied in the public sector on small-scale initiatives? In this article in Software Development Times, Cherie Jarvis examines how the US Forest Service created an interactive visitor's map in one year with a cross-functional Scrum Team of 9. Note: Even on large transformation projects it is best to first ensure that even just one team can deliver a release quality MVP which meets its Definition of Done.

FEBRUARY 9 2018: “Obama’s Agile Startup 18F Has Shrunk By Half Under Trump”, writes Mark Sullivan in Fast Company. The "startup-like group" was founded in 2014 to help U.S. government agencies adopt agile practices. Meanwhile another government IT office, the U.S. Digital Service, continues to recruit new members amid concerns about further cuts to Obama-era tech efforts.

DECEMBER 20 2017: According to a recent Forrester report summarized by ZDNet, " The State Of Agile In 2017: Agile At Scale", 44% of respondents said that their firms lack skilled product owners from the business. Furthermore, 31% said that PO's didn't spend enough time with development teams. Note: Although Proxy Product Ownership can be used to ease the demands made of a Product Owner's time, the PO must alway remains accountable for the value delivered. He or she must therefore ensure that any proxy has the skills and the availability to perform the role.

NOVEMBER 8 2017: Most agencies are failing to certify implementation of incremental development on major software projects, despite a FITARA requirement that they do so, reports Federal Computer Week. A lack of expertise, overtaxed staff, inefficient governance and procurement delays are cited as contributory factors. Note: Although the value of an increment is alluded to, the standard only appeared to require a six month delivery cycle. The Agile Manifesto describes delivering working software at a frequency from a couple of weeks to a couple of months, with a preference to the shorter timescale.

OCTOBER 31 2017: There are three practices and techniques that could that can help address the challenges of scaling agile development, according to this article in Federal Computer Week. Enabling cultural evolution, combining emergent design with intentional architecture, and adopting agile governance are critical to aligning agile development efforts with mission objectives.

OCTOBER 3 2017: In a recent McKinsey survey into agile transformation, respondents who described their business environments as "unstable" were more likely to claim that their organizations had begun a transformation attempt. However, few such enterprise transformations would appear to have been completed. Only 4 percent of all respondents said that their organizations had fully implemented agile practice, while another 37 percent claimed that enterprise transformation is in progress.

SEPTEMBER 11 2017: ANZ Bank is restructuring to create '150 start-ups', according to the Sydney Morning Herald. Note: The bank is broadly attempting to emulate an approach which has come to be known as the Spotify model.

AUGUST 22 2017: IBM has said it will use Agile approaches not just as a methodology for developing software but as “an engine for business transformation”. Diginomica reports that to this end, IBM has spent $380 million in setting up a number of “agile hubs” in six strategic centres in Austin, Texas, San Francisco, Atlanta, New York City, Cambridge, Massachusetts, and Raleigh, North Carolina. Note: The need to improve collaboration and to Fail Early Fail Fast are evidently significant factors.

AUGUST 3 2017: Software created in corporate "innovation centers" often fails to make the break into mainstream development. In this CIO Magazine article, TD Ameritrade CIO Vijay Sankaran explains how he intends to break that cycle. Sankaran says he has turned to agile practices, design thinking and lean startup methodologies for quickly building adaptive products. "There is an opportunity to leverage digital technologies to continue to disrupt" he claims. "If you don't do the devops piece in parallel where you have automated testing and continuous integration and you can deploy packages automatically you create a bottleneck in a different place". Note: Sankaran says this has allowed the innovation center to deliver prototypes in one to three months and minimally viable commercial products in only six months.

JUNE 21 2017: The Department of Homeland Security has canceled a large-scale project to make agile software development services easier for agencies to buy, reports Nextgov.com. The initiative let agencies depart from the old acquisition process which could take months or years to work through bids. However, the new system proved too complex to be viable, with 100 companies participating in half-day technical challenges which had not been adequately prepared for, resulting in a "logistical nightmare". Bidders who didn't make the cut were then in a position to claim that the acquisition process was unfair.

MAY 3 2017: The UK 'wastes billions every year' on failed agile projects, according to this article in IT Pro. It reports that "More than half of CIOs think the methodology has been discredited and is just an IT fad". Reference is made to a survey in which it has been found that "Just under one third (32%) of projects are failing because the teams are too geographically dispersed, while 34% have failed because the teams didn't plan before getting started or didn't plan sufficiently as the project developed". Note: Geographical dispersion is a symptom of the unbounded team antipattern while a failure to prepare suggests uncommitment on the part of stakeholders who may be too busy. Inspection and adaptation underpins an agile way of working, where replanning ought to occur on an ongoing basis.

APRIL 3 2017: The Register reports that "hundreds of millions of pounds" have been wasted on a supposedly agile delivery program for digitizing the UK's judicial court services. "Particular scorn has been directed at the so-called 'agile experts' who have been in charge of managing the programme. 'There is no plan, no artefacts, no direction, just constant excuses,' said one insider. 'How they can still be in place as well as still being allowed to recruit 'experts' with absolutely no delivery after 30 months is scandalous.'" The Register also cites an insider as saying that "if the programme were following proper agile principles, by this stage there ought to be at least a dozen meaningful services available across the criminal justice system". Note: A failure to plan and release value incrementally is contrary to an agile way-of-working, as there can be no empirical evidence of progress. The result in this case has been an escalation of commitment when controlled failure may have been the wiser option.

MARCH 1 2017: Eastern Foundry, a collaborative workplace for Federal Government contractors started by military veterans, has spoken out in defense of the 18F initiative. Writing in Nextgov, they point out that although the 18F initiative has struggled, "there is a cost to opposing innovation and forcing reform to move at the pace of government acquisition...18F was created out of a massive procurement failure, and it has moved mountains to help agencies and the nation avoid such costly errors".

FEBRUARY 17 2017: A report by diginomica asserts that although growth has been modest, large enterprises are nonetheless buying in to the agile development approach. In this article Jon Reed looks at Computer Economics’ new survey on agile development, and gains insight into the continued relevance of agile practice and why barriers to adoption persist. Note: Two essential patterns are elicited as a takeaway: "Agile is really about iterative cycles. When you combine agile with a minimum viable product approach, you have a huge edge building relevant software that customers/users actually feel like using".

JANUARY 26 2017: Mobomo, a Washington DC supplier to the U.S Government, has explained its use of agile techniques in the Federal space. In this interview in Application Developer Magazine, they explain the key roles and benefits and the new disciplines which are needed. Note: Many good patterns are identified here, including Product Ownership, Teamwork, Servant Leadership, and Swarm.

JANUARY 12 2017: A new CA Technologies (NASDAQ:CA) global study indicates that there are great advantages to be had in using agile and Devops techniques together. Mature adopters "realized significant increases of up to 52 percent in customer satisfaction and up to 50 percent in employee productivity", according to this report in BusinessWire. Note: Those advanced users recognized that agile and DevOps practices have to be used jointly before benefits can accrue. Just as DevOps is about changing organizational culture - and is not simply a matter of automation - so agile delivery requires teamwork that bridges the "development-operations divide" and optimizes the value stream.

DECEMBER 20 2016: Software development and operations teams working for the Federal Government need better collaboration and use of automated testing. Writing in CollabNet, Thomas Hooker says that "Agencies using traditional Waterfall methods for software development will need to strongly consider shifting their strategy to Agile practices in order to scale and streamline the software delivery lifecycle". The driver is new standards for compliance, governance, and security. He says these demand that "the bar should be set high when it comes to the security and quality of technology". Note: Many agencies struggle to adopt agile practices and resort to the Waterscrumfall antipattern rather than sponsor the deep and pervasive change which is required.

NOVEMBER 10 2016: The digital recruitment head at the Department for Business, Energy and Industrial Strategy (BEIS), has resigned. In a damning blog post picked up and reported by Government Computing, Ann Kempster criticized "...a lack of appetite in departments for real, meaningful transformation. I’m seeing a lack of effective leadership right from the very top of the civil service...There is a lack of vision, lack of ambition and lack of any sort of a plan anywhere". Note: The agile transformation crisis will continue in Government until appropriate sponsorship and vision is in place for the deep and pervasive change that is clearly needed.

NOVEMBER 3 2016: Bipartisan congressional support for the Modernizing Government Technology Act of 2016 (MGT Act) has improved efforts to modernize federal IT, according to Claire Bailey in InsideTechTalk. Concurring with Federal Computer Week that "modernizing big federal IT systems isn’t really about the technology—it’s about solving user’s problems effectively", she says that the need to actively involve customers in agile practice - and to be prepared to fail early and fast - are important steps. Note: Effective Product Ownership underpins successful customer liaison, especially at scale where many stakeholders can be involved. The Controlled Failure pattern should also be mastered if the Death March antipattern is to be avoided.

JULY 18 2016: The Department of Homeland Security agency has only partially implemented an agile way of working, a congressional audit has found. While the Citizenship and Immigration Services department claimed to have adopted agile software development techniques for its huge, $3.1 billion technology transformation from paper to digital, success has been patchy. According to an article in FedScoop, User Story prioritization has been accomplished reasonably well, but integration of deliverables has been poor. Note: The timely release of fully integrated increments of release quality underpins agile practice. Without this the value stream will be inhibited and release orchestration will be compromised. Note also that the audit itself set an expectation that release planning ought to be complete prior to initiating development, which is itself contrary to the agile principles that are supposedly being adopted.

MAY 12 2016: The BBC has been criticised for thinking that agility means "making it up as you go along". According to an article in The Register, the National Audit Office found significant shortcomings in the organization's project for customising the user experience of the MyBBC portal. “MyBBC is an ‘agile’ project that was designed to define benefits as the project progressed," the NAO is quoted as saying, "but two years into the project it still was not clear what the BBC expected MyBBC to achieve overall”. The report goes on to claim that benefits were only defined late in the day, two years after the project had started. Note: The BBC Trust is quoted as saying that the project "did not define its its expected benefits upfront" because it was "an ‘agile’ project where benefits were to be defined as the project progresse[d].” This is a misunderstanding of agile practice, since a Minimum Viable Product ought to be delivered regularly and incrementally, where each MVP is framed in order to validate or invalidate a clear and testable hypothesis. Failure to do so can encourage Vanity Metrics, an antipattern in which success is not measured objectively, but rather by attributes that show the initiative in the best light.

MARCH 8 2016: There is no linear relationship between the number of validated hypotheses and a team’s subsequent success, according to an article in Harvard Business Review. Ted Ladd provides thought-provoking evidence that although "the Lean Startup method works", and that validated learning helps teams perform "almost three times better in the pitch competition than teams that did not test any hypotheses", more is not necessarily better. In essence, he has determined that the relationship between validated learning and subsequent success is non-linear. "There was no linear relationship between the number of validated hypotheses and a team’s subsequent success" he observes. "I also found that teams that conducted both open-ended conversations and more formalized experiments with customers actually performed worse in the competition than teams that conducted either one or the other during the early stages of venture design". Note: The author has cited possible explanations for this such as erosion of confidence due to rapid changes in direction, or managers running out of patience for continued testing. Both are plausible. Validated learning may be applied as an enabling constraint, but not as a substitute for having a sponsored vision.

JANUARY 25 2016: CompuWare CEO Chris O’Malley has expressed optimism that mainframe development can bridge the chasm to agile practice. The main challenge he sees is changing the culture. “The gravity against change is huge”, he said. He explained that cultural change is hard because team members will always come up with a "rational" reason that their particular area can't change…even if they see the necessity of change for the rest of the organization. Note: The application of the Transformation pattern requires clear executive sponsorship so that deep and pervasive change can be effected across an enterprise. Employees must be left in no doubt that their own behaviours and working practices are expected to change too.

JANUARY 6 2016: According to Hewlett Packard, just one year after its birth the fledgling Impact Hub Berlin "was a single email away from closure". The Berlin team had to accept they'd failed to implement their own model: "We teach all the time that we should prototype and we didn't even prototype", admitted CEO Nele Kapretz. The trick they used to recover was to "think like a scientist and treat every iteration of your idea as a chance to test a hypothesis". Note: this application of the Controlled Failure pattern saved the business by closing the validated learning loop in a timely manner.

Earlier stories from the news archive[...]