Agile Methods‎ > ‎

Agile Quality Assurance Initiative


Our company started an initiative to implement agile quality assurance best practices in all software development projects. Our teams have extensive experience with traditional quality assurance approaches based on the idea "test the deliverables". They have also experience in integrated quality assurance tools such as Test Driven Development, Defect Driven Development, Pair Programming, Coding Dojos, and agile Scrum development methods. This approach is based on the idea "deliver quality products". 

Below the definition of quality insurance taken from wikipedia. The major element in the definition is the statement "in advance".

"Quality Assurance is a part and consistent pair of quality management proving fact-based external confidence to customers and other stakeholders that a product meets needs, expectations, and other requirements. QA assures the existence and effectiveness of procedures that attempt to make sure - in advance - that the expected levels of quality will be reached" Wikipedia

The vision of our initiative is stated on the bbv Software service company internal wiki.

The agile quality insurance group is the knowledge group for agile quality insurance and testing. One major goal is to increase our knowledge and "best practices" for agile quality assurance. Through continuous improvement we will ameliorate the internal and external quality of our internal projects and customer projects. Themes are for example clean code, ATDD, TDD, CI, continuous deployment, static metrics, dynamic metrics, coding guidelines, reviews, walkthroughs, pair check-in, architecture guidelines such as SOLID, etc.-

We want to find an ideal approach to integrate agile quality assurance and agile testing skills in our Scrum teams. Once the approaches are identified they should be part of the standard approach how bbv teams realize projects. All gathered information and activities of this group should be published on the company wiki. 


Test Driven Development - TDD -

Test Driven Development (TDD) is a software development process that relies on the repetition of a very short development cycle: first the developer writes a failing automated test case that defines a desired improvement or new function, then produces code to pass that test and finally refactors the new code to acceptable standards. Kent Beck, who is credited with having developed or 'rediscovered' the technique, stated in 2003 that TDD encourages simple designs and inspires confidence.[Wikipedia]
Please read the benefits chapters to understand that TDD is cheaper than traditional testing and delivers a product with higher quality

Acceptance Test Driven Development - ATDD -

The goals of acceptance tests is to drive the development and verify the correct realization of stories.

The practices of ATDD include:

    • Establishing the goals of different stakeholders required for a vision to be implemented
    • Drawing out features which will achieve those goals using feature injection
    • Involving stakeholders in the implementation process through outside–in software development
    • Using examples to describe the behavior of the application, or of units of code
    • Automating those examples to provide quick feedback and regression testing
    • Using 'should' when describing the behavior of software to help clarify responsibility and allow the software's functionality to be questioned
    • Using 'ensure' when describing responsibilities of software to differentiate outcomes in the scope of the code in question from side-effects of other elements of code.
    • Using mocks to stand-in for collaborating modules of code which have not yet been written [Wikipedia]

Automated Acceptance Testing (ATDD) is a key agile development practice that can also be very beneficial for more traditional projects. Developers write automated functional tests that are specified collaboratively with the business. Key benefits include [Wakaleo]:

    1. Reduced risk due to a better understanding of the requirements
    2. Better team communication and an unambiguous definition of what needs to be done
    3. Issues are detected and fixed faster
    4. Faster releases due to comprehensive automated regression tests
    5. Objective, real-time measures of progress


  • DevOpsis an emerging set of principles, methods and practices for communication, collaboration and integration between software development (application/software engineering) and IT operations (systems administration/infrastructure) professionals.

Findings 2011

End of year 2011 we published a company internal questionary how agile quality assurance and agile testing principles are used on a daily basis in software development projects. The findings are used to define an improvement program for all software projects developed internally and at customer sites. The responses are compiled from the ten projects which provided information. We had five projects developed under .NET/C##, three projects developed under Java, and two projects developed in C++. As a remark currently we do not distinguish industrial applications from commercial finance oriented solutions.
The majority of the projects are green field projects, only two projects stated that they are brown field projects.

Wide opportunities and threats are identified over all the sent responses describing software projects.
  • Opportunities
    • Our company is a geographical leader for agile Scrum development and Extreme Programming approaches
    • All collaborators are trained in agile techniques
    • Almost all collaborators are certified Scrum master or product owners and have access to advanced training and coaching
  • Threats
    • In 2011 the company could not fulfill requests for Scrum development - Scrum coaches and experienced Scrum masters -
    • In 2011 the company lost leads in the area of agile quality assurance and agile testing
    • Major competitors are trying to establish themselves as the Swiss agile Scrum specialists
The strengths and weaknesses were defined below on a more detailed level.

 Theme Dimension  .NET / C#  Java/ JEE  C++/
 Static Tools Programming environment and development tools
  •  Visual Studio + Resharper
  • Eclipse 
  • Visual Studio 
  • Eclipse 
Strength: Our organization is mature. Training should be provided to all juniors for effective use of modern IDE and standard extensions.
  Coding guidelines
  •  StyleCop and FxCop
  •  Java Coding Guidelines
  Strength: Our organization is mature. Approaches should be defined how brown field projects are transformed to respect coding guidelines where adequate.
  Static syntax quality metrics
  •  Style Cop
  • PMD
  • Checkstyle 
  • Sonar (history)
  •  Clockwork
  • Misra
Improvement: Tools are known but not all projets use them. Not all projects use the results as quality gate.
  Static semantic quality metrics
  •  FxCop
  • NDepend
  • Structure101
  •  Findbugs
  • JDepend
  • Sonar (history)
  Improvement:  Tools are known but not all projects use them. Not all projects use the results as quality gate.
  • Agile architecture and refactoring
  • Agile principles such as SOLID
  • Clean Code
  •  Resharper
  • Restructure101
  •  Eclipse 
 Improvement:  Our organization is professional.  Clean code is insured through pair programming, pair checkin, coding dojo. Agile architecture principles are not lived in all projects.
 Dynamic Tools Test driven development and mocking
  •  xUnit
  • TFS test
  • Moq
  • FakeItEasy
  •  Junit
  • Mockito
  Strength: Our organization is professional. Training should be provided to seniors to increase efficiency and to juniors to push awareness of these approaches.
  Acceptance test driven development
  •  MSpec
  • SpecFlow
  • none
  •  QTP
  • SLCE (propriatary)
Weakness: major weakness in the area of performing the tests, adapting the RE process and missing KPI.
  Code coverage measurement
  •  dotCover
  • NCover
  •  Cobertura
  Improvement:  weakness
 Environment Continuous integration
  •  Teamcity
  • TFS
  • Jenkins 
  •  ElectricCloud
Strength: Our organization is mature
  Continuous deployment
  • Teamcity
  •  Jenkins
  Improvement:  No clear approach and best practices could be identified in the company. Some projects have already reached professional level.
  Quality gate
  • through Teamcity 
  • through TFS
  •  through Jenkins
  Improvement: Not all projects have systematic pair checkin, defined process for done stories, or defined release process.
 Development Approach Scrum approach (management aspects)
  • sprint duration is 2 weeks
  • stories and RE are written ad hoc
  •  TFS Scrum
  • Analog Board
  •  Analog Board
  •  Analog Board
Strength: Our organization is professional. All developers know about Scrum. But not all projects are practicing Scrum. The majority of the teams also have working knowledge of Kanban approach.
Subtle aspects such a team organization, team responsibility, scrum master responsibilities, team code ownership are not well understood in all projects.
  Extreme programming approach (technical aspects)   Strength: Our organization is professional. All developers know about clean code, TDD, pair programming, pair checkin, and coding dojo.
  Stories or requirements with testable acceptance criteria   Improvement: Projects are learning how to write such stories and requirements. The dependencies between backlog, requirements and acceptance tests should be better understood. Improvements are necessary.
  Tracking of defects and cycle time for defects
  •  TFS
  • HP Quality Center
  • Jira 
  • Bugzilla
  •  Jira
  • Bugzilla
Weakness: Issue management tools provide support. Not all projects have a clear tracking of issues to releases.
  Key performance indicators to track qualitynonenone none Weakness: Projects do not have a culture for KPI - Key Performance Indicator -. The focus on ROI as required in Scrum is also weak.
  Risk Managementnonenone none Weakness: Risk management for project should be standard.
  Project and release planning
  • analog roadmap
  • analog current release plan
none none Weakness: Agile project and release planning should be standard.
  Quality assurance role in team nonenone none  The TechDay 2011 keynote clearly showed the weaknesses in our development approach.

Based on the above findings the agile quality assurance group will define measures and prioritize them.

Under definition by the agile quality assurance group members. The identified area are
  1. KPI and Quality Gate
  2. Scrum project management with release planning, risk management with mitigation measures, continuous improvement
  3. Agile architecture and refactoring
  4. ATDD, documentation, and Quality Gate
  5. Quality assurance roles in the Scrum team
  6. Continuous Deployment and DevOps aspects
  7. ...

The weaknesses and improvements identified in the area of requirements engineering should be corrected in the requirement engineering task force. Tracking of progress will still be done in our group due to the quality assurance component.
The weaknesses and improvements identified in the area of Agile Scrum and Kanban development process should be corrected in the Scrum master and product owner task force. Tracking of progress will still be done in our group due to the quality assurance component.

Themes for Kick-Off Meeting Quality Assurance Improvements

The data collected from the questionary and the feedback from the experts members of the agile quality assurance and testing group give hints for the following themes
  1. Clean Code and writing of source code
    1. Coding guidelines
    2. Static syntax metrics
    3. Static semantic metrics
    4. Continuous integration
    5. Refactoring
  2. TDD and eXtreme Programming
    1. Test Driven Development - TDD, mock, impact on the architecture -
    2. Coding dojos, definition of done for internal quality
  3. ATDD, BDD and acceptance quality
    1. ATDD and BDD tools
    2. Writing conventions for stories to insure implementable, testable and complete acceptance criteria
  4. Agile software architecture
    1. Software architecture for TDD/ATDD/Clean Code projects
    2. Architecture refactoring
    3. Technical project documentation: architecture, installation, user manual, issue-TDD-release
    4. Impact on TDD, ATDD, BDD
  5. Requirement Engineering and Product Owners (RE/PO Interest Group - Stefan Widmer / Raphael Auf der Mauer)
    1. Agile quality assurance influence on stories, requirements and the role of product owner
    2. Quality assurance for stories and requirements
    3. Agile portfolio management and agile architecture - Enterprise Architecture -
    4. Refactoring aspects
    5. Change management and traceability in a regulatory environment
  6. Process improvements (Process/Scrum Interest Group - Alan Ettlin / Heads)
    1. Scrum usage
    2. Quality assurance roles in a Scrum team
    3. Project Management Office PMO with releases planning, project planning and risk management,  and Scrum of Scrums
    4. KPI and quality gates - process, ROI, tools -
    5. Bridge between issue management and development process - Scrum, Kanban, traceability -


Here a set of books discussing various aspects of agile quality assurance and testing. For more details see books I recommend often.
  • Agile Testing: A Practical Guide for Testers and Agile Teams, Lisa Crispin and Janet Gregory, Addison-Wesley 2009
  • Refactoring Improving the Design of Existing code, Martin Fowler, Addison-Wesley 1999
  • Working Effectively with Legacy Code, Michael C. Feathers, Prentice-Hall 2005
  • Test Driven Development by Example, Ken Beck, Addison-Wesley 2003
  • Clean Code: A Handbook of Agile Software Craftsmanship, Robert C. Martin, Prentice Hall 2009
  • The Clean Coder: A Code of Conduct for Professional Programmers, Robert C. Martin, Prentice Hall 2011
And one reference to help us introduce improvements in organization
  • Fearless Change: Patterns for Introducing New Ideas, Mary Lynn Manns & Linda Rising, Addison-Wesley 2005 
Blogs discussion agile quality insurance and agile testing are