Specification by Example

實例化需求

2019/03/14 (調整內容)
2020/05/03 (增加連結)
2021/12/12 (更新投影片)

概念

實例化需求(Specification by Example, SbE)是希望透過在描述需求時舉出實例，一方面讓需求更清楚，另一方面也有利於驗收測試。一般而言，User story都只是講個大概的需求及價值，需要利用實例化需求來讓需求更為清楚，也可以利用實例化需求來進一步撰寫測試案例。Adzic (2011)認為SbE其實就是Behavior Driven Development (BDD)的核心內容，但是為了避免BDD定義歧異的問題，不採用BDD這個名詞。也不想讓讀者誤以為測試是整本書的重點，所以，也不採用Test Driven Development (TDD)或Acceptance Test Driven Development (ATDD)等名詞。

Adzic (2011)認為SbE有兩種流行的模型:Acceptance Test Driven Development (ATDD)及Behavior Driven Development (BDD)，以ATDD為中心的模式，重視自動化測試，以BDD為中心的模式，重視於制定系統行為的情境。Adzic (2011)認為，如果團隊在導入SbE時，面對很多品質上的問題，建議採用以ATDD為主軸的開發模式，當一切運作順暢時，就可以採用BDD的模式。

SpecificationByExample.pptx

2021/12/12 (更新投影片)

  • Specification by Example

    • 主要優勢

    • 關鍵程序模式

      • 從目標中獲取範圍 (Deriving scope from goal)

      • 透過協作制定需求規格 (Specifying collaboratively)

      • 舉例說明 (Illustrating using example)

      • 提煉需求規格 (Refining the specification)

      • 不改變需求規格的情況下進行自動化驗證 (Automating validation without changing specification)

      • 頻繁驗證 (Validating frequently)

      • 演變出活的說明文件系統 (Evolving a living documentation system)

    • 活的說明文件

      • 我們需要權威的說明文件

      • 測試可以是好的說明文件

      • 根據可執行的需求規格建立說明文件

程序模式

Adzic (2011)提出一個程序模式 (詳參: 對實例化需求方法的整理與思考)

  • 從目標中獲取範圍 (Deriving scope from goal)

    • Adzic (2011)強調要以解決業務問題或達成業務目標為出發點

    • Adzic (2011)提到uSwitch的團隊的做法是: User Story中的"I am"及"So that"由使用者提供，而"I want"由開發者去思考，就是希望使用者專注於業務問題或業務目標，讓開發者去思考解決方案

  • 透過協作制定需求規格 (Specifying collaboratively)

    • Adzic (2011)強調需求規格是開發人員、測試人員與企業使用者共同開發，所以，需求規格要能讓企業使用者看得懂

    • Adzic (2011)說明協作方式很多(如:大型的全體工作坊、小型的工作坊、結對編寫、非正式交談)，要依適用情境去選擇方式。

  • 舉例說明 (Illustrating using example)

    • Adzic (2011)提到範例必須精確、完整、真實且容易理解

  • 提煉需求規格 (Refining the specification)

    • Adzic (2011)強調這些測試案例不只是要給機器看的，也是給人看的，最終的實例應該是精確的、可測的、不言自明的、聚焦的、以領域語言(業務使用者看得懂的方式)所編寫

    • Adzic (2011)強調測試腳本不是需求規格

    • Adzic (2011)建議使用Given-When-Then來描述規格情境

  • 不改變需求規格的情況下進行自動化驗證 (Automating validation without changing specification)

    • Adzic (2011)強調這些測試案例不只是要給機器看的，也是給人看的，跟自動化測試還是有所差距的。

    • Adzic (2011)說明了雖然建議自動化驗證，但有些情況下，自動化是不值得或不可行(如:實用性、趣味性)的。

    • Adzic (2011)提出自動化測試可以有兩個方向:UI、資料庫

      • Adzic (2011)提出UI自動化測試的三個階層 (詳參: How to implement UI testing without shooting yourself in the foot)

        • 業務規則階層 (Business rule/functionality level)

        • 使用者工作流程階層 (User interface workflow level)

        • 技術行為階層 (Technical activity level)

        • Adzic (2011)認為規格應該以業務規則階層來描述，自動化層應該是結合技術行為階層上編寫的區塊來處理使用者工作流程階層

      • Adzic (2011)建議可以在真實資料庫裡找出一些對應測試案例的對應資料

  • 頻繁驗證 (Validating frequently)

    • Adzic (2011)強調這些測試案例不只是要給機器看的，也是給人看的，所以跟一些自動化測試為基礎的迴歸測試是有所差距的。

    • Adzic (2011)對於頻繁驗證會面對的兩大問題(穩定性及快速回饋)提出了一些建議，另外，針對未通過的測試也提出一些建議

  • 演變出活的說明文件系統 (Evolving a living documentation system)

    • Adzic (2011)提出一些組織說明文件的方法: 依正在執行的user story、依照功能區域、依照UI的導覽路線、按照業務程序，要依適用情境去選擇方式。

    • Documentation and Developer Onboarding

      • When I go into a new environment, I find the following types of documentation to be the most useful for starting to understand what’s going on:

        • A Context Diagram for each major system, or each major component of the system I will be supporting

        • Some sort of lines-and-boxes drawing of the main components of the system, showing which ones touch which other ones.

      • When it’s time to get down to it and do some work on the code, I’d like to find:

        • Executable test suites at multiple levels of abstraction, properly isolated

        • Descriptive and meaningful commit messages in the version control system

        • If supported for the language, some sort of special source comments that IDEs pick up and show you in context while you’re working, like rdoc (Ruby), javadoc (Java), or documentation comments (C#).

  • Specification by Example, 10 years later: What's changed since the Specification by Example book came out.

    • Relationship between product quality and using examples

      • The results show that using examples for acceptance criteria is strongly correlated with higher product quality.

    • Given-When-Then won, by a huge margin (71%)

    • Tooling has stabilised

      • Opensource based on given-when-then (Cucumber, Specflow,…) (82%)

      • Who is primarily responsible for automating testing based on the examples?

        • Developers who implement the related feature (48%)

        • Testers inside the team (31%)

        • Any delivery team member - nobody is primarily responsible (11%)

        • Test automation specialists outside the delivery team but in the same organisation (9%)

        • Someone external/outsourced/specialist testing organisation (1%)

    • Collaboration won over testing

      • Who is primarily responsible for defining the acceptance criteria?

        • Collaborative: Delivery team together with business representatives (47%)

        • Product owners or business analysts (from the delivery team) (26%)

        • Collaborative: Whole delivery team (without business representatives) (10%)

        • Customer/Business representatives/sponsors (outside delivery team) (6%)

        • Developers (6%)

        • Testers (4%)

        • UX people (1%)

      • How do business representatives engage with examples in specifications or tests?

        • Actively participate in defining (40%)

        • They mostly do not care about the examples (26%)

        • Do not participate in defining, but review with the team (23%)

        • Do not participate in defining, but review without the team (8%)

        • Just read the test results (4%)

    • About a third of people who use examples do not automate them

      • Do you use a tool to automate testing based on the examples from specifications?

        • YES (71%)

        • NO (29%)

      • Automation in this case strongly correlates to improvements in the self-reported product quality estimate.

    • Task tracking tools won at the expense of good documentation

      • What is the primary source of truth for work requirements?

        • Work or Task tracking tool (eg Jira) (57%)

        • Dedicated individual documents (eg Word files) (15%)

        • Text files in version control (eg feature files or fitnesse pages committed to git) (12%)

        • Wiki pages outside code version control systems (eg Confluence) (1%)

        • Dedicated requirements management tool (eg RISE) (3%)

      • What is the primary source of truth for work requirements? Using Examples

        • Work or Task tracking tool (eg Jira) (55%)

        • Dedicated individual documents (eg Word files) (14%)

        • Text files in version control (eg feature files or fitnesse pages committed to git) (15%)

        • Wiki pages outside code version control systems (eg Confluence) (9%)

        • Dedicated requirements management tool (eg RISE) (3%)

      • What is the primary source of truth for work requirements? Automating Examples

        • Work or Task tracking tool (eg Jira) (52%)

        • Dedicated individual documents (eg Word files) (11%)

        • Text files in version control (eg feature files or fitnesse pages committed to git) (20%)

        • Wiki pages outside code version control systems (eg Confluence) (10%)

        • Dedicated requirements management tool (eg RISE) (3%)

  • Specification By Example and Product Quality

    • A Little History

      • 1999 — Kent Beck (Extreme Programming Explained: Embrace Change)

      • 2004 — Martin Fowler — Specification by Example

      • 2005 — Lisa Crispin — Customer Test-Driven Development

      • 2006 — Dan North — Introducing BDD

      • 2008 — Elisabeth Hendrickson — Driving Development with Tests: ATDD and TDD

      • 2009 — Gojko Adzic — Bridging the Communication Gap: Specification by Example and Agile Acceptance Testing

      • 2010 — Lisa Crispin — ATDD X BDD X Specification By Example

      • 2010 — Craig Larman — Specification By Example

      • 2011 — Gojko Adzic — Specification by example: How successful teams deliver the right software

    • Applying SBE in Practice

      • Derive the scope from the objectives

      • Business Objectives -> Derive the Scope -> Create the Stories

      • Specify Collaboratively

      • Illustrate Using Examples

      • Automate Specifications

      • Validate Frequently

  • 從【一例一休】牽拖Specification By Example

  • 從【一例一休】續談測試案例設計

工具

參考資料

# Imperative example in the Cucumber book Wynne & Hellesøy (2012)

Scenario: Redirect user to originally requested page after logging in

Given a User "dave" exists with password "secret"

And I am not logged in

When I navigate to the home page

Then I am redirected to the login form

When I fill in "Username" with "dave"

And I fill in "Password" with "secret"

And I press "Login"

Then I should be on the home page


# Declarative example from Aslak Hellesøy: The training wheels came off

Scenario: Redirect user to originally requested page after logging in

Given I am an unauthenticated guest

And I have a valid user account

And I attempt to access restricted content

When I log in

Then I have access to the restricted content

    • Declarative User Stories Translate to Good Cucumber Features (2015)

      • In order to achieve that goal, however, Cucumber scenaios need to be written in a certain way. They need to read as conversational requirements, not a QA test plan check list. This style of writing has a name: declarative.

    • Test Case Desing Approach: imperative vs declarative (2015)

      • Prons and cons of imperative approach:

        • More detail about steps

        • A kind of micro (steps) management of test cases

        • More feed backs when failures occur

        • You can use the features of Cucumber more efficiently

        • Gives more information to consumer for whole steps

        • You can produce more reusable steps

        • Not suitable for generalisation

        • More documentary work

        • A bit confusing if you are checking the result of a long list

      • Prons and cons of declarative approach:

        • Good for cross-platform testing

        • Good for generalization

        • Good if we are interested in summary of test case results

        • Group of actions, easy to manage

        • More automation code

        • Not good for reusable steps

    • Imperative Vs. Declarative style of writing Gherkin scenarios (2017)

      • The best way is: Remove ambiguities but keep scenario short