軟體測試

2021/10/23 (增加連結)
2022/01/05 (增加連結)

軟體測試

軟體測試是個很重要也很複雜的工作 (詳參:Software testing)，有不同種類的測試，目前大部分的測試已經發展出自動化工具，也使得測試工作越來越容易了。

• Test smart: How to select testing techniques for agile team

  • Guide development vs. Critique the product

    • Guide development

      • These are the tests that are performed before coding happens or concurrently as coding proceeds

    • Critique the product

      • Those that critique (evaluate) the product after coding is complete

  • Business facing vs. Technology facing

    • Business facing

      • tests that are readable by business stakeholders and could be performed by testers, product owners, designers

    • Technology facing

      • tests that are written by or with the help of technical team members

• Software QA in Scrum

測試的類型

廖家盛 (2015)整理了常見的測試類型:

• 單元測試:單一程式的測試

• 功能測試:模組或功能階層的測試

• 整合測試:系統整體的測試，有些公司會有不同層級的整合測試，將最後的測試稱為系統測試

• 探索性測試:除了已規劃的測試案例之外，進行未規劃的測試

• 壓力測試:了解系統所能承受的壓力，以及在壓力下的表現

• 容錯測試:了解系統面對未預期錯誤的容錯程度

• 使用者介面測試:使用者透過系統介面進行測試

• 安全性測試:進行資訊安全的相關測試

• 回歸測試:新增或更動功能時，測試過去已經測試且未被更動的功能

測試方式與工具

廖家盛 (2015)整理五個個案公司進行的測試及方式，發現大部分的公司還是以人工測試為主，也整理了一些個案公司所使用的測試工具:

功能性測試

• Why We Quit Unit Testing Classes to Focus On a Behavioral Approach

• Unexpected Lessons From 100% Test Coverage

  • While getting to 100 involved making some sacrifices, it also allowed me to:

    • fix a lot of bugs,

    • catch and fix redundancies in my code,

    • reason about my unreachable code,

    • delineate thoughtfully between error cases and assertions, and

    • thoughtfully shift run-time assertions into compile-time guarantees.

  • Ultimately, my sense is that 100% test coverage is definitely a useful exercise to do at least once. It’s probably also a useful exercise to do more than once.

• 單元測試 (Unit Test)

  • JUnit:很多軟體，如Eclipse已經內建JUnit

    • JUnit入門

    • 在Eclipse中使用JUnit4进行单元测试（初级篇） (此文主要參考: Get Acquainted with the New Advanced Features of JUnit 4)

    • 開源框架：JUnit Gossip

    • JUnit tutorial in Eclipse part 1 : javavids (play list共9部影片)

    • Testing Java with Visual Studio Code

  • Visual Studio單元測試功能

    • ASP.NET MVC 單元測試系列 (7)：Visual Studio Unit Test

    • 對程式碼進行單元測試

    • Visual Studio 2017 的 Live Unit Testing

    • Visual Studio 2017 你有寫單元測試嗎？

  • NUnit

  • 5 Questions Every Unit Test Must Answer

    • What are you testing?

    • What should it do?

    • What is the actual output?

    • What is the expected output?

    • How can the test be reproduced?

  • Rethinking Unit Test Assertions

  • Waiter, There’s a Database in My Unit Test!

  • 13 Tips for Writing Useful Unit Tests

    • Test One Thing at a Time in Isolation

    • Follow the AAA Rule: Arrange, Act, Assert

    • Write Simple “Fastball-Down-the-Middle” Tests First

    • Test Across Boundaries

    • If You Can, Test the Entire Spectrum

    • If Possible, Cover Every Code Path

    • Write Tests That Reveal a Bug, Then Fix It

    • Make Each Test Independent

    • Name Your Tests Clearly and Don’t Be Afraid of Long Names

    • Test That Every Raised Exception Is Raised

    • Avoid the Use of Assert.IsTrue

    • Constantly Run Your Tests

    • Run Your Tests as Part of Every Automated Build

  • Stop writing unit tests: How to avoid the daily pain in the ass and become more efficient

    • Why writing unit tests sucks

      • You actually have to write them

      • You manually have to define your inputs and outputs

      • You have to maintain twice as much code

    • While the record and replay approach has already gained some attention in UI development (there exist some open source frameworks), testing the replay against a recorded return value is uncommon.

  • Endless Unit Testing Power in Your Pocket

  • Unit Testing: Best Practices To Follow

    • Only test your code

    • Unit tests should be automated

    • Test one thing at a time

    • Write your tests during development

      • Enter TDD

    • Only test the public interface

    • Dependency Injection is your friend, embrace it

    • Use mocks, stubs, spies and dummies to control logic flow

    • Don’t test external calls

• 使用者介面測試

  • Visual Studio介面測試功能

    • 使用 UI 自動化來測試您的程式碼

  • 使用 Microsoft Test Manager 執行探勘測試

  • 在 Microsoft Test Manager 中自動化測試案例

  • GUI Testing Tutorial: User Interface (UI) TestCases with Examples

  • How to Get Rid of Frontend bugs

    • How Serious is Frontend Bugs

    • Prevention Techniques

      • Monitoring

      • CI/CD

      • Code Quality

      • Static Testing

      • Automated Testing

      • Battle Testing

      • Ownership and Passion

• Web使用者介面測試

  • Katalon (based on Selenium)

    • Introduction to Web Testing

  • Selenium

    • 持續整合 - 透過 Selenium 實現自動化測試 (2015)

    • 網頁應用程式測試工具Selenium IDE介紹 (2011)

    • Selenium Tutorial for Beginners: Learn WebDriver in 7 Days

  • TestCafe vs Selenium : Which is better?

    • Selenium is simple as well as easy to learn. It has been around for quite some time now and has loyal testers in the QA industry.

    • TestCafe does not require any external plugins to run tests on different browsers which makes it easier for the testers to eliminate the effort of setting up plugins.

      • TestCafe is new to automation tool but with enhanced features. However, it only supports one programming language that is JavaScript, whereas Selenium supports all the main programming languages such as Python, PHP or ruby etc.

  • Selenium, can it withstand the next generation test tooling?

    • Selenium, of course, remains a great tool to use. When properly set up and configured, they provide us with reliable and maintainable test suites that do the trick. It gives flexibility with regards to being able to add stuff, however this requires a good amount of knowledge to do so. Especially when it comes to reliability, Selenium has the advantage of having proven themselves over a few years.

    • Cypress momentarily only supports chrome as a browser, but have Firefox, IE11 and edge on their roadmap. The cool thing about Cypress is that it’s actually a complete framework. It contains Mocha, Chai, Sinon and Chai-JQuery. Starting to write tests is something you can start on immediately after installing.

    • TestCafe actually recognizes any browser installed on the target device which makes it very easy to get proper browser support. It’s also quite easy to install, configure and get started.

• API Test

  • Automate Your API Testing With CI Pipelines

非功能性測試

• 效能測試

  • PageSpeed Insights

  • Pingdom Website Speed Test

  • WebPageTest

  • Load Focus

• 壓力測試

  • 7 Essential Cloud-based Load Testing Tools

    • Load Impact

      • free for 25 current users

      • free trial: 50 current users, 50 free tests

    • Flood IO

      • unlimited concurrent users in free plan with few limitation (5 hours free trial)

    • Loader

      • lets you run a load test for one minute with 1000 user request in the free account

    • Load Storm

      • allows you to stress test against your web application with 10 users for 10 minutes

    • Blazemeter

      • you can perform load for up to 50 users and end of the test

    • Load Focus

      • Free website speed test

      • Perform test with 500 concurrent users with a free account from either US or Ireland location (free trial for 14 days)

    • OctoPerf

      • 50 concurrent users/test

  • JMeter

    • JMeter

    • JMeter 入門

    • JMeter - 壓力測試 - 簡單版

    • JMeter - 壓力測試 - 進階版

    • JMeterTutorial (5個影片)

    • JMeter Tutorial for Beginners: Learn in 7 Days

  • Visual Studio壓力測試功能

    • 如何進行網站壓力測試：以不動產交易實價查詢服務網為例

    • 使用 Visual Studio 進行 Web 效能和負載測試

  • The Grinder

    • 性能测试工具Grinder

  • LoadNinja by SmartBear

• 安全測試

  • Checkmarx

    • Checkmarx 源碼安全檢測工具

    • 自動化檢測工具掃瞄漏洞快又準 評選仔細比一比

  • 善用檢測工具幫你預先找出軟體安全性漏洞

  • 10 Types of Application Security Testing Tools: When and How to Use Them

    • Static Application Security Testing (SAST)

    • Dynamic Application Security Testing (DAST)

    • Origin Analysis/Software Composition Analysis (SCA)

    • Database Security Scanning

    • Interactive Application Security Testing (IAST) and Hybrid Tools

    • Mobile Application Security Testing (MAST)

    • Application Security Testing as a Service (ASTaaS)

    • Correlation Tools

    • Test-Coverage Analyzers

    • Application Security Testing Orchestration (ASTO)

  • Testing Tools (Open Web Application Security Project, OWASP)

  • Top 11 Open Source Security Testing Tools for Web Applications

    • Wapiti

    • Zed Attack Proxy

    • Vega

    • W3af

    • Skipfish

    • Ratproxy

    • SQLMap

    • Wfuzz

    • Grendel-Scan

    • Arachni

    • Grabber

  • 8 Open source security testing tools to test your website

    • Vega

    • ZED Attack Proxy (ZAP)

    • Wapiti

    • W3af

    • Iron Wasp

    • SQLMap

    • Google Nogotofail

    • BeEF (Browser Exploitation Framework)

  • 15 Most Powerful & Reliable Security Testing Tools 2018

    • Metaspoilt

    • Wireshark

    • W3af

    • CORE Impact

    • Netsparker

    • Burpsuite

    • Cain & Abel

    • Acunetix

    • Retina

    • Canvas

    • Nmap

    • Dradis

    • Security Onion

    • Nikto

    • Vega

  • 19 Powerful Penetration Testing Tools: Security Testing and Hacking Tools

    • Netsparker

    • Acunetix

    • Metasploit

    • Wireshark

    • w3af

    • Kali Linux

    • Nessus

    • Burpsuite

    • Cain & Abel

    • Zed Attack Proxy (ZAP)

    • John The Ripper

    • Retina

    • Sqlmap

    • Canvas

    • Social Engineer Toolkit

    • Sqlninja

    • Nmap

    • BeEF

    • Dradis

常用的測試工具

• An Overview of JavaScript Testing in 2019 (**非常詳盡的文章**)

• Best software testing tools (2018)

  • qTest

  • Katalon Studio (Free)

  • Selenium (Open Source)

  • Unified Functional Testing (UFT) (Formerly known as HP QuickTest Professional)

  • TestComplete

• 9 automation testing tools to consider for web applications in 2018

  • UI automation testing frameworks

    • Selenium WebDriver

    • Protractor

    • Cucumber (for BDD)

  • Desktop automation tools for enterprises

    • TestComplete

    • UFT Pro (LeanFT)

    • Eggplant Functional

  • Cloud testing platforms

    • Screenster

    • Ghost Inspector

    • Usetrace

• Best Automation Testing Tools for 2018 (Top 10 reviews)

  • Selenium (open source)

  • Katalon Studio (free)

  • Unified Functional Testing (UFT)

  • Watir (open source)

  • IBM Rational Functional Tester

  • TestComplete

  • TestPlant eggPlant

  • Tricentis Tosca

  • Ranorex

  • Robot framework (open source)

Debug

• How to Fix Bugs

測試計畫

測試計畫大概的內容

• 測試目的

• 測試範圍

• 測試完成標準

• 測試時程

• 資源需求

  • 人力需求

  • 設備需求

    • 硬體環境

  • 軟體環境

• 測試案例

• 不列入本次系統測試部分

• 測試完成須交付之文件

• 參考文件

要如何進行功能性測試

• 撰寫測試個案

  • 測試個案名稱、測試步驟、預期結果

• 測試結果

  • 通過、不通過(發現之問題)

要如何進行非功能性測試

• 壓力測試

  • 壓力測試之目標 (上線人數、反應時間)

  • 壓力測試之環境

• 安全測試

  • 安全測試之目標 (哪些安全測試項目)

    • 可參考: Web Application Security Testing Cheat Sheet

  • 安全測試之環境

參考資料

• Why I now appreciate testing, and why you should, too.

• Katalon Studio vs Selenium based open source frameworks

• The Wide World of Software Testing

  • Unit Testing

  • Integration Testing

  • End to End Testing

  • Performance Testing

  • Load Testing

  • User Acceptance Testing

• Use decision tables to write better tests faster

• Mocking is a Code Smell

• Writing Automated Acceptance Tests Using Serenity and the Screenplay Pattern

  • Serenity is a powerful library for writing automated user acceptance tests. It uses test results to generate world-class test reports that document and describe what your application does and how it works.

    • Serenity can also be used with BDD tools such as JBehave or Cucumber.

  • In the Screenplay pattern, tests are presented from the perspective of the user. A user that interacts with the web app is called an actor. Everything revolves around actors in the Screenplay pattern.

• Write tests. Not too many. Mostly integration.

• 9 Excuses Why Programmers Don’t Test Their Code

  • “My Code Works Fine — Why Should I Even Bother Testing It?”

  • “This Piece of Code Is Untestable”

  • “I Don’t Know What to Test”

  • “Testing Increases the Development Time, and We’re Running Out of Time”

  • “The Requirements Are No Good”

  • “This Piece of Code Doesn’t Change”

  • “I Can Test This Way Faster If I do It Manually”

  • “The Client Only Wants to Pay for Deliverables”

  • “This Piece of Code Is So Small … It Won’t Break Anything”