Automated Software Testing as a Service

Abstract:

This paper makes the case for TaaS—automated software testing as a cloud-based service. We present three kinds of TaaS: a “programmer’s sidekick” enabling developers to thoroughly and promptly test their code with minimal upfront resource investment; a “home edition” on-demand testing service for consumers to verify the software they are about to install on their PC or mobile device; and a public “certification service,” akin to Underwriters Labs, that independently assesses the reliability, safety, and security of software.

TaaS automatically tests software, without human involvement from the service user’s or provider’s side. This is unlike today’s “testing as a service” businesses, which employ humans to write tests. Our goal is to take recently proposed techniques for automated testing—even if usable only on toy programs—and make them practical by modifying them to harness the resources of compute clouds. Preliminary work suggests it is technically feasible to do so, and we find that TaaS is also compelling from a social and business point of view.

Introduction:

Software quality assurance is in dire need of substantial progress. Software testing is resource-hungry, time-consuming, labor-intensive, and prone to human omission and error. Despite massive investments in quality assurance, serious code defects are routinely discovered after software has been released [16], and fixing them at so late a stage carries substantial cost [13]. Thorough testing of large, complex software involves great effort, and the software industry still employs relatively primitive testing techniques. The current software business model forces software users to take on faith that the vendor has performed thorough testing before shipping. Yet, given the difficulty of thoroughly testing software, such trust is typically misplaced. There exists no objective way to assess the reliability of a software product, therefore the main competitive metric is often performance and functionality. There is no independent certification body to guarantee that every vendor employs state-of-the-art testing.

We need a “disruptive technology” to substantially improve software quality. Various studies have found the average bug density in production-ready software to have stayed relatively constant over time, while average code volume of software has increased along an exponential curve [13], with the net effect that the number of bugs per product is increasing. It is therefore necessary to quickly find a way of reducing bug density by at least an order of magnitude. A promising direction is to reduce reliance on human labor through automated testing techniques, and recent proposals [4, 5, 2, 1] have made promising progress along these lines. Alas, they are still not ready to handle real-sized software (1 million lines of code or more), mainly due to high CPU and memory requirements. We believe cloud computing can come to the rescue.

Conclusion:

In this paper we made the case for TaaS—automated software testing as a cloud-based service. We presented three classes of testing services: TaaSD for developers to more thoroughly test their code, TaaSH for end users to check the software they install, and TaaSC certification services that enable consumers to choose among software products based on the products’ measured reliability. We argued that the combination of recent advances in test automation and the availability of compute clouds can offer unprecedented levels of testing quality. We find TaaS to be compelling from both technical and non-technical points of view. By simultaneously empowering consumers to make educated choices and also enabling developers to build better products, TaaS has the ingredients to indeed help reduce bug density by an order of magnitude.