Acknowledging the Elephant in the Room: Brittle Test Cases

Atif Memon, University of Maryland


For years we have talked about regression testing in terms of test selection, prioritization, minimization, and reduction. We have focused on test cases that remain executable from one version of the software to the next. What about test cases that break---are no longer executable---after the software changes? As a community, we have ignored the issues surrounding such brittle test cases. As we transition to an event-driven execution model for our software, our test cases are increasingly composed of sequences of method-calls, user- and system-generated events, messages, and web service requests. These test cases are extremely brittle, easily breaking when a software change disallows certain method sequences, workflows, and message protocols. Given their large numbers, we can no longer afford to ignore brittle test cases.

We started, in the year 2001, to develop techniques to handle brittle test cases for GUI-based software. Since then, several other researchers have applied our techniques to non-GUI software. In this talk, I will give an overview of some of these techniques. We have created robust, high-level representations of brittle test cases, which remain usable across software versions, and are used to regenerate the test cases on demand. We have developed model-based test generation techniques that allow us to treat test cases as throwaway objects. Their brittleness is inconsequential because all test cases are discarded. New test cases are quickly regenerated for each software version. We have also developed search-based techniques to repair broken test cases. I will provide empirical evidence showing the effectiveness of these techniques, as well as deeper insights into characteristics of test cases that make them brittle.


Atif Memon in an Associate Professor in the Department of Computer Science at the University of Maryland, College Park.  He is the founder and head of the Event Driven Software Lab (EDSL), whose members study issues of design, development, quality assurance, and maintenance of such software applications.  Atif has designed and developed the model-based GUI testing software, GUITAR, which operates on Android, iPhone, Java Swing, .NET, Java SWT, and web systems, and levarages a resource cloud for test automation.  He is also the Serial Editor of Advances in Computers, published by Elsevier and a member of the Editorial Board of the Journal of Software Testing, Verification & Reliability (STVR), published by John Wiley & Sons, Ltd.  Atif is also an elected member of the Steering Committee of the International Conference on Software Testing, Verification and Validation — ICST, the largest conference on software testing and the conference of co-location for this workshop.