Time-Space Efficient Regression Testing for Configurable Systems

Sabrina Souto, Marcelo d’Amorim


Configurable systems are those that can be adapted from a set of options. They are prevalent and testing them is important and challenging. Existing approaches for testing configurable systems are either unsound (i.e., they can miss fault-revealing configurations) or do not scale.
This paper proposes EvoSPLat, a regression testing technique for configurable systems. EvoSPLat builds on our previously- developed technique, SPLat, which explores all dynamically reachable configurations from a test. EvoSPLat is tuned for two scenarios of use in regression testing: Regression Configuration Selection (RCS) and Regression Test Selection (RTS). EvoSPLat for RCS prunes configurations (not tests) that are not impacted by changes whereas EvoSPLat for RTS prunes tests (not configurations) which are not impacted by changes. Handling both scenarios in the context of evolution is important.
Experimental results show that EvoSPLat is promising. We observed a substantial reduction in time (∼22%) and in the number of configurations (∼45%) for configurable Java programs. In a case study on a large real-world configurable system (GCC), EvoSPLat reduced ∼35% of the running time. Comparing EvoSPLat with sampling techniques, 2-wise was the most efficient technique, but it missed two bugs whereas EvoSPLat detected all bugs four times faster than 6-wise, on average.

JSS 2016 submission - Artifacts
If you have any question, please contact sabrinadfs@gmail.com.

  1. For SPLs
  2. For GCC

1. Experiments for SPLs

Evaluation Environment
The SPLs evaluation was performed against a machine with the following specification:
- Architecture: X86_64
- OS: Mac OS X Lion 10.7.5
- Processor: 2,4 GHz Intel Core i5
- Memory: 4 GB
- Java version: 1.7.0_65

Running Evo-SPLat
Here is how to run the Evo-SPLat evaluation against one of the subjects we used <subject_name> (replace occurrences of <subject_name> with one of the subjects we used: companies, desktopsearcher, elevator, email, gpl, jtopas, minepump, notepad, prevayler, sudoku, xstream, zipme).
The instructions are for Unix/MacOS, but can easily be changed for Windows:
Download the tool and subjects: evo-splat.tar.gz
1) tar -zxvf evo-splat.tar.gz
2) cd evo-splat

EXPERIMENT1: Complete Re-execution (Time Reduction)
Run the tests from the specified subject with Evo-SPLat producing a file ../evo-splat/src-subjects/<subject_name>/results.txt, containing the results for experiment 1 (Table 3 in the document):
1) chmod +x runexp1.sh
2) ./runexp1.sh <subject_name>

EXPERIMENT2: Partial Re-execution (Space Reduction)
Run the tests from the specified subject with Evo-SPLat producing a file ../evo-splat/src-subjects/<subject_name>/results.txt, containing the results for experiment 2 (Table 4 in the document):
1) chmod +x runexp2.sh
2) ./runexp2.sh <subject_name>

2. Experiments for GCC
Download the modified tool for GCC: evo-splat-gcc

We used the following versions: