Random Testing Termination Criteria
Random testing is a software testing technique through which programs are tested by generating and executing random inputs. Because of its unstructured nature, it is difficult to determine when to stop a random testing process. Faults may be missed if the process is stopped prematurely, and resources may be wasted if the process is run too long. In this article, we propose two promising termination criteria, “All Equivalent” (AEQ) and “All Included in One” (AIO), applicable to random testing. These criteria stop random testing once the process has reached a code-coverage-based saturation point after which additional testing effort is unlikely to provide additional effectiveness. We model and implement them in the context of a general random testing process composed of independent random testing sessions. Thirty-six experiments involving GUI testing and unit testing of Java applications have demonstrated that the AEQ criteria is generally able to stop the process when a code coverage equal or very near to the saturation level is reached, while AIO is able to stop the process earlier in cases it reaches the saturation level of coverage. In addition, the performance of the two criteria has been compared against other termination criteria adopted in the literature.
Support Material : Experiment on GUI Testing of Java Applications
This experiment has been carried out on the following set of Java GUI based applications.
By clicking on the name of each application, you can download the source code of the tested application and the test harness that has been realized to test it.
A1
A2
A3
A4
A5
A6
A7
A8
A9
A10
A11
A12
A13
A14
A15
A16
A17
A18
In details, in the folder called source you can find the source code of the application under test, while in the folder called executable you can find the material needed to automatically test it.
In the folder called executable, you can find two batch files executable by windows shell, respectively called ripperciclico.bat and ripper.bat.
All the tests have been carried out in Windows 7 and 10 environments, but the batch files could be easily transformed in order to execute them in Unix environment.
To start a testing session, you have to execute the ripperciclico.bat shell file or, if this file does not exist, the ripper.bat file.
By executing ripperciclico.bat, 3 testing sessions will be carried out. In order to change the number of sessions, you can edit the number 3 in the following row of ripperciclio.bat.
for /L %%n in (1,1,3) do call ripper.bat %%n
Alternatively, it is possible to change the value of the parameter num_executions into the ripper.properties file.
The number of events constituting a testing session can be modified by editing the num_events parameter into the ripper.properties file.
During the execution of the tests, the mouse is directly driven by the testing tool, so the interaction with the computer should be avoided (running the experiment in a Virtual Machine is preferable).
The main outputs generated by the testing process are the following:
- coverage.em, metadata file generated by the Emma instrumentation tool
- coverage.ec, files corresponding to each of the executed testing sessions, in the coverage folder (you can set the name of this folder in the ripper.properties file).
Please refer to emma documentation to know how it is possible to visualize the obtained coverage.
- fired_events xml files, reporting information about the randomly executed user events;
- tempi.txt, reporting information about the execution time of the testing process;
- unhandled_exceptions.txt, reporting a log of crashes and other unhandled exceptions occurred during the testing process.
Support Material : Unit Testing of Java Libraries Supported by Randoop
The second experiment has been carried out on the following set of Java classes.
The complete set including all the resources needed to execute the testing process can be downloaded by this link (152Mb).
A1 A4J
A2 AspectJ
A3 Battlecry
A4 Calendar
A5 Classviewer
A6 Dcparseargs
A7 Diffi
A8 Falselight
A9 Http-utils
A10 Jargs
A11 Javax
A12 Jclo
A13 Jgaap
A14 Jipa
A15 Noen
A16 SaxPath
A17 Sugar
A18 Tullibee
The starting point of the process is the batch file starter6Randoop.bat. it has been developed and tested on Windows 7 and 10. In order to know the syntax and to start the testing process for all the 18 applications available, you can use the batchtest6.0.bat batch files that sequentially open series of testing sessions on the 18 considered applications.
The two main parameters of starter6Randoop are, respectively, the name of the jar of the application under test (the jar file has to be placed in the jars folder), the number of testing sessions to be executed and the time (in seconds) of each randoop test.
In this implementation the testing process continues indefinitely and have to be manually stopped by the user. It is possible to continue an interrupted testing process by means of the starter6RandoopRecever.bat batch file that needs as parameters the name of the folder with the already executed tests and the number of executed cycle of sessions.
For each application under test, in the jar folder there is, too, a text file containing the list of classes that have to be considered for coverage evaluation. The subfolder lib of the folder jars include libraries by which the tested applications depend.
The output of the testing processes are included in folders starting with the name of the application under test, followed by date and time of the start of the test. In this folder there are ec and es files reporting the emma coverage information corresponding to the executed tests and their cumulative coverage. The txt files provide summary of the obtained coverage. In particular, in the sessioncoverage.txt file there is a report of the trend of the cumulative coverage for each testing session. The unioncoverage.txt text file reports the trend of the cumulative coverage obtained by the union of all the executed testing sessions.
Publications