Our experiments involves the executions of three tools: SpecFuzzer, GAssert and EvoSpex. Each tool was executed 10 times for each subject. Below we describe how to run each of them, and then how to interpret and analyze the results.
SpecFuzzer can be executed from the folder specfuzzer. As shown with the motivating example in the home page, for each subject a setup must be performed before running SpecFuzzer. Thus, essentially we need to perform two steps:
$ ./experiments/setup.sh <subject_arguments>
$ ./experiments/run-specfuzzer.sh <subject_arguments>
where <subject_arguments> contains the name of the folder where the source code is located, the fully qualified name of the class under test and the name of the target method. For instance, for method SimpleMethods.getMin , the two steps are the following:
$ ./experiments/setup.sh simple-examples_getMin examples.SimpleMethods getMin
$ ./experiments/run-specfuzzer.sh simple-examples_getMin examples.SimpleMethods getMin
Again, 10 executions will be performed, and for each execution i, the output files of interest are the following:
experiments/simple-examples_getMin/output/SimpleMethods-getMin-specfuzzer-i.assertions
experiments/simple-examples_getMin/output/SimpleMethods-getMin-specfuzzer-i-buckets.assertions
The first file contains the assertions without performing the filtering step, and the second file contains the assertions after performing the filtering step with our mutation and buckets based mechanism.
The file experiments/subjects contains the list of arguments for all the subject methods we analyzed in our paper.
To run GAssert, we followed the instructions provided by the authors in the replication package associated with the paper presenting the tool. As some of our subjects were already included in GAssert cases, we only added the ones that we took from the EvoSpex paper. These subjects were added following the instructions provided by the authors, and then we edited the scripts to also run the tool on the cases we added. To run GAssert on all the subjects perform the following steps:
$ cd $GASSERTDIR/scripts
$ ./run_evaluation.sh
IMPORTANT: the script below will run 10 executions for each subject, and each execution will take 90 minutes.
To run GAssert on single case study, for instance the SimpleMethods.getMin case, run the following script:
$ ./run_gassert.sh GASSERT simple-examples_getMin 90 daikon.assertions
The folder $GASSERTDIR/subjects contains all the subjects we analyzed.
To run EvoSpex, first we need to download the tool with the following commands:
$ pip3 install zenodo_get
$ cd /tools
$ zenodo_get 4458256
These commands will download the replication package associated with the paper presenting the tool. Once downloaded, perform these steps:
$ tar -xvf evospex-all.tar.gz
$ cd evospex-all
$ tar -xvf evospex.tar.gz
$ tar -xzvf object-generation.tgz
At this point, EvoSpex is ready to be executed. For every subject, a setup must be performed before actually running EvoSpex. For instance, to run EvoSpex for method SimpleMethods.getMin execute the following commands from the specfuzzer folder:
$ ./experiments/setup-evospex.sh simple-examples_getMin examples.SimpleMethods getMin
$ ./experiments/run-evospex.sh simple-examples_getMin examples.SimpleMethods getMin
The script will perform 10 executions of EvoSpex for the specified subject. The rest of the case studies can be executed by using the proper arguments described in file experiments/subjects.