Welcome to the homepage of the artifact for the paper "Combining Symbolic Execution and Search-Based Testing for Programs with Complex Heap Inputs", accepted at ISSTA 2017. The artifact comes in the shape of a VirtualBox virtual machine file (file Sushi.ova) with Ubuntu 16.10 and the software and data necessary to replicate the experiments reported in the paper.
You have two options. You can download the artifact evaluated and accepted by the committee at ISSTA 2017 from the ACM Digital Library: click here and scroll down, then click on the "Auxiliary archive" link. Otherwise, you can download a more up-to-date version of the Sushi.ova virtual machine here: In this case the results of the experiments might slightly differ from that contained in the paper, but the setup should be more hassle-free. In both cases beware: It is a huge (~5 GB) download.
This artifact allows you to:
Inspect and execute all the test suites that we computed in our experiments and that produce the test and coverage data reported in Table 1 and Table 2 in the paper. See details below at section Paper test data.
Replicate a subset of the experiments reported in the paper with our prototype SUSHI and EvoSuite on a platform with moderate computational resources, that is, a virtual machine configured with 4G of RAM memory and 2 virtual CPUs. See details below at section Launching the experiments.
Replicate all the experiments with SUSHI and EvoSuite, provided that you throw in sufficient physical resources. See details below at section Remarks.
Due to platform and licensing reasons we are currently unable to provide a replication environment for the experiments with Pex and Seeker.
With the default setting of the provided virtual machine you will be able to launch only SUSHI instantiated with accurate data structure invariants (referred to as SUSHIA in the paper) and EvoSuite to generate the test suites referred in Table 1 and Table 2 at the columns labeled as SUSHIA and EvoSuite, respectively, but you will likely observe worse performance than the data in Table 1, and you will not be able to replicate the SUSHIP experiments (SUSHI instantiated with partial invariants). This is due to the limited computational resources of the virtual machine. We provide this setting to make it easy for the readers to reproduce a relevant portion of the experimental data even if they can only work with machine with moderate computational resources, such as, their own laptops. Replicating the experiments requires you to run SUSHI on hardware with at least 32G of RAM memory and 32 CPU cores, a configuration that VirtualBox currently is unable to manage (but might in the future).
Note that we did not stress-test the setting provided with this virtual machine. Should you experience failures (e.g., SUSHI terminates but does not emit any test case) increase the memory of the VM and/or the number of virtual CPUs. Of course your hardware should provide an adequate amount of physical resources.
Install the latest version of VirtualBox, available here. If at the first launch VirtualBox asks you whether you want to install the Oracle Extension Pack, answer yes. Then, create a new virtual machine by clicking the New button on top of the VirtualBox main window, and when the wizard asks for a virtual hard drive select to use an existing virtual hard drive, and provide the path to the Sushi.ova file. Then boot the machine. It should log in automatically to the default user. Its credentials are:
username: sushi password: sushi
These credentials have root access, so you can use them to install/uninstall packages or update the distribution, should you ever need to do that, or to install the guest additions for the version of VirtualBox you are running (this virtual machine has guest additions for VirtualBox 5.1.14).
The home directory of the sushi user (/home/sushi) contains several subdirectories. The most relevant are
/home/sushi/bin: it contains all the scripts to automatically launch the experiments
/home/sushi/ISSTA2017: it contains SUSHI, EvoSuite and the experiment subjects:
/home/sushi/ISSTA2017/sushi: the binary files of SUSHI
/home/sushi/ISSTA2017/sushi-experiments: the source and binary files for the data structures, GanttProject and TSAFE experiments
/home/sushi/ISSTA2017/sushi-experiments-closure01: same for Google Closure bug 01
/home/sushi/ISSTA2017/sushi-experiments-closure72: same for Google Closure bug 72
More subdirectories will be explained later.
In this virtual machine you can launch SUSHI and EvoSuite experiments. Open a terminal and issue a command with the following structure
$ sushi-[avl|caching|dllhard|closure01|closure02|gantt|treemap|tsafe]- [accurate|partial] <name> $ evosuite-[avl|caching|closure01|closure02|dllhard|gantt|treemap|tsafe] <name>
where [...] indicates a set of alternatives and <name> is a string that identifies different repetitions of the same experiment. For instance, if you want to launch for three times SUSHI over the AvlTree experiment, two times with accurate invariants and one time with partial invariants, you may issue the following commands:
$ sushi-avl-accurate 1a $ sushi-avl-accurate 2a $ sushi-avl-partial 1p
The execution will be silent: the output of SUSHI is redirected to a file dump.txt. You can monitor the execution by opening another terminal and inspecting the content of dump.txt. This can be found in the following out directories:
For SUSHI experiments avl, caching, dllhard, gantt, treemap and tsafe, in the directory /home/sushi/ISSTA2017/sushi-experiments/sushi-out/[AVL|CACHING|GANTT|HARD|TREEMAP|TSAFE]/<name>;
For SUSHI experiment closure01, in the directory /home/sushi/ISSTA2017/sushi-experiments-closure01/sushi-out/<name>;
For SUSHI experiment closure72, in the directory /home/sushi/ISSTA2017/sushi-experiments-closure72/sushi-out/<name>;
For EvoSuite experiments, in the directory /home/sushi/ISSTA2017/evosuite-out/[AVL|CACHING|CLOSURE01|CLOSURE72|GANTT|HARD|TREEMAP|TSAFE]/<name>.
For example, you can inspect the dump.txt for the SUSHI AvlTree experiment 1a by opening a console and issuing the command:
$ tail -f /home/sushi/ISSTA2017/sushi-experiments/sushi-out/AVL/1a/dump.txt.
A dump.txt file produced by a SUSHI run has the following structure:
17-02-20 12:14:54.626 INFO - This is Sushi, version 0.1... ... 17-02-20 12:14:54.718 INFO - Phase 1: executing tool JBSEMethods ... 17-02-20 12:15:05.740 INFO - Phase 2: executing tool Merger 17-02-20 12:15:05.791 INFO - Targets to cover: 57, traces to explore: 224 17-02-20 12:15:05.792 INFO - Phase 3: executing tool Minimizer 17-02-20 12:15:05.861 INFO - Phase 4: executing tool JBSETraces ... 17-02-20 12:15:06.872 INFO - Phase 5: executing tool Javac ... 17-02-20 12:15:08.991 INFO - Phase 6: executing tool Evosuite ...
After the initial salutations you will find rows that set the start of each SUSHI phase. Phase 1 is symbolic execution, phase 2 merges the results of symbolic execution and produces some temporary files that will be used in later phases. It also prints statistics on how many targets are there in the software under test to be covered, and how many paths did symbolic execution explore. Phase 3 selects a set of traces that covers all the targets and has minimal cost according to a path complexity heuristics. Phase 4 and 5 produce the path condition distance calculators that will be fed Evosuite as fitness functions. Phase 6 runs Evosuite. For convenience the sushi-*-accurate experiments will also log the command lines used by SUSHI to invoke the instances of Evosuite. The sushi-*-partial experiments have a seventh phase, LoopMgr, that checks whether all the targets have been covered, and in the negative case repeat the phases from 3 to 7 until all the targets are covered or all the traces have been explored.
The generated tests are found in the following directories:
/home/sushi/ISSTA2017/evosuite-test for the tests generated by EvoSuite;
/home/sushi/ISSTA2017/sushi-experiments/sushi-test for avl, caching, dllhard, gantt, treemap and tsafe subjects, generated by SUSHI;
/home/sushi/ISSTA2017/sushi-experiments-closure01/sushi-test for closure01, generated by SUSHI;
/home/sushi/ISSTA2017/sushi-experiments-closure72/sushi-test for closure72, generated by SUSHI.
Beware: All the repetitions of a same SUSHI/EvoSuite experiment write in the same directory. Thus a repetition of an experiment will overlap, and possible overwrite, the tests produced by earlier repetitions. Clean the directories before generating new tests.
The tests are generated in EvoSuite format, where each run of EvoSuite produces one scaffolding class and one test suite containing all the test cases. The EvoSuiteexperiments will produce exactly this format. SUSHI instead will produce many suites each containing exactly one test, so if a run of SUSHI produces, e.g., 10 test cases then you will find 10 scaffolding classes and 10 test suites with 1 test case each. These files have names structured as follows:
<class name>_<method name>_PC_<number>_<number>_Test_scaffolding.java (the scaffolding), and
<class name>_<method name>_PC_<number>_<number>_Test.java(the suite).
where <class name> is the name of the class under test, <method name> is the name of the method under test, and PC_<number>_<number> identifies the trace along which the test executes (you don’t need this information so we will not elaborate on it further).
You have six commands to compile the tests:
$ javac-test-[closure01|closure72|others]-[evosuite|sushi] <tests>
where <tests> is a list of .java files. The variants javac-test-others-* are used to compile the tests for the avl, caching, dllhard, gantt, treemap and tsafe subjects. The variants javac-test-*-evosuite are used to compile the test generated with EvoSuite, while the variants javac-test-*-sushi are used to compile the test generated with SUSHI.
Note that the scaffolding and test suite classes are declared in the same package as the class under test so they can access the package-level members of the class under test. This means, for example, that the .java files generated by SUSHI with the test suites and scaffoldings for the avl_tree.AvlTree class under test are in the /home/sushi/ISSTA2017/sushi-experiments/sushi-test/avl_tree directory. If you want to compile and execute them you must issue the command:
$ javac-test-experiments-sushi /home/sushi/ISSTA2017/sushi-experiments/sushi-test/avl_tree/*.java
Once compiled the tests it is possible to print the coverage data with the commands:
$ coverage-[avl|caching|closure01|closure02|dllhard|gantt|treemap|tsafe]
The command will produce an output with this format:
17-02-14 14:16:43.571 INFO - Analyzed data for 14 test cases 17-02-14 14:16:43.574 INFO - In total covered 40 (out of 40) branches
For convenience we included all the test suites that we generated in the experiments described in the paper, that is, the test suites generated with our prototype SUSHI, instantiated both with accurate (SUSHIA) and partial (SUSHIP) invariants, and with all competing tools discussed in the paper, that is, JBSE/DHM with partial invariants, JBSE/DHM with accurate invariants, Pex, Seeker and EvoSuite. The tests are in the following directories:
/home/sushi/ISSTA2017/sushi-experiments/sushi-test-paper for avl, caching, dllhard, gantt, treemap and tsafe
/home/sushi/ISSTA2017/sushi-experiments-closure01/sushi-test-paper for closure01
/home/sushi/ISSTA2017/sushi-experiments-closure72/sushi-test-paper for closure72
To visualize the coverage data you must first compile the tests with the command
$ javac-test-paper
and then issue the command
$ coverage-paper [dhm-partial-ok|dhm-partial-ok+fp|sushi-partial|dhm-accurate| sushi-accurate|pex|seeker|evosuite]?
where [...]? is an optional indicator of which combination of tool and invariant one wants to consider. If none is provided all the combinations will be displayed, amounting at showing all the data in Table 1 and Table 2 of the paper. For example, the command
$ coverage-paper sushi-partial
will produce the output:
## sample 17-02-14 15:10:06.297 INFO - Analyzed data for 2 test cases 17-02-14 15:10:06.301 INFO - In total covered 2 (out of 2) branches ## treemap 17-02-14 15:10:07.274 INFO - Analyzed data for 50 test cases 17-02-14 15:10:07.278 INFO - In total covered 138 (out of 164) branches ... ## closure01 17-02-14 15:10:10.904 INFO - Analyzed data for 4 test cases 17-02-14 15:10:10.908 INFO - In total covered 11 (out of 279) branches ## closure72 17-02-14 15:10:11.717 INFO - Analyzed data for 2 test cases 17-02-14 15:10:11.720 INFO - In total covered 16 (out of 23) branches
that correspond to the data in columns 1, 5 and 10 in Table 1. The test suites that reveal the bugs in the subjects tsafe, closure01 and closure72, will produce additional messages in the output about the violated assertions. Beware: The binaries of the compiled tests are all put in the same directory. Therefore, whenever you issue the javac-test-paper command you can no longer visualize the coverage of the tests that you generated. At the purpose, you need to recompile them with javac-test-[closure01|closure72|experiments]. Conversely, every time you invoke javac-test-[closure01|closure72|experiments] you lose the binaries of the paper test data, so you need to recompile with javac-test-paper before you can print the paper coverage data. Incoherent data will be shown if you don't follow this protocol.
The copy of SUSHI you find in the virtual machine is complete, and you can use it to run SUSHI on subject programs different from those included. While this goes beyond the scope of experiment replication we report some brief instructions on how to configure SUSHI for analyzing subject programs different from the ones provided.
SUSHI is a Java application with many external dependencies, some of which native. We provide a sushi command that sets the classpath and library path and launches the Java application: Thus you may launch SUSHI with syntax
$ sushi -params_modifier_path <path> -params_modifier_class <className>
where <className> is the name of a Java class containing the parameters for a SUSHI execution and <path> is the classpath of the parameter class. If, for example, we inspect the content of the file /home/sushi/bin/sushi-avl-partial we will find:
#!/bin/sh ... ... sushi -params_modifier_path /home/sushi/ISSTA2017/sushi-experiments/bin -params_modifier_class avl_tree.settings.AvlTreeParametersPartial ...
The parameter class must be a subclass of the class sushi.configure.ParametersModifier contained in /home/sushi/ISSTA2017/sushi/sushi-lib.jar. In the previous example the source code of the parameters class avl_tree.settings.AvlTreeParametersPartial is found in /home/sushi/ISSTA2017/sushi-experiments/sushi-src/avl_tree/settings. By reading the source code of the class you may notice that the bulk of it is the definition of the method public void modify(Options p), that overrides a method with same signature in the superclass. Most parameters are defined here by suitably invoking setter methods of p. The most important are:
p.setEvosuitePath, p.setSushiLibPath, p.setZ3Path and p.setJREPath: used to set the paths to some of the external tools that SUSHI invokes. You may used the same constants used in the `AvlTreeParametersPartial` class. These constants are defined in the file /home/sushi/ISSTA2017/sushi-experiments/sushi-src/common/Settings.java. Note however that all the tools are contained in the /home/sushi/ISSTA2017/sushi directory.
setClassesPath: used to set the classpath of the software to test it accepts a comma-separated list of java.nio.file.Path objects.
setTargetClass: the name in internal classfile format of the class to test: SUSHI will generate tests for all the methods in the class.
setJBSEBudget, setMinimizerBudget, setEvosuiteBudget, setGlobalBudget: the time budget, in seconds, for the symbolic execution, trace selection and search phases, and for the whole run of SUSHI, respectively.
setCoverage: what SUSHI should attempt to cover. Please leave this parameter to its default value Coverage.BRANCHES.
setOutDirectory: a java.nio.file.Path to a directory where the generated tests will be put.
setTmpDirectoryBase: a java.nio.file.Path to a temporary directory; SUSHI needs to create many files for its intermediate results, and will put them in a subdirectory of this directory having as name the date and time it was launched.
setParallelismEvosuite: the maximum number of concurrent Evosuite search threads that may be active at the same time.
setRedundanceEvosuite: the number of concurrent evosuite search threads that must search the same trace, to obviate for Evosuite randomness (usually a value of 1 is OK).
The virtual machine can be accessed from the host via SSH and SFTP. A port forwarding rule sends everything you put on your host's port 3022 on the guest's port 22, so you just have to run the commands:
ssh -p 3022 sushi@localhost sftp -P 3022 sushi@localhost
(note that it is -p for SSH and -P for SFTP).
SUSHI is an open source tool, so the most adventurous users may want to entirely skip the virtual machine setup and use the bleeding-edge version of SUSHI. To this purpose we made available both SUSHI and all the experiments as a set of five Github projects. The first two projects are the tool itself:
The remaining three projects contain the code of the experiments reported in the paper:
Instructions to set up everything are available in the README files at these repositories. Our advice is to check out the projects under a same Eclipse workspace and use the Eclipse launch configurations contained in the experiments projects to run the experiments. Note that the launch scripts will not be available.