In this page we describe how to perform the experiment to assess the quality of the test suites produced with LI as well as LI+NN. The analysis is performed in terms of pruning precision by counting the amount of spurious objects (with respect to the ones produced by BLISS) and in terms of branch coverage.
The following steps shows how to analyze the quality of a test suite produced with the LI technique for the method AvlTree.insert and a scope of 5.
1. Follow the instructions from the example, but for method insert instead of bfs and scope 5 instead of 8, without step 5.
2. In class AvlTree.java, before the invocation to method countStructure(X), include the two following lines:
X.printCurrentPath();
X.dumpStructure(X);
3. Redirect the execution output to a file, and perform the symbolic execution (step 5) in the example.
4. Finally, go to the folder jpf-symbc-bounded and, assuming that your output file is symbexec.txt, run the following script:
$ python analyze-covered-branches.py symbexec.txt neuralnetwork/symbolicexec/bliss/avltree-05-insert-bliss-uniquestructs.txt
This is script will determine the amount of spurious structures produced when symbolic execution finishes, and determine the branch coverage of the produced suite. If everything was executed correctly, the output will be:
SE output file: symbexec.txt
Bliss valid structures: neuralnetwork/symbolicexec/bliss/avltree-05-insert-bliss-uniquestructs.txt
> Computing coverage
> Results
Structures: 165
Branches: 14
Valid structures: 29
Covered branches: 10
Percentage: 71.4
Spurious structures: 136
Covered branches: 14
Percentage: 100.0
All structures: 165
Covered branches: 14
Percentage: 100.0
To analyze this same method but with the LI+NN technique, follow these same instructions but considering this example.
For other scopes or methods, follow the analogous instructions using the corresponding file in step 4 (the one from the folder neuralnetwork/symbolicexec/bliss).