Experiment

Methodology

Experiment are carried out in large instances of a virtual machine at Amazon EC2 with the following configuration (m1.large): 7.5 GB RAM, 4 EC2 Compute Units (2 virtual cores with 2 EC2 Compute Units each), 35 GB instance storage. For each of 30 runs of each experiment with each application under test (AUT), we run it for 24 hours of time limit (which is chosen experimentally) to establish what coverage can be achieved for this AUT. That is, the total execution time is 1,440×24=34,560 hours. With the cost of $0.48 per instance per hour as of September, 2011, the total cost of this experiment is over USD $16,500 not to mention the additional cost for disk space, I/O requests, and the internet traffic. We determine the maximum coverage for each run within each experiment that was reached after running for 24 hours, and then we select the minimum of the maximum coverages reached for a set of 30 runs for an experiment. This way we experimentally establish the ceiling for test coverage so that we can retrieve times it takes to reach this coverage in each run.

Benchmark

We used 12 Java programs, whose size ranges from 300 LOC to over 1 MLOC. The characteristics of the benchmarks are in the following table.

Result

Here is the table summarizing the results of the experiments. The results show that CarFast outperforms evaluated competitive approaches with most subject applications. See details in the paper.