Runtime Analysis

Since the cost of mutation testing for traditional software is a major concern, we explain the time cost of DeepMutation++ in terms of 1)mutant generation phase and 2)robustness analysis phase.

Models used:

FNN - LeNet-5 (9 layers, 236 neurons, and 169226 parameters)

RNN - simple LSTM (3 layers and 2561051 parameters).

1)Mutant Generation Time Cost

DeepMutation++ adds time calculation component into mutant generation function, so we just need to run mutant generation command directly to record the time cost. We use the follows command to record the time cost of mutant generation.

python generator.py -model_path mnist_lenet5.h5 -data_type mnist -threshold 0.9 -operator 1 -ratio 0.01 -save_path lenet5-WS0.01 -num 200 -standard_deviation 0.5

Table I: mutant generation time cost

Table I shows the results. In the FNN table, the first column shows the mutation ratio, note that the layer-level operators just add one layer to the original model so they don't need the ratio options. The rest values in FNN table presents the time cost (seconds) for one mutant generation, i.e, seconds/mutant.

In the RNN table, the values in WGF and WPR columns are calculated in the same way as FNN. Note that, WGF and WPR both use the mutation ratio 0.03. The remain results are calculated in the manner of seconds/time step.

2)Robustness Analysis Time Cost

The same as mutant generation, the robustness analysis code fragment also contains time calculation component. We just run the analysis command to record time cost.

python sort_data.py -model_path mnist_lenet5.h5 -mutants_path lenet5-mutants -save_path result/

Table II: robustness analysis time cost

Table II shows the time cost of analyzing FNN model robustness against inputs and RNN model robustness at segments.

FNN row - time cost (seconds) for analyzing one input on 200 mutants.

RNN row - time cost (seconds) for analyzing 80 segments with one time mutation.

Page updated

Google Sites

Report abuse