Examples of DeepHunter Fuzz Testing Results
In this page, we show some (1) failed test examples including initial seeds and final failed tests, as well as (2) the detailed mutation transformation process to generate the failed tests by DeepHunter. For each dataset, we first give some failed tests. Then, we show the detailed intermediate transformation results, describing how the failed tests are generated from from the very initial seeds.
In the detailed results, the transformation process shows some interesting and non-trivial intermediate results generated by DeepHunter (coverage guided fuzz testing). For example, consider the following detailed mutation example 1 for MNIST, it is actually very interesting that a final failed test "3" (i.e., from step 1 -> step 2 -> .... ->step 14->failed) is generated by surrounding the initial tests "3" by some dots, which forms a shape similar to a dotted rectangle (or zero). It demonstrates the advantage of the fuzz testing with coverage guidance; such non trivial tests could be very difficult to be generated by a simple random mutation without guidance.
*Initial represents the initial seed (i.e., original seeding images from the test data); Step n means the sequential mutation process on the given initial input; Failed represents the generated failed tests. Note that, except for the last failed test, all the the intermediate results are benign tests. During the mutation, some images look so similar, making them imperceptible to human eyes, but they are in indeed different.
MNIST:
Initial Seeds
Failed Tests
Mutation Detailed Example 1:
Initial
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9
Step 10
Step 11
Step 12
Step 13
Step 14
Failed
Mutation Detailed Example 2:
Initial
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Step 7
Step 8
Step 9
Step 10
Failed
Mutation Detailed Example 3:
Initial
Step 1
Step 2
Step 3
Step 4
Step 5
Failed
CIFAR-10:
Initial Seeds
Failed Tests
Mutation Detailed Example 1:
Initial
Step 1
Step 2
Step 3
Failed
Mutation Detailed Example 2:
Initial
Step 1
Step 2
Failed
Mutation Detailed Example 3:
Initial
Step 1
Step 2
Step 3
Failed
ImageNet:
Initial Seeds
Failed Tests
Mutation Detailed Example 1:
Initial
Step 1
Step 2
Failed
Mutation Detailed Example 2:
Initial
Step 1
Step 2
Step 3
Step 4
Step 5
Step 6
Failed
Mutation Detailed Example 3:
Initial
Step 1
Step 2
Step 3
Step 4
Failed