Concolic testing is popular in unit testing because it can detect bugs quickly in a relatively small search space. But, in system-level testing, it suffers from the symbolic path explosion problem, and often misses bugs. To resolve this problem, we have developed a focused compositional concolic testing technique, FOCAL, for effective bug detection. Focusing on a target unit failure v (a crash or an assert violation) detected by concolic unit testing, FOCAL generates a system-level test input that validates v. This test input is obtained by building and solving symbolic path formulas that represent system-level executions raising v. FOCAL builds such formulas by combining function summaries one by one backward from a function that raised v to main. If a function summary ϕ_a of function a conflicts with the summaries of the other functions, FOCAL refines ϕ_a to ϕ′_a by applying a refining constraint learned from the conflict. In the experiments, FOCAL showed high system-level bug detection ability by detecting 71 out of the 100 real-world target bugs in the SIR benchmark programs while other relevant cutting edge techniques (i.e., AFL-fast, KATCH, Mix-CCBSE) detected at most 40 bugs. Also, FOCAL detected 13 new crash bugs in popular file parsing programs.
We provide a benchmark on 100 real-world crash bugs of the 7 C programs (bash-2.0, flex-2.4.3, grep-2.0, gzip-1.0.7, make-3.75, sed-1.17, and vim-5.0) in the SIR benchmark. The crash bug data were collected and organized after examining all bug reports of the last 13 to 25 years (i.e. after examining bug fix commits reported since 1993 until 2018).
1. FOCAL_crash_bug_description.xlsx: This excel file describes crash bug information including faulty statements and corresponding patch information.
We provide detailed experiment results.
1. FOCAL_experiment_results.xlsx: This excel file contains detailed experiment results.
2. FOCAL_known_crash_bug_system_tests.zip: This archive contains FOCAL-generated system test drivers and system test inputs for the target programs of the FOCAL real-world crash bug benchmark.
3. The responses of the original developers for the new bugs detected by FOCAL: This excel file contains
CAUTION: Following the URLs to bug reports in this excel file may reveal author identity.
If you have any questions, please send an e-mail to focalfse2019@gmail.com