RQ3: Study of CRS Tests
Based on the collected tests, we further study the effectiveness of existing tests in detecting collected bugs and the potential challenges in CRSs. In general, the testing process involves running the target program with given test cases and using test oracles to determine if any bugs occur. Therefore, our study focuses on three main questions: (1) whether the target functions with bugs can be tested, (2) whether the test cases can trigger the buggy code when the functions are executed, and (3) whether the test oracles can identify the bugs when the buggy code is covered.
Findings
Common testing methods, such as unit tests, integration tests, and fuzz tests, are insufficient for detecting the collected bugs, with only a small portion (21.21%) being detectable by existing tests. The reasons for the missing bug detection include the lack of test drivers (41.96%), test oracles (9.32%), and test inputs (27.97%).
Percentage of bugs with uncovered buggy functions in different root causes
Percentage of bugs without testing oracles in different symptoms