Juliet Testcases

In this Appendix section, we present supplementary materials in accompanying with our main text. In Sec. 5.1, we present the FP/FN root cause distributions about the CWE121 and CWE190 cases. Here, we report the FP/FN root cause distributions of all CWE cases. The FP cases for both tools have been discussed in Sec. 5.1. As for the FN cases, Joern manifests overall consistent distributions of FN root causes across all CWEs. This illustrates that incomprehensive queries (FN1) is the primary root cause to Joern’s FNs for most CWEs. As introduced in Sec. 4, for each CWE, JULIET contains up-to thousands of vulnerable program variants. This makes developing comprehensive queries to cover all vulnerability variants highly challenging. In contrast, we find that CodeQL has relatively more diverse distributions across different CWEs. For instance, for the CWE570, our manual investigation shows that all FNs are due to incomprehensive queries (FN1). This is primarily due to the query for CWE570 only detects the always-false expression when the expression is used in if condition and the body of if is about handling allocation error. CWE253 shares the same result and the reason is discussed in Sec. 5.1. For the CWE457, three categories of FNs(FN3, FN4 and FN5) do not exist because no such test cases are included in JULIET. 

CWE Complementarity of various tools in detecting vulnerabilities