Mapping Vulnerability Data in Benchmarks and Rules of  Tools to CWE

As described in our paper, we use CWE as a reference to map detection rules of SAST tools and the vulnerabilities in dataset. The reasons for it are as follows:

(1) Actually, different SAST tools use different identifiers for the types of vulnerabilities they support. Insider, for example, use CWEs, while others introduce their own vulnerability identifiers. These different identifiers make it difficult to automatically assess whether a tool is referring to the correct vulnerability type, which would allow for a more rigorous evaluation. 

(2) Besides, there exists a case that several rules pointing to the same tool may correspond to the same vulnerability. For this, we give an example for CodeQL, in which several rules point to CWE-664, especially for those related to path traversal. We aim at mapping rules of selected tools to the same granularity, so we choose the CWE classes as a  reference.

Therefore, to enable us to automate the evaluation of the tools studied, we consider mapping detecting rules and the vulnerabilities in the dataset (CVE/CWE IDs) directly to CWE “Pillar” classes of the CWE-1000 view, which is a view provided for researchers. 

Mapping vulnerability data to CWE. Note that vulnerabilities in our two benchmarks have been mapped to CWE Weaknesses, we thereby mapped them to CWE Classes in the CWE-1000 View according to their rule documentation.

Mapping detecting rules to CWE. Since these tools have mapped their detecting rules to CWE Weaknesses except for SpotBugs, we need to further map them to CWE Classes. However, for SpotBugs, we mapped its rules to CWE Weaknesses and CWE Classes manually by involving three authors to perform the mapping independently. In this process, if their mapping results are consistent, the result will be accepted directly. Otherwise, a discussion is required until the mapping results of the three authors coincide. For the mapping from CWE Weaknesses to CWE Classes, a CWE Class is thereby considered supported if the documentation states that the respective tool implements a security check for at least one of the CWE Weaknesses in that specific class.

The detailed CWE mapping results of each selected SAST tools are as follows: