Dataset Construction

1. Popularity: Popularity is quantified by GitHub stars, and prior usage in vulnerability research to ensure community relevance and maintenance sustainability. 

2. Vulnerability Density: Ensuring the selected projects have a substantial number of reported CVEs. Make the evaluation results have real safety value.

3. Domain Diversity: Covering a range of application areas, Ensure the comprehensiveness and robustness of the evaluation. 

Finally, we selected 9 project as below.

For each selected project, we manually collect fixing commit of vulnerability from Jan 2020 to Dec 2024. Patch identification is primarily based on CVE references from NVD, and is cross-referenced with official security advisories or project-maintained vulnerability logs.

Finally, we selected 1,128 CVEs with 1,542 patches. Note that a single CVE vulnerability may have multiple patches, hence the two quantities are not consistent. 

Our strict multi-stage annotation process proceeds as follows: 

1. First, Two independent annotators first identify the vulnerability's root cause by analyzing commit messages, vulnerability reports (if available), and CVE descriptions. 

2. Then, the two authors look for the previous modification commit of the vulnerable statement and determine whether that modification commit introduced the vulnerability. After that, we might obtain multiple inducing commits, and we consider the earliest commit as the vulnerability-inducing commit.

3. Finally, versions that include the vulnerability-inducing commit but do not include the fixing commit are considered vulnerability-affected versions. This is equivalent to the data annotation methods of V-SZZ. When there is a discrepancy in the vulnerability-affected version between the two annotators, a third annotator repeats the above steps. The correct vulnerability-affected version is finally determined through a discussion among all three. In this process, a total of 134(11.9%) vulnerabilities showed inconsistencies in annotations, and the Cohen’s Kappa for our dataset is 0.83.

Notably, our annotation methodology also drew upon the practices of open-source software developers, and the following details our email correspondence with them.