Collection of Defects4C_bug

Overview

Our approach, defects4C_bug, focuses on extracting bug commits from widely used GitHub C/C++ repositories using a set of predefined bug-related keywords. We then develop a meticulously designed unit test pair verification algorithm to identify corresponding test cases for each bug fix. To ensure the benchmark quality, we incorporate a three-stage human annotation process, executed by three security experts. This process is crucial for filtering out false positives – instances where commit messages contain bug-related keywords but the code changes are not related to fixing bugs or security issues. Our thorough and systematic approach has yielded 248 bugs as well as their corresponding unit tests for verifying the existence of the vulnerability. Additionally, we have categorized these bugs into four distinct groups, as classified by security experts, to support and enhance future research using our dataset.

To understand the effectiveness of existing APR techniques in fixing C/C++ bugs, we conducted an empirical study using our Defects4C benchmark. The study focuses on the performance of LLM-based APR techniques, incorporating leading large language models such as GPT-3.5 turbo, GPT-4 various iterations of CODELLAMA, and WizardCoder. These models are tested in both single-round and conversation-based program repair scenarios. Our findings reveal a notable gap in the performance of state-of-the-art LLM-based APRs on C/C++ bugs compared to their success with the Defects4J benchmark. This discrepancy underscores the urgent need for APR techniques tailored specifically for C/C++ fault repair. Our newly developed Defects4C, with its high-quality and comprehensive dataset, is poised to be a resource for future research in testing and repairing C/C++ programs.

Contents

• collect and validate from Bigquery+GitHub

• Unit Test Pair verification

Collect validly content from Bigquery+GitHub

2.  Use the GitHub API to filter out invalid commit 

Because the commits obtained through BigQuery queries may become invalid over time due to repository ownership transfer, archiving, and other reasons, we filter out these invalid commits. Through our rigorous deduplication process to remove duplicate commits, we obtain 9M commits.

The corresponding code can find from crawler source code with path at process_github_commit/run.sh

3.  Modification Only at function level statement

Although keyword filtering can help us obtain potential bug commits, these commits may consist of multiple file modifications which are difficult to analyze and unsuitable for APR evaluation. Thus, we only keep the commits that have one function content changed for use. Furthermore, some of the selected commits lack an expert-written test suite for validation. We also filter out them to ensure the dataset is runnable and verifiable. Finally, we obtain  76K commits for usage. 

In particular, we use the tree-sitter to parse the function definition from the source code, where the node type is "function_definition" and "function_declarator". we also tried clang to extract function statements, however, clang needs the header file and its other dependence to parse specifically for C++ files, when the source code and header file are separated into two files, i.e. the member function implemented in xx.cpp and defined in xx.h, the clang parse cannot precisely parse function statement in this case.  

Unit Test Pair verification

With collecting 76K commits from above, each of them has the test suite which consists of multiple test cases. To accurately identify which test case in the test suite is paired with the current changed code, we design our unit test pair verification algorithm, which is based on two basic observations:  

• for a bug fix, there typically exists a unit test that passes on the corrected code version but fails on the buggy version in a single commit. 

• if any changes only happen at the inner side of the function statement, this fix may largely resolve the buggy statement and be less related to the compiler or environment perturbation.