Recent works have confirmed that LLMs have strong capacities in repairing buggy code. To evaluate their performance on our collected C/C++ benchmark, we select several representative LLMs for evaluation. In particular, the evaluation settings are categorized into single-round and conversation-based repair with different evaluation metrics for evaluation. 

For the conversation-based repair experiment described in section 6.2 of the paper, we delineate the details of our experimental settings. Here we introduce two hyperparameters, m and n, representing the maximum number of repair attempts and the maximum conversation length in each attempt, with values of m and n being set as 10 and 3, respectively. Specifically, one repair attempt consists of three continuous conversations. The aim is not only to resolve failures but also to evolve its performance automatically by iteratively investigating failure points that occurred during the same attempt phase. 

The following illustrates a conversation-based prompt, given the the buggy function F_n and error message M_n represents the nth conversation in one attempt.  

At the beginning of this attempt,  the F_0 and M_0 are extracted from our dataset concatenated to construct the prompt. At the patch's verification phase, for example, the first conversation phase, the LLM outputs a patch and evaluates with corresponding Unit Test cases getting a Error Message M_1, if it can pass all the test cases, this patch is considered plausible then the conversation stops and the repair process ends, otherwise this patch is invalid, and we will updated prompt format with its error message M_1 and updated buggy function F_1, to build a new prompt for the continuous conversation. Following this rationale, after the completion of three iterations of conversation, the repair will reset to the initial prompt  again and start a new attempt loop.