Comment Information: measures the sufficiency and clarity of the instructions contained in the comment regarding the code change, which reflects the difficulty for the contributor or a model to refine the code.

For example, a comment like ``There are spaces missing'' is more informative than ``This function name does not describe well what it does.''  We followed the definition of comment information from, and divided the comment information into three levels:

• Level 1 (Vague Question): The review comment only gives a general direction for modification (e.g., ``we should maintain the consistency of variable naming'') without clear suggestions for changes.

• Level 2 (Vague Suggestion): The review comment provides specific suggestions for modification (e.g., ``changing it with camel case style''), but does not directly specify the location of the code that should be modified.

• Level 3 (Concrete Suggestion): The review comment includes explicit requests for adding or modifying code snippets (e.g., ``changing the variable name 'testfile' to 'testFile''') or explicitly identifies code snippets to be removed.

Figure 3 illustrates the results of ChatGPT on different comment relevance and information levels. The figure highlights that ChatGPT performs the best when the comments are classified as perfect relevance, outperforming both partial and not relevance levels. In addition, ChatGPT performs the best on reviews that contain concrete suggestion information, while performing similarly for vague suggestions and vague questions. The results imply that the quality of data significantly impacts ChatGPT's performance, as reviews with low relevance and low information do not provide enough context and information for ChatGPT to make accurate predictions.

Code Change Category is used to measure the intention of the code changes. We defined the categories based on our annotations. There are 4 major categories, including Documentation Category, Feature Category, Refactoring Category, and Documentation-and-Code Category.

• Documentation Category represents code changes that only add, modify, or remove documentation. Modifications according to conventions (Documentation-conventions) may also involve additions, modifications, or deletions, but we separated them for easier analysis of the unique challenges it poses to the model's prediction of revised code.

• Feature Category represents code changes in terms of functional logic, such as adding, modifying, or removing code.

• Refactoring Category refers to non-functional code refactoring, including renaming code entities (Refactoring-rename), swapping two code snippets (Refactoring-swap), and updating code based on coding standards (Refactoring-conventions).

• Documentation-and-Code Category represents code changes that include both documentation and code modifications.

The table above contains 400 data cases, which were randomly selected from Codereview and CodereviewNew datasets with 200 cases each. In addition to the data such as EM and EM_Trim, BLEU, and BLEU_Trim, the table also includes manual classifications of relevance, information, and category. Relevance_user1 represents the evaluation of the first expert on relevance, Relevance_user2 represents the evaluation of the second expert on relevance, and Relevance represents the score merged after discussion by the two experts.

Due to limited table space, old code and new code are not listed in the table, but they can be viewed from the original data based on the Original_id. The original data for codereview type is in codereview.jsonl, and the original data for codereview_new type is in codereview_new.jsonl.