Embedded Files
How helpful is the code generated by ChatGPT in assisting developers with refactoring tasks?
How helpful is the code generated by ChatGPT in assisting developers with refactoring tasks?
Motivations
Motivations
The growing use of ChatGPT by developers calls for a thorough analysis of its performance in practical settings. Although prior research has mostly concentrated on controlled settings, it is necessary to assess ChatGPT's performance in real-world settings. Refactoring well not only makes code easier to read and maintain, but it also lowers the number of errors and raises the caliber of software as a whole. We hope to give developers useful information about maximizing their usage of ChatGPT by evaluating how well it performs in aiding with refactoring activities, thereby saving time and effort on repetitive coding jobs. When deciding when and how to incorporate ChatGPT into their workflows, developers can make well-informed decisions based on their understanding of the environment in which it works best, utilizing ChatGPT's full potential to speed up development. process.
Results
Results
Taxonomy of the topics relate to modified codes when asking ChatGPT
The above taxonomy divides the four different categories of topics for modified instances. This taxonomy can help future developers improve their conversation between ChatGPT to minimize the conversation while needing to modify lesser codes. First, 3.57% developers give the exact code and then ask for advice on how to make it better, adding the recommendations right into the code they have already provided. Second, 53.57% developers look for generic guidance or methods and then modify them to meet their own projects, which leads to changed code. Thirdly, 21.42% developers tweak some functions to better fit their project requirements while keeping the general structure of the created code. Lastly, to maintain uniformity and clarity across the current codebase, a category is devoted to modifications in variable names. This taxonomy highlights the typical ways that code modifications are made, showing how developers can customize ChatGPT's output to suit their requirements.
The left chart shows the percentage of "Exact" versus "Modified" cases for the "commit" category when developers utilized ChatGPT for code refactoring is shown in the pie chart. The statistics showed that a vast majority of cases (97%) were precise, indicating that developers accepted ChatGPT's code without making any changes. Conversely, just a small percentage of (3%) of the cases needed to be changed. This high percentage of precise matches implies that ChatGPT's suggestions were very accurate and successful for the 'commit' category, meaning that developers were able to use the generated code for their purposes without the need for additional modifications.
The left chart shows the percentage of 'Exact' versus 'Modified' cases for the 'File' category when developers utilized ChatGPT for code refactoring is shown in the pie chart. 31% of the cases in this category were precise, meaning that developers used ChatGPT's code exactly as it was written. On the other hand, 69% of the cases needed to be changed, indicating that a significant amount of the code that was generated needed to be adjusted before being incorporated into the projects of the developers. The increased proportion of altered occurrences in the 'File' category implies that although ChatGPT offers helpful foundations, developers frequently need to modify the code to better suit their unique needs. It may be necessary to make structural, functional, or variable name changes to bring the code into compliance with the project requirements and the current codebase. This finding highlights the importance of developer intervention in refining ChatGPT's output to ensure optimal performance and suitability for their unique coding contexts.
For the 'Issue' category, the pie chart displays a similar proportion of 'Exact' versus 'Modified' occurrences, with each representing 50% of cases. This shows the developers thought there was an equal chance that the code produced by ChatGPT would be utilized exactly as it was or would need to be modified to fix problems. The evenly distributed data indicates that although ChatGPT frequently offers precise and practical code solutions, there are times when developers must make modifications to better suit the particular scenario or find a more workable alternative. This can entail modifying routines, reworking the logic, or changing variable names to better fit the project's needs and the current codebase. The uniform distribution demonstrates ChatGPT's adaptability in producing highly adaptive and flexible initial solutions, contingent on the developer's needs and the complexity of the issue at hand.
The pie chart illustrates the proportion of 'Exact' versus 'Modified' instances for the 'pull request' category when developers used ChatGPT for code refactoring. In this category, 62% of the instances were exact, meaning that developers adopted the code generated by ChatGPT without any modifications. In contrast, 38% of the instances required modifications. This higher percentage of exact matches suggests that ChatGPT's recommendations were largely effective and aligned with the developers' needs for pull requests, which often involve reviewing and merging changes into the main codebase. The remaining 38% of modified instances indicate that while ChatGPT provides a strong foundation, developers sometimes need to tweak the generated code to better fit the specific requirements or standards of their projects.
Page updated
Report abuse