Experiment

Approach

Our research was run on the DevGPT dataset, which is an amalgamation of 29,778 prompts and responses sourced from open source GitHub Repositories, including links to source code, commits, issues, pull requests, discussions, and Hacker News Threads. In order to focus on code refactoring, we filtered out all of the instances with the key word "refactor." We began processing this subset by removing duplicate instances, invalid links, and irrelevant conversations. The final curated dataset consisted of all unique, "refactor"-related DevGPT `Commit,' `File,' `Pull Request,' and `Issue' instances. To analyze our data, we categorized the instances into 4 groups: `Exact,' `Modified,' `No,' and `Supplementary Info' (see definitions below). For each research question, we considered a subset of these categories. For RQ1, we focused on `Yes' instances, as well as further analyzing the `Modified' instances by creating a taxonomy. RQ2 considered all instances and divided them into prompt categories for further analysis. Finally, RQ3 only considered  conversations that produced code, meaning `Supplementary Info' instances were disregarded.

Definitions:

'Yes' instances: The code provided in the ChatGPT was utilized in the GitHub project.

'No' instances: The code provided by ChatGPT was not utilized at all in the GitHub project.

'Supplementary Info' instances: The ChatGPT conversation did not produce code, but the content was utilized in some other form, such as the README, an article, or for reference..

'Exact' instances: Subset of 'Yes,' the code produced was not modified at all before merging to the repository.

'Modified' instances: Subset of 'Yes,' the code produced needed to be altered before merging to the repository.