After observing the kinds of questions developers would ask ChatGPT in various circumstances through the development process, we asked if the language ChatGPT was responding to would be a sensible variable to measure within the study to help developers understand how to further utilize this tool in their development process. For example, if developers were more likely to ask ChatGPT for advice with procedural programming languages, it would be a practical conclusion of this study to recommend developers to seek the aid of ChatGPT when working with languages like C or Pascal rather than scripting languages.
Displayed to the left is a chart outlining the amount of languages utilized by developers within our selected sample of the DevGPT dataset, as well as the frequency of the appearance of each language. The 'Other' category is comprised of languages with only one instance in the dataset. All languages are labeled and analyzed further in each category's breakdown of their respective included languages seen below.
Results
This chart showcases the different languages that developers use in their commits and demonstrates CSS as a stand-out choice among others. In the case of CSS, there were 138 unique occurrences where developers adopted ChatGPT’s recommendations into their final commit, out of which 120 were exact matches while 2 had slight modifications and 3 were not adopted. The high number of instances with CSS implies that developers perceived ChatGPT’s guidance as highly effective and precise when dealing with tasks related to CSS refactoring. The Commit category heavily embraces CSS because it has a direct bearing on the visual presentation and user interface— aspects that see frequent updates with finer refinements. The small nature of CSS changes makes them more incremental; hence, ideal for constant commits.
This chart displays the wide variety of languages featured in the File category, with the majority of files being in CSS, C++, JavaScript, and TypeScript, contrasting the sparse distribution throughout the other languages. CSS and C++ led in popularity with 5 instances each, both having a 1:4 exact to modified ratio. The projects in 'Files' are often web pages or applications, which explains the high frequency of CSS, JavaScript, and TypeScript projects. The fact that the only two unmerged instances were in JavaScript and TypeScript, a superset of JavaScript, could indicate that ChatGPT is not as effective in helping with refactoring tasks with these languages due to the extensive documentation and nuanced nature of JavaScript and larger-scale applications done with TypeScript.
The next figure displays the wide variety of languages used within the issue category, showing the continued interest in utilizing ChatGPT as a refactoring tool from developers when it comes to issues in their Github Repositories. While the number of instances in this category are limited, there is a preference for languages that support object-oriented programming, namely TypeScript and Python. This could lead to the presumption that developers are more likely to turn to ChatGPT when faced with an instance in the `Issues' category that deals with object oriented programming. Creating an `Issues' in Github is a common procedure for developers throughout the design process, particularly lengthy ones, so that they can keep track of parts of the code that they plan to work on at a later time. Thus, a possible reason for the abundance of instances that focused on a language that supports object oriented programming could be that 'Issues' instances in Github are more likely to be seen in repositories that plan to contain an extensive amount of code.
The final figure displays the variety of languages featured in the Pull Request category, clearly showing a favor for TypeScript among the other selected languages. The category of Pull requests involves developers suggesting changes to an existing repository, most of the time to solve existing errors or bugs in the code. A possible reason why TypeScript is the favorite in this category among the many other languages is that it is used to develop large-scale applications, a process that often leads to oversights or seemingly inexplicable error messages. Therefore, it would make sense for developers to suggest ChatGPT refactoring through Pull Requests, as TypeScript is a commonly used collaborative language where developers will likely turn to outside resources such as ChatGPT for aid.