Empirical Studies on Regression Bugs
Three Sources of Game Software
A typical video game development usually includes preparing art-effect resources(e.g., the video and 3D model), designing gameplay rules (e.g., storylines and characters), and achieving basic utility code (e.g., maintaining the status of a player and facilitating network communications). Here, we define a video game can generally be represented as a 3-tuple (R, S, D), where R is a set of art resource files, S is a set of source code files that implement the basic functions, and D is a set of design files. A game software implements a set of game tasks, where each game task uses a flowchart in a design file to define the detailed process of the task.
The Evolution of Game Software
The evolution of the game involves the changes of all the three sources. The table below shows the results of an empirical study we conducted on the evolution of 3 industrial games in one month from 12/01/2020 to 01/01/2021. For example, for Game-1, the art files were changed in 9,939 (~67.31%) commits. These changes can be about any visual elements or audios of the game such as model replacement and scene improvement, which can all affect smooth game playing. The flowcharts and code were changed in 2,139 and 2,678 commits respectively. For flowcharts, the average change per commit is 3.2 nodes and for code, an average of 2.2 methods were changed in a commit.
Commit statistics in three industrial games in one month
Empirical Studies on Regression Bugs
We selected a large-scale industrial game as a subject. We collected 2,763 bugs, started from August, 2020, back to more than two years until June 2018. We worked with developers in the game company and manually studied which game file changes caused such bugs as well as the importance of the bugs, marked as High, Normal and Low priorities in the figure. The results show that the changes to three sources all can lead to bugs. To test the new game version for detecting such bugs, the company usually re-runs all tests or runs only some in an ad-hoc fashion for efficiency. To address this issue and improve the effectiveness, RTS techniques that can systematically select necessary test cases are needed.
Regression bugs in a video game
We observe that majority of the regression bugs (93.34%) are caused by the change of the design files. Only 2.46% and 4.20% regression bugs are caused by the changes in the source code and the art resources, respectively. This can be quite different from traditional software in that the regression bugs are often located in the game logic and flowchart defined in design files rather than located in the code level implementations. We also find that most regression bugs in the design files have relatively high priority (45.86%) and normal priority (53.67%).
Furthermore, we investigated the records of these bugs and aim to understand how these bugs are detected. We found that 91 (3.29%) bugs are reported by the users, 2,633 (95.29%) bugs are detected from the in-house quality assurance testing (e.g., by manually written scripts and manual game-playing), 19 (0.68%) bugs are detected by the automated tools and 20 (0.72%) bugs are unknown. To detect such regression bugs, the automatic regression testing is required.
Simplified Example of a Video Game Task
Here is an example in figure shown below to find and start the conversation with Mike, the player needs to perform a sequence of actions such as walking on the map, looking at a person and confirming that it is Mike.
To implement the task, we develop art resources, source code and flowchart. Specifically, the code implements Player and Terrain classes. For example, Player.UseItem supports the player to use his/her items. Player.GetLocation gets the location of a player in the map. The resource includes the game art provided by artists, e.g., the apple tree and the map. The flowchart records the workflow of the task. First, the designer uses Player.UseItem to check whether the player has ``seeds''. Then, it calls Player.GetLocation to check whether the player finds some cropland; if so, the UI will show the seed sowing process via terrain.ShowAppearance. After 60 seconds, it shows the apple tree growing on the cropland. At each step of the flowchart, the corresponding code will be invoked and the art resources will be fetched. The flowchart combined with code and art resources is automatically compiled to the game executable.
In the above figue, in 3. Flowchart, the yellow box indicates that we added a new node terrain.GetStatus("seed") in the new version. This change fixes a game logic bug. In the game, after the players obtain the seeds, they can send the seeds to others when they are in the cropland (this step is defined in another game task and not shown in the figure. If we don't check the existence of seeds before growing them using terrain.GetStatus("seed"), the seeds will incorrectly grow in the terrain.