GAMERTS
A Regression Testing Framework for Video Games
GameRTS: A Regression Testing Framework for Video Games
2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE)
Jiongchi Yu, Yuechen Wu, Xiaofei Xie*, Wei Le, Lei Ma, Yingfeng Chen*, Yujing Hu, Fan Zhang
Data: Due to the policies and regulations of the company, we cannot make all detailed data and code available. We tried our best to provide more detailed data in this website including 1) the initial regression tests for each task 2) the constructed graph based on the traces of the initial regression tests, 3) the affected states and transitions in the graph, 4) the selected traces (i.e., the selected regression tests) for v0->v1 and v1->v2, 5) the detailed results about the context sensitivity of game behaviours and 6) the videos of the new detected bugs.
More details can be found:
1) Initial regression tests, state transition graphs including the affected states/transitions
Abstract
Game software is widely used worldwide for both daily entertainment and professional competition with huge value. Continuous game quality assurance is of great importance to satisfy the increasing demands of users. To respond to game issues reported by users timely, game companies often create and maintain a large number of releases, updates and tweaks in a short time. Regression testing is an essential technique adopted to detect regression issues during the evolution of the game software. However, due to the special characteristics of game software, traditional regression testing techniques are not directly applicable.
To bridge this gap, in this paper, we perform an early exploratory study to investigate the challenges in regression testing of games. In particular, we first performed empirical studies to better understand the game development process, bugs introduced during game evolution and context sensitivity. Based on the findings, we proposed the first regression test selection (RTS) technique for game software. In particular, we model the test suites of game software as a Markov Decision Process (STG) and then propose the RTS on the STG. We establish the dependencies between the STG and game files, including game design files, game resources and source code, and perform change impact analysis to identify the states/actions (in the STG) that potentially execute such changes.
We implemented our framework in a tool, named GameRTS, and evaluated its usefulness on 10 tasks of a large-scale commercial game, including a total of 1,429 commits over three versions. The experimental results demonstrate the usefulness and effectiveness of GameRTS in RTS of real-world games. For most tasks, GameRTS only selected one trace from STG, which can significantly reduce the testing time. Furthermore, GameRTS does not miss any regression bugs. Compared with the file-level RTS, GameRTS selected less states/actions/traces (i.e., 10.44%, 23.97%, 6.85%). In addition, GameRTS identified 2 critical bugs in the latest version of the game.
Overview of GameRTS
GameRTS takes several steps to help game developers find the most urgent test tasks under less testing consumption.