Regression Test Properties

Context Relevence Experiment

The (in)sensitivity of context determines the granularity of regression test selection analysis. If the context is sensitive, we need a fine-grained dependency analysis and test selection such that test cases with different contexts can be selected. If insensitive, then we can consider only selecting minimal test cases that cover all changes for efficiency. Figure 2 shows the idea of our evaluation. Given two test cases (𝑇 1 and 𝑇 2) that can reach the same state 𝑠2 from different playing histories (i.e., different context), we evaluate the impact of different contexts at state 𝑎0 on future states by cross execution. Consider 𝑇 1 that covers the state sequence (. . . , 𝑠0, 𝑠2, 𝑠3, . . . , 𝑠𝑚), we try to execute a sequence of actions (𝑇 1′) that include the actions of 𝑇 1 before the common state 𝑠2 (context from 𝑇 1) and the following actions of 𝑇 2 after 𝑠2. If 𝑇 1′ can be replayed successfully, then we say the context of 𝑇 1 has little effect on the future execution (i.e., insensitive). Similarly, we try to use the context of 𝑇 2 and replay the future actions of 𝑇 1. We measure the context sensitivity based on the success rate of the cross execution. The higher the success rate, the less sensitive the context. 

We collect 10 real-world tasks from a large-scale study video game which will also be used in our evaluation. For each task, we first randomly generate 50 test cases. Then we identify all intersection states among the 50 test cases. Each intersection state has 𝑀 predecessors and 𝑁 successors where 𝑀 > 1, denoted as (𝑀, 𝑁) intersection. Finally, we identify an average of 119.8 (𝑀, 𝑁)intersections for each task, including 56.9 (𝑀, 1) intersections. 

Statistic of Different Crossing Nodes and Testing Traces 

Crossing Nodes Distribution among Different Tasks

Crossing Nodes Selection

We selected 50 pairs of traces for the cross execution, where each pair has at least one intersection (𝑁 > 1). Note that the selection strategy considers the distance between the initial state and the intersection, which can affect the length (difficulty) of the cross execution. For example, the length of (𝑠2, . . . , 𝑠𝑛) could affect the difficulty of replaying 𝑇 1′. We try to balance the distance of the selected intersection. Then we perform the cross execution for each pair of test cases. Our results show that the success rate of cross execution is 100%, which indicates the context insensitivity of game execution.