RQ4: falsification

The experimental results of falsification are presented in Table 9. We run 30 falsification trials for each falsification method and report the number of successfully falsified trials, i.e., FSR, as the indicator of the effectiveness of falsification. The method with the highest FSR is highlighted. For methods with the same FSRs, the better one is determined by the falsification time, with shorter times indicating a higher efficiency. From the results, we notice that: 

• In general, Mosaic outperforms the other four baseline falsification techniques (with higher FSRs) in 22 out of 23 cases. Specifically, Mosaic surpasses Breach with CMAES, GA, and SA, and S-TaLiRo with SA and SOAR, with an average FSR of 23.88, 6.96, 6.39, 12.30, 8.57, and 8.35, respectively. This demonstrates the effectiveness of Mosaic in finding more falsifying inputs in AI-CPSs. 

• Additionally, the choice of optimization algorithm in Mosaic makes a difference. Mosaic-SA achieves the best falsification performance among all falsification algorithms in 8 cases, while Mosaic-CMAES exhibits the highest FSRs in seven benchmarks. This suggests that developers should experiment with various optimization algorithms to obtain the most efficient falsification outcomes. 

• We also notice that classical falsification methods struggle to falsify CPSs with AI components. For instance, both Breach and S-TaLiRo are hard to find falsifying signals for AFC-DNN2 with φ2, WTK-DDPG with φ1, SC-DDPG with φ1  and SC-PPO with φ1  . 

• Random fails to outperform any other methods in the falsification trials. This indicates that Random, which relies solely on random exploration in the input space, is the least effective falsification approach. 

We also compare the falsification time cost of Mosaic, Breach, and S-TaLiRo, over seven benchmark systems, as shown in Figure 6. We notice that Mosaic achieves a lower median falsification time than Breach and S-TaLiRo across all systems. For example, in ACC, the median falsification time of Mosaic is 88.5, which is lower than the Breach (103.85) and S-TaLiRo (127.3). Moreover, Breach is with a relatively larger falsification time in most systems, e.g., WTK and SC. This is mainly due to the fact that Breach cannot find any falsification trial within the specified time limit (600 seconds).