RQ2: Can interactive pattern sequences help
to improve transformation precision?
Embedded Files
Experiment Design
To address RQ2, we conduct an ablation study to assess the usefulness of the extracted functional scenarios, especially the interactive pattern sequences, for the critical scenario generation process. For this purpose, we implement a variant of LeGEND, called LeGEND−. In LeGEND−, only one LLM is used to generate logical scenarios directly from accident reports, without extracting the functional scenarios. We run LeGEND− and LeGEND based on the same set of accident reports, and use the same number of simulations as the budget for searching concrete scenarios. We then compare the performances of them using the metrics in RQ1.
Comparison Results
During the execution, LeGEND− identifies 28 critical scenarios revealing potential system faults, with the first collision occurring in the 81-𝑠𝑡 simulation. These critical scenarios are categorized into 2 distinct types, achieving a type exposure rate of 7.1% (min 5.5% and max 8.3%). Both types were discovered within the first 982 simulations. In comparison, LeGEND is able to generate more critical scenarios and expose more diverse types of scenarios. Notably, all types of critical scenarios discovered by LeGEND− are also detected by LeGEND in the 118-𝑡ℎ and 938-𝑡ℎ simulations. This demonstrates that LeGEND not only identifies more critical scenarios but also uncovers a broader spectrum of distinct types, compared to LeGEND−.
During the execution, LeGEND− identifies 28 critical scenarios revealing potential system faults, with the first collision occurring in the 81-𝑠𝑡 simulation. These critical scenarios are categorized into 2 distinct types, achieving a type exposure rate of 7.1% (min 5.5% and max 8.3%). Both types were discovered within the first 982 simulations. In comparison, LeGEND is able to generate more critical scenarios and expose more diverse types of scenarios. Notably, all types of critical scenarios discovered by LeGEND− are also detected by LeGEND in the 118-𝑡ℎ and 938-𝑡ℎ simulations. This demonstrates that LeGEND not only identifies more critical scenarios but also uncovers a broader spectrum of distinct types, compared to LeGEND−.
To investigate why LeGEND− exposes fewer types of critical scenarios than LeGEND, we further conduct a detailed comparison of the logical scenarios generated by the two approaches. As a result, we conclude the following root causes:
• The logical scenario has an incorrect trigger sequence for the maneuvers of NPC vehicles, represented by C1;
• The logical scenario lacks necessary maneuvers of NPC vehicles that facilitate a violation, represented by C2;
• The logical scenario includes redundant maneuvers that could affect the substitution of the ego vehicle,
represented by C3;
• The logical scenario contains parameter ranges that could not accurately reflect the interactions
indicated from the original report, represented by C4.
We find that C4 is the main cause for 44.4% (4 of 9) of the cases. Moreover, C1, C2 and C4 have a direct relation to the interactive actions between vehicles described in the original accident reports, accounting for 77.8% (7 of 9) of the causes. These findings emphasize the necessity of a semi-formal functional scenario, particularly the extraction of interactive pattern sequences, in improving the performance of LeGEND.
Google Sites
Report abuse