RQ3: How accurate are LLM1 and LLM2 for the transformation tasks in LeGEND?
Embedded Files
Experiment Design
To address RQ3, we investigate the accuracy of LLM1 for the extraction in Phase 1 and LLM2 for the conversion in Phase 2. We answer RQ3 empirically by a user study, since it is difficult to assess the accuracy of the transformations. Specifically, we perform an online survey with 10 graduate students recruited from our department of computer science. Among the participants, six people have more than two years’ research experience in the field of autonomous driving.
In the study, since it takes one user about 5 minutes to finish the questions regarding one accident report, we randomly selected 5 accident reports from our database as test seeds. We then ask each participant to read each accident report and express their opinions on the statements regarding the extracted functional scenarios and converted logical scenarios. Participants are asked to rate their agreement on a 5-point Likert scale, ranging from “Strongly Disagree” to “Strongly Agree”. In addition, we give participants an opportunity to provide supplementary feedback, such as brief textual comments, particularly for the instances on which their opinions are not strong.
Statistical Results
Table 4 lists the statements and the selected accident reports utilized in the study. We construct five statements specifically for evaluation purposes. The statements related to the intermediate representation cover aspects such as road structures (Statement #1), initial vehicle actions (Statement #2), and the interactive patterns (Statement #3). The statements related to logical scenarios focus on the correctness of the logical scenario template (Statement #4) and the parameter ranges (Statement #5). For the evaluation, we select 5 accident reports, with the main distinguishing features summarized in Table 4.
Figure 6: Results over different cases
The questionnaire used in our user study is available at Google Form. Fig. 6 presents the results of participants’ responses across the five cases. The data indicates that participants generally agree that LeGEND accurately extracts the road structures and initial vehicle actions. Regarding the extracted interactive pattern sequences across the five cases, participants still agree but less strongly for 𝐶𝑎𝑠𝑒2 and 𝐶𝑎𝑠𝑒4. By the analysis of their feedback, we find that this discrepancy arises because the large language model (LLM) can generate new actions for vehicles when specific actions are not explicitly described in the accident report. This result aligns with the design choice of LeGEND, as LLMs are generative models characterized by their high creativity. Moreover, while the template conversion process is deemed accurate by participants, the parameter filling process is found to be less precise. This is attributed to the current limitations in the reasoning capabilities of LLMs.
The questionnaire used in our user study is available at Google Form. Fig. 6 presents the results of participants’ responses across the five cases. The data indicates that participants generally agree that LeGEND accurately extracts the road structures and initial vehicle actions. Regarding the extracted interactive pattern sequences across the five cases, participants still agree but less strongly for 𝐶𝑎𝑠𝑒2 and 𝐶𝑎𝑠𝑒4. By the analysis of their feedback, we find that this discrepancy arises because the large language model (LLM) can generate new actions for vehicles when specific actions are not explicitly described in the accident report. This result aligns with the design choice of LeGEND, as LLMs are generative models characterized by their high creativity. Moreover, while the template conversion process is deemed accurate by participants, the parameter filling process is found to be less precise. This is attributed to the current limitations in the reasoning capabilities of LLMs.
Google Sites
Report abuse