Approach

2. Individual Module Oracle

2.1 Perception Module: We adopt a weighted Intersection over Union (IoU) to measure the errors in the perception module, which is a widely recognized metric in object detection

2.2 Prediction Module: To avoid the effect of upstream perception module, we adopt a perception-biased trajectory to measure the errors in the Prediction module

Consequently, we can measure the error of the Prediction module by comparing the predicted trajectories with the perception-biased trajectories

2.3 Planning Module: We measure errors in the Planning module by assessing if the planned trajectories collide with objects detected by the upstream modules.

2.4 Control Module: We use naive control results diff to measure the performance of the control module