This page provides a brief overview of the basic process required to reproduce the results of our experiments as described in our research paper. In the meantime, we have made our replication package available to the public via a GitHub repository with the aim of establishing a basis for researchers to validate and further explore.
This section covers the V2XGen's installation steps, a quick start demo, experiment reproduction steps, and the project structure.
Code Repository： https://github.com/Cooperative-Perception/V2XGen
All experiments are conducted on a server with an Intel i7-10700K CPU(3.80 GHz), 32 GB RAM, and an NVIDIA GeForce RTX 4070 GPU (12GB VRAM).
1. Download dataset
You need to check the V2V4real website and download the test datasets test1, test2, test3.
2. Format conversion
In order to be compatible with the semantic segmentation model, you need to convert the .pcd data format to .bin, only (x,y,z) needs to be converted here.
You can refer to the rq_tools.py in the path for format conversion.
3. Semantic segmentation
We used the SalsaNext semantic segmentation model to process the point cloud data.
Dataset folder structure
The final dataset folder structure should as follows:
.
├── 0 ego vehilce dataset with a mix of all scenarios
│ ├── labels vehilce labels
│ ├── predictions semantic segmentation result for each data
│ ├── road_pcd road split result, the initial is empty
│ ├── pcd .pcd data
│ └── velodyne .bin data
└── 1 cooperative vehilce dataset with a mix of all scenarios
├── labels
├── predictions
├── road_pcd
├── pcd
└── velodyne
Create conda environment, python >= 3.7
conda create -n v2xgen python=3.7
conda activate v2xgen
Pytorch installation, pytorch >= 1.12.0
conda install pytorch==1.12.0 torchvision=0.13.0 cudatoolkit=11.3 -c pytorch -c conda-forge
spconv 2.x installation
pip install spconv-cu113
Run the following command to install the dependencies.
pip install -r requirements.txt
You need to download the cooperative 3D detection models here, and unzip them in the model folder.
model
├── attfuse
├── early_fusion
├── late_fusion
├── PointPillar_Fcooper
├── PointPillar_V2VNet
└── PointPillar_V2XViT
You can see how each V2XGen transformation is used in the examples directory.
python examples/v2xgen_demo.py -s ${scene_id} -t ${transform}
scene_id in 1-9
choose transformation of insert, delete, translation, scaling and rotation.
Our five transform operations are integrated from insert and delete operations, you can refer to obj_transformation_demo.py to customize the transformation functions.
1. Visualize the result of transformation
python rq1/rq1_vis.py -s ${scene_id} -t ${transform}
scene_id in 1-9
choose transformation of insert, delete, translation, scaling and rotation.
2. Generate data for frd eval
python rq1/rq1_frd.py -m ${method}
method: number of data transformations, such as 1, 2, 3...'
The data generated in each round is divided into three parts according to different scenes from 1 to 9, including initial data, V2XGen transform data and Baseline transform data. After generate data, you need to use the project lidargen to verify the authenticity of the generated data by evaluating the results of the indicator FRD.
# in project lidargen, you need to modify the data read path in path
# "lidargen/rangenetpp/lidar_bonnetal_master/train/tasks/semantic/modules/kittiparser.py"
python lidargen.py --fid --exp kitti_pretrained --config kitti.yml
1. Split dataset
Half of the sequences are randomly selected and saved as a training set for retrain and a test set for testing.
python rq2/rq2_dataset_split.py -d ${dataset_path}
2. Generate data for select
RQ2 requires three transformations of the train dataset, the resulting dataset v2x_gen used for data selection and models retrain.
python rq2/rq2_gen.py -m ${times} # times in [1, 2, 3]
3. Visulize
If you want to visualize the transformation result, you need to comment out the visual annotations of the core/obj_insert.py and core/delete.py.
python rq2_rq2_vis.py -s ${scene_id}
4. Evaluate train dataset and select data based on the method
The experiment consisted of six models, and needed to choose different fusion methods late/early/intermediate according to the models.
python opencood/rq_eval/rq2_inference.py --dataset_dir ${dataset}/rq2/rq2_gen --model_dir model/early_fusion --fusion_method early
1. Retrain for performance inprovement
We selected 10% and 15% data from the v2x_gen dataset to retrain different models.
python opencood/rq_eval/rq3_train.py --dataset_dir ${dataset}/rq2/rq2_select --model_dir model/early_fusion --method v2x_gen --scale 0.15
method: we choose v2x_gen data select method
scale: the scaling of choosed data in v2x_gen dataset
2. Eval the result of retrain models
We evaluate the results of the retraining models based on three defined metrics AP_50, occlusion error and long-distance error.
python opencood/rq2/rq3_inference.py --scale 0.1 --method ori --dataset_dir ${dataset}/rq2/rq2_select/v2x_gen/0.15/early_fusion --model_dir model/early_fusion --fusion_method early
${dataset}/rq2/rq2_select/v2x_gen/0.15/early_fusion: you can choose 0.1 scale and other retrain models
.
├── _assets object dataset
├── build
├── config common, dataset and lidar configuation
├── core V2XGen transformation core code
├── model RQ evaluation and retrain models
├── opencood RQ evaluation and retrain code
├── utils common utils
├── examples examples of each transformation
├── rq1 rq1 data generation and visulization
├── rq2 rq2 data generation and visulization
├── rq_tools.py init dataset
├── logger.py log
├── requirements.txt
├── copy_pcd_files.py
├── setup.py
└── visual.py
├── README.md