Yi-Chen Chiu and Jui-Chiu Chiang
Fig. 1: Overview of our task-oriented compression framework.
Connected autonomous vehicles increasingly rely on cloud or edge servers to exchange perception information for collaborative scene understanding and decision-making. However, transmitting dense 3D semantic scene representations requires substantial communication bandwidth, posing a major challenge for real-time vehicle-to-everything (V2X) systems. Although image and video coding for machines (ICM/VCM) has been extensively studied for 2D imagery, task-oriented compression for structured 3D semantic scene representations remains largely underexplored.
Our work presents a task-oriented compression framework for 3D semantic Gaussians in semantic occupancy prediction. Unlike conventional 3D Gaussian Splatting (3DGS) compression methods that optimize rendering quality for human perception, the proposed framework focuses on preserving the feature information required for downstream machine perception tasks. LiDAR priors are incorporated to establish a semantic Gaussian representation, after which irregular semantic Gaussians are reorganized into spatially correlated 2D attribute maps using Parallel Linear Assignment Sorting (PLAS) and compressed with HEVC. In addition, an Attribute-Adaptive Quantization (AAQ) strategy allocates bits according to the sensitivity of semantic and geometric attributes, improving coding efficiency while maintaining occupancy prediction accuracy.
Evaluations indicate that our method secures a compression ratio exceeding 10$\times$ without compromising the mean Intersection over Union (mIoU). These results highlight the framework's capability to alleviate bandwidth constraints, proving its potential for real-time collaborative perception deployments in V2X ecosystems.
The task-oriented compression framework designed for 3D semantic Gaussian in semantic occupancy prediction is described in this section. Fig.1 summarizes the pipeline, which is organized into three successive stages. First, given synchronized multi-view images and LiDAR point clouds, a LiDAR-initialized GaussianFormer-2 backbone produces a set of 3D semantic Gaussian carrying geometric and semantic attributes. Second, Parallel Linear Assignment Sorting (PLAS) rearranges these unordered primitives into spatially coherent two-dimensional attribute maps, which are then handed to a High Efficiency Video Coding (HEVC) encoder operating under an Attribute-Adaptive Quantization (AAQ) strategy. Finally, the reconstructed Gaussian is rasterized onto a voxel grid by Gaussian-to-voxel splatting to yield the semantic occupancy prediction. To handle dynamic driving scenes, an inter-frame coding scheme built on a deterministic Morton-code initialization of PLAS is further introduced in. Unlike rate-distortion-optimized 3DGS compression, which optimizes for photo realistic rendering, the proposed framework preserves the feature-level information required by the downstream task rather than pixel-level visual fidelity.
A. Lidar-Guided Semantic Gaussian Generation
We represent the scene as a set of semantic Gaussian. Each primitive is defined by its center position, covariance (parameterized by scaling and rotation), opacity, and a semantic feature vector. To generate these primitives, we extend the GaussianFormer-2 architecture by incorporating LiDAR data to provide robust geometric grounding. While the baseline GaussianFormer-2 initializes queries using image-based depth estimation, it is prone to generating geometric outliers due to inherent monocular depth ambiguity. To mitigate this, we integrate raw LiDAR point clouds, filtered to the perceptible range, and employ Farthest Point Sampling (FPS) to select $N$ representative anchor points, ensuring uniform spatial coverage across the scene. These points are assigned as the initial positions of the queries. These accurate geometric priors accelerate cross-attention convergence and ensure primitives are initialized near physical object surfaces.
B. Gaussian Reorganization via Sorting
To map unordered 3D Gaussians into structured 2D grids while preserving local attribute smoothness, we adopt Parallel Linear Assignment Sorting (PLAS). This process begins by constructing a sorting feature for each Gaussian, which serves as the high-dimensional descriptor used to determine its optimal position on the 2D grid. While the PLAS designed for visual reconstruction typically utilizes 3D position and RGB color, our semantic occupancy task requires a focus on categorical consistency. We therefore define by concatenating the normalized 3D position and the full semantic feature vector. This transformation converts unstructured 3D data into a spatially coherent 2D representation, significantly enhancing the coding efficiency of standard video codecs.
C. Quantization Sensitivity of Semantic Gaussian Attributes
Uniform quantization treats all attributes equally, failing to account for their varied impact on task performance. To optimize our bit-allocation strategy for semantic Gaussian compression, we conduct a sensitivity analysis to evaluate how different attributes affect occupancy prediction. We categorize these into geometry attributes, comprising scale, rotation, and opacity; and semantic attributes, consisting of the feature vector. Throughout these experiments, we maintain lossless for the center position to ensure good geometry precision.
D. Gaussian-to-Voxel Splatting
To generate dense occupancy predictions, we strictly follow the probabilistic superposition model from GaussianFormer-2. The reconstructed Gaussians are rasterized onto voxel grid by accumulating opacity-weighted semantic features from intersecting primitives. This discretization allows direct comparison with ground-truth labels for mIoU evaluation.
A. Dataset
All experiments are carried out on nuScenes. The benchmark records urban driving with a complete surround sensor rig---six cameras spanning the full 360 degree horizon together with a $32$-beam LiDAR---and annotates keyframes at 2Hz. Such a configuration is a prerequisite for our setting: the vision-only backbone consumes calibrated multi-view imagery, and the LiDAR-guided initialization additionally requires the point cloud registered into the same coordinate frame.
Because nuScenes itself supplies only sparse LiDAR returns and 3D bounding boxes, we adopt the protocol of GaussianFormer-2 and train and evaluate against the densified occupancy annotations released by SurroundOcc. Those annotations accumulate the sparse per-frame labels into a 200x200x16 voxel volume that covers [-50,50][-50,50][-5,3] m with 0.5m cells. Of the annotated scenes, 700 are reserved for training and 150 each for validation and testing, over a 17-class semantic vocabulary. The class distribution is heavily long-tailed, as is typical of driving data: drivable surface and manmade structures account for the vast majority of voxels, while pedestrians, bicycles and traffic cones---the categories that matter most for safety---occupy a vanishing fraction. It is precisely this imbalance that makes the mean-over-classes metric defined below a stricter measure of perception quality than any voxel-averaged score.
B. Rate-Distortion Performance
Fig. 2 illustrates the rate-distortion (R-D) curves. The three dashed horizontal lines indicate the performance upper bounds of the uncompressed baselines for varying Gaussian counts. The proposed method significantly outperforms the uniform compression baselines in all densities. Notably, in high-fidelity scenarios, our method achieves a compression ratio exceeding 10 times while maintaining a mIoU comparable to the uncompressed upper bounds.
To further quantify the coding gain, we compute the Bjøntegaard Delta Rate (BD-Rate). As shown in Table 1, our adaptive approach achieves a 91.95% bitrate reduction compared to the uncompressed baseline, and a 17.4% gain over the uniform-QP baseline. Furthermore, the bitstream breakdown in Fig. 3 validates these findings, confirming that our adaptive QP strategy effectively minimizes the bit-rate of semantic Gaussian features while maintaining task performance.
Fig. 2: Rate-distortion performance under varying primitive numbers.
Table. 1: BD-Rate (%) comparison.
Fig. 3: Bitstream distribution comparison.
C. Visualization
Fig. 4 presents a visual comparison between the GaussianFormer-2 and our result. As observed, the vision-only baseline suffers from severe depth ambiguity, resulting in fragmented geometry and missing structures, particularly evident in the incomplete road surfaces. In contrast, our method leverages LiDAR priors to resolve these spatial inconsistencies. Our approach reconstructs a dense, structurally coherent scene that effectively fills the geometric gaps of the baseline, achieving higher fidelity to the ground truth.
Fig. 4: Visual comparison of semantic occupancy prediction on the nuScenes validation set.