GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention

Under Review

Lingjun Zhao1✉, Sizhe Wei1, James Hays1, Lu Gan1
✉Corresponding Author 1Georgia Institute of Technology
Paper (Coming Soon) arXiv (Coming Soon) Code (Coming Soon)
GaussianFormer3D

We propose a LiDAR-camera fusion-based semantic occupancy prediction framework named GaussianFormer3D. We use 3D Gaussians instead of dense grids to reduce memory consumption and enhance algorithm efficiency. GaussianFormer3D achieves comparable performance to state-of-the-art multi-modal occupancy methods with reduced memory usage.

Abstract

3D semantic occupancy prediction is critical for achieving safe and reliable autonomous driving. Compared to camera-only perception systems, multi-modal pipelines, especially LiDAR-camera fusion methods, can produce more accurate and detailed predictions. Although most existing works utilize a dense grid-based representation, in which the entire 3D space is uniformly divided into discrete voxels, the emergence of 3D Gaussians provides a compact and continuous object-centric representation. In this work, we propose a multi-modal Gaussian-based semantic occupancy prediction framework utilizing 3D deformable attention, named as GaussianFormer3D. We introduce a voxel-to-Gaussian initialization strategy to provide 3D Gaussians with geometry priors from LiDAR data, and design a LiDAR-guided 3D deformable attention mechanism for refining 3D Gaussians with LiDAR-camera fusion features in a lifted 3D space. We conducted extensive experiments on both on-road and off-road datasets, demonstrating that our GaussianFormer3D achieves high prediction accuracy that is comparable to state-of-the-art multi-modal fusion-based methods with reduced memory consumption and improved efficiency. Code will be released upon publication.

Method

We propose a novel multi-modal Gaussian-based semantic occupancy prediction framework. By integrating LiDAR and camera data, our method significantly outperforms camera-only baselines with similar memory usage.

We design a voxel-to-Gaussian initialization module to provide 3D Gaussians with geometry priors from LiDAR data. We also develop an enhanced 3D deformable attention mechanism to update Gaussians by aggregating LiDAR-camera fusion features in a lifted 3D space.

We present extensive evaluations on two on-road datasets, nuScenes-SurroundOcc and nuScenes-Occ3D, and one off-road dataset, RELLIS3D-WildOcc. Results show that our method performs on par with state-of-the-art dense grid-based methods while having reduced memory consumption and improved efficiency.

GaussianFormer3D Method

Quantitative Results

3D semantic occupancy prediction performance on the on-road nuScenes-SurroundOcc validation set.

nuScenes-SurroundOcc

3D semantic occupancy prediction performance on the on-road nuScenes-Occ3D validation set.

nuScenes-Occ3D

3D semantic occupancy prediction performance on the off-road RELLIS3D-WildOcc validation and test sets.

RELLIS3D-WildOcc

Efficiency evaluation and comparison on the nuScenes-SurroundOcc validation set during inference.

efficiency

Qualitative Results

Visualization on the on-road nuScenes-Occ3D validation set.

nuScenes-Occ3D-vis

Visualization on the off-road RELLIS3D-WildOcc dataset.

RELLIS3D-WildOcc-vis

BibTeX

@article{zhao2024gaussianformer3d,
  author    = {Zhao, Lingjun and Wei, Sizhe and Hays, James and Gan Lu},
  title     = {GaussianFormer3D: Multi-Modal Gaussian-based Semantic Occupancy Prediction with 3D Deformable Attention},
  year      = {2025},
  note      = {Preprint available soon on arXiv.}
}