Temporally Compressed 3D Gaussian Splatting for Dynamic Scenes

Saqib Javed†, Ahmad Jarrar Khan†, Corentin Dumery, Chen Zhao, Mathieu Salzmann
†Equal Contribution
EPFL EPFL EPFL
Paper arXiv
,

Abstract

Recent advancements in high-fidelity dynamic scene reconstruction have leveraged dynamic 3D Gaussians and 4D Gaussian Splatting for realistic scene representation. However, to make these methods viable for real-time applications such as AR/VR, gaming, and rendering on low-power devices, substantial reductions in memory usage and improvements in rendering efficiency are required.

While many state-of-the-art methods prioritize lightweight implementations, they struggle in handling scenes with complex motions or long sequences. In this work, we introduce Temporally Compressed 3D Gaussian Splatting (TC3DGS), a novel technique designed specifically to effectively compress dynamic 3D Gaussian representations.

TC3DGS selectively prunes Gaussians based on their temporal relevance and employs gradient-aware mixed-precision quantization to dynamically compress Gaussian parameters. It additionally relies on a variation of the Ramer-Douglas-Peucker algorithm in a post-processing step to further reduce storage by interpolating Gaussian trajectories across frames.

Our experiments across multiple datasets demonstrate that TC3DGS achieves up to 67x compression with minimal or no degradation in visual quality.

Method

Method
Our approach involves a temporally consistent masking strategy to select relevant 3D Gaussians across frames. The masked Gaussians are then pruned and quantized using a gradient-based, parameter-aware bit-precision quantization scheme, which optimizes memory usage while preserving scene fidelity.

Keypoint Interpolation

We propose a novel keypoint interpolation method that leverages a variant of the Ramer-Douglas-Peucker algorithm to interpolate Gaussian trajectories across frames. This allows us to further reduce storage requirements while maintaining the visual quality of the scene. Our method can achieve much interpolation error than uniform sampling while using similar or lower number of keypoints.

Max Keypoints     3

Visual Comparisons

Panoptic Dataset

Ours vs Dynamic3DGaussians

Ours vs 4DGaussians

Ours vs STG

Ours vs Ground Truth

Technicolor Dataset

Ours

Ground Truth

Ours

Ground Truth

Neural 3D Video Dataset

Ours

Ground Truth

Ours

Ground Truth

BibTeX

@misc{javed2024temporallycompressed3dgaussian,
        title={Temporally Compressed 3D Gaussian Splatting for Dynamic Scenes}, 
        author={Saqib Javed and Ahmad Jarrar Khan and Corentin Dumery and Chen Zhao and Mathieu Salzmann},
        year={2024},
        eprint={2412.05700},
        archivePrefix={arXiv},
        primaryClass={cs.CV},
        url={https://arxiv.org/abs/2412.05700}, 
  }