DUSTER: A Deduplication Framework for Efficient Point Cloud Storage and Retrieval

Abstract: The rise of embodied AI, exemplified by autonomous driving, has led to a rapid surge in the volume of point cloud (PC) data, highlighting the need for a dedicated storage and management system. In such scenarios, LiDAR sensors continuously generate PC frames at a stable frequency. As consecutive frames often capture overlapping regions, they exhibit spatiotemporal redundancy, resulting in duplicated 3D points in both the relative coordinate system (centered on the sensor) and the absolute coordinate system (aligned with global positioning). To address this issue, we propose DUSTER, a deduplication framework for efficient point cloud storage and retrieval. DUSTER considers the entire PC sequence for joint deduplication and employs a segmentation module to determine which frames should be grouped and processed together. For each identified segment, a deduplication module effectively detects and eliminates redundant points across the grouped PC frames. Finally, an indexing module manages the deduplicated data and reconstructs the original PC frames on demand during retrieval. Experimental results show that DUSTER achieves up to 42% reduction in the number of stored PC points and up to 80% reduction in disk space usage. Moreover, it supports efficient random and sequential PC frame retrieval in terms of I/O cost, while maintaining deep model prediction accuracy on reconstructed PC frames with minimal degradation.