SegmentAnything3D (SAM3D) tackles a thorny problem in 3D vision: how to segment 3D point clouds without training a dedicated 3D model. Instead of collecting expensive 3D annotations or training from scratch, SAM3D transfers Meta’s Segment Anything Model (SAM)—a strong 2D image segmentation foundation model—into the 3D domain by projecting its 2D masks into 3D space using depth data. This zero-shot approach sidesteps 3D training complexities and leverages advances in 2D foundation models for open-world 3D perception.
bridging 2d foundation models to 3d point cloud segmentation
SAM3D is a Python-based research project from Shanghai AI Lab and HKU that adapts SAM for 3D point cloud segmentation without any 3D-specific training. It is designed to work with RGB-D data, specifically the ScanNet v2 dataset, which provides RGB frames with aligned depth maps and camera poses.
The core pipeline runs SAM on each 2D RGB frame to generate segmentation masks. These 2D masks are then projected into 3D point clouds by combining pixel-wise mask labels with depth information, lifting the segmentation from image pixels to actual 3D points in space.
Once projected, the resulting segmented point clouds from multiple views undergo a merging process. This consists of a bidirectional-group-overlap algorithm that identifies overlapping regions between adjacent point clouds, followed by a graph-based region merging step that refines boundaries and consolidates segments across views.
Under the hood, SAM3D builds on the Pointcept perception codebase. It requires compiling custom CUDA kernels optimized for point cloud operations, enabling efficient handling of large-scale 3D data sets. The project is research-focused, presented as an ICCV 2023 workshop paper, demonstrating the feasibility of this 2D-to-3D transfer approach for open-world 3D segmentation.
technical strengths and tradeoffs in zero-shot 3d segmentation
SAM3D stands out because it accomplishes zero-shot 3D segmentation without any 3D model training. This is a big deal given the high annotation cost and complexity of 3D datasets. Instead, it fully relies on the mature 2D SAM model and complements it with a depth-based projection and merging pipeline.
The bidirectional-group-overlap merging algorithm is a key technical highlight. It detects overlapping regions between point clouds from different views by analyzing the spatial and mask label overlaps, then merges these regions to create coherent 3D segments. This graph-based merging helps reduce fragmentation and enforces spatial consistency.
However, the approach has limitations. The quality of 3D segmentation depends heavily on the quality of 2D masks and depth accuracy. Noise in depth maps or errors in 2D segmentation propagate into the 3D projection, potentially causing artifacts or over-segmentation.
Another tradeoff is that SAM3D currently supports ScanNet v2 data format out of the box, so adapting it to other RGB-D datasets requires data preprocessing and calibration. The custom CUDA kernels improve performance but add a compilation step that could be a barrier for some users.
The codebase is surprisingly clean and modular given its research focus, with clear separation between mask projection, merging algorithms, and CUDA-accelerated point operations. The reliance on the Pointcept codebase means users familiar with that ecosystem will find it easier to navigate.
quick start with sam3d.py and scanNet data
Installation requires Python 3.8 and Conda for environment management. The CUDA kernels must be compiled with an appropriate TORCH_CUDA_ARCH_LIST for your GPU architecture, as SAM3D relies on CUDA-accelerated point cloud operations.
conda create -n sam3d python=3.8 -y
conda activate sam3d
# docker & multi GPU arch
TORCH_CUDA_ARCH_LIST="ARCH LIST" python setup.py install
After installation, the main entry point is sam3d.py. This script expects ScanNet v2 formatted RGB-D sequences, where each frame has an RGB image, a depth map, and camera pose information.
Typical usage involves pointing sam3d.py at a directory of ScanNet scenes. The pipeline will run SAM on each RGB frame, project the masks to 3D using depth, then merge point clouds across frames to produce segmented 3D point clouds.
Arguments include specifying input and output paths, and optionally tuning parameters for the merging thresholds. The documentation and comments in sam3d.py provide guidance on these parameters. Users should ensure their data is aligned and calibrated to ScanNet standards for best results.
practical verdict: who should explore sam3d
SAM3D is relevant for researchers and engineers working on 3D perception who want to experiment with zero-shot segmentation without investing in costly 3D annotations or training. It demonstrates a valuable design pattern: reusing powerful 2D foundation models for 3D tasks via geometric projections and smart merging.
That said, SAM3D is currently a research prototype, and its reliance on ScanNet data limits immediate out-of-the-box applicability to other datasets or real-world setups. The CUDA kernel compilation step and dependency on Pointcept codebase might raise the entry barrier for some.
For production-grade 3D segmentation or real-time applications, dedicated 3D-trained models may still outperform SAM3D. But if you want to explore zero-shot 3D segmentation leveraging the rapid progress in 2D foundation models, SAM3D is a concrete and technically interesting starting point.
The codebase provides a clean, modular pipeline that could be adapted or extended for other RGB-D datasets or improved depth fusion strategies. It’s worth understanding even if you don’t adopt it directly.
In sum, SAM3D is a solid example of cross-modality transfer in perception research, using geometry and graph-based methods to bridge 2D and 3D segmentation with minimal training overhead.
→ GitHub Repo: Pointcept/SegmentAnything3D ⭐ 1,364 · Python