ReconViaGen tackles the challenge of reconstructing detailed 3D objects from multi-view images using a carefully designed two-stage generative diffusion pipeline. What sets it apart is the split between producing a coarse initial 3D shape and then refining it with a structured latent diffusion stage. This layered approach is reminiscent of modern video generation workflows but applied here in three dimensions, balancing reconstruction quality with computational efficiency.
multi-view 3d reconstruction via a two-stage diffusion pipeline
At its core, ReconViaGen is a Python-based project built on PyTorch, designed for multi-view 3D object reconstruction. It was developed by the GAP-LAB at CUHK Shenzhen and accepted at ICLR 2026, highlighting its academic rigor.
The architecture centers around two main stages:
Surface Stage (SS): This initial phase produces a coarse, sparse voxel-based 3D geometry representation from multiple input images. The goal is to quickly generate a plausible base shape capturing the rough surface and structure of the object.
Structured Latent Stage (SLat): Building on the coarse output, this stage refines the voxel geometry using a structured latent diffusion model. It enhances geometric detail and accuracy, focusing on higher fidelity reconstruction.
The model is trained on the ProObjaverse-300K dataset, a large-scale collection of 3D object images, which supports learning a diverse set of object shapes and appearances.
A notable feature in the repo’s v0.5 branch is the integration of TRELLIS.2, a multi-view fusion method. TRELLIS.2 enables the generation of high-resolution meshes with physically based rendering (PBR) materials, pushing the output quality beyond voxel grids to more realistic, textured 3D models.
Under the hood, the repo relies on CUDA 12.1 for GPU acceleration, with PyTorch 2.4.0 as the deep learning framework. The inference pipeline is optimized to run on consumer GPUs — the base reconstruction requires less than 18GB of VRAM for 16 images, while the refinement stage with SLat pushes the requirement to under 24GB. Training, however, is resource-intensive and requires an 8× A100 (80GB) GPU setup.
The codebase is modular, supporting checkpoint swapping and experimentation with different components of the pipeline, which is useful for research and extension.
architectural strengths and tradeoffs
What distinguishes ReconViaGen is its two-stage diffusion approach that mirrors trends in generative modeling, where a problem is decomposed into generating a coarse structure and then refining details. This is beneficial because generating a high-fidelity 3D model directly from images is challenging and computationally expensive.
The Surface Stage’s sparse voxel representation is a practical tradeoff. Sparse voxels reduce memory footprint and speed up initial reconstruction, but their coarse nature limits detail. The Structured Latent stage compensates by refining this geometry with a learned latent diffusion prior, improving accuracy without starting from scratch.
Integrating TRELLIS.2 in the v0.5 branch adds multi-view fusion capabilities. This enables the system to produce meshes with PBR materials, which is crucial for applications needing realistic rendering, like AR/VR or digital asset creation. However, this also increases the complexity and GPU memory requirements.
The repo’s reliance on high VRAM GPUs for training (8× A100 80GB) makes it less accessible for those without access to large-scale compute clusters. Inference is more manageable on consumer hardware, but still requires substantial GPU memory (18-24GB VRAM), which is beyond typical mid-range GPUs.
From a code quality perspective, the repo is Pythonic and built around PyTorch, using CUDA 12.1 for acceleration. The modular checkpoint design improves DX for researchers wanting to test variations or fine-tune components. The inclusion of Gradio demos in the repo is a nice touch, allowing quick interactive experimentation without heavy setup.
The tradeoff is clear: the pipeline balances quality, flexibility, and resource demands in a way that suits research and advanced prototyping rather than lightweight production use.
quick start
Getting started with ReconViaGen involves cloning the repo, setting up a conda environment, and installing dependencies that include PyTorch 2.4.0 with CUDA 12.1 support. The repo provides a setup script to handle this.
# Clone the repo
git clone --recursive https://github.com/GAP-LAB-CUHK-SZ/ReconViaGen.git
cd ReconViaGen
# Create conda environment and install dependencies
. ./setup.sh --new-env --basic --xformers --flash-attn --spconv --mipgaussian --kaolin --nvdiffrast --demo
The setup script automates environment creation and installs essential components such as xformers, flash attention, spconv, mip gaussian filtering, Kaolin (3D deep learning tools), and nvdiffrast (NVIDIA differentiable rasterizer), along with launching a demo interface.
This straightforward installation process allows you to quickly get the inference pipeline running on a supported GPU, with demos to visualize multi-view reconstructions.
verdict
ReconViaGen is a sophisticated research-grade 3D reconstruction pipeline with a thoughtful two-stage generative diffusion architecture. Its modular design and integration of multi-view fusion techniques position it well for exploration by researchers and practitioners interested in multi-view generative models or high-fidelity 3D reconstruction.
The main limitation is its hardware demands: training requires powerful multi-GPU setups, and inference still needs a high-memory GPU, which can be a barrier for some. The codebase is clean and modular but geared toward experimental use rather than turnkey production deployment.
If you work with 3D generative models, diffusion-based reconstruction, or want to experiment with multi-view fusion in a GPU-accelerated Python environment, ReconViaGen is worth your time. It offers a solid foundation to build on or adapt for your own 3D reconstruction needs, especially if you have access to the required hardware resources.
The repo’s combination of a two-stage pipeline, multi-view fusion, and practical VRAM requirements for inference strikes a reasonable balance between research complexity and real-world usability.
→ GitHub Repo: GAP-LAB-CUHK-SZ/ReconViaGen ⭐ 593 · Python