NavDP flips the script on robot navigation by applying diffusion policies—usually reserved for manipulation tasks—to real-time mapless navigation. Its standout feature is how it uses privileged information during training to close the sim-to-real gap without ever touching real-world robot data. This repo from Shanghai AI Lab packs not just the core NavDP model but also a robust IsaacSim-based benchmark suite that tests navigation in cluttered, realistic indoor scenes.
what NavDP does: mapless navigation powered by diffusion policies with privileged guidance
NavDP is an end-to-end system designed for robot navigation without relying on traditional mapping or localization. Instead of building explicit maps, it learns policies that navigate directly from raw visual inputs. The core innovation is leveraging a diffusion policy architecture, which has been more common in robotic manipulation, for navigation tasks. This approach provides a probabilistic framework that can model multi-modal actions and complex decision spaces.
The model incorporates privileged information during training—additional data available in simulation but not at deployment time—to guide learning and improve sim-to-real transfer. This is crucial because it allows the system to generalize without requiring real-world robot data, a significant hurdle in robot navigation.
The repo also includes the InternVLA-N1 System-1 Benchmark, built on NVIDIA’s IsaacSim 4.2.0 and IsaacLab 1.2.0 platforms. This benchmark suite encompasses a variety of navigation tasks:
- No-goal exploration
- Point-goal navigation
- Image-goal navigation
These tasks are tested across diverse indoor environments including cluttered home and commercial scenes, providing a realistic challenge for navigation algorithms.
The benchmark supports 8 baseline methods such as DD-PPO, iPlanner, ViPlanner, GNM, ViNT, NoMad, NavDP itself, and LoGoPlanner. It features a decoupled HTTP API evaluation framework, allowing flexible integration and method comparison.
A key highlight is the system’s cross-embodiment generalization, meaning the learned policies can transfer across different robot platforms simulated without real-world data. This is enabled by a highly efficient simulation data generation pipeline.
The project also offers an open-sourced deployment process based on LeKiwi hardware, demonstrating how to bring the simulation-trained policies into a real robot setup.
technical strengths and design tradeoffs in NavDP
What sets NavDP apart is its use of diffusion policy for navigation. Diffusion models provide a way to represent complex, multi-modal distributions over actions, which suits the uncertainty and variability in navigation decisions. Applying this concept, previously more common in manipulation, to navigation is a notable architectural choice.
The integration of privileged information during training acts as a form of supervised guidance. This helps to overcome the typical sim-to-real gap that plagues robotic systems trained solely in simulation. Without any real-world robot data, NavDP still manages to generalize well, which is impressive.
The codebase is organized to support both the NavDP model and the extensive benchmark infrastructure. The benchmark is tightly coupled with IsaacSim and IsaacLab, leveraging these NVIDIA platforms’ advanced physics and rendering capabilities to create realistic simulation environments.
However, this dependency on IsaacSim/IsaacLab also introduces limitations:
- Requires complex setup of specific simulation environments.
- Not trivial to run on standard hardware without GPU support compatible with IsaacSim.
- The system’s deployment is currently tied to LeKiwi hardware, which may not be broadly accessible.
The benchmark’s HTTP API decoupling provides good modularity and extensibility, facilitating integration of other navigation methods for evaluation. The inclusion of multiple baselines allows for thorough comparison.
Overall, the code quality is solid with clear separation between model, benchmark, and deployment components. The simulation data pipeline is optimized for efficiency, enabling extensive data generation without real-world collection.
quick start: setting up NavDP and its benchmark
If you want to try NavDP yourself, the repo provides explicit installation steps for the model and the benchmark environment. Here they are verbatim:
# Clone the repository and navigate to the NavDP baseline folder
git clone https://github.com/InternRobotics/NavDP
cd NavDP/baselines/navdp/
# Create and activate a conda environment with Python 3.10
conda create -n navdp python=3.10
conda activate navdp
# Install Python dependencies
pip install -r requirements.txt
# Install IsaacSim 4.2.0 and related extensions
pip install --upgrade pip
pip install isaacsim==4.2.0.2 isaacsim-extscache-physics==4.2.0.2 isaacsim-extscache-kit==4.2.0.2 isaacsim-extscache-kit-sdk==4.2.0.2 --extra-index-url https://pypi.nvidia.com
# Check the IsaacSim installation
isaacsim omni.isaac.sim.python.kit
# Install IsaacLab 1.2.0
git clone https://github.com/isaac-sim/IsaacLab.git
cd IsaacLab/
git checkout tags/v1.2.0
# Check the IsaacLab installation
./isaaclab.sh -p source/standalone/tutorials/00_sim/create_empty.py
Additional baseline methods are supported with their checkpoints and HTTP API integration, but may need extra dependencies as per their individual instructions.
verdict: who should look at NavDP and its benchmark
NavDP is a solid resource if you are working on robot navigation and interested in exploring diffusion policy methods beyond manipulation tasks. Its sim-to-real approach without real-world data is particularly relevant for research labs and advanced robotics teams aiming to reduce expensive physical data collection.
The benchmark infrastructure is comprehensive and realistic, making it a good choice for anyone wanting to evaluate navigation policies in cluttered indoor environments with rich simulation. However, the reliance on specific NVIDIA simulation platforms and specialized hardware means it’s less accessible for quick experiments or setups without powerful GPUs.
The code is well organized, but setting up the full environment requires patience and familiarity with IsaacSim and IsaacLab. If you’re looking for a plug-and-play navigation solution, this might not be it, but for building on cutting-edge sim-to-real navigation research with transparent baselines, NavDP offers a strong foundation.
In short, NavDP is worth diving into if you have the simulation infrastructure and want to experiment with diffusion policies applied to navigation, especially with a focus on bridging the sim-to-real gap without real robot data.
Related Articles
- A hands-on guide to classical autonomous vehicle control algorithms in Python — Explore a Python repo implementing classical autonomous vehicle algorithms as transparent simulations. Covers localizati
- Inside Genie Sim 3.0: LLM-driven embodied AI simulation with high-fidelity 3D scenes — Genie Sim 3.0 is an open-source platform combining 3D Gaussian Splatting and LLM-driven scene generation for embodied AI
- Inside the ZED SDK: GPU-accelerated spatial perception for stereo cameras — Explore the ZED SDK, a C++ library for real-time stereo vision, SLAM, and spatial mapping with GPU acceleration and zero
- NVIDIA NeMo Agent Toolkit: Enhancing multi-agent workflows with performance primitives and observability — NVIDIA NeMo Agent Toolkit adds performance primitives, profiling, and runtime intelligence to multi-agent workflows alon
- Deploying RL-trained motion tracking policies on legged robots with motion_tracking_controller — motion_tracking_controller is a C++ ROS 2 package deploying RL-trained motion tracking policies on legged robots with ON
→ GitHub Repo: InternRobotics/NavDP ⭐ 629 · Python