Spider is a physics-based motion retargeting framework that tackles one of robotics’ thornier problems: converting raw human motion capture from video into trajectories that a robot can physically execute. It supports a wide range of robot embodiments, from humanoids to dexterous hands, and integrates tightly with physics simulators to produce optimized, realistic robot motions. This makes it a noteworthy tool for anyone working on robot learning, manipulation, or sim-to-real transfer.
modular physics pipeline for human-to-robot motion retargeting
At its core, Spider decomposes the retargeting task into a clear modular pipeline that processes raw human video data step-by-step to generate robot-executable trajectories.
The pipeline stages include:
- decompose_fast: Quickly extract raw human pose and motion data from video.
- detect_contact: Identify contact points like hands touching objects or the ground.
- generate_xml: Convert the processed human motion data into XML robot models suitable for simulators.
- ik_fast: Run a fast inverse kinematics solver to align robot joints with the target poses.
- retarget: Use physics-informed optimization to refine these trajectories, ensuring physical plausibility and robot feasibility.
Spider supports over 9 robot platforms and more than 6 datasets right out of the box, showing its versatility. It also provides native integration with popular simulators like Mujoco Warp (MJWP), Genesis, and IsaacGym, which are widely used in robotics research. This multi-simulator support is crucial for sim-to-real workflows where optimized trajectories can be validated and transferred to real robots.
The stack is primarily Python 3.12-based, relying heavily on numerical and physics libraries, and designed to run efficiently with options for faster approximations at each pipeline stage, balancing speed and accuracy.
what sets spider apart: modular tradeoffs and sim-to-real focus
Spider’s standout technical feature is its modular decomposition with configurable fast vs original methods at each step. This design lets users tailor the pipeline to their needs — for example, using faster but less precise decomposition and IK when prototyping, versus the original methods for higher fidelity when deploying.
This tradeoff approach is practical because real-time or near-real-time retargeting often needs speed, while offline trajectory optimization benefits from accuracy. The codebase reflects this with clear separation of concerns and well-documented modules under the hood.
Another strength is the physics-informed optimization step. Instead of naive retargeting that might produce kinematically plausible but physically impossible motions, Spider runs constrained optimization incorporating contact forces and joint limits. This ensures the resulting robot trajectories are executable in real-world physics, which is vital for sim-to-real transfer.
The integration with multiple simulators further shows the framework’s maturity. Each simulator has different capabilities and performance characteristics, and Spider’s abstraction lets users switch or combine them without major rewrites.
The code quality appears solid with a Pythonic structure, use of type hints, and reasonably clean interfaces. However, targeting Python 3.12 and dependencies on specific simulator environments might limit immediate adoption in some projects. Also, while the modular pipeline is powerful, it does require understanding of robotics concepts like inverse kinematics and contact detection to fully leverage.
quickstart: cloning datasets and environment setups
Spider provides two main quickstart options, either using the uv environment manager or conda. Both assume you have cloned example datasets using Git LFS:
sudo apt install git-lfs
git lfs install
git clone https://huggingface.co/datasets/retarget/retarget_example example_datasets
Quickstart with uv
Make sure uv uses Python 3.12, then run:
uv sync
If you have the example datasets cloned, you can skip preprocessing and run Spider on a processed trial.
Quickstart with conda
conda create -n spider python=3.12
conda activate spider
python -m pip install --upgrade pip
python -m pip install -r requirements.txt
python -m pip install --no-deps -e .
Run Mujoco Warp example:
python examples/run_mjwp.py
Additional environment setups are noted for dexmachina and HDMI uv environments, mostly for using Spider’s optimization component standalone or integrated with other projects.
verdict: a solid, modular physics retargeting framework for robotics researchers
Spider stands out as a well-engineered, modular physics-based retargeting framework that covers an important robotics use case: making human motion data usable for diverse robot platforms.
Its strengths lie in the clear pipeline decomposition, configurable speed vs accuracy tradeoffs, physics-aware optimization, and support for multiple simulators. This makes it valuable for researchers and practitioners focused on robot learning, dexterous manipulation, and sim-to-real transfer.
The main limitations are its reliance on Python 3.12 (which may require environment setup work), the complexity of understanding the robotics concepts it encapsulates, and dependency on external simulators.
If you’re working on robot motion retargeting or sim-to-real pipelines and want a robust starting point with support for multiple robots and datasets, Spider is worth exploring. The code is accessible, and the provided quickstart commands get you running with real data quickly. Just be prepared for some learning curve on the robotics pipeline details and simulator integration.
→ GitHub Repo: facebookresearch/spider ⭐ 443 · Python