Multimodal large language models (MLLMs) are pushing AI beyond text, combining visual understanding with language reasoning at scale. InternVL from OpenGVLab stands out by openly sharing not only model weights but also training code, data pipelines, and evaluation scripts. Its latest iteration, InternVL3.5, showcases a unique CascadeRL training pipeline that integrates offline and online reinforcement learning — a rare glimpse into production-scale RL fine-tuning for multimodal models.
What InternVL delivers and how it works under the hood
InternVL is a family of multimodal large language models that fuse vision transformers (InternViT) with large language models in a hybrid architecture. This design allows processing and reasoning over both images and text inputs, tackling benchmarks like MMMU (multimodal understanding), MathVista (mathematical reasoning), DocVQA (document question answering), and Video-MME (video multimodal evaluation).
The models range from compact 1B-parameter variants to massive 241B-parameter models, including mixture-of-experts (MoE) versions such as the 20B-A4B. This spectrum covers use cases from efficient deployment to research at the frontier.
Key architectural components include InternViT, a vision transformer optimized for extracting rich visual features, combined with large language models that handle multimodal reasoning. This hybrid approach balances visual feature extraction and language understanding effectively.
Performance-wise, InternVL2.5-78B hits over 70% accuracy on the MMMU benchmark, matching GPT-4o. Other variants like InternVL2-8B-MPO achieve 67.0 on MathVista, and InternVL2-40B scores 61.2 to 64.4 on Video-MME depending on frame count. These numbers highlight that open-source models can compete with closed commercial offerings in multiple challenging tasks.
What sets InternVL apart technically
Several technical innovations differentiate InternVL:
Mixed Preference Optimization (MPO): This fine-tuning approach aligns model outputs with human preferences, boosting performance by about 2 points on average across model sizes on the OpenCompass benchmark. MPO improves alignment without requiring prohibitively expensive RL training.
CascadeRL: A two-stage reinforcement learning pipeline combining offline and online RL. Offline RL leverages large-scale preference-labeled datasets, while online RL refines the model through interaction. This hybrid approach balances sample efficiency and adaptability, a tradeoff often tricky in RLHF (reinforcement learning with human feedback) for multimodal models.
VisualPRM: A process reward model tailored for visual tasks. It enhances reward signal quality during RL training by focusing on the visual reasoning process rather than just end results. This subtle but impactful distinction helps the model learn better representations and decision-making strategies.
Multimodal Test-Time Scaling: Dynamic scaling techniques during inference optimize performance across different input modalities and contexts, allowing the model to adapt resource usage effectively.
Variable Visual Position Encoding: This technique improves how the model handles varying visual input sizes and resolutions, which is critical for real-world applications where image dimensions are not fixed.
Unlike many closed-source competitors, OpenGVLab open-sources not only model weights but also the training code, data construction pipelines (including MMPR-v1.2, MMPR-Tiny, VisualPRM400K), and evaluation scripts. This transparency is invaluable for reproducibility and further research.
The tradeoffs are clear: the models require substantial compute resources, especially at the large scales, and the complexity of the CascadeRL pipeline means a steeper learning curve for practitioners. However, the modularity and open data help ease experimentation.
Quick start with InternVL models
The repo provides straightforward Python examples for using InternVL models via Hugging Face transformers. Here’s how to run visual feature extraction with InternViT-6B:
import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor
model = AutoModel.from_pretrained(
'OpenGVLab/InternViT-6B-448px-V2_5',
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
trust_remote_code=True).cuda().eval()
image = Image.open('./examples/image1.jpg').convert('RGB')
image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternViT-6B-448px-V1-5')
pixel_values = image_processor(images=image, return_tensors='pt').pixel_values
pixel_values = pixel_values.to(torch.bfloat16).cuda()
outputs = model(pixel_values)
For cross-modal retrieval with InternVL-14B models:
import torch
from PIL import Image
from transformers import AutoModel, CLIPImageProcessor
from transformers import AutoTokenizer
model = AutoModel.from_pretrained(
'OpenGVLab/InternVL-14B-224px',
torch_dtype=torch.bfloat16,
low_cpu_mem_usage=True,
trust_remote_code=True).cuda().eval()
image_processor = CLIPImageProcessor.from_pretrained('OpenGVLab/InternVL-14B-224px')
tokenizer = AutoTokenizer.from_pretrained(
'OpenGVLab/InternVL-14B-224px', use_fast=False, add_eos_token=True)
tokenizer.pad_token_id = 0 # set pad_token_id to 0
images = [
Image.open('./examples/image1.jpg').convert('RGB'),
Image.open('./examples/image2.jpg').convert('RGB'),
Image.open('./examples/image3.jpg').convert('RGB')
]
prefix = 'summarize:'
texts = [
prefix + 'a photo of a red panda', # English
prefix + '一张熊猫的照片', # Chinese
prefix + '二匹の猫の写真' # Japanese
]
pixel_values = image_processor(images=images, return_tensors='pt').pixel_values
pixel_values = pixel_values.to(torch.bfloat16).cuda()
input_ids = tokenizer(texts, return_tensors='pt', max_length=80,
truncation=True, padding='max_length').input_ids.cuda()
These examples give a quick entry point into experimenting with InternVL models, whether for visual feature extraction or multimodal retrieval.
Verdict: who should explore InternVL
InternVL is a solid resource for researchers and practitioners working on multimodal AI who want access to state-of-the-art models and training pipelines with full transparency. The open-sourcing of training data and code sets it apart in a field where many competitors remain closed. The CascadeRL pipeline is especially worth understanding if you’re interested in RLHF at scale for multimodal models.
That said, the hardware requirements and training complexity mean it’s not a plug-and-play solution for all. Smaller variants like Mini-InternVL 4B offer a good tradeoff between performance and resource footprint, delivering 90% of larger model capabilities at a fraction of the size.
If you’re building applications that combine vision and language and want to avoid black-box closed models, InternVL offers a rare window into top-tier open multimodal LLMs with practical training and evaluation tools. The repository is a good starting point for hands-on experimentation and advancing research in this fast-evolving area.
→ GitHub Repo: OpenGVLab/InternVL ⭐ 10,037 · Python