Noureddine RAMDI / UI-Voyager: Self-evolving AI agent for Android GUI automation with SSIM-based trajectory correction

Created Mon, 04 May 2026 10:23:03 +0000 Modified Sat, 23 May 2026 20:41:27 +0000
1021 Words
github-stars python ai-agent android-automation reinforcement-learning self-evolving gui-automation

ui-voyager/UI-Voyager

UI-Voyager is a rare example of an AI agent that learns to interact with mobile GUIs at scale by teaching itself from both its successes and failures. Achieving an 81.0% success rate on the AndroidWorld benchmark, it surpasses human-level performance on this task. What sets it apart is how it extracts training signals not just from successful attempts but cleverly mines errors using image similarity to pinpoint where things went wrong.

What UI-Voyager does and how it works

UI-Voyager is a Python-based AI agent designed to automate tasks on Android user interfaces. At its core, it controls Android emulators to perform GUI operations, learning a policy to navigate and manipulate mobile apps effectively.

The system is built around a large 4 billion parameter model that is trained in a two-stage pipeline to improve its interaction policy iteratively. This pipeline consists of:

  • Rejection Fine-Tuning (RFT): The model generates trajectories of interactions, which are filtered through a rule-based verifier to keep only high-quality successful sequences for supervised fine-tuning. This ensures the training data is robust.

  • Group Relative Self-Distillation (GRSD): This stage identifies “fork points” where successful and failed trajectories diverge by comparing screenshots using Structural Similarity Index Measure (SSIM). It then corrects erroneous actions in failed trajectories by learning from the successful ones, effectively turning failures into learning opportunities.

The model is served via the vLLM framework, exposing an OpenAI-compatible API, making it straightforward to integrate or experiment with. Evaluation is performed on parallel Android emulators, simulating diverse environments to validate generalization.

Technically, the stack includes Python for model training and orchestration, usage of Android Virtual Devices (emulators) for environment simulation, and vLLM for efficient large model serving. The approach avoids reliance on human-labeled data by using rule-based verification and self-distillation, which is significant for scaling training without expensive annotation.

The training pipeline and self-evolving mechanism

What distinguishes UI-Voyager is its two-stage training pipeline that mines the maximum signal from all experiences, including failures — a contrast to many reinforcement learning systems that discard failed attempts.

Rejection Fine-Tuning (RFT) acts as a quality gate. The model’s generated trajectories are passed through a rule-based verifier to reject low-quality or incorrect sequences. This yields a curated dataset of successful interactions, enabling more reliable supervised fine-tuning. It’s a pragmatic approach to maintain training data quality without manual human labeling.

Group Relative Self-Distillation (GRSD) is the more novel aspect. When the agent fails, rather than discarding those rollouts, it compares the failed trajectory screenshots to those of successful ones using SSIM to find the exact point (fork point) where the failure happened. This precision allows the system to generate corrective training signals that refine the policy at these failure points. By iteratively applying GRSD, the agent self-evolves, improving its policy continuously.

This method of treating failures as a source of corrective feedback rather than noise is clever and cost-effective. It leverages image similarity (SSIM) to align trajectories without requiring manual annotation, which is often a bottleneck in training such agents.

The codebase reflects this pipeline clearly, with separate modules handling verification, trajectory processing, and model updates. The integration with Android emulators is also well encapsulated, allowing for parallel evaluation and data collection.

The tradeoff here is complexity: setting up Android emulators and managing multiple parallel evaluation environments requires infrastructure and can be resource intensive. The model itself is large (4B parameters), so serving and inference need a capable GPU environment.

Quick start

The project provides detailed steps to get started with evaluation, assuming you have an Android emulator environment ready.

1. Prepare an Android emulator (AVD)

You must have an Android Virtual Device (AVD) available for emulator startup. For AVD creation and emulator setup, follow the AndroidWorld installation guides linked in the repo.

By default, the scripts assume:

  • AVD_NAME=AndroidWorldAvd
  • emulator binary at /root/android/emulator/emulator

Override these if your setup differs.

2. Install dependencies

pip install -r androidworld/requirements.txt
python3 android_env/setup.py install

3. Start model API service with vLLM

Download the model from HuggingFace:

huggingface-cli download --resume-download MarsXL/UI-Voyager --local-dir /path/to/ui-voyager

Deploy the model using vLLM:

vllm serve /path/to/ui-voyager \ 
  --served-model-name UI-Voyager \
  --host 0.0.0.0 \
  --port 8080 \
  --tensor-parallel-size 1

The default YAML config uses:

  • llm.base_url: http://localhost:8000
  • llm.model: UI-Voyager

4. Start evaluation (parallel emulators)

NUM_WORKERS=4 CONFIG_NAME=UI-Voyager MODEL_NAME=UI-Voyager ./run_android_world.sh

5. Monitor and stop

After evaluation starts, the script outputs the main PID, log file path, and output directory.

To stop a running evaluation:

./stop_android_world.sh /path/to/eval_results/<MODEL_NAME>/logs/<timestamp>

Or kill manually if needed:

kill "$(cat eval_results/<MODEL_NAME>/logs/<timestamp>/eval.pid)"

verdict

UI-Voyager is a solid example of using self-supervised, self-correcting AI to automate complex mobile GUI tasks without human annotation. Its 81.0% success on AndroidWorld shows the strength of its training approach, especially the SSIM-based Group Relative Self-Distillation that turns failures into a rich corrective signal.

That said, the setup is non-trivial, requiring Android emulator infrastructure and capable GPU resources for serving the large model. It’s best suited for researchers and engineers interested in autonomous mobile agents, reinforcement learning from imperfect data, and large model serving.

If you’re tackling mobile GUI automation or interested in training AI agents that learn from their mistakes without labeled data, UI-Voyager is worth exploring. The pipeline design and integration with vLLM provide a practical blueprint for similar projects.

→ GitHub Repo: ui-voyager/UI-Voyager ⭐ 67 · Python