Introduction: The Rise of GUI Agents

Modern computing is dominated by graphical user interfaces across devices—mobile, desktop, and web. Automating tasks in these environments has traditionally been limited to scripted macros or brittle, hand-engineered rules. Recent advances in vision-language models offer the possibility of agents that can understand screens, reason about tasks, and execute actions like humans. However, most approaches have either relied on closed-source models or have struggled with generalizability, reasoning fidelity, and cross-platform robustness.

A team of researchers from Alibaba Qwen introduces GUI-Owl and Mobile-Agent-v3 to tackle these challenges. GUI-Owl is a native, end-to-end multimodal agent model, built on Qwen2.5-VL and post-trained on large-scale, diverse GUI interaction data. It unifies perception, grounding, reasoning, planning, and action execution within a single policy network, enabling robust cross-platform interaction and explicit multi-turn reasoning. The Mobile-Agent-v3 framework leverages GUI-Owl as a foundational module, orchestrating multiple specialized agents (Manager, Worker, Reflector, Notetaker) to handle complex, long-horizon tasks with dynamic planning, reflection, and memory.

Architecture and Core Capabilities

GUI-Owl: The Foundational Model

GUI-Owl is designed to handle the heterogeneity and dynamism of real-world GUI environments. It is initialized from Qwen2.5-VL, a state-of-the-art vision-language model, but undergoes extensive additional training on specialized GUI datasets. This includes grounding (locating UI elements from natural language queries), task planning (breaking down complex instructions into actionable steps), and action semantics (understanding how actions affect the GUI state). The model is fine-tuned via a mix of supervised learning and reinforcement learning (RL), focusing on aligning its decisions with real-world task success.

Key Innovations in GUI-Owl:

• Unified Policy Network: Integrates perception, planning, and execution into a single neural network, allowing for seamless multi-turn decision-making and explicit intermediate reasoning.
• Scalable Training Infrastructure: A cloud-based virtual environment spanning Android, Ubuntu, macOS, and Windows generates high-quality interaction data through rigorous correctness evaluation.
• Diverse Data Synthesis: Employs various data synthesis strategies for robust grounding and reasoning.
• Reinforcement Learning Alignment: Refines GUI-Owl via a scalable RL framework, supporting asynchronous training and a novel “Trajectory-aware Relative Policy Optimization” (TRPO).

Mobile-Agent-v3: Multi-Agent Coordination

Mobile-Agent-v3 is a general-purpose agentic framework designed for complex, multi-step, and cross-application workflows. It breaks tasks into subgoals, dynamically updates plans based on execution feedback, and maintains persistent contextual memory. The framework coordinates four specialized agents:

• Manager Agent: Decomposes high-level instructions into subgoals and updates plans based on results.
• Worker Agent: Executes the most relevant actionable subgoal given the current GUI state.
• Reflector Agent: Evaluates the outcome of each action and generates diagnostic feedback.
• Notetaker Agent: Persists critical information across application boundaries.

Training and Data Pipeline

A significant bottleneck in GUI agent development is the lack of high-quality, scalable training data. The GUI-Owl team addresses this with a self-evolving data production pipeline:

• Query Generation: Models realistic navigation flows and user inputs, synthesizing natural instructions validated against real app interfaces.
• Trajectory Generation: Produces a sequence of actions and state transitions through interaction with a virtual environment.
• Trajectory Correctness Judgment: A two-level critic system evaluates each step and overall trajectory using textual and multimodal reasoning.
• Guidance Synthesis: Synthesizes step-by-step guidance from successful trajectories to aid agent learning.
• Iterative Training: Newly generated successful trajectories are added to the training set for model retraining.

Benchmarking and Performance

GUI-Owl and Mobile-Agent-v3 are rigorously evaluated across GUI automation benchmarks, covering grounding, single-step decision-making, question answering, and end-to-end task completion.

Grounding and UI Understanding

On grounding tasks—locating UI elements from natural language queries—GUI-Owl-7B outperforms all open-source models of comparable size. For example, on the MMBench-GUI L2 benchmark, GUI-Owl-7B scores 80.49, while GUI-Owl-32B achieves 82.97, both ahead of the competition. These results demonstrate that GUI-Owl’s grounding capabilities are both broad and deep.

Comprehensive GUI Understanding and Single-Step Decision Making

MMBench-GUI L1 evaluates UI understanding and single-step decision-making. Here, GUI-Owl-7B scores 84.5 (easy), 86.9 (medium), and 90.9 (hard), far outpacing all existing models. This indicates robust reasoning about interface states and actions.

End-to-End and Multi-Agent Capabilities

GUI-Owl-7B scores 66.4 on AndroidWorld and 34.9 on OSWorld, while Mobile-Agent-v3 achieves 73.3 and 37.7, respectively—a new state-of-the-art for open-source frameworks. The multi-agent design proves effective on long-horizon, error-prone tasks.

Real-World Deployment

GUI-Owl supports a rich, platform-specific action space, making it readily deployable in real environments. The agent’s reasoning and decision process is transparent, improving robustness and enabling integration into larger multi-agent systems.

Conclusion: Toward General-Purpose GUI Agents

GUI-Owl and Mobile-Agent-v3 represent a significant leap toward general-purpose, autonomous GUI agents. By unifying perception, grounding, reasoning, and action into a single model, the research team has achieved state-of-the-art performance across both mobile and desktop environments.