Mirage: Multimodal Reasoning in VLMs Without Rendering Images

The target audience for the research on Mirage consists of AI researchers, business managers in tech companies, and developers focused on enhancing visual language models (VLMs). Their primary pain points include the challenges associated with VLMs’ reliance on text for reasoning, which limits their effectiveness in tasks requiring visual thinking. These professionals aim to develop solutions that improve the reasoning capabilities of VLMs while balancing computational efficiency and accuracy.

Many in this audience are interested in the latest advancements in AI, particularly in multimodal reasoning, and the practical applications of these technologies in business settings. They prefer clear, technical communication that includes peer-reviewed statistics and case studies demonstrating the real-world impact of these innovations.

Understanding the Limitations of Current VLMs

While VLMs excel at interpreting both text and images, their reasoning abilities are often confined to text alone. This limitation hampers their performance in tasks that require visual thinking, such as spatial puzzles. Humans naturally visualize solutions rather than articulating every detail, but VLMs struggle with this cognitive process. Although some recent models can generate both text and images, the focus on image generation can undermine their reasoning capabilities. Additionally, generating images does not facilitate step-by-step visual reasoning, presenting a significant challenge in unlocking the full potential of VLMs for complex, visually grounded tasks.

Methodologies for Enhanced Multimodal Reasoning

Chain-of-Thought (CoT) prompting encourages models to tackle problems step by step, utilizing examples with intermediate explanations. This concept has been adapted for multimodal tasks, integrating visual information into the reasoning flow. Techniques such as ICoT embed image regions within text sequences, while Visual CoT employs visual annotations to enhance spatial understanding. Some recent models capable of generating both text and images simultaneously require substantial supervision and incur high computational costs. Researchers are also exploring internal reasoning embeddings within models, using special tokens or latent representations to guide reasoning without explicit steps.

Introducing Mirage: A New Framework

Researchers from the University of Massachusetts Amherst and MIT propose Mirage, a framework that enables VLMs to integrate visual reasoning directly into their text outputs without generating full images. Instead, the model incorporates compact visual cues derived from its hidden states. Mirage is trained in two phases: initially with both text and visual supervision, followed by text-only guidance. Reinforcement learning further refines its reasoning skills, allowing VLMs to think more like humans and improving their performance on complex multimodal tasks.

Training and Evaluation of Mirage

Mirage employs a two-stage training process. The first phase grounds compressed visual features, known as latent tokens, within the reasoning process using helper images and joint supervision. The second phase relaxes this constraint, enabling the model to generate its latent tokens independently to guide reasoning. A final reinforcement learning stage enhances performance by rewarding accuracy and structured thought processes.

The model was evaluated on four spatial reasoning tasks, including visual puzzles and geometry problems, using a dataset of 1,000 training samples. To support reasoning, it generates synthetic helper images and thought steps, mimicking human cognitive strategies like sketches and cues. The model consistently outperformed both text-only and multimodal baselines, excelling in tasks requiring extensive planning, such as maze solving. A smaller version of the model also demonstrated strong results, indicating the robustness of the approach. Ablation studies confirmed that grounding latent visual tokens initially, followed by flexible training, is critical for success.

Conclusion

In summary, Mirage introduces a lightweight approach inspired by human mental imagery that allows VLMs to reason visually without generating actual images. By interleaving compact visual cues with text during decoding, the model learns to reason multimodally through a two-phase training process. Although tested on spatial reasoning tasks, the method consistently outperformed traditional text-only models. Challenges remain in scaling to additional tasks and enhancing the quality of synthetic training data.

Check out the Paper and GitHub Page. All credit for this research goes to the researchers of this project.

Sponsorship Opportunity

Reach the most influential AI developers worldwide. 1M+ monthly readers, 500K+ community builders, infinite possibilities. Explore Sponsorship