GLM-4.1V-Thinking: Advancing General-Purpose Multimodal Understanding and Reasoning

Vision-language models (VLMs) are essential in modern intelligent systems, facilitating a comprehensive understanding of visual content. The complexity of multimodal intelligence tasks has expanded significantly, encompassing scientific problem-solving and the development of autonomous agents. Current demands on VLMs have surpassed basic visual content perception, with a growing emphasis on advanced reasoning capabilities.

Recent studies indicate that long-form reasoning and scalable reinforcement learning (RL) can significantly enhance the problem-solving abilities of large language models (LLMs). However, existing efforts primarily target specific domains to improve VLM reasoning. The open-source community currently lacks a multimodal reasoning model that outperforms traditional non-thinking models of comparable parameter scale across diverse tasks.

Researchers from Zhipu AI and Tsinghua University have introduced GLM-4.1V-Thinking, a VLM aimed at advancing general-purpose multimodal understanding and reasoning. This model incorporates Reinforcement Learning with Curriculum Sampling (RLCS) to unlock its full potential, leading to improvements in STEM problem-solving, video understanding, content recognition, coding, grounding, GUI-based agents, and long document understanding.

The open-sourced GLM-4.1V-9B-Thinking sets a new benchmark among similarly sized models, delivering competitive, and in some cases superior, performance compared to proprietary models like GPT-4o on challenging tasks such as long document understanding and STEM reasoning.

Core Components

GLM-4.1V-Thinking comprises three core components:

• Vision encoder
• MLP adapter
• LLM decoder

The model utilizes AIMv2-Huge as the vision encoder and GLM as the LLM, replacing the original 2D convolutions with 3D convolutions for temporal downsampling. It integrates 2D-RoPE to support arbitrary image resolutions and aspect ratios, processing extreme aspect ratios over 200:1 and high resolutions beyond 4K. Researchers extend RoPE to 3D-RoPE in the LLM to enhance spatial understanding in multimodal contexts. For temporal modeling in videos, time index tokens are added after each frame token, with timestamps encoded as strings to help the model understand real-world temporal gaps between frames.

Training Methodology

During pre-training, the researchers employed a variety of datasets, combining large academic corpora with interleaved image-text data rich in knowledge. By including pure text data, the model’s core language capabilities are preserved, resulting in better pass@k performance than other state-of-the-art pre-trained base models of similar size. The supervised fine-tuning stage transforms the base VLM into one capable of long chain-of-thought (CoT) inference using a curated long-CoT corpus across verifiable tasks, such as STEM problems, and non-verifiable tasks like instruction following. Finally, the RL phase employs a combination of RLVR and RLHF to conduct large-scale training across all multimodal domains, including STEM problem-solving, grounding, optical character recognition, GUI agents, and more.

Performance Metrics

GLM-4.1V-9B-Thinking outperforms all competing open-source models under 10B parameters in General Visual Question Answering (VQA) tasks covering both single-image and multi-image settings. It achieves the highest performance on challenging STEM benchmarks, including MMMU_Val, MMMU_Pro, VideoMMMU, and AI2D. In the Optical Character Recognition (OCR) and Chart domains, the model sets new state-of-the-art scores on ChartQAPro and ChartMuseum. For Long Document Understanding, GLM-4.1V-9B-Thinking surpasses all other models on MMLongBench, while establishing new state-of-the-art results in GUI Agents and multimodal coding tasks. Lastly, the model demonstrates robust video understanding performance, outperforming benchmarks such as VideoMME, MMVU, and MotionBench.

Conclusion and Future Directions

In conclusion, GLM-4.1V-Thinking represents a significant advancement in general-purpose multimodal reasoning. Its 9B-parameter model outperforms larger models exceeding 70B parameters. However, several limitations persist, including inconsistent improvements in reasoning quality through RL, instability during training, and challenges with complex cases. Future developments should focus on enhancing supervision and evaluation of model reasoning, with reward models assessing intermediate reasoning steps while detecting hallucinations and logical inconsistencies. Additionally, exploring strategies to prevent reward hacking in subjective evaluation tasks is crucial for achieving general-purpose intelligence.

Check out the Paper and GitHub Page. All credit for this research goes to the researchers of this project.

Sponsorship Opportunity

Reach the most influential AI developers worldwide. 1M+ monthly readers, 500K+ community builders, infinite possibilities. Explore Sponsorship