itinai content

←back to Blog

VL-Cogito: Advancing Multimodal Reasoning with Progressive Curriculum Reinforcement Learning

VL-Cogito: Advancing Multimodal Reasoning with Progressive Curriculum Reinforcement Learning

Understanding the Target Audience

The primary audience for VL-Cogito includes AI researchers, business leaders in technology, and educators interested in the advancements of multimodal reasoning and reinforcement learning. Their pain points revolve around the challenges of integrating diverse data sources, improving model accuracy, and addressing the limitations of existing AI systems. They seek to enhance their understanding of complex AI frameworks and are particularly interested in practical applications that can drive business innovation.

Core Innovations

VL-Cogito introduces a unique approach to multimodal reasoning through the Progressive Curriculum Reinforcement Learning (PCuRL) framework, designed to systematically address the instability and domain gaps prevalent in this area. The framework includes two significant innovations:

  • Online Difficulty Soft Weighting (ODSW): This mechanism dynamically assigns weights to training samples based on their difficulty level and the model’s capabilities. It allows the model to progress through tasks of varying complexities, ensuring that each prompt contributes meaningfully to gradient updates.
  • Dynamic Length Reward (DyLR): Unlike traditional static length rewards, DyLR calculates an ideal target length for each prompt based on the average length of correct rollout samples. This promotes concise reasoning for simpler tasks and encourages deeper exploration for more complex ones.

Training Pipeline

VL-Cogito’s reinforcement learning (RL) post-training begins with the Qwen2.5-VL-Instruct-7B backbone, without requiring initial supervised fine-tuning (SFT). The PCuRL process is divided into three sequential RL stages: easy, medium, and hard. In each stage:

  • The dataset is shuffled to expose the model to various generalization challenges.
  • ODSW biases gradient updates towards the target difficulty for that stage.
  • In the hard stage, DyLR promotes adaptive reasoning chain expansion.

Technical Setup

VL-Cogito utilizes the following technical specifications:

  • Optimizer: AdamW
  • Learning Rate: 1e-6
  • DeepSpeed: ZeRO3
  • Rollout Batch Size: 512
  • Global Batch Size: 128
  • Sequence Length: 4,096
  • KL Divergence Loss: 1e-3
  • Response Samples per Prompt: 16
  • Temperature: 1.0
  • Reward Hyperparameters: α=1, β=0.5, γ=1, w=0.25 (penalty for zero-accuracy prompts)

Dataset Curation and RL Data Sampling

The training set comprises 23 open-source multimodal datasets across six task categories: Mathematical Reasoning, Logical Reasoning, Counting, Science Reasoning, Chart Understanding, and General Image Understanding. All samples are reformulated to open-ended QA formats to avoid superficial multiple-choice cues. Difficulty sampling ensures that only genuinely challenging tasks remain.

Evaluation and Benchmark Results

VL-Cogito has been benchmarked against various general-purpose and reasoning-oriented MLLMs across ten tasks, including Geometry@3K, MathVerse, and ScienceQA. The model demonstrates significant accuracy gains over its backbone:

  • +7.6% on Geometry@3K
  • +5.5% on MathVista
  • +4.9% on LogicVista
  • +2.2% on ScienceQA
  • +4.5% on EMMA
  • +3.8% on MMStar

VL-Cogito achieves state-of-the-art results in 6 out of 10 benchmarks, particularly excelling in rigorous math and scientific tasks.

Insights and Impact

VL-Cogito’s systematic PCuRL pipeline offers several key insights:

  • Intermediate difficulty prompts optimize model progress.
  • Exposure to challenging tasks enhances deep reasoning capabilities.
  • Combining correctness, format, and length of rewards yields nuanced reasoning outputs.
  • No-SFT cold-start RL is feasible and effective.

Conclusion

VL-Cogito’s architecture and training innovations establish a new benchmark for multimodal reasoning across diverse applications. The design and empirical validation of progressive curriculum RL with dynamic length rewards provide a roadmap for robust reasoning in multimodal models.

For further exploration, visit the GitHub Page for Tutorials, Codes, and Notebooks and follow us on Twitter.