itinai content

←back to Blog

Meta FAIR Released Code World Model (CWM): A 32-Billion-Parameter Open-Weights LLM, to Advance Research on Code Generation with World Models

Meta FAIR Released Code World Model (CWM): A 32-Billion-Parameter Open-Weights LLM to Advance Research on Code Generation with World Models

Understanding the Target Audience

The target audience for the Meta FAIR Code World Model (CWM) primarily includes:

  • Researchers and Academics: Individuals focused on advancing AI and machine learning, particularly in code generation and software engineering.
  • Software Engineers: Professionals interested in leveraging AI tools for code generation, debugging, and enhancing productivity.
  • Data Scientists: Experts who analyze and interpret data, looking for innovative models to improve coding practices.
  • AI Enthusiasts: Individuals keen on exploring new AI models and their applications in real-world scenarios.

Common pain points include:

  • Difficulty in generating accurate and context-aware code.
  • Challenges in debugging and understanding code execution.
  • Need for scalable and efficient AI models for practical applications.

Goals of the audience involve:

  • Improving code generation accuracy and efficiency.
  • Enhancing understanding of code execution through AI.
  • Exploring new methodologies in AI-driven software development.

Preferred communication styles are typically technical and concise, favoring data-driven insights and practical applications.

Overview of CWM

Meta FAIR has introduced the Code World Model (CWM), a 32-billion-parameter dense decoder-only LLM designed to enhance code generation through world modeling. This model is trained on execution traces and long-horizon agent-environment interactions, moving beyond static source text.

Key Features of CWM

The CWM incorporates innovative learning techniques by predicting execution:

  • Mid-Training on Observation-Action Trajectories: CWM is trained on two significant families of trajectories:
    • Python interpreter traces capturing local variable states after each executed line.
    • Agentic interactions within Dockerized repositories, documenting edits, shell commands, and test feedback.
  • Executable Repository Images: The research team created executable images from thousands of GitHub projects, collecting approximately 3 million trajectories across 10,000 images and 3,150 repositories.

Model Specifications

CWM is a dense, decoder-only Transformer model with:

  • 64 layers
  • GQA (48Q/8KV)
  • SwiGLU
  • RMSNorm
  • Scaled RoPE

The attention mechanism alternates between local 8,000 and global 131,000 sliding-window blocks, allowing an effective context of 131,000 tokens. The training employs document-causal masking.

Training Process

The training process consists of three phases:

  • Pre-training: 8 trillion tokens (code-heavy) at 8,000 context.
  • Mid-training: An additional 5 trillion tokens with long-context (131,000) using Python execution traces and ForagerAgent data.
  • Post-training: 100 billion tokens for instruction and reasoning, followed by multi-task reinforcement learning across various coding environments.

Performance Benchmarks

The CWM has demonstrated competitive performance with the following benchmarks:

  • SWE-bench Verified: 65.8% pass rate (with test-time scaling).
  • LiveCodeBench-v5: 68.6%; LCB-v6: 63.5%.
  • Math-500: 96.6%; AIME-24: 76.0%; AIME-25: 68.2%.
  • CruxEval-Output: 94.3%.

Importance of World Modeling in Code Generation

The CWM emphasizes two critical capabilities:

  • Execution-Trace Prediction: CWM predicts stack frames and executed lines at each step, functioning as a «neural debugger» for grounded reasoning.
  • Agentic Coding: The model engages in multi-turn reasoning with tool use against real repositories, generating end-to-end patches verified by hidden tests.

Conclusion

The CWM represents a significant advancement in grounded code generation, linking a 32 billion parameter dense transformer to execution-trace learning and agentic patching. Meta has made intermediate and post-trained checkpoints available under the FAIR Non-Commercial Research License, facilitating reproducible research in long-context, execution-aware coding.

For further details, refer to the original publication.