This AI Paper from Alibaba Introduces Lumos-1: A Unified Autoregressive Video Generator Leveraging MM-RoPE and AR-DF for Efficient Spatiotemporal Modeling

Autoregressive video generation is a rapidly evolving research domain that focuses on synthesizing videos frame-by-frame using learned patterns of both spatial arrangements and temporal dynamics. Unlike traditional video creation methods, which may rely on pre-built frames or handcrafted transitions, autoregressive models aim to generate content dynamically based on prior tokens. This approach is similar to how large language models predict the next word, offering the potential to unify video, image, and text generation under a shared framework by utilizing the structural power of transformer-based architectures.

Challenges in Spatiotemporal Modeling

One major problem in this space is accurately capturing and modeling the intrinsic spatiotemporal dependencies in videos. Videos contain rich structures across both time and space, and encoding this complexity so models can predict coherent future frames remains a challenge. When these dependencies are not modeled well, it leads to broken frame continuity or unrealistic content generation. Traditional training techniques like random masking often fail to provide balanced learning signals across frames, causing prediction to become too easy when spatial information from adjacent frames leaks.

Introducing Lumos-1

The research team from Alibaba Group’s DAMO Academy, Hupan Lab, and Zhejiang University introduced Lumos-1, a unified model for autoregressive video generation that adheres closely to large language model architecture. Unlike previous tools, Lumos-1 eliminates the need for external encoders and changes very little in the original LLM design. The model employs MM-RoPE, or Multi-Modal Rotary Position Embeddings, to address the challenge of modeling video’s three-dimensional structure. It also utilizes a token dependency approach that preserves intra-frame bidirectionality and inter-frame temporal causality, aligning more naturally with video data behavior.

Technical Innovations

In MM-RoPE, researchers expand existing RoPE methods to balance frequency spectrum for spatial and temporal dimensions. Traditional 3D RoPE misallocates frequency focus, leading to detail loss or ambiguous positional encoding. MM-RoPE restructures allocations so that temporal, height, and width each receive balanced representation. To address loss imbalance in frame-wise training, Lumos-1 introduces AR-DF, or Autoregressive Discrete Diffusion Forcing, which employs temporal tube masking during training. This ensures even learning across the video sequence and allows high-quality frame generation without degradation.

Performance and Training Efficiency

Lumos-1 was trained from scratch on 60 million images and 10 million videos using only 48 GPUs, which is considered memory-efficient given the training scale. The model achieved results comparable to top models in the field, matching EMU3’s results on GenEval benchmarks and performing equivalently to COSMOS-Video2World on the VBench-I2V test. It also rivaled OpenSoraPlan’s outputs on the VBench-T2V benchmark. These comparisons demonstrate that Lumos-1’s lightweight training does not compromise competitiveness. The model supports text-to-video, image-to-video, and text-to-image generation, showcasing strong generalization across modalities.

Conclusion

This research not only identifies and addresses core challenges in spatiotemporal modeling for video generation but also showcases how Lumos-1 sets a new standard for unifying efficiency and effectiveness in autoregressive frameworks. By successfully blending advanced architectures with innovative training, Lumos-1 paves the way for the next generation of scalable, high-quality video generation models and opens up new avenues for future multimodal research.

Check out the Paper and GitHub. All credit for this research goes to the researchers of this project.

Would you take the risk to miss this AI Dev Newsletter read by Devs and Researchers from NVIDIA, OpenAI, DeepMind, Meta, Microsoft, JP Morgan Chase, Amgen, Aflac, Wells Fargo, and hundreds more?