How Radial Attention Cuts Costs in Video Diffusion by 4.4× Without Sacrificing Quality

Introduction to Video Diffusion Models and Computational Challenges

Diffusion models have made significant advances in generating high-quality, coherent videos, building on their success in image synthesis. However, the extra temporal dimension in videos increases computational demands, particularly as self-attention scales poorly with sequence length. This poses challenges for training and running these models efficiently on longer videos. Approaches like Sparse VideoGen utilize attention head classification to accelerate inference but often struggle with accuracy and generalization during training. Other methods replace softmax attention with linear alternatives, which can require substantial architectural changes. Recent research inspired by the natural energy decay of signals over time in physics suggests new, more efficient modeling strategies.

Evolution of Attention Mechanisms in Video Synthesis

Early video models enhanced 2D architectures by adding temporal components, while newer approaches, such as DiT and Latte, have improved spatial-temporal modeling with advanced attention mechanisms. Although 3D dense attention achieves state-of-the-art performance, its computational cost increases rapidly with video length, making long video generation expensive. Techniques such as timestep distillation, quantization, and sparse attention help alleviate this burden but often neglect the unique structure of video data. Alternatives like linear or hierarchical attention improve efficiency but typically struggle to maintain detail or scale effectively in practice.

Introduction to Spatiotemporal Energy Decay and Radial Attention

Researchers from MIT, NVIDIA, Princeton, UC Berkeley, Stanford, and First Intelligence have identified a phenomenon in video diffusion models called Spatiotemporal Energy Decay. This principle observes that attention scores between tokens decline as spatial or temporal distance increases, which mirrors the natural fading of signals over time. In response to this, they proposed Radial Attention, a sparse attention mechanism with O(n log n) complexity. This approach employs a static attention mask, allowing tokens to primarily attend to nearby ones, with the attention window decreasing over time. This method enables pre-trained models to generate videos up to four times longer, significantly reducing training costs by 4.4× and inference time by 3.7× while preserving video quality.

Sparse Attention Using Energy Decay Principles

Radial Attention relies on the insight that attention scores in video models decrease with increasing spatial and temporal distance, termed Spatiotemporal Energy Decay. Instead of attending to all tokens uniformly, Radial Attention strategically minimizes computation where attention is weaker. It introduces a sparse attention mask that decays exponentially outward in both space and time, focusing on the most relevant interactions. This results in an O(n log n) complexity, making it significantly faster and more efficient than dense attention. Additionally, with minimal fine-tuning using LoRA adapters, pre-trained models can effectively adapt to generate much longer videos.

Evaluation Across Video Diffusion Models

Radial Attention has been evaluated on three leading text-to-video diffusion models: Mochi 1, HunyuanVideo, and Wan2.1, showcasing both speed and quality enhancements. When compared to existing sparse attention baselines like SVG and PowerAttention, Radial Attention offers improved perceptual quality and considerable computational gains, achieving up to 3.7× faster inference and 4.4× lower training costs for extended videos. It scales effectively to 4× longer video lengths and maintains compatibility with existing LoRAs, including style-specific adaptations. Notably, LoRA fine-tuning with Radial Attention has shown to outperform full fine-tuning in certain cases, highlighting its effectiveness and resource efficiency for high-quality long-video generation.

Conclusion: Scalable and Efficient Long Video Generation

In summary, Radial Attention is a sparse attention mechanism designed to manage long video generation in diffusion models with enhanced efficiency. Inspired by the observed decline in attention scores with increasing spatial and temporal distances, this approach mimics natural decay to reduce computational load. Utilizing a static attention pattern with exponentially shrinking windows, it achieves performance improvements of up to 1.9× while supporting videos up to 4× longer. With lightweight LoRA-based fine-tuning, it significantly cuts training costs by 4.4× and inference costs by 3.7×, all while maintaining video quality across multiple state-of-the-art diffusion models.

For more information, check out the Paper and GitHub Page. All credit for this research goes to the researchers of this project. You can also follow us on Twitter, YouTube, and Spotify. Don’t forget to join our 100k+ ML SubReddit and subscribe to our newsletter.