itinai content

←back to Blog

EPFL Researchers Introduce MEMOIR: A Scalable Framework for Lifelong Model Editing in LLMs

EPFL Researchers Introduce MEMOIR: A Scalable Framework for Lifelong Model Editing in LLMs

Understanding the Target Audience

The target audience for the MEMOIR framework primarily includes AI researchers, data scientists, and business leaders interested in the practical applications of large language models (LLMs). These individuals are typically involved in the development and implementation of AI technologies within their organizations. Their pain points include:

  • Difficulty in keeping LLMs updated with current knowledge.
  • Challenges related to biases and inaccuracies in model outputs.
  • High costs and inefficiencies associated with traditional fine-tuning methods.

Their goals are to:

  • Enhance the reliability and accuracy of AI models.
  • Implement scalable solutions for continuous model updates.
  • Reduce operational costs while maintaining model performance.

Interests include advancements in AI methodologies, practical applications of LLMs in business, and innovative techniques for model management. Communication preferences lean towards concise, technical content that provides actionable insights and empirical data.

The Challenge of Updating LLM Knowledge

Large language models (LLMs) have demonstrated exceptional performance across various tasks due to extensive pre-training on large datasets. However, these models often produce outdated or inaccurate information and may reflect inherent biases when deployed. Continuous updates to their knowledge base are essential. Traditional fine-tuning methods can be costly and prone to catastrophic forgetting, which has led to the exploration of lifelong model editing as a more efficient and localized approach to updating model knowledge.

To ensure accurate predictions, each edit must exhibit reliability, generalizability, and localization. Non-parametric methods achieve precise localized edits but lack generalization, while parametric methods provide better generalization at the cost of catastrophic forgetting.

Limitations of Prior Model Editing Techniques

Previous research has investigated sparse neural activations in continual learning. Techniques such as PackNet and Supermasks-in-Superposition allocate distinct parameter subsets for each task. Gradient-based methods like GPM and SPARCL enhance efficiency through orthogonal updates but are limited to continual learning scenarios. Parametric approaches, including ROME, MEMIT, and WISE, modify weights through locating-then-editing strategies or auxiliary modules but struggle with forgetting over extended edit sequences. Non-parametric methods like GRACE and LOKA store knowledge externally to preserve original weights, allowing for precise local edits, but they depend on exact input matches, limiting their generalization capabilities.

Introducing MEMOIR: A Structured Approach to Model Editing

Researchers from EPFL, Lausanne, Switzerland, have developed MEMOIR (Model Editing with Minimal Overwrite and Informed Retention), which strikes an optimal balance between reliability, generalization, and locality for large-scale edits. MEMOIR introduces a memory module consisting of a fully-connected layer within a single transformer block where all edits occur. This framework addresses catastrophic forgetting by assigning distinct parameter subsets for each edit and retrieving them during inference to activate only relevant knowledge for specific prompts. Additionally, MEMOIR employs structured sparsification with sample-dependent masks during editing, activating only prompt-specific parameter subsets and distributing new knowledge across the parameter space to minimize overwriting.

Evaluation and Experimental Results

MEMOIR operates through a residual memory framework during inference, where the edited output integrates original layer outputs with residual memory outputs. It is evaluated against several baselines, including GRACE for external knowledge storage, DEFER for inference-time routing, and various memory-based methods. Experiments are conducted on four autoregressive language models: LLaMA-3-8B-Instruct, Mistral-7B, LLaMA-2-7B, and GPT-J-6B, providing a comprehensive evaluation across different models and scales.

On the ZsRE question-answering dataset, MEMOIR achieves an average metric of 0.95 on LLaMA-3 with 1000 edits, outperforming all prior methods by a margin of 0.16. Similar results are observed with Mistral, where MEMOIR again achieves the highest average score, demonstrating its robustness across various LLMs. Furthermore, MEMOIR maintains optimal balanced performance with increasing edit volumes for hallucination correction using the SelfCheckGPT dataset. It sustains saturated locality scores under challenging scenarios of 600 edits while achieving perplexity metrics 57% and 77% lower than WISE on LLaMA-3 and Mistral, respectively.

Conclusion and Future Directions

In summary, MEMOIR is a scalable framework for lifelong model editing that effectively balances reliability, generalization, and locality through innovative sparsification techniques. The method retrieves relevant updates via sparse activation pattern comparison, allowing edits to generalize to rephrased queries while preserving model behavior on unrelated prompts. However, limitations exist, such as the modification of only single linear layers, which may restrict the handling of long-horizon edits or knowledge requiring broader model changes. Future directions involve extending the approach to multiple layers, hierarchical editing strategies, and applications to multi-modal or encoder-decoder models beyond the current focus on decoder-only transformers.

Check out the Paper. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 100k+ ML SubReddit and Subscribe to our Newsletter.