IBM and ETH Zürich Researchers Unveil Analog Foundation Models to Tackle Noise in In-Memory AI Hardware

Understanding the Target Audience

The target audience for this research includes AI researchers, data scientists, and business leaders in technology sectors focused on AI and machine learning. These individuals are typically engaged in developing or implementing AI solutions and are interested in advancements that can enhance computational efficiency and model performance.

Pain Points

• Challenges with noise in analog computing affecting model accuracy.
• Limitations of current hardware in scaling large language models (LLMs).
• Need for energy-efficient solutions for edge devices.

Goals

• To leverage advanced AI models that can operate effectively on compact hardware.
• To improve the robustness of AI models against noise and variability.
• To explore new architectures that can support larger models without compromising performance.

Interests

• Innovations in AI hardware and software integration.
• Research findings that can be applied to real-world business challenges.
• Collaborative efforts between academia and industry to advance AI technologies.

Communication Preferences

The audience prefers clear, concise, and technical communication that includes data-driven insights and practical applications. They value peer-reviewed research and case studies that demonstrate the effectiveness of new technologies.

Overview of Analog Foundation Models

IBM researchers, in collaboration with ETH Zürich, have introduced a new class of Analog Foundation Models (AFMs) aimed at addressing the noise issues inherent in Analog In-Memory Computing (AIMC) hardware. AIMC has the potential to significantly enhance efficiency by enabling the execution of models with a billion parameters in a compact footprint suitable for embedded or edge devices. However, noise has been a critical barrier, as matrix-vector multiplications performed directly within non-volatile memory (NVM) devices often result in non-deterministic errors that hinder the performance of existing models.

The Importance of Analog Computing for LLMs

Unlike traditional computing methods using GPUs or TPUs, AIMC performs matrix-vector multiplications directly within memory arrays, eliminating the von Neumann bottleneck and significantly improving throughput and power efficiency. Previous studies indicated that combining AIMC with 3D NVM and Mixture-of-Experts (MoE) architectures could theoretically support trillion-parameter models on compact accelerators, making large-scale AI feasible beyond data centers.

Challenges in Implementing AIMC

The primary challenge in utilizing AIMC is the presence of noise. AIMC computations are affected by device variability, DAC/ADC quantization, and runtime fluctuations, which can degrade model accuracy. Unlike quantization on GPUs, where errors are predictable, analog noise is stochastic and unpredictable. While earlier research adapted smaller networks like CNNs and RNNs (less than 100M parameters) to tolerate such noise, LLMs with billions of parameters have struggled under AIMC constraints.

Addressing Noise with Analog Foundation Models

The IBM team has developed AFMs that incorporate hardware-aware training to prepare LLMs for analog execution. Their training pipeline includes:

• Noise injection during training to simulate AIMC randomness.
• Iterative weight clipping to stabilize distributions within device limits.
• Learned static input/output quantization ranges aligned with real hardware constraints.
• Distillation from pre-trained LLMs using 20B tokens of synthetic data.

These methods, implemented with AIHWKIT-Lightning, enable models like Phi-3-mini-4k-instruct and Llama-3.2-1B-Instruct to maintain performance comparable to weight-quantized 4-bit / activation 8-bit baselines under analog noise. Evaluations across reasoning and factual benchmarks indicate that AFMs outperform both quantization-aware training (QAT) and post-training quantization (SpinQuant).

Compatibility with Digital Hardware

Interestingly, AFMs also demonstrate strong performance on low-precision digital hardware. Because AFMs are trained to withstand noise and clipping, they manage simple post-training round-to-nearest (RTN) quantization more effectively than existing methods. This adaptability makes them valuable not only for AIMC accelerators but also for standard digital inference hardware.

Scalability of Performance

Yes, performance can scale with increased compute at inference time. Researchers tested compute scaling on the MATH-500 benchmark, generating multiple answers per query and selecting the best using a reward model. AFMs exhibited better scaling behavior than QAT models, with accuracy gaps diminishing as more inference compute was allocated. This aligns with AIMC’s strengths in low-power, high-throughput inference rather than training.

Future Implications for AIMC

This research represents the first systematic demonstration that large LLMs can be adapted to AIMC hardware without significant accuracy loss. While training AFMs is resource-intensive and reasoning tasks like GSM8K still reveal accuracy gaps, the findings mark a significant milestone. The combination of energy efficiency, robustness to noise, and compatibility with digital hardware positions AFMs as a promising avenue for scaling foundation models beyond the limitations of GPU technology.

Further Reading

For more detailed insights, refer to the research paper and explore the GitHub page for tutorials, codes, and notebooks.