Google DeepMind Finds a Fundamental Bug in RAG: Embedding Limits Break Retrieval at Scale

Retrieval-Augmented Generation (RAG) systems commonly rely on dense embedding models that map queries and documents into fixed-dimensional vector spaces. A recent research study from the Google DeepMind team highlights a fundamental architectural limitation that cannot be resolved solely by larger models or improved training.

What Is the Theoretical Limit of Embedding Dimensions?

The core issue lies in the representational capacity of fixed-size embeddings. An embedding of dimension d cannot represent all possible combinations of relevant documents once the database surpasses a critical size. This principle is supported by findings in communication complexity and sign-rank theory.

For embeddings of size 512, retrieval fails around 500K documents. For 1024 dimensions, the limit extends to about 4 million documents. For 4096 dimensions, the theoretical ceiling is 250 million documents. These estimates are derived under optimal conditions, where vectors are directly optimized against test labels. However, real-world language-constrained embeddings tend to fail even earlier.

How Does the LIMIT Benchmark Expose This Problem?

To empirically test this limitation, the Google DeepMind team introduced the LIMIT (Limitations of Embeddings in Information Retrieval) benchmark dataset designed to stress-test embedders. LIMIT includes two configurations:

• LIMIT full (50K documents): In this large-scale setup, even strong embedders struggle, with recall@100 often dropping below 20%.
• LIMIT small (46 documents): Despite the simplicity of this small setup, models still fail to solve the task. Performance varies widely but remains unreliable, with the best models achieving:
• Promptriever Llama3 8B: 54.3% recall@2 (4096d)
• GritLM 7B: 38.4% recall@2 (4096d)
• E5-Mistral 7B: 29.5% recall@2 (4096d)
• Gemini Embed: 33.7% recall@2 (3072d)

Even with just 46 documents, no embedder achieves full recall, indicating that the limitation is not solely due to dataset size but the inherent design of single-vector embedding architecture.

In contrast, BM25, a classical sparse lexical model, does not face this ceiling since sparse models operate in effectively unbounded dimensional spaces, allowing them to capture combinations that dense embeddings cannot.

Why Does This Matter for RAG?

Current RAG implementations typically assume that embeddings can scale indefinitely with more data. The research from Google DeepMind clarifies that this assumption is flawed: embedding size inherently constrains retrieval capacity. This limitation impacts:

• Enterprise search engines managing millions of documents.
• Agentic systems that rely on complex logical queries.
• Instruction-following retrieval tasks, where queries dynamically define relevance.

Even advanced benchmarks like MTEB fail to capture these limitations as they test only a narrow subset of query-document combinations.

What Are the Alternatives to Single-Vector Embeddings?

The research team suggests that scalable retrieval solutions must move beyond single-vector embeddings:

• Cross-Encoders: Achieve perfect recall on LIMIT by directly scoring query-document pairs, but at the cost of high inference latency.
• Multi-Vector Models (e.g., ColBERT): Provide more expressive retrieval by assigning multiple vectors per sequence, enhancing performance on LIMIT tasks.
• Sparse Models (BM25, TF-IDF, neural sparse retrievers): Scale better in high-dimensional search but lack semantic generalization.

The key insight is that architectural innovation is necessary, rather than simply increasing the size of embedders.

Key Takeaway

The analysis demonstrates that dense embeddings, despite their success, are bound by a mathematical limit: they cannot capture all possible relevance combinations once corpus sizes exceed limits tied to embedding dimensionality. The LIMIT benchmark illustrates this failure concretely:

• On LIMIT full (50K docs): recall@100 drops below 20%.
• On LIMIT small (46 docs): even the best models max out at approximately 54% recall@2.

Classical techniques like BM25, as well as newer architectures such as multi-vector retrievers and cross-encoders, remain essential for developing reliable retrieval engines at scale.

For further reading, check out the research paper. Explore our GitHub Page for tutorials, codes, and notebooks. Follow us on Twitter, and don’t forget to join our 100k+ ML SubReddit and subscribe to our newsletter.