RAG Is Not a Silver Bullet

Retrieval-augmented generation has become the default answer to a common question: “How do I make an LLM answer questions about my data?”

Embed your documents. Store the vectors. Retrieve relevant chunks at query time. Feed them to the LLM. Get grounded answers.

It’s a compelling architecture. It’s also the architecture most teams reach for without questioning whether it’s the right fit, and then spend months debugging when the answers aren’t good enough.

RAG works. But it doesn’t work the way most people think it does, and it fails in ways that are easy to miss.

How RAG Actually Fails

The pitch for RAG is straightforward: give the LLM access to your data, and it will answer questions accurately. The reality is more complicated.

The Retrieval Step Is the Bottleneck

Most RAG quality problems aren’t generation problems. They’re retrieval problems. The LLM is perfectly capable of generating a good answer. It just didn’t receive the right context to do so.

This happens because semantic search (the vector similarity lookup at the core of most RAG systems) optimizes for relevance, not correctness. A chunk can be semantically similar to the query without containing the actual answer.

Consider a user asking: “What’s our refund policy for enterprise customers?”

Semantic search might return:

A chunk about the general refund policy (relevant, partially correct)
A chunk about enterprise pricing tiers (semantically similar, wrong topic)
A chunk about consumer refund policy (contains “refund” and “policy”, wrong customer segment)

The LLM synthesizes these chunks and produces a confident, well-structured answer that blends the general refund policy with enterprise pricing details. It sounds right. It’s not.

This is the quiet failure pattern applied to retrieval. The system doesn’t crash or error. It just gives a subtly wrong answer.

Chunking Is an Art, Not a Science

How you split documents into chunks determines what the retrieval step can find. And there’s no universal right answer.

Chunks too small: Individual sentences lose context. The retrieval finds the right sentence but the LLM can’t make sense of it without the surrounding paragraphs.

Chunks too large: The retrieval returns broad sections that contain the answer buried in noise. The LLM has to find the needle in the haystack, and sometimes it picks up the hay instead.

Chunks that split at wrong boundaries: A policy document split mid-paragraph can separate a condition from its exception. The LLM sees “refunds are available within 30 days” without seeing “except for custom orders” in the next chunk.

Most teams start with a fixed chunk size (500–1000 tokens) and never revisit it. But chunk size should be tuned to your data and your queries, not set once and forgotten.

The “I Don’t Know” Problem

RAG systems are bad at knowing when they don’t know. If the retrieval step returns chunks (and it almost always will, because semantic search always finds something), the LLM will generate an answer from whatever it received.

This means RAG systems rarely say “I don’t have information about that.” Instead, they extrapolate from tangentially related content and present the result with the same confidence as a well-grounded answer.

Detecting these cases requires explicit engineering: confidence scoring, retrieval quality thresholds, or LLM-as-judge evaluation. It’s not something that comes for free with the architecture.

When RAG Is the Right Choice

RAG genuinely shines in specific scenarios:

Large, relatively stable document collections. Knowledge bases, documentation sites, policy libraries, product catalogs. If you have hundreds or thousands of documents and users need to query across them, RAG handles this well.

Questions that have clear, localized answers. “What’s the return policy?” has an answer that lives in one place. RAG is great at finding and surfacing that answer.

When freshness matters more than depth. RAG lets you update the knowledge base without retraining or fine-tuning the model. Swap in new documents, re-embed, and the system reflects the latest information immediately.

When you need citations. RAG naturally supports showing users where the answer came from, which builds trust and allows verification.

When RAG Is the Wrong Choice

Questions That Require Reasoning Across Documents

“How has our pricing strategy evolved over the past three years?” isn’t answered by any single chunk. It requires synthesizing information scattered across multiple documents, understanding temporal relationships, and drawing conclusions.

RAG retrieves a handful of chunks. If the answer requires connecting dots across twenty documents, those dots won’t all fit in the retrieval window.

Alternative: For cross-document reasoning, consider pre-computing summaries, building structured knowledge representations, or using multi-step retrieval that iteratively gathers information.

Structured Data Queries

“How many enterprise customers signed up last quarter?” is a database query, not a retrieval task. RAG systems are text-in, text-out. If the answer lives in a database, SQL is more reliable than semantic search.

Alternative: Text-to-SQL or structured tool-use patterns let the LLM query databases directly, with the precision that comes from structured queries.

Small, Well-Defined Knowledge Sets

If your entire knowledge base is a single 10-page document, you don’t need RAG. Just put the document in the context window. Modern models handle 100k+ tokens. The overhead of embedding, indexing, and retrieval isn’t justified when the entire corpus fits in a single prompt.

Alternative: Direct context injection. Simpler, faster, and more reliable for small document sets.

Tasks That Need Deterministic Answers

If the answer must be exactly right every time, as with compliance checks, legal determinations, or financial calculations, RAG’s probabilistic retrieval introduces unacceptable risk. The retrieval might miss the relevant chunk, the LLM might misinterpret it, or the chunking might split a critical clause.

Alternative: Rule-based systems, structured lookups, or LLM-assisted workflows where the final answer is validated against a deterministic source.

Making RAG Better

If RAG is the right architecture for your use case, here’s how to make it actually work well.

Hybrid Retrieval

Don’t rely on semantic search alone. Combine it with keyword search (BM25 or similar) and re-rank the results. Semantic search captures meaning; keyword search captures exact terminology. Together, they cover each other’s blind spots.

A query for “SOC 2 compliance requirements” benefits from keyword matching (“SOC 2” is a specific term) combined with semantic understanding (“compliance requirements” could be phrased many ways in your documents).

Query Transformation

The user’s query is rarely the optimal search query. Transform it before searching:

Query expansion: “What’s the refund policy?” → search for “refund policy” AND “return policy” AND “money back guarantee”
Hypothetical document embedding (HyDE): Ask the LLM to write a hypothetical answer, then use that as the search query. This often retrieves better chunks because the hypothetical answer is closer in embedding space to the actual document.
Decomposition: Break complex queries into sub-queries. “Compare our enterprise and startup pricing” becomes two searches: one for enterprise pricing, one for startup pricing.

Better Chunking Strategies

Move beyond fixed-size chunks:

Semantic chunking: Split on topic boundaries rather than token counts. Use heading structure, paragraph breaks, or topic modeling.
Overlapping chunks: Include overlap between chunks so boundary information isn’t lost.
Parent-child chunks: Retrieve at the paragraph level but include the parent section for context. This gives the LLM both precision and context.
Document-specific strategies: A legal contract should be chunked differently than a FAQ page. Tune chunking to document type.

Retrieval Quality Scoring

Don’t blindly pass every retrieved chunk to the LLM. Score retrieval quality and filter:

Relevance threshold: If the best chunk’s similarity score is below a threshold, respond with “I don’t have information about that” instead of guessing.
Chunk deduplication: Remove near-duplicate chunks that waste context window space.
Re-ranking: Use a cross-encoder or a separate LLM call to re-rank chunks by actual relevance to the query, not just vector similarity.

Evaluation Is Non-Negotiable

RAG systems need the same evaluation infrastructure as any other LLM feature. Arguably more, because there are two failure points (retrieval and generation) instead of one.

Build a golden dataset of question-answer pairs. Track retrieval precision (did the right chunks get retrieved?) separately from generation quality (did the LLM produce the right answer from the chunks?). When quality drops, this separation tells you where to look.

The Honest Assessment

RAG is a good architecture. It’s not a magic architecture. The gap between “we set up a RAG pipeline” and “our RAG pipeline reliably answers user questions” is measured in months of engineering work: chunking tuning, retrieval optimization, evaluation infrastructure, and edge case handling.

If you’re building a knowledge-grounded AI feature, start by asking whether RAG is actually the right pattern for your specific use case. Sometimes the answer is a simpler architecture. Sometimes it’s a more complex one. And sometimes RAG is exactly right, but only if you’re prepared to invest in making it reliable, not just functional.

The retrieval is the product. The generation is just the presentation layer. Get the retrieval right, and the rest follows. Get it wrong, and no amount of prompt engineering will save you.