Your team built a RAG prototype in two weeks. In the demo it worked. In production, users complain about irrelevant answers, token consumption spikes and nobody trusts the responses. It's not an LLM problem. It's a retrieval architecture problem.
Retrieval-Augmented Generation promised to solve LLM hallucinations by connecting them to real data. And it does, when the architecture is right. But most enterprise implementations fall into the same three mistakes that turn RAG from an investment into a budget sink.
According to recent data, 80% of enterprise RAG projects experience critical failures in production. 42% of AI projects failed in 2025, representing $13.8 billion in enterprise spend at risk. And the root cause isn't technological: it's the gap between a prototype that impresses and a system that works at scale.
Mistake #1: Retrieval without data curation
The first mistake is the most basic and the most expensive: connecting an LLM to a knowledge base without curating the data that feeds it. Garbage in, garbage out, but now with a token cost for every garbage query.
In practice, this shows up in complaints like: "I asked about the updated vacation policy and it gave me the Q1 draft", or "It says we don't have a returns policy when we updated it two months ago". The LLM answers with what it finds. If it finds outdated, duplicated or contradictory documents, the answer reflects that chaos.
The critical data point: 80% of RAG failures originate in chunking decisions, not in retrieval or generation. Naive fixed-size chunking reaches a faithfulness score of 0.47-0.51. Optimized semantic chunking reaches 0.79-0.82. That difference is what separates a useful system from one that generates distrust.
How to avoid it
- Active curation: Don't index everything. Filter out obsolete documents, remove duplicates, normalize formats before ingesting.
- Semantic chunking: Split documents by units of meaning (sections, thematic paragraphs), not by character count.
- Enriched metadata: Each chunk should carry date, source, version and category. Without metadata, the retriever can't prioritize the right document.
Mistake #2: Ignoring re-ranking and relevance validation
The second mistake is trusting the retriever blindly. A vector search returns the semantically "closest" chunks, but semantic closeness is not the same as relevance to the user's question.
Without re-ranking, the LLM receives context that is vaguely related but not directly useful. The result: generic answers, inflated context that consumes unnecessary tokens, and user trust that drops fast.
Hybrid architecture: the solution that works
The architecture we see working in production combines three layers:
- Hybrid retrieval (dense + sparse): Vector search (embeddings) to capture semantic meaning + BM25 for exact term matching. Reciprocal Rank Fusion (RRF) combines both rankings into one.
- Semantic re-ranking: A transformer model (ColBERT, Cohere Rerank, or similar) reorders the results by real relevance to the query. It filters out the noise before it reaches the LLM.
- Relevance validation: A minimum relevance score for each chunk. If no result passes the threshold, the system answers "I don't have enough information" instead of hallucinating.
User query
│
├─→ Dense retrieval (embeddings) ──→ Top 20 candidates
│
├─→ Sparse retrieval (BM25) ──────→ Top 20 candidates
│
└─→ Reciprocal Rank Fusion ───────→ Top 20 fused
│
Re-ranker (ColBERT)
│
Top 5 relevant
│
Score > threshold?
│ │
Yes No
│ │
LLM generates "I don't have enough
answer information to answer" Mistake #3: Simple RAG when the case requires agency
The third mistake is applying naive RAG to problems that require multi-step reasoning. Questions like "What's the difference between our 2024 travel policy and our 2025 one?" need to retrieve two documents, compare them and synthesize the differences. A simple RAG retrieves chunks and passes them to the LLM with no intermediate reasoning capability.
But the opposite mistake also exists: teams that jump straight to Agentic RAG when a simple pipeline would suffice. Every agentic step consumes additional tokens. Every tool call adds latency and cost.
The compound cost: In a system with cascading failures where each layer has 95% precision, total reliability drops to 81%. Every agentic layer you add multiplies the failure points and token consumption. In 2024, 90% of agentic RAG projects failed in production precisely because of this compound effect.
When to scale from simple to agentic RAG
| Query type | Architecture | Why |
|---|---|---|
| Direct factual question | Simple RAG | One retrieval + one generation. No agentic overhead. |
| Comparison across documents | Agentic RAG | Needs multiple retrievals and intermediate reasoning. |
| Summary of a document | Simple RAG | The document is retrieved and summarized in one step. |
| Cross-source analysis | Agentic RAG | Requires planning which sources to query and in what order. |
| FAQ about internal policies | Simple RAG | Direct answers with low cost per query. |
| Research with multi-hop reasoning | Agentic RAG | The answer depends on chaining intermediate findings. |
Benchmark tools: measure before going to production
One of the most expensive patterns we see is teams deploying RAG to production without an evaluation framework. They discover the failures when users complain, not when they could have prevented them.
RAGAS (Retrieval-Augmented Generation Assessment Suite) has become the de facto standard for RAG evaluation, with more than 400,000 monthly downloads and 20 million evaluations executed. What's relevant for teams that want to avoid wasting budget: RAGAS evaluates quality without the need for manual labels.
The 4 metrics you should measure before production
- Faithfulness: Is the answer faithfully based on the retrieved context, or does it add invented information? It's the anti-hallucination metric.
- Context Precision: Are the retrieved chunks relevant to the question? A low score means you're sending noise to the LLM and paying tokens for useless context.
- Context Recall: Is all the information needed to answer being retrieved? A low score means the retriever isn't finding what it should.
- Answer Relevance: Does the generated answer actually address the user's question? It measures the gap between intent and output.
Rule of thumb: If your faithfulness score is below 0.75, don't put the system in production. Every point below is a hallucination your users will find, and every hallucination erodes trust faster than it costs to fix it.
ROI: when RAG justifies the investment
RAG isn't always the right answer. Sometimes fine-tuning is more efficient. Sometimes a traditional search with good UX is enough. The question isn't "can we implement RAG?" but "does RAG give us a return on investment in this case?"
RAG: positive ROI vs overhead
Positive ROI
- Knowledge base that changes frequently (not viable with fine-tuning)
- High volume of repetitive queries over internal documentation
- Need for traceability (knowing which document each answer comes from)
- Multiple data sources that must be queried in a unified way
Probably overhead
- Static, small knowledge base (fine-tuning is more efficient)
- Few queries per day (the infrastructure cost isn't amortized)
- Unstructured data with no possibility of curation (permanent garbage in)
- The team doesn't have the capacity to maintain the pipeline in production
Conclusion: RAG works when the architecture is honest
RAG isn't magic. It's an architecture with components that can fail independently and whose cost accumulates at each layer. The teams that get real value from RAG are the ones that treat every component with the same rigor as any production system:
- They curate data before indexing, not after users complain.
- They implement re-ranking instead of trusting the retriever blindly.
- They choose the right amount of complexity for each query type, without agentic over-engineering when a simple RAG is enough.
- They measure before deploying with RAGAS or another evaluation framework.
71% of organizations use GenAI regularly, but only 17% attribute more than 5% of their EBIT to GenAI. The gap between using AI and getting a return from it closes with the right architecture, not with more tokens.
A well-designed RAG reduces costs and improves answers. A badly designed RAG only multiplies the token bill.
Jordi García is Tech Lead at onext. He works on bringing AI into governed production across development and product teams —with Spec-Driven Development, context engineering and human verification at every step— and authors onext's technical insights on the method, quality and cost of applied AI.
LinkedIn →