Every RAG tutorial teaches the same pipeline: split documents into chunks, embed them, retrieve the top-k nearest neighbours, stuff them into the prompt. It works beautifully on the tutorial's dataset. Then you point it at a real company's documents and accuracy falls off a cliff. Having built retrieval systems across support, legal, and internal-knowledge use cases, we see the same failure modes every time.
The five ways naive RAG breaks
- Tables become word soup. Fixed-size chunking shreds tables, and an embedded fragment of a pricing table matches almost nothing. Structure-aware ingestion — keeping tables intact, carrying headings as metadata — fixes more accuracy problems than any model upgrade.
- Semantic search misses exact terms. Embeddings are great at "how do I cancel my plan" but terrible at part numbers, error codes, and clause references. Production retrieval is hybrid: dense vectors plus keyword (BM25) plus metadata filters, merged and reranked.
- Versions contradict each other. The 2024 policy and the 2026 policy both live in the corpus, and cosine similarity has no opinion about which is current. Freshness and supersedence have to be modelled explicitly — retrieval filters by validity, not just relevance.
- Permissions are an afterthought. If HR documents and public FAQs share one index without per-user filtering, your chatbot is one question away from a leak. Access control must be inherited from source systems and enforced at retrieval time, not in the prompt.
- Nobody measures anything. "It seems to answer well" is how RAG systems go to production and quietly hallucinate for months. Without an evaluation set of real questions scored for retrieval hit-rate and answer faithfulness, you cannot tell tuning from vibes.
The stack that holds up
The production version isn't exotic, it's just complete: structure-preserving ingestion pipelines, hybrid retrieval with reranking, metadata for freshness and permissions, citations on every answer, and a regression suite of real questions that runs on every change. Each piece is unglamorous; together they're the difference between a demo and a system your support team trusts.
Retrieval quality is an engineering property, not a model property. The model can only be as accurate as the passages you hand it.
We describe how we build this — including permission-aware retrieval and evaluation — on our RAG chatbot development page. And if your agents need to act on that knowledge, not just answer from it, see how we combine retrieval with agentic systems.