RAG in practice: why retrieval matters more than the model

An assistant answering from your own documentation has two parts: retrieval that finds passages, and a model that turns them into an answer. When the answer is bad, it is usually the first part.

Measure retrieval before swapping models

The decisive question: is the correct answer even among the returned passages? That metric can be checked without a model at all. If it sits at 60 %, no language model on earth will produce reliable answers from it.

1def hit_rate(cases, retrieve, k: int = 5) -> float:
2    """Share of queries where the correct passage lands in the top k."""
3    hits = 0
4    for case in cases:
5        passages = retrieve(case.query, k=k)
6        if any(p.id == case.expected_id for p in passages):
7            hits += 1
8    return hits / len(cases)
9
10
11# Targets from practice
12#   > 90 %   solid foundation
13#   70–90 %  review chunking and metadata
14#   < 70 %   rebuild the index

[Image: Four connected boxes: query, retrieval, reranking, answer — The four stages of a query. Most quality problems appear between step two and three.]

The most common causes

• Chunks too large: a 4,000-character block dilutes the vector into uselessness.
• Missing metadata: without product, version and date, retrieval hits outdated revisions.
• Purely semantic search: for error codes and part numbers, classic keyword matching clearly beats vector search.
• No reranking: a cross-encoder sorts the first twenty candidates far better than raw similarity.

Only once the hit rate is right does work on the answering part pay off: enforce citations, allow uncertainty, and let the model say the information is not there.