What is RAG, and when should a business use it?

How RAG works, step by step

A language model only knows what it learned in training and what it receives in its context window. RAG takes care of the second part: for every question, it finds what matters and puts it in the context. The work splits into preparing the data, once and on every change, and answering, on every question.

1. Ingestion and chunking: Documents are extracted, cleaned and split into passages (chunks), sized and overlapped so an idea is not cut in half.
2. Embeddings: An embedding model turns each passage into a numeric vector that represents its meaning.
3. Indexing: Vectors, text and metadata (source, date, permissions) go into a vector store, such as pgvector on PostgreSQL.
4. Hybrid search: The question is matched by meaning (vectors) and by keywords, which finds paraphrases as well as exact codes, acronyms and names.
5. Reranking: A model reorders the candidates by relevance to the question, and only the best ones enter the context.
6. Generation with citations: The model answers from the passages it received and shows which document each statement came from.

When a business should use RAG

RAG is the natural choice when the answer depends on the company's knowledge rather than the model's general knowledge. The most common signs:

• Your own knowledge base: Manuals, policies, contracts, tickets or technical documentation the model has never seen.
• Information that changes: Catalogs, rules and procedures updated often; you reindex instead of retraining the model.
• Answers that must be audited: The person asking needs to see the source, as in legal, compliance, technical support and customer service.
• A large corpus: The content does not fit in the context window, or would fit only at a high cost per question.

When the data is tabular, such as orders, balances or metrics, retrieval is usually a structured query (SQL or an API) rather than a similarity search. In Contas Casal, Master Timm's own product, the assistant runs on a structured RAG pipeline: intent routing, 10 parallel SQL retrieval tools and a semantic cache.

RAG, fine-tuning or an agent: what each one solves

The three are often confused, but they change different parts of the system. RAG changes what the model knows at question time. Fine-tuning changes how the model behaves: tone, format or a narrow task. An agent changes what the model can do, by calling tools over several steps.

• RAG: Answer what the current return policy is, citing the document and its version.
• Fine-tuning: Sort tickets into internal categories, when an eval shows a gain over prompt and context.
• Agent: Look up the order in the ERP, check the policy and open the return, with approval on sensitive actions.

They combine: an agent can have RAG search as one of its tools and decide when to use it. The choice between RAG and fine-tuning for company data is covered in detail on the AI & LLM engineering page, linked right below.

Why RAG fails, and how to prevent it

When RAG fails, the cause is usually retrieval, not the model: if the right passage never reaches the context, the model answers with what it has, and the mistake looks like a correct answer.

• Badly cut chunks: The rule sits in one passage and the exception in another; splitting by the structure of the document, with overlap, fixes much of it.
• Different vocabulary: People phrase things one way and documents another; hybrid search and rewriting the question help.
• Stale or duplicate sources: Two versions of the same policy compete; date and version metadata and a cleaned-up corpus prevent the mix-up.
• Ignored permissions: Search returns a passage the person may not see; the access filter has to apply inside the search itself.
• Hidden instructions: A document carries text that tries to change how the model behaves (indirect prompt injection); retrieved content is treated as data, never as a command.
• Calculation questions: Sums, averages and rankings call for SQL, not text similarity.

How to measure RAG quality

RAG is measured in two parts, because each one fails in its own way: retrieval and the answer. Both use the same evaluation dataset, with real questions, the expected answer and the document that should be found.

• Retrieval: The right passage shows up among the top results (recall@k), and irrelevant ones stay out.
• Faithfulness: Every statement in the answer is supported by the retrieved passages, with nothing made up.
• Relevance and citation: The answer addresses the question and points to the right source.
• Operations: Latency, tokens and cost per question, tracked through traces in production.

The evaluations combine code checks, sampled human review and LLM-as-a-judge, and they run on every change to chunking, the embedding model, the prompt or the language model.