What is the difference between RAG and fine-tuning?

Retrieval-augmented generation (RAG) looks up relevant passages from your documents at the moment a question is asked and gives them to the model to answer from. Fine-tuning changes the model itself by training it further on examples you provide. RAG is about what the model knows right now. Fine-tuning is about how the model behaves: its format, tone and handling of a narrow task.

Can fine-tuning teach an LLM my company's facts?

Not reliably. Fine-tuning is good at teaching patterns, such as a house style, an output format or how to classify a certain kind of request. It is a poor way to store facts, because the model cannot tell you where an answer came from, it may blend or misremember details, and every change to the facts means training again. For facts, retrieval is the better tool.

Is RAG or fine-tuning better for data privacy?

RAG is usually easier to control. Your documents stay in a store you manage, you can delete a document and it stops appearing in answers, and you can filter results by each user's permissions. Once data is trained into a model's weights, you cannot remove one document or restrict it to certain users, and the model may repeat training data to anyone who asks the right question.

Do I still need RAG now that models have very long context windows?

Sometimes not. If your material is small, fairly stable and everyone may see all of it, putting it directly into the prompt can be simpler and good enough, especially with prompt caching to reduce cost. Once the material is large, changes often, or different users may see different documents, retrieval becomes the practical choice, because it sends only what is relevant and permitted.

How do you evaluate a RAG system?

Build a set of real questions with known correct answers and the documents that support them. Then check two things separately: whether retrieval brought back the right passages, and whether the answer is correct and supported by those passages. Include questions the documents cannot answer, to confirm the system says so. Run the set before launch and after every change.

RAG vs Fine-Tuning for Your Company Data

RAG vs fine-tuning comes down to this: retrieval-augmented generation (RAG) gives a language model your information at the moment it answers, while fine-tuning changes how the model behaves by training it on examples. If you want a custom chatbot with company data that answers from current documents and shows its sources, you almost always want RAG. If you want the model to follow a particular format, tone or narrow task more consistently, fine-tuning can help. Many production systems use RAG for facts and, occasionally, fine-tuning for behaviour.

That covers the decision for most teams. The rest of this post explains why, compares the two side by side, and spends most of its time on the part that actually decides whether the project works: the ways RAG fails and how to test for them.

The plain explanation

A general-purpose language model knows a great deal about the world and nothing about your business. It has never seen your returns policy, your product specs or last month's price list. There are two broad ways to close that gap.

What RAG does

Retrieval-augmented generation works like an open-book exam. Before the model answers, the system searches your documents for the passages most likely to be relevant, then passes those passages to the model along with the question and an instruction along the lines of "answer using only this material, and say which parts you used".

To make that search work, your documents are split into smaller pieces, usually called chunks, and indexed so they can be found by meaning as well as by keywords. When a question arrives, the best-matching chunks are retrieved, sometimes re-ranked for relevance, and handed to the model.

The model itself does not change. If a document is updated, you update the index, and the next answer reflects it.

What fine-tuning does

Fine-tuning is more like training a new hire on a stack of worked examples. You take an existing model and train it further on pairs of inputs and ideal outputs: a support message and the reply you would want, a messy document and the clean structured record you would extract from it.

The result is a model that behaves differently. It picks up the pattern, the format and the tone. What it does not do well is memorise facts in a way you can rely on, cite, or update. It learns how to answer, not what is currently true.

RAG vs fine-tuning: a side-by-side comparison

	RAG	Fine-tuning
Freshness	As fresh as your index. Update a document and the next answer can use it.	Frozen at training time. New facts mean training again.
Cost profile	Ongoing cost for search infrastructure and longer prompts on every request. Little upfront cost.	Upfront cost to prepare data and train, plus retraining when things change. Prompts can be shorter afterwards.
Accuracy on facts	Strong, when retrieval finds the right passage. Answers can cite sources you can check.	Weak and hard to verify. The model may blend or misremember details and cannot point to a source.
Style and format control	Moderate. Instructions and examples in the prompt go a long way.	Strong. Good at consistent formats, tone and narrow, repeated tasks.
Data privacy	Documents stay in a store you control. You can delete them and filter by user permissions.	Data becomes part of the model's weights. You cannot remove one document or restrict who sees it.
Effort	Most effort goes into document preparation, search quality and evaluation.	Most effort goes into collecting and cleaning enough good training examples.

Two things stand out. Retrieval wins on almost everything to do with facts, and fine-tuning wins on consistency of behaviour. They are solving different problems, which is why "which one" is often the wrong question.

There is also a third option worth naming. If your material is small, stable and everyone may see all of it, current models can often take it directly in the prompt. That skips the retrieval system entirely. It stops being practical as the material grows, changes often, or needs different permissions for different people.

When to combine them

Combining the two makes sense when you have both a facts problem and a behaviour problem, and prompting alone has not fixed the behaviour.

Say an insurer wants an assistant that drafts replies to policyholder questions. The facts, such as policy wording, current excesses and claim status, must come from live documents and systems, so that is retrieval. The replies also need to follow a strict structure that compliance has signed off, in a particular tone, every time. If careful instructions and examples in the prompt still produce drift, a fine-tuned model can make that structure more consistent, while retrieval keeps supplying the facts.

The order matters. Build the retrieval and the evaluation first, try to get the behaviour you want through prompting, and only fine-tune if a measurable gap remains. Fine-tuning is the harder step to undo, and it is easy to spend effort on it before discovering the real problem was that retrieval was returning the wrong passages.

Where RAG development goes wrong

Most RAG systems that disappoint in production fail in one of a few predictable ways. None of them is about the model.

Bad chunking

If documents are split carelessly, the passage that answers a question can end up cut in half, or separated from the heading that gives it meaning. A table split across two chunks, a clause separated from the exception in the next paragraph, a product spec that no longer says which product it belongs to.

The fix is to split along the structure of the document, such as sections, headings and table boundaries, rather than at a fixed character count, and to carry useful context like the document title and section name with each chunk. Then test it, because the right approach varies by document type.

A stale index

The documents change, the index does not. Someone updates the pricing sheet, but the assistant keeps quoting last quarter's numbers because the re-indexing job failed quietly three weeks ago, or was never set up.

Treat the index like any other data pipeline. Re-index on change where you can, on a schedule where you cannot, remove documents that have been deleted at the source, and alert when a sync fails. Showing the source and its last-updated date alongside each answer also helps users spot stale information.

No evaluation

This is the most common failure, and it hides the others. The team tries a dozen questions, the answers look good, and it ships. Later, someone changes the chunk size or swaps the model, and nobody can say whether things got better or worse.

Without a fixed test set, every change is a guess. The next section covers what that test set should contain.

Missing access control

A company-wide assistant can only be as safe as its retrieval. If the index holds HR files, board papers and customer contracts, and the search does not check who is asking, any employee can get the contents of any document by asking the right question.

The fix is to filter retrieval by the user's permissions before anything reaches the model, ideally by carrying the source system's permissions into the index. Asking the model to "not reveal" restricted material is not a control. If a passage reaches the prompt, assume it can reach the answer.

Search that misses the obvious

Meaning-based search is good at paraphrases and poor at exact identifiers. A question about part number "AX-4471" or a specific clause number may not find the right chunk. Combining meaning-based search with keyword search, and re-ranking the results, usually fixes this.

How to evaluate answers

A useful evaluation set is not large. A few dozen to a few hundred real questions, collected from the people who will use the system, is enough to start. For each one, record the correct answer and the document or passage that supports it.

Then check each stage separately:

Retrieval. Did the right passage appear in what was retrieved? If not, no model can answer correctly, and the fix is in chunking or search, not the prompt.
Faithfulness. Is everything in the answer supported by the retrieved passages? An answer that is correct but unsupported is a lucky guess, and the next one may not be.
Correctness. Does the answer match the known correct answer, including numbers, names and conditions?
Refusal. Include questions the documents cannot answer. The right response is to say so, not to improvise.
Permissions. Include questions asked as users who should not see certain documents, and confirm those documents never appear.

A second model can grade answers against your reference answers at scale, which makes it practical to run the whole set on every change. Spot-check its grading by hand, especially early on, because an automated grader can be wrong in consistent ways. After launch, keep adding real questions that went badly, so the test set grows to reflect how the system is actually used.

If your source data is scattered across systems or needs cleaning before any of this works, that is usually the first job. Our data and analytics work often starts there, and the AI integration side covers connecting an assistant to the systems you already run. If the assistant has to live inside a product you already run, see how to add AI to existing software.

A practical way to start

Pick one well-defined body of documents and one group of users. Collect fifty real questions they ask today, with the answers. Build retrieval over those documents, measure it against those questions, and fix whatever stage fails first. Only consider fine-tuning once retrieval is solid and a specific behaviour problem remains that prompting cannot solve.

If you want help scoping it, our generative AI and LLM development team can look at your data and use case. Book a free 30-minute discovery call and you will get a written estimate within two business days.

RAG vs Fine-Tuning: How to Put Your Own Data Into an LLM