RAG: Core Concepts

I want to defend RAG from what I suspect is about to happen to it. Because I think it's actually an elegant solution to a genuine problem.

I have no objection to RAG. My objections are reserved for the people who learn the acronym and consider that sufficient. Retrieval-Augmented Generation is, at its core, a straightforward idea: the model doesn't know everything, so you go and fetch the relevant bits before asking it to respond.

Which solves the hallucination problem, at least partially. A model that's grounded in retrieved facts has considerably less room to invent things with confidence.

Considerably less room, yes. Not zero room. People hear 'RAG reduces hallucinations' and conclude they've built a trustworthy system. They have built a system that hallucinates with better source material, which is an improvement, but not a guarantee.

That's fair, but the lesson isn't claiming it's a guarantee. It's saying grounding the model in external knowledge makes outputs more accurate and relevant. Which is demonstrably true.

Demonstrably true when the retrieval is good. This is the part people skip over entirely. They treat the retrieval step as though it's automatic — as though asking a database a question and getting back the right context is something that simply happens, like weather.

Whereas in practice, what you retrieve is everything. Garbage in, generated garbage out, just with footnotes.

There it is. The footnotes give it an air of credibility that the underlying retrieval may not have earned. I find that more alarming than a straightforward hallucination, not less.

I think you're being slightly uncharitable to the core concept, though. The analogy that works for me is long-term memory. The model's weights are what it learned in training — fixed, potentially outdated. RAG is how it looks things up before answering. That's not a minor feature, that's solving a structural limitation.

The memory analogy is reasonable. I would push back on one thing: it implies the model is doing the looking up, exercising some kind of judgment about what to retrieve. It isn't. You are. The retrieval pipeline is yours. The model receives what you hand it and works with that. People find this surprising because the system feels seamless.

And that matters practically because it means the expertise required hasn't disappeared — it's just moved. You're not writing training data anymore, but you are responsible for what gets retrieved and how.

Precisely. The other thing worth noting is the adaptation argument — that updating a knowledge base is faster than retraining a model. This is true, and it's genuinely one of RAG's most underappreciated advantages. If your information changes frequently, fine-tuning a model to keep up is not a serious option. Updating a document store is.

Which means RAG isn't just a workaround for hallucinations. It's an architectural choice that makes AI systems actually maintainable in production. That's a different and arguably more important point.

It is. I'll concede that the lesson makes this point adequately. What it does not make adequately is the point that RAG introduces its own failure modes — retrieval latency, context window limits, retrieved content that's accurate but irrelevant to what the user actually needed. These are not edge cases. They are Tuesday.

Core concepts lesson, Sterling. It's not obligated to cover every failure mode in the first fifteen minutes.

No, but students who leave thinking RAG is a solved problem will be disappointed by Tuesday. Students who leave knowing it's a powerful pattern with real engineering demands on the retrieval side will be usefully prepared. There is a difference, and I think you know I'm right.

I do. I just didn't want you to have the last word without at least making you work for it.