In This Handbook

What is Agentic RAG?

Retrieval-Augmented Generation, or RAG, is one of the most practical ways to make large language models useful in real systems. It works by giving the model relevant context at the time of the question, instead of forcing it to rely on whatever it learned during training. When it works, it feels like you gave the model an open book and told it to cite the right page.

But once you move beyond straightforward Q&A, standard RAG starts to show cracks. The issue is not retrieval itself. The issue is that the retrieval process is usually fixed. It runs the same way every time, even when the question clearly demands a different strategy.

Agentic RAG is the shift from “retrieve once, answer once” to “reason first, retrieve as needed, use tools, then answer.” It keeps the value of RAG, but upgrades the workflow so it can handle ambiguity, multi-step questions, and real-world messiness.

What standard RAG actually solves

LLMs are strong at language and reasoning, but they have limitations that become obvious in professional use:

They do not have access to your private knowledge base by default.
Their training knowledge is not guaranteed to be current.
They can produce confident answers even when evidence is missing.

RAG addresses these issues by introducing a simple idea: retrieve relevant information from an external source, attach it to the prompt, and require the model to answer from that context.

Where standard RAG starts to break

RAG is often described as simple and reliable. The hidden assumption is that the user’s question can be answered with one retrieval pass and one generation pass.

That assumption fails in three common situations.

Multi-step questions

Some questions require decomposition. For example: “Compare our contractor onboarding process with the full-time employee workflow and recommend which one applies to a remote hire in Europe.”

A single retrieval query often pulls mixed context. The model then has to untangle it, decide what matters, and produce a structured comparison in one go. You may get an answer, but it is often incomplete, inconsistent, or poorly organized.

Tool-dependent questions

Some questions require more than your document store. For example: “Summarize our Q3 report and tell me how today’s stock price compares to the last earnings day.”

The document summary is in your knowledge base. The stock price is not. Standard RAG has no built-in place to fetch live data, so the system either guesses or ignores part of the request.

Ambiguous questions

Some questions are underspecified and require clarification. For example: “Tell me about the Titan project.”

If Titan refers to multiple initiatives, retrieval will likely return a blended set of chunks. The model may answer confidently using the wrong context, or combine multiple unrelated projects into a single response.

In these cases, the pipeline did exactly what it was designed to do. The retrieved and generated. The problem is that it never paused to decide whether the retrieval strategy was correct.

The Shift: From a pipeline to a decision process

Agentic RAG introduces a simple but powerful change.

Instead of treating retrieval as a single mandatory step, retrieval becomes one option in a broader decision loop.

An agentic system can:

Decide whether it should retrieve at all
Choose which retrieval query to run
Retrieve multiple times with different intents
Use other tools when retrieval is not enough
Ask a clarification question before retrieving
Verify whether the retrieved context is sufficient

This is the core difference: standard RAG retrieves once because it is the pipeline. Agentic RAG retrieves because the system decided retrieval is the next best action.

The problem you will hit next

Let’s end with a scenario that tends to show up as soon as you deploy a RAG system to real users.

You build a RAG assistant for internal policy and technical documentation. Early feedback is positive.

Then these tickets start arriving:

“It answered confidently, but cited the wrong policy.”
“It mixed two similarly named projects and produced a blended response.”
“It retrieved irrelevant chunks and ignored the key page.”
“It missed the one paragraph that contained the exception.”
“It could not ask clarifying questions, so it guessed.”

Now you have a decision to make.

Previous Lesson Next Lesson

Fahim Ul Haq

I’m Fahim, a software engineer specializing in generative AI, system design, and responsible AI adoption.

View Profile

Generative AI