GraphRAG, Agentic RAG and Long Context: What Comes After Basic RAG

Basic retrieval-augmented generation (RAG) splits documents into chunks, finds the chunks most similar to a question, and hands them to a language model. It works well for direct questions with answers in one place. It struggles with questions that span many documents, need relationships between facts, or depend on context a chunk lost when it was cut out.

Four newer patterns address those gaps: better retrieval (hybrid search, contextual retrieval and reranking), GraphRAG, agentic RAG, and simply putting more into a long context window. Each solves a different problem and has a different cost.

This guide explains each pattern, what the research shows, when to use which, and how to upgrade an existing RAG system without rebuilding it. If you are still deciding between RAG and fine-tuning, start with our post on RAG vs fine-tuning.

Why basic RAG hits a ceiling

Most RAG failures fall into four groups:

• Lost context. A chunk says "the revenue grew 3%" but not which company or quarter, because that was in a heading two pages earlier.
• Vocabulary mismatch. The user asks about "heart attack", the document says "myocardial infarction", and similarity search misses it, or the reverse with exact codes and IDs.
• Global questions. "What are the recurring complaints across all feedback this year?" has no single chunk that answers it.
• Multi-hop questions. "Which suppliers of the delayed items also had quality issues last year?" requires linking facts across documents.

Each pattern below targets one or more of these.

Pattern 1: Better retrieval before anything fancy

The cheapest upgrade is usually improving retrieval itself.

Hybrid search combines vector similarity with keyword search (BM25). Keyword search catches exact terms, codes and names that embeddings blur.

Contextual retrieval adds a short, generated description of where each chunk sits in its document before indexing it. Anthropic's contextual retrieval write-up reported that on its benchmarks this reduced the top-20 retrieval failure rate from 5.7% to 3.7% with contextual embeddings, to 2.9% when combined with contextual BM25 (a 49% reduction), and to 1.9% with reranking added (67%). With prompt caching, the one-time cost of generating the context was estimated at about $1.02 per million document tokens.

Reranking retrieves a larger candidate set and uses a second model to reorder it, at the cost of extra latency per query.

Pattern 2: GraphRAG for connected and global questions

GraphRAG, popularised by Microsoft Research, uses a language model to extract entities and relationships from documents, builds a knowledge graph, groups the graph into communities and writes summaries for each. Questions can then be answered from community summaries (good for global "what are the main themes" questions) or from the local neighbourhood of an entity (good for multi-hop questions).

The catch was cost: extracting entities and summarising every community with an LLM made indexing expensive. Microsoft's LazyGraphRAG addresses this by using cheaper language processing to build the graph and deferring LLM work until query time. Microsoft reported indexing costs identical to vector RAG and 0.1% of full GraphRAG, and comparable answer quality to GraphRAG's global search for global questions at more than 700 times lower query cost. Since June 2025 it has been available in Microsoft Discovery and in public preview in Azure Local services.

GraphRAG fits well when:

• Your documents are about connected entities: suppliers, patients and providers, contracts and parties, products and components.
• Users ask "across everything" questions such as themes, trends and patterns.
• You can tolerate more engineering and a graph store to maintain.

Pattern 3: Agentic RAG

In agentic RAG, retrieval becomes a tool the model chooses to use, rather than a fixed step before generation. The agent can rewrite the question, search several sources, read results, notice gaps, search again, and decide when it has enough. It can also choose between tools: a vector index for policy text, a SQL query for numbers, a graph lookup for relationships.

Strengths:

• Handles multi-hop and comparative questions by breaking them into steps.
• Combines structured data (databases) and unstructured data (documents) naturally.
• Can say "I could not find this" after a real search instead of answering from a weak chunk.

Costs and risks:

• More model calls per question, so higher latency and cost.
• Harder to evaluate, because the path varies between runs.
• Retrieved content becomes untrusted input that can carry prompt injection, so tool permissions matter. See our prompt injection defence guide.

Pattern 4: Long context instead of retrieval

Context windows of hundreds of thousands to a million tokens make a tempting shortcut: skip retrieval and put everything in the prompt. For small knowledge bases this is sensible. Anthropic suggests that a knowledge base under about 200,000 tokens (roughly 500 pages) can simply be included in the prompt, with prompt caching to cut cost and latency.

But more context is not free accuracy. Chroma's July 2025 context rot study tested 18 models and found performance becomes less reliable as input length grows, even on simple tasks. Similar but wrong "distractor" passages made it worse. On a long-memory benchmark, every model performed significantly better with a focused prompt of about 300 tokens than with the full prompt of about 113,000 tokens containing the same answer.

Comparing the patterns

A decision guide

1. Is the whole knowledge base small and stable (under roughly 200,000 tokens)? Try long context with caching first.
2. Otherwise, start with hybrid search plus contextual retrieval and reranking. Measure.
3. Do users ask global or relationship questions that fail? Add a graph layer, starting with a lazy approach.
4. Do questions need several steps or several data sources, including databases? Wrap retrieval as tools in an agent.
5. Re-measure after each step. Stop when quality meets the bar; each step adds cost and maintenance.

A worked example: a hospital policy and SOP assistant

A multi-specialty hospital wants staff to ask questions about clinical SOPs, NABH documentation, HR policies and pharmacy formulary rules, around a few thousand pages in total.

• Start with hybrid search and contextual retrieval over the documents. Most questions ("what is the hand-off process for ICU transfers?") are answered from one place.
• Questions like "which SOPs reference the high-alert medication list?" are relationship questions. A lightweight graph of documents, departments and referenced items handles them.
• Questions like "how many beds were blocked for more than 24 hours last week, and which policy covers escalation?" need both the HMS database and a policy document. An agent with a read-only SQL tool and the document index answers it.

Each layer is added only when evaluation shows the previous one is not enough. Our RAG development services team builds systems this way, often on top of the data already in a custom hospital management system.

Keeping advanced RAG fresh and correct

Advanced patterns add maintenance that basic RAG teams often underestimate.

• Incremental indexing. Documents change. Contextual descriptions, graph entities and community summaries must update when a source changes, without re-indexing everything. Design the pipeline around change detection from the start.
• Access control. A graph or agent that combines documents can reveal information a user should not see, such as an HR policy draft linked to a public SOP. Filter by the user's permissions at retrieval time, not after generation.
• Versioning. When a policy is revised, the old version should not keep answering questions. Store effective dates and prefer the current version.
• Citations. Show users which documents and sections an answer came from. It builds trust and makes errors easy to report.
• Cost monitoring. Agentic and graph queries vary widely in cost. Track cost per question by type and cap the number of retrieval steps per question.

Evaluating advanced RAG

Measure retrieval and generation separately:

• Retrieval recall: for each test question, did the documents containing the answer appear in the results?
• Faithfulness: are the answer's claims supported by retrieved text?
• Answer quality: does it fully answer the question, judged against expected criteria?
• Cost and latency per question: especially for agentic and graph patterns.

Our guide to LLM evaluation and observability covers building the test set and graders.

How to start upgrading an existing RAG system

1. Collect 50-100 real questions, including ones users say failed, with the documents that answer them.
2. Measure current retrieval recall and answer quality.
3. Add hybrid search and reranking; re-measure.
4. Add contextual chunk descriptions; re-measure.
5. Categorise remaining failures: global, multi-hop, structured data, or missing documents.
6. Address each category with the matching pattern, and keep the rest of the system unchanged.

Conclusion

The best RAG system in 2026 is rarely a single technique. It is solid hybrid retrieval with contextual chunks and reranking, plus a graph layer or an agent only where the questions demand it, and long context reserved for small or single-document jobs.

Research has made the trade-offs clearer: graph indexing no longer has to be expensive, and longer context windows do not guarantee better answers. Measure your own failures, then add the pattern that fixes them.