Agentic RAG Failure Modes¶

TL;DR

• Agents break in predictable places: query understanding, routing, retrieval, generation, citations, tool calls, SQL generation, security, memory, and long-horizon or multi-agent runs.
• Always debug upstream first. What looks like a generator hallucination is often a retriever failure in disguise.
• Use this page as a field manual. For every failure mode below, you will find the symptom you see, the root cause, the eval that measures it, and the guardrail or fix that stops it.

• You will learn:
  • How to separate retrieval bugs from generation bugs
  • Barnett et al.'s seven RAG failure points
  • Stage-by-stage failures for RAG, tools, SQL, security (including MCP and agent memory), and multi-turn chat
  • Failures that appear only in long-horizon, autonomous, and multi-agent runs
  • How to trace a broken answer back to its root cause
• Prerequisites:
  • Why Evals and Guardrails
  • The Evals & Guardrails Map
• Related guides:
  • Threat Model & OWASP
  • Eval Selection Decision Tree
  • Glossary

1. Where failures happen¶

flowchart LR
    Q["1. Query understanding"] --> R["2. Routing / planning"]
    R --> RET["3. Retrieval + chunking"]
    RET --> GEN["4. Generation"]
    GEN --> CIT["5. Citations"]
    R --> TOOL["6. Tool calls"]
    R --> SQL["7. Text-to-SQL"]
    TOOL --> GEN
    SQL --> GEN
    SEC["8. Security: injection, leakage"] -.-> Q
    SEC -.-> RET
    SEC -.-> TOOL
    SEC -.-> GEN
    MT["9. Multi-turn: context loss, memory"] -.-> Q
    MT -.-> GEN

Follow two rules whenever you debug a bad answer:

1. Find the first upstream error. If the retriever never fetched the right policy page, editing the generator prompt will not fix the answer.
2. Separate "missing information" from "misused information." Compare context recall, faithfulness, and answer correctness side by side to see whether retrieval or generation failed:

Context recall here is measured against the reference answer. So a stale document usually shows up as the first row (Low / High / Low): the outdated document was retrieved and the current one was missed. See Section 4 for a worked example, and End-to-End RAG Evals for how to compute these three metrics.

2. The seven classic RAG failure points (Barnett et al., 2024)¶

Barnett et al. (Seven Failure Points When Engineering a Retrieval Augmented Generation System, arXiv:2401.05856) identified seven failure points from three RAG case studies. We tag these as FP1–FP7 in the tables below and expand them to cover tools, SQL, security, and multi-turn agents.

3. Stage-by-stage taxonomy¶

Use the tables below to diagnose traces during error analysis. Every table lists:

• Symptom: What you see in the trace or user bug report.
• Root cause: Why the system behaved that way.
• Detect (Eval): Which check measures how often this happens.
• Mitigate (Guardrail or Fix): How to block it at runtime or fix it in code.

3.1 Query understanding¶

3.2 Routing and planning¶

3.3 Retrieval and chunking¶

3.4 Generation¶

3.5 Citations¶

3.6 Tool calls and agent behavior¶

Multi-agent failures: If you run multiple collaborating agents, see Cemri et al. (2025), Why Do Multi-Agent LLM Systems Fail? (arXiv:2503.13657). It proposes the MAST taxonomy: 14 failure modes in three groups. The groups are specification and system-design issues, inter-agent misalignment, and task verification (for example, no verification, incomplete verification, or stopping too early). See §3.11 for the multi-agent and long-horizon failures that matter most in practice.

3.7 Text-to-SQL¶

3.8 Security¶

3.9 Multi-turn conversation¶

3.10 System and operations¶

3.11 Long-horizon, autonomy, and multi-agent¶

These failures appear when an agent runs for many steps, runs without a human watching, or works with other agents. They apply to every archetype in Agent Archetypes, not only to RAG.

4. Worked example: tracing one bad answer upstream¶

Suppose an employee asks Acme Internal Assistant:

• Question: "How many PTO days do I get after 5 years?"
• System Answer: "You get 20 days of PTO per year. [Source: HR-Handbook-2022]"
• Actual Rule: Employees get 25 days under HR-Policy-2024.

Here is how you trace that bug upstream:

1. Check generation first: Did the generator hallucinate? No. Its answer of "20 days" is 100% faithful to HR-Handbook-2022.
2. Check the prompt context: Was HR-Policy-2024 in the prompt sent to the LLM? No.
3. Check the retriever: Did the retriever find HR-Policy-2024? It ranked at #9, and top-k was set to 5.
4. Identify the root cause: This is FP2 (Missed top-ranked doc) combined with Stale documents in the index. The outdated 2022 handbook was never archived, and its longer text scored higher in vector search.

How you fix and prevent it:

• Engineering fix: Add status="current" metadata during ingestion, filter out archived handbooks at retrieval time, and add a reranker.
• Offline eval: Add 15 time-sensitive policy questions to your golden dataset and measure Recall@5 plus answer correctness.
• Why the runtime guardrails did not catch it: The answer was polite, well-cited, and 100% faithful to the retrieved chunk, so a faithfulness guardrail passed it. A faithfulness check cannot know that a source is outdated. Two things can catch it: a reference-based eval (or SME review) that knows the right answer, and a reference-free metadata check at runtime that blocks citations to documents marked superseded or past their effective_date. Acme had neither.

5. How to use this page during error analysis¶

Treat these tables as a starter menu, not a checklist you must implement all at once:

1. Read at least ~100 traces on your own first and write plain-English notes on the first error in each trace (open coding).
2. Group your notes into 5–10 buckets and match them to the failure names on this page where they fit.
3. Prioritize using frequency × severity:

6. Quick reference: Failure → Eval → Guardrail¶

Key takeaways¶

• Debug upstream first. Many "hallucinations" are actually retrieval or context-packing failures (FP1–FP3).
• Read Context Recall, Faithfulness, and Answer Correctness together. That three-number pattern immediately tells you whether retrieval, generation, or your source documents are broken.
• Faithfulness checks cannot catch stale documents. If the retriever fetches an outdated 2022 policy, the answer will look 100% faithful. Use runtime metadata checks (status, effective_date) to block superseded sources, and offline reference-based evals with domain experts to find the cases that metadata misses.
• Block high-severity risks with code, not prompts. Enforce RBAC permissions, read-only SQL, and tool confirmation gates in application code outside the LLM.
• Treat tools, tool descriptions, and memory as untrusted input. Agents add new attack paths (MCP tool poisoning, memory poisoning, goal hijack). Use the lethal trifecta and the Agents Rule of Two to decide where a human must approve.
• Do not trust an agent's own report of success. Check the final state, protect what you grade, and treat messages from other agents as untrusted input.

Further reading¶

• Barnett et al., 2024, Seven Failure Points When Engineering a Retrieval Augmented Generation System
• Cemri et al., 2025, Why Do Multi-Agent LLM Systems Fail?
• Liu et al., 2023, Lost in the Middle: How Language Models Use Long Contexts
• Greshake et al., 2023, Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection
• Gao et al., 2023, Retrieval-Augmented Generation for Large Language Models: A Survey
• OWASP, 2026, Top 10 for LLM Applications 2026 · Top 10 for Agentic Applications 2026 · 2025 → 2026 mapping
• Invariant Labs, 2025, MCP Security Notification: Tool Poisoning Attacks
• Model Context Protocol, Specification · Security Best Practices
• Simon Willison, 2025, The lethal trifecta for AI agents
• Meta, 2025, Agents Rule of Two: A Practical Approach to AI Agent Security
• Anthropic, 2025, Effective context engineering for AI agents
• METR, 2025, Recent Frontier Models Are Reward Hacking
• Simon Willison, Prompt injection series
• Yu et al., 2018, Spider: Text-to-SQL benchmark · Li et al., 2023, BIRD benchmark · Zhong, Yu & Klein, 2020, Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• PostgreSQL docs, EXPLAIN · Data-modifying statements in WITH
• Hamel Husain, A Field Guide to Rapidly Improving AI Products
• Next guide: Glossary