The Evals & Guardrails Map¶

TL;DR

• Five questions classify every check: When does it run? What is its scope? How is it computed? Does it need reference data? And for guardrails, how does it act?
• Three loops keep an agent reliable: Offline evals gate your releases, runtime guardrails protect live requests, and online evals turn production traffic into new test data.
• Use this page as your compass. Learn the five axes once, then use the master tables in Section 6 to jump to the exact guide you need.

• You will learn:
  • The five axes of evals and guardrails
  • Where every metric and guardrail fits
  • How datasets feed offline evals and how production feeds datasets
  • How to choose fail-open or fail-closed behavior and stay inside a latency budget
  • How to deploy one check in three places (with Python code)
• Prerequisites: Why Evals and Guardrails
• Related guides:
  • Failure Modes
  • Glossary
  • Eval Selection Decision Tree
  • Guardrail Selection Decision Tree
  • Metric Cheatsheet

0. The whole picture on one page¶

flowchart TD
    subgraph DATA["Datasets - 01-datasets"]
        SME["SME golden datasets"]
        SYN["Synthetic datasets"]
        PROD["Curated production samples"]
    end

    subgraph OFF["Offline evals - pre-deploy - 02-offline-evals"]
        E2E["End-to-end RAG evals"]
        COMP["Component evals: retrieval, tools, SQL, generation, citations"]
        JUDGE["LLM-as-a-judge + human review"]
    end

    subgraph RUN["Runtime - in the request path - 04-guardrails"]
        IN["Input guardrails"]
        APP["Agentic RAG app"]
        CTX["Retrieval/context guardrails"]
        TOOL["Tool/action guardrails"]
        OUT["Output guardrails"]
    end

    subgraph ON["Online evals & observability - 03-online-evals"]
        TRACE["Traces"]
        OEV["Async online evals: deterministic + sampled LLM judges"]
        FB["User feedback"]
        MON["Monitoring, alerting, drift"]
    end

    SME --> OFF
    SYN --> OFF
    PROD --> OFF
    OFF -->|"release gate"| APP
    IN --> APP --> CTX --> APP
    APP --> TOOL --> APP
    APP --> OUT
    RUN --> TRACE
    TRACE --> OEV --> MON
    FB --> MON
    OEV -->|"failures + edge cases"| PROD
    FB -->|"thumbs-down traces"| PROD
    MON -->|"new failure modes"| OFF

Read this diagram as three connected loops:

1. Pre-deploy loop (top): Datasets feed offline evals. Passing scores allow your code to ship.
2. Runtime protection (middle): Guardrails inspect inputs, retrieved context, tool calls, and outputs on every live request.
3. Learning flywheel (bottom): Live requests produce traces and user feedback. Online evals flag bad traces, which you label and add back to your datasets.

1. Axis A: When (offline vs online vs runtime guardrails)¶

Every check runs at one of three times: before deployment, after a live response, or during a live request.

Three useful bridges between these modes:

• Shadow mode: Run a new model, prompt, or guardrail on live traffic without showing its output to users. You get real-world scores with zero user risk. (A shadow model must not call side-effecting tools such as send_email; stub or block its writes.)
• Canary rollout: Route 1–5% of live traffic to the new version and watch online evals before rolling out to 100%.
• Red teaming: Attack the system offline to build an adversarial dataset, then use that dataset in CI to test your runtime guardrails. See Threat Model & OWASP.

2. Axis B: Scope (end-to-end vs component)¶

An agentic RAG system is a chain of steps. You can score the entire chain from question to final answer, or you can isolate and score one step at a time.

flowchart LR
    Q["User query"] --> QR["Query rewriter"]
    QR --> RT["Router / planner"]
    RT --> RET["Retriever"]
    RET --> RR["Reranker"]
    RR --> GEN["Generator"]
    RT --> SQLG["SQL generator"]
    SQLG --> EXEC["SQL execution"]
    EXEC --> GEN
    RT --> TC["Tool caller"]
    TC --> GEN
    GEN --> CIT["Citation step"]
    CIT --> A["Answer / action"]

Rule of thumb: Use end-to-end evals to see if your system is healthy. Use component evals to see where it is broken. Run both.

2.1 Component-by-component lookup¶

3. Axis C: Method (how the check is computed)¶

You can build a check using five different methods. Always start at the top of this table and move down only when a simpler method cannot test your rule.

Golden rule of grading: Use code whenever possible. Use LLM judges when you need semantic understanding. Use humans to define "good" and to calibrate your LLM judges. As a rule of thumb, trust a judge only when it reaches TPR ≥ 0.85, TNR ≥ 0.85, and Cohen's κ ≥ 0.60 on a held-out split of at least 100 human-labeled items with at least 30–50 Fails (aim for ≥ 0.90 on high-risk slices). Re-validate after any change to the judge model or prompt. See LLM-as-a-Judge.

4. Axis D: Ground truth (reference-based vs reference-free)¶

Some checks compare the system's output against a known-correct reference answer (ground truth). Other checks grade the output using only the question and the retrieved context.

Why you need both: An answer can be 100% faithful to an outdated 2021 document and still be 100% wrong. Conversely, an answer can be correct by luck from pre-training memory while ignoring your documents. Combine reference-free checks (faithfulness) with reference-based checks (answer correctness). See End-to-End RAG Evals.

5. Axis E: Guardrail mode (blocking vs non-blocking, sync vs async)¶

When you run a check at runtime as a guardrail, you must make two design choices:

1. Blocking vs. non-blocking: Does it change what the user gets, or does it just log an alert?
2. Synchronous vs. asynchronous: Does the request wait for the check to finish?

Four practical runtime patterns:

• Parallel guardrail: Start your input guardrail and your LLM call at the same time. If the guardrail fires, cancel the LLM call before showing anything to the user. Use this only for text-only turns. In parallel mode the agent may already have spent tokens or called a tool (such as create_ticket or send_email) before the guardrail fires. For tool-enabled turns, run the input guardrail in blocking mode first (for example, run_in_parallel=False in the OpenAI Agents SDK).
• Streaming tripwire: Stream tokens to the user while a background check watches the output. If the check detects a leak, cut the stream and replace the message.
• Staged rollout (shadow → flag → block): Deploy every new guardrail in non-blocking shadow mode first (log only). Measure how often it false-triggers on real users. Then move to flag (log and alert, or show a warning) and finally to block.
• Tiered checking: Run a fast regex or small classifier synchronously on every request. Call a slower LLM judge only when the fast classifier is unsure.

Every guardrail is a classifier, so it has a false-positive rate (it blocks good requests) and a recall (the share of bad requests it catches; attack success rate = 1 − recall on an attack set). Measure both offline before you switch to block. Sections 5.2 and 5.3 cover what to do when a guardrail fails and how latency adds up.

5.1 Where guardrails sit in the pipeline¶

flowchart LR
    U["User input"] --> G1{"1. Input guardrails: injection, PII, topic, rate limit"}
    G1 -->|deny| X1["Polite refusal"]
    G1 -->|allow| P["Planner / router"]
    P --> R["Retriever"]
    R --> G2{"2. Context guardrails: RBAC filter, indirect injection, freshness, relevance cutoff"}
    G2 --> LLM["Generator"]
    P --> S["SQL generator"]
    S --> G3{"3. SQL guardrails: SELECT-only, table allowlist, row-level security"}
    G3 --> DB[("Warehouse")]
    P --> T["Tool caller"]
    T --> G4{"4. Tool guardrails: JSON schema, role authZ, egress allowlist, human confirmation, step limit"}
    G4 --> API["Tools / APIs"]
    DB --> LLM
    API --> LLM
    LLM --> G5{"5. Output guardrails: faithfulness, PII, toxicity, valid citations, schema"}
    G5 -->|allow / modify| OUT["Final response"]
    G5 -->|deny| X2["Safe fallback"]

5.2 Fail open or fail closed?¶

A guardrail can time out or crash. Decide in advance what happens then:

• Fail closed (block, or fall back to a safe answer) for tool-enabled turns, Tier 3 (high-risk) intents such as HR and finance, and all access-control checks. If an injection classifier times out on a tool-enabled turn, block the turn or continue in a no-tools mode.
• Fail open, and flag the trace only for read-only, low-risk turns, where a short outage of a quality check is less harmful than blocking every user.

5.3 Latency adds up¶

Sync guardrails run in series on the request path, so their latency adds up. Rules of thumb for each stage: input checks 50–150 ms, retrieval and context checks 10–50 ms (plus 50–150 ms if you run a chunk-level injection classifier), tool-call gates 5–50 ms, and output checks 100–800 ms. Keep the total guardrail overhead at or below about 20–30% of end-to-end latency. Run independent checks in parallel where it is safe (Section 5 explains when parallel is unsafe). See Guardrails Architecture for the full latency budgets and failure-mode design.

6. Master lookup tables¶

Use these two tables to find any metric or guardrail in the repository.

Legend:

• When:
  • Off = Offline eval
  • On = Online eval
  • GR = Runtime guardrail (✅ = standard, ◐ = possible on a sample or with latency trade-offs, ❌ = not applicable)
• Ref:
  • Yes = Needs reference data
  • No = Reference-free
  • Opt = Both versions exist
  • For Ref = Yes rows, On = ◐ means the check runs online only on a labeled sample of traces, after a delay
• Method:
  • Code (deterministic code; Code (metadata), Code (policy), and Code (canary token) name what the code checks)
  • Rules / Regex (pattern lists and regular expressions, a kind of deterministic code)
  • Policy (a written business rule, such as "confirm before any write", enforced by the app)
  • DB roles (database permissions, such as a read-only role and row-level security)
  • Stat (statistical/IR math; Stat (model-based) needs an embedding or encoder model)
  • Clf (ML classifier; Clf (NER) = named-entity recognition, Clf (NLI) = entailment model)
  • Judge (LLM judge)
  • Human

6.1 Evals lookup table¶

6.2 Guardrails lookup table¶

Not sure which ones to pick first? Use the Eval Selection Decision Tree and the Guardrail Selection Decision Tree.

7. Datasets: the fuel for offline evals¶

Every offline eval requires dataset columns that match its inputs:

You will build these datasets from three main sources:

1. SME Golden Datasets: Written or verified by domain experts. Highest quality, slowest to build.
2. Synthetic Datasets: Generated by LLMs from your documents and schemas. Fast way to test edge cases before launch.
3. Production Samples: Real user traces pulled from production logs and labeled by experts.

See Dataset Strategy for how to size and combine them.

8. The production data flywheel¶

Your eval suite should improve automatically as real users interact with your system:

flowchart LR
    P["1. Production traffic"] --> T["2. Traces + guardrail logs + user feedback"]
    T --> S["3. Sample failures, thumbs-down, and guardrail blocks"]
    S --> EA["4. Error analysis: open + axial coding"]
    EA --> L["5. Label with SMEs"]
    L --> D["6. Update versioned datasets"]
    D --> OE["7. Run offline evals in CI"]
    EA --> NE["Build new checks or guardrails"]
    NE --> OE
    OE --> REL["8. Ship better release"]
    REL --> P

Three habits keep this flywheel spinning:

1. Treat every guardrail block and thumbs-down click as a candidate test row.
2. Keep your regression dataset separate from your capability dataset. Regression rows test bugs that must never break again; capability rows test hard problems you are still trying to solve.
3. Scrub PII before saving production traces into datasets. See Production Data Flywheel.

9. One check, three deployments: code example¶

Let's take a single check ("Is every claim in the answer supported by the retrieved context?") and run it in all three places:

The code below uses only the standard library (Python 3.9+). app and the judges are your own functions. app(question, strict=False) returns (answer, chunks), where each chunk is a dict with doc_id and text. Each judge returns (passed, score, reason). To try the code, stub them (for example, def judge(answer, texts): return (True, 0.95, "ok")).

import random
from dataclasses import dataclass
from typing import Callable, Optional

@dataclass
class Verdict:
    passed: bool
    score: Optional[float]
    reason: str

# A judge returns (passed, score, reason). Each judge applies its OWN calibrated
# threshold, because an LLM judge, a cheap judge, and an NLI model use different scales.
Judge = Callable[[str, list[str]], tuple[bool, float, str]]

def check_faithfulness(answer: str, chunks: list[dict], judge: Judge) -> Verdict:
    """One shared check function used across offline, online, and runtime."""
    passed, score, reason = judge(answer, [c["text"] for c in chunks])
    return Verdict(passed=passed, score=score, reason=reason)

# 1. Offline eval: score a dataset in CI
def run_offline_eval(dataset_rows: list[dict], app, strong_judge: Judge) -> float:
    if not dataset_rows:
        raise ValueError("Dataset is empty; refusing to report a pass rate.")
    verdicts = [
        check_faithfulness(*app(row["question"]), strong_judge)
        for row in dataset_rows
    ]
    return sum(v.passed for v in verdicts) / len(verdicts)

# 2. Online monitor: score a sample of traces, plus every thumbs-down
def _score_trace(trace: dict, cheap_judge: Judge, log_metric, enqueue_for_review) -> Verdict:
    verdict = check_faithfulness(trace["answer"], trace["chunks"], cheap_judge)
    log_metric("faithfulness_pass", int(verdict.passed), trace_id=trace["id"])
    if not verdict.passed:
        enqueue_for_review(trace["id"], verdict.reason)  # labeling queue
    return verdict

def on_trace_logged(trace, cheap_judge, log_metric, enqueue_for_review,
                    sample_rate: float = 0.05, rng=random.random):
    """Runs async after the response. Samples 1-5% of traffic."""
    if rng() < sample_rate:
        return _score_trace(trace, cheap_judge, log_metric, enqueue_for_review)
    return None

def on_feedback(trace, feedback: str, cheap_judge, log_metric, enqueue_for_review):
    """Feedback arrives after the trace is logged, so score 100% of thumbs-down here."""
    if feedback == "thumbs_down":
        return _score_trace(trace, cheap_judge, log_metric, enqueue_for_review)
    return None

# 3. Runtime guardrail: sync and blocking on Tier 3 (HR, finance) routes
def _abstain(chunks: list[dict]) -> str:
    sources = ", ".join(sorted({c["doc_id"] for c in chunks})) or "none"
    return ("I could not verify an answer in the documents you can access. "
            f"Sources I checked: {sources}.")

def answer_with_guardrail(question: str, app, fast_judge: Judge, is_high_risk: bool) -> str:
    answer, chunks = app(question)
    if not is_high_risk:
        return answer  # lower tiers: score async with on_trace_logged instead
    for attempt in range(2):  # first answer, then one stricter regeneration
        if attempt == 1:
            answer, chunks = app(question, strict=True)
        try:
            verdict = check_faithfulness(answer, chunks, fast_judge)
        except Exception:  # judge timeout or crash: fail closed on Tier 3
            return _abstain(chunks)
        if verdict.passed:
            return answer
    return _abstain(chunks)

You can use this exact three-way pattern for PII checks, SQL safety checks, tool schema checks, and citation checks.

10. Where to go next¶

Key takeaways¶

• Classify any check on five axes: When (offline, online, runtime), Scope (end-to-end vs. component), Method (code, math, classifier, LLM judge, human), Reference data (reference-based vs. reference-free), and Guardrail mode (blocking vs. non-blocking, sync vs. async).
• Pair end-to-end evals with component evals. End-to-end evals tell you if the user got a good answer; component evals tell you which step broke.
• Only reference-free checks work as runtime guardrails. Reference-based checks need labeled references, so they run offline or online on a labeled sample of traces.
• Start with deterministic code checks. Reach for LLM judges only when code cannot test the rule, and always validate judges against human labels on a held-out split.
• Roll out new guardrails as shadow → flag → block. Measure false positives on real traffic before blocking live users.
• Decide failure behavior in advance. Fail closed on tool-enabled and Tier 3 turns; fail open (and flag) only on read-only, low-risk turns. Do not run a guardrail in parallel with an agent that can call side-effecting tools.

Further reading¶

• Es et al., 2023, Ragas: Automated Evaluation of Retrieval Augmented Generation · RAGAS documentation
• Zheng et al., 2023, Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena
• Shankar et al., 2024, Who Validates the Validators?
• Hamel Husain, Your AI Product Needs Evals
• Eugene Yan, Patterns for Building LLM-based Systems & Products
• OWASP, 2026, Top 10 for LLM Applications 2026
• OpenAI, Agents SDK: Guardrails (parallel vs blocking execution)
• Zhang et al., 2019, BERTScore: Evaluating Text Generation with BERT
• Zhong, Yu & Klein, 2020, Semantic Evaluation for Text-to-SQL with Distilled Test Suites
• Anthropic, Demystifying evals for AI agents
• Next guide: Agentic RAG Failure Modes