End-to-End RAG Evals¶

TL;DR

• Start with the RAG triad. Measure context relevance, groundedness, and answer relevance without any ground-truth labels.
• Add reference-based metrics for truth. Use answer correctness and context recall on a golden dataset to verify factual accuracy.
• Evaluate abstention on unanswerable queries. Measure both missed abstentions and false refusals on out-of-scope or restricted questions.
• Diagnose failures with the 2×2 matrix. Cross-reference context recall and faithfulness to isolate retrieval bugs from generation bugs.

• You will learn:
  • The RAG triad and RAGAS-style metrics (v0.4 names)
  • Faithfulness when the answer uses tool outputs
  • Abstention evaluation
  • Scorecard design
  • Retrieval × generation 2×2 diagnosis
• Prerequisites: Why evals and guardrails, Agentic RAG failure modes, Dataset strategy
• Related: Retrieval metrics, LLM-as-a-judge, Citation adherence

1. The anatomy of a RAG sample¶

Every E2E RAG metric uses a subset of five core artefacts:

flowchart LR
    Q["Question q"] --> RET["Retriever"]
    RET --> C["Contexts C"]
    C --> GEN["Generator"]
    Q --> GEN
    GEN --> A["Answer a"]

    Q -.->|context relevance| C
    C -.->|groundedness or faithfulness| A
    Q -.->|answer relevance| A
    REF["Reference answer a*"] -.->|answer correctness| A
    REF -.->|context recall| C

Reference-based vs reference-free¶

Recommendation: Reference-free metrics are necessary, but they are not sufficient on their own. A faithfulness score of 1.0 only proves that the answer agrees with the retrieved context. Build a reference-based set of 50–100 SME questions to verify that your answers are factually right (see SME golden datasets).

2. The RAG triad (TruLens)¶

TruLens introduced the RAG triad: three reference-free checks that cover each edge of the q → C → a triangle. When all three checks pass, your pipeline blocks the main hallucination paths.

Framework terminology mapping:

RAGAS context precision without reference (ContextUtilization) is not a pure q → C check. It asks whether each chunk was useful for the generated answer a, so a wrong answer moves the score.

RAGAS versions. RAGAS v0.4 added a collections API (from ragas.metrics.collections import ...) with async ascore() methods that return a MetricResult. Imports from ragas.metrics still work but are deprecated and will be removed in v1.0. Some metrics were removed (for example AspectCritic; use the @discrete_metric decorator). Pin the RAGAS version and the judge model, because prompts change between versions and scores are not comparable across versions. See the v0.3 → v0.4 migration guide.

Start with the RAG triad because it requires no ground-truth labels. Add reference-based metrics as soon as you build a golden dataset.

3. Metric cards¶

3.1 Faithfulness / groundedness¶

• What it measures: The fraction of claims in the answer that the retrieved context supports. This metric measures support, not factual truth.
• Inputs required: q, a, C. No ground truth required.

• How to compute it:

  1. Decompose claims: Prompt an LLM to split a into atomic, self-contained statements. Resolve all pronouns (for example, turn "It was founded in 2010" into "Acme was founded in 2010").
  2. Verify claims: Check whether C entails each claim. Use an LLM judge or a specialized NLI model such as a cross-encoder or Vectara's HHEM.

  3. Aggregate the score:

     faithfulness = |claims supported by C| / |claims in a|
     

• Score range & interpretation: 0 to 1, where higher is better. A score of 1.0 means every claim traces back to the retrieved context.

• Worked example:
  • q: "How many days of parental leave do employees in India get?"
  • C: ["Employees in India are entitled to 26 weeks of paid maternity leave.", "Paternity leave in India is 10 working days."]
  • a: "Mothers get 26 weeks of paid leave, fathers get 10 working days, and both can extend by 4 weeks unpaid."
  • Claims: (1) mothers get 26 weeks paid ✅, (2) fathers get 10 working days ✅, (3) both can extend 4 weeks unpaid ❌.
  • Result: faithfulness = 2/3 ≈ 0.67.
• Pitfalls and gotchas:
  • Grounded does not mean correct. If the retriever returns a superseded 2021 policy, a faithful answer is still factually wrong.
  • Claim decomposition quality drives the score. Long answers generate more claims and increase judge variance. Very short answers like "Yes." may produce zero claims. Track zero-claim answers separately as None rather than inflating your mean score.
  • Common-knowledge claims may fail verification. A strict judge marks "Paris is in France" as unsupported if the context omits it. Decide upfront whether your enterprise assistant may use parametric knowledge.
  • Paraphrases and rounding confuse judges. Expressions like "~26 weeks" or "six months" can trigger false negatives under strict entailment. Spot-check judge disagreements regularly.
• When to use it: Run faithfulness on every RAG answer as your primary reference-free hallucination check. In an agentic app, the answer can also use tool outputs and memory, so build C from the whole trajectory (see Faithfulness in agentic runs below).
• Implemented in: RAGAS (Faithfulness), TruLens (groundedness), DeepEval (FaithfulnessMetric), Phoenix (hallucination eval), promptfoo (context-faithfulness).

def faithfulness(claim_verdicts: list[bool]) -> float | None:
    """claim_verdicts[i] = True if claim i is entailed by the retrieved context."""
    if not claim_verdicts:
        return None  # track "no claims" separately rather than inflating the score
    return sum(claim_verdicts) / len(claim_verdicts)

Judge prompt sketch (verification step):

You are given CONTEXT and a list of STATEMENTS.
For each statement, answer "yes" only if it can be directly inferred from CONTEXT
without outside knowledge. Otherwise answer "no". Give a one-sentence reason first.
Return JSON: [{"statement": ..., "reason": ..., "verdict": "yes"|"no"}]

Faithfulness in agentic runs: build C from the trajectory¶

In an agentic RAG app, the final answer can draw on more than one search_docs call. It can also use run_sql rows, a get_ticket result, earlier turns, and memory. If C holds only the last search result, a correct answer that quotes a SQL result scores as unfaithful. If C holds everything, the judge sees too much text and misses errors.

• Build C from every tool output that the agent saw before the answer, in trace order, with a source label ([search_docs#2], [run_sql#1]). Keep tool error messages: an answer that ignores an error is a false-success failure.
• Add the conversation turns and memory items that the answer may rely on. Keep them separate from tool outputs, so you can tell "supported by a tool" from "supported only by what the user said".
• Exclude content that the agent should not trust as evidence, such as instructions inside a fetched web page. Score it with the indirect prompt injection tests instead.
• Truncate long tool outputs per source, not at the end of C, so that late tool calls are not cut off.
• Log which source supports each claim. This also gives you citation checks (06).

RAGAS MultiTurnSample stores the full message list, including ToolMessage outputs, for agent metrics. For faithfulness, flatten the tool messages into retrieved_contexts yourself. Trajectory-level checks (tool choice, order, final state) are in agent and trajectory evals.

3.2 Answer relevancy (response relevancy)¶

• What it measures: How directly the answer addresses the question. It penalizes incomplete, evasive, or padded answers, but it does not measure factual correctness.
• Inputs required: q, a (some RAGAS versions also use C). Requires an LLM to generate questions and an embedding model, but no ground truth.

• How to compute it (RAGAS reverse-question method):

  1. Prompt an LLM to generate N questions that the answer a would directly answer (typically N = 3).
  2. Embed each generated question g_i and the original question q.

  3. Compute the mean cosine similarity:

     answer_relevancy = (1/N) * Σ_i cos( E(g_i), E(q) )
     

  RAGAS also checks whether the answer is noncommittal (such as "I'm not sure" or "It depends") and assigns a score of 0.

• Alternative (TruLens / DeepEval method): Prompt an LLM judge to rate relevance directly from q and a. DeepEval splits the answer into statements and computes the fraction of statements relevant to the question.

• Score range & interpretation: Roughly 0 to 1. Because embedding cosine similarities rarely reach 0 or 1, absolute values depend on your embedding model. Compare relative scores across variants instead of setting a universal threshold.
• Worked example: Suppose q is "What is the VPN client for macOS?" and a is "Use GlobalProtect. Install it from Self Service." The generated questions ("Which VPN client should Mac users install?", "Where do I get the macOS VPN client?") match q and score high. If a says "IT Security reviews VPN policies annually", the generated questions drift and produce a lower score.
• Pitfalls and gotchas:
  • Confidently wrong answers can score high. The metric rewards answers that restate the question, even when the facts are wrong.
  • Noncommittal rules penalize valid abstentions. An honest "I don't have information about that" receives a 0 score. Exclude unanswerable questions from this metric and evaluate them with abstention correctness.
  • Scores shift across embedding models. Pin your embedding model version.
• When to use it: Use it as a fast triad check to catch topic drift and verbosity. Never use it as a proxy for factual correctness.
• Implemented in: RAGAS (AnswerRelevancy in the v0.4 collections API; legacy alias ResponseRelevancy), TruLens (answer relevance), DeepEval (AnswerRelevancyMetric), promptfoo (answer-relevance).

3.3 Answer correctness¶

• What it measures: How closely the generated answer agrees with the reference answer. This metric directly measures whether the answer is factually right.
• Inputs required: q, a, a*. Requires ground truth.

• How to compute it (RAGAS method): Combine factual correctness and semantic similarity using a weighted average:

  1. Decompose both a and a* into atomic claims.
  2. Classify each claim into one of three buckets:
     • TP: A claim in a that a* supports.
     • FP: A claim in a that a* does not support (extra or incorrect information).
     • FN: A claim in a* that is missing from a.

  3. Compute the factual F1 score:

     F1_factual = TP / (TP + 0.5 * (FP + FN))
     

  4. Compute semantic similarity S = cos(E(a), E(a*)) (see section 3.4).

  5. Combine both scores (RAGAS defaults to 0.75 factual and 0.25 semantic):

     answer_correctness = w_f * F1_factual + w_s * S       # w_f + w_s = 1
     

• Score range & interpretation: 0 to 1. The factual F1 score drives the result, while semantic similarity smooths over valid paraphrases.

• Worked example:
  • a*: "Expense reports must be submitted within 30 days. Receipts are required for amounts over $25."
  • a: "Submit expense reports within 30 days. Receipts are needed for any amount. Approval is by your manager."
  • Classification: TP = 1 ("within 30 days"), FP = 2 ("any amount" contradicts the reference, "manager approval" is absent from the reference), FN = 1 ("receipts over $25").
  • F1_factual = 1 / (1 + 0.5 * (2 + 1)) = 1 / 2.5 = 0.40.
  • With S = 0.88, answer_correctness = 0.75 * 0.40 + 0.25 * 0.88 = 0.30 + 0.22 = 0.52.
• Pitfalls and gotchas:
  • Extra true facts count as false positives if the reference omits them. Write comprehensive references, or configure FactualCorrectness to use precision-only or recall-only mode.
  • Reference answers grow stale when company policies change. Version your golden dataset alongside your document corpus.
  • LLM judges are overkill for short factoids. Use normalized exact match or numeric tolerance for single numbers, dates, or names.
• When to use it: Use answer correctness as the primary metric for offline regression suites with golden datasets. You cannot run it on unlabeled production traffic.
• Implemented in: RAGAS (FactualCorrectness with mode="f1", "precision", or "recall", which is on the current RAGAS metrics list; the weighted AnswerCorrectness described above is still available but no longer listed), Phoenix (QA correctness eval), promptfoo (factuality), DeepEval (GEval with a correctness rubric).

def factual_f1(tp: int, fp: int, fn: int) -> float:
    denom = tp + 0.5 * (fp + fn)
    return tp / denom if denom else 0.0

def answer_correctness(tp: int, fp: int, fn: int, sem_sim: float,
                       w_factual: float = 0.75) -> float:
    return w_factual * factual_f1(tp, fp, fn) + (1 - w_factual) * sem_sim

print(answer_correctness(tp=1, fp=2, fn=1, sem_sim=0.88))  # 0.52

3.4 Answer semantic similarity¶

• What it measures: Meaning-level similarity between a and a*.
• Inputs required: a, a*. Requires ground truth plus an embedding model or cross-encoder.

• How to compute it:

  semantic_similarity = cos( E(a), E(a*) ) = (E(a) · E(a*)) / (‖E(a)‖ ‖E(a*)‖)
  

  You can optionally apply a threshold to convert the score into a binary pass/fail result. Cross-encoder models score (a, a*) jointly and usually outperform bi-encoder cosine similarity. For token-level matching, see BERTScore in 03 · Generation quality and safety evals.

• Score range & interpretation: Modern embedding models compress most pairs into a narrow high band (such as 0.70 to 0.95). Small score changes matter, and absolute thresholds do not transfer across embedding models.

• Worked example: "Refunds take 5–7 business days" vs "Expect your refund within a week" produces high similarity. However, "Refunds take 5–7 business days" vs "Refunds take 50–70 business days" also produces high similarity.
• Pitfalls and gotchas: Embeddings often miss negations, number changes, and entity swaps. Never use semantic similarity as your sole correctness metric.
• When to use it: Use it as a smoothing term inside answer correctness or as a cheap pre-filter.
• Implemented in: RAGAS (SemanticSimilarity), promptfoo (similar), DeepEval (custom metrics).

3.5 Context precision¶

• What it measures: Whether your retriever ranks relevant chunks above irrelevant chunks. It acts as a rank-aware signal-to-noise ratio for the context window.
• Inputs required: q, ordered C, and either a* (with reference) or a (without reference, which RAGAS calls context utilization). Ground truth is optional.

• How to compute it:

  1. For each chunk at rank k, prompt an LLM judge to score relevance v_k ∈ {0,1}. Ask "Was this chunk useful in arriving at the reference answer?" (or "...the given answer?" without a reference).

  2. Compute rank-weighted average precision over the retrieved list:

     precision@k = (Σ_{i=1..k} v_i) / k
     
     context_precision@K = Σ_{k=1..K} ( precision@k × v_k ) / (number of relevant chunks in top K)
     

• Score range & interpretation: 0 to 1. A score of 1.0 means all relevant chunks appear at the top of the list. Placing the same chunks lower in the ranking reduces the score.

• Worked example: Suppose K = 4 and the relevance vector is [1, 0, 1, 0].
  • precision@1 = 1/1 = 1.0 (v₁ = 1) → contributes 1.0
  • precision@3 = 2/3 ≈ 0.667 (v₃ = 1) → contributes 0.667
  • Sum = 1.667 across 2 relevant chunks → 0.83.
  • Reordering the list to [0, 0, 1, 1] yields (1/3 + 2/4) / 2 ≈ 0.42.
• Pitfalls and gotchas:
  • The reference-free variant anchors on a. If the generated answer is wrong, the judge evaluates chunk usefulness against a wrong target.
  • The denominator only counts retrieved relevant chunks. It ignores relevant chunks that your retriever missed completely. Always pair context precision with context recall.
  • Judge calls scale with K. Scoring 500 questions at K = 10 requires 5,000 judge calls. Cache judgments by (q, chunk_id).
• When to use it: Use it when tuning rerankers, chunk sizes, or cutoff K. If you already have labeled chunk IDs, use deterministic retrieval metrics from 02 · Retrieval metrics instead.
• Implemented in: RAGAS (ContextPrecisionWithReference, ContextPrecisionWithoutReference / ContextUtilization, and the non-LLM NonLLMContextPrecisionWithReference), DeepEval (ContextualPrecisionMetric).

def context_precision(relevance: list[int]) -> float:
    hits, total = 0, 0.0
    for k, v in enumerate(relevance, start=1):
        if v:
            hits += 1
            total += hits / k
    return total / hits if hits else 0.0

assert round(context_precision([1, 0, 1, 0]), 2) == 0.83
assert round(context_precision([0, 0, 1, 1]), 2) == 0.42

3.6 Context relevance¶

• What it measures: The fraction of retrieved content that is relevant to the question, ignoring chunk rank and reference answers.
• Inputs required: q, C. No ground truth required.

• How to compute it: Prompt an LLM judge to score each chunk for relevance to q (binary or 0–3 scale), then average across chunks. DeepEval extracts individual statements from each chunk and computes relevant statements / total statements.

  context_relevance = (1/|C|) * Σ_c relevance(q, c)
  

• Score range & interpretation: 0 to 1. Low scores indicate a noisy context window that increases distraction and hallucination risk.

• Pitfalls and gotchas: Long chunks with a single relevant sentence receive inconsistent scores. Define clearly in your rubric whether a chunk must contain relevant info or mostly consist of relevant info.
• When to use it: Use it for online monitoring as the retrieval leg of the RAG triad. Prefer context precision whenever chunk rank matters.
• Implemented in: TruLens (context relevance), DeepEval (ContextualRelevancyMetric), RAGAS (Nvidia ContextRelevance), Phoenix (retrieval relevance eval).

3.7 Context recall¶

• What it measures: Whether the retrieved context contains all the facts needed to produce the reference answer.
• Inputs required: q, C, a*. Requires ground truth (a reference answer, not labeled chunk IDs).

• How to compute it:

  1. Decompose a* into atomic claims.

  2. Judge whether C supports each reference claim:

     context_recall = |reference claims attributable to C| / |reference claims|
     

  If you have labeled relevant chunk IDs, use deterministic recall@k instead.

• Score range & interpretation: 0 to 1. Low context recall caps your maximum achievable answer correctness because the generator cannot use missing facts.

• Worked example: Suppose a* states: "Laptops are refreshed every 3 years. Engineers may request a refresh at 2 years. Requests go through the IT portal." If C includes the 3-year rule and portal steps but omits the engineer exception, context recall = 2/3 ≈ 0.67.
• Pitfalls and gotchas:
  • Unindexed facts cap recall below 1.0. If an SME writes a reference answer from memory using facts absent from the corpus, no retriever can achieve 1.0.
  • Recall ignores noise. Retrieving 50 chunks inflates recall, so always pair it with context precision.
• When to use it: Use context recall as your primary retrieval metric when you have reference answers but lack chunk-level labels.
• Implemented in: RAGAS (ContextRecall, LLM and non-LLM variants), DeepEval (ContextualRecallMetric), promptfoo (context-recall).

3.8 Context entities recall¶

• What it measures: The fraction of named entities (people, products, dates, IDs, amounts) in the reference answer that appear in the retrieved context.
• Inputs required: C, a* (or C*). Requires ground truth.

• How to compute it:

  GE = entities extracted from reference
  CE = entities extracted from retrieved contexts
  context_entities_recall = |CE ∩ GE| / |GE|
  

  Extract entities with an LLM or NER model, and normalize aliases (such as "Q3 FY24" vs "third quarter of fiscal 2024").

• Score range & interpretation: 0 to 1. Once you extract entities, this check is cheaper and more deterministic than claim-level recall.

• Worked example: The reference contains {"GlobalProtect", "macOS 14", "Self Service"}. The retrieved context mentions GlobalProtect and Self Service, giving a score of 2/3.
• Pitfalls and gotchas: Finding an entity in a chunk does not guarantee the chunk states the right fact about that entity. Alias normalization requires careful tuning.
• When to use it: Use it for fact-dense domains like product catalogs, HR policies, finance, and incident tickets. Skip it for conceptual how-to questions.
• Implemented in: RAGAS (ContextEntityRecall).

3.9 Noise sensitivity¶

• What it measures: How often the generator produces incorrect claims because irrelevant or confusing chunks distracted it. Lower is better.
• Inputs required: q, C, a, a*. Requires ground truth.

• How to compute it: Decompose a into claims. Check whether each claim agrees with a* and trace which retrieved chunks entailed it:

  noise_sensitivity = |incorrect claims in a| / |claims in a|
  

  RAGAS provides two modes. The relevant mode counts incorrect claims drawn from relevant chunks (misreading good context). The irrelevant mode counts incorrect claims drawn from irrelevant chunks (distraction from noise).

• Score range & interpretation: 0 to 1, where lower is better.

• Pitfalls and gotchas: Claim-to-chunk attribution relies on an LLM judge. Validate judge attribution accuracy on a manual sample.
• When to use it: Run noise sensitivity when you increase K or add noisy data sources like team wikis. You can also inject known-irrelevant chunks to stress-test generator robustness.
• Implemented in: RAGAS (NoiseSensitivity).

3.10 Response completeness¶

• What it measures: Whether the answer covers every required key point in the reference answer.
• Inputs required: q, a, a* (or an explicit SME checklist of key facts). Requires ground truth.

• How to compute it: Have SMEs list the must-have key points for each question. Prompt an LLM judge to check which key points appear in a:

  completeness = |required key points covered by a| / |required key points|
  

  This metric isolates the false-negative (FN) side of answer correctness. RAGAS FactualCorrectness in recall mode computes a close equivalent.

• Score range & interpretation: 0 to 1. Track completeness alongside a conciseness metric (03 · Generation quality and safety evals) so the model does not game recall by writing walls of text.

• Worked example: Suppose q is "How do I access the finance warehouse?" Required points: (1) use access portal, (2) manager approval, and (3) data-handling training. An answer that mentions only 1 and 2 scores 0.67.
• Pitfalls and gotchas: Key-point checklists take SME time to write. Keep each list short (3–5 must-have items).
• When to use it: Use completeness for multi-part, procedural, and policy questions. Skip it for simple single-fact lookups.
• Implemented in: RAGAS (FactualCorrectness in recall mode), DeepEval (GEval with a completeness rubric), custom judges.

3.11 Abstention correctness ("I don't know")¶

• What it measures: Whether the system declines to answer when it should (unanswerable, out-of-scope, or restricted by RBAC) and answers when it can.
• Inputs required: q, a, and a dataset label answerable ∈ {yes, no} per question. Requires ground-truth labels. Include 10–20% unanswerable questions covering missing corpus facts, external companies, future events, and RBAC-restricted documents.

• How to compute it: Classify each response as abstained or answered using a regex or lightweight LLM classifier. Build a 2×2 confusion matrix:

  Ground truth System action	System answered	System abstained
  Answerable	Answered (score correctness next)	False abstention (over-refusal)
  Unanswerable	Hallucinated answer (worst case)	Correct abstention

  abstention_recall     = correct abstentions / unanswerable questions      # did we decline when we should?
  abstention_precision  = correct abstentions / all abstentions             # when we declined, was it right?
  false_abstention_rate = false abstentions / answerable questions
  

• Score range & interpretation: 0 to 1 for each metric. Always report both recall and false-abstention rate. Optimizing abstention recall alone creates an unhelpful bot that refuses every question.

• Worked example: Your dataset has 80 answerable and 20 unanswerable questions. The system abstains on 15 unanswerable and 6 answerable questions. Here, abstention_recall = 15/20 = 0.75, abstention_precision = 15/21 ≈ 0.71, and false_abstention_rate = 6/80 = 0.075.
• Pitfalls and gotchas:
  • Exclude unanswerable questions from standard answer relevancy. Reference-free relevancy penalizes valid refusals.
  • Test RBAC boundaries explicitly. The assistant must refuse "What is the CFO's salary?" for unauthorized roles even when the document exists in the corpus. Enforce access control at the retrieval layer (see access control), and verify the user-facing refusal here.
  • Define rules for partial abstentions. Decide how your rubric grades responses like "I couldn't find X, but here is Y."
• When to use it: Include an unanswerable slice (10–20% of samples) in every enterprise RAG evaluation suite. See synthetic datasets for ways to generate unanswerable questions.
• Implemented in: Custom evaluators, or DeepEval GEval and promptfoo llm-rubric assertions.

def abstention_metrics(rows: list[dict]) -> dict:
    """rows: [{"answerable": bool, "abstained": bool}, ...]"""
    unans = [r for r in rows if not r["answerable"]]
    ans = [r for r in rows if r["answerable"]]
    abst = [r for r in rows if r["abstained"]]
    correct_abst = sum(1 for r in unans if r["abstained"])
    return {
        "abstention_recall": correct_abst / len(unans) if unans else None,
        "abstention_precision": correct_abst / len(abst) if abst else None,
        "false_abstention_rate": sum(r["abstained"] for r in ans) / len(ans) if ans else None,
        "hallucination_on_unanswerable": (len(unans) - correct_abst) / len(unans) if unans else None,
    }

3.12 Latency and cost as E2E metrics¶

Always evaluate quality metrics alongside latency and cost budgets. A variant that adds 2 points of correctness by doubling p95 latency often degrades the user experience.

• What it measures: Time-to-first-token (TTFT), end-to-end latency, input/output token counts, LLM/tool call counts, and dollar cost per request.
• Inputs required: Execution traces from the evaluation run. No ground truth required.
• How to compute it: Record per-request timings and token counts from your tracing layer (see tracing and OpenTelemetry). Report p50, p95, and p99, not just means. Sum tokens × price_per_token across models plus tool and infrastructure costs.
• Score range & interpretation: Measured in seconds and dollars. Set explicit budgets per use case (for example, TTFT under 1–2 s and full answer p95 under 10 s for interactive chat).
• Pitfalls and gotchas: Offline latency differs from production latency because of cold caches, rate limits, and concurrency. Use offline timings for relative comparisons between variants, and exclude LLM-judge costs from your application cost total.
• When to use it: Include latency and cost on every scorecard. Gate releases on relative regressions (such as failing the build if p95 latency increases by more than 20%).
• Implemented in: Langfuse, LangSmith, Phoenix, Braintrust, and promptfoo (latency and cost assertions).

4. Other E2E evaluation frameworks worth knowing¶

• RAGAS (Es et al., 2023) introduced reference-free LLM metrics for faithfulness, answer relevance, and context relevance. It popularized the claim-decomposition and reverse-question recipes covered in section 3.
• ARES (Saad-Falcon et al., 2023) evaluates context relevance, answer faithfulness, and answer relevance by fine-tuning lightweight judge models on synthetic data. It then uses prediction-powered inference (PPI) with a small human-labeled sample to compute confidence intervals. Even if you do not use ARES directly, follow its core principle: validate automated judges against human labels and report confidence intervals (see LLM-as-a-judge).
• TruLens packages the RAG triad into reusable feedback functions that run directly on application traces.

5. Building a scorecard¶

Do not collapse all metrics into a single composite number too early. Build a scorecard table where each metric has a clear direction, a release gate, and a slice breakdown.

5.1 Recommended default scorecard¶

Treat these thresholds as starting heuristics rather than fixed standards. Have SMEs label at least 100 outputs as acceptable or unacceptable (oversample so that at least 30–50 are unacceptable), then pick thresholds that best separate the two groups on a held-out split.

A fixed tolerance such as "baseline − 2 points" is only valid if you state the sample size and the run-to-run noise. On 200 items, a pass rate of 0.80 has a 95% Wilson interval of about ±0.055, so a 2-point drop is usually noise. Re-run the baseline on the same dataset and compare with a paired test (McNemar for pass/fail, paired bootstrap for continuous scores). See running offline experiments.

5.2 Aggregation rules¶

• Report failure rates alongside means. A mean faithfulness of 0.93 can hide 5% of answers scoring below 0.30. Track the percentage of samples that fall below an acceptable floor.
• Break down scores by slice. Group results by question type (factoid, procedural, comparison, multi-hop), source system, user role, language, and answerability. Most regressions happen inside a single slice.
• Gate releases on per-sample pass rates. Define a clear pass rule such as pass = correctness >= 0.7 AND faithfulness >= 0.8 AND not hallucinated_on_unanswerable. Stakeholders understand pass rates much better than weighted averages.
• Keep raw metrics visible even if leadership requests a single weighted composite score.
• Report statistical uncertainty. On a 200-sample dataset, a 2-point difference is often random noise. Report each pass rate with its sample size and a Wilson interval (code below). When you test many slices, correct for multiple comparisons (for example Benjamini–Hochberg). See running offline experiments for significance testing.

def per_sample_pass(s: dict) -> bool:
    if not s["answerable"]:
        return s["abstained"]
    return (not s["abstained"]
            and s["answer_correctness"] >= 0.7
            and (s["faithfulness"] is None or s["faithfulness"] >= 0.8))

def wilson_interval(passes: int, n: int, z: float = 1.96) -> tuple[float, float]:
    """95% Wilson score interval for a pass rate (better than p ± z·SE at small n or p near 0/1)."""
    if n == 0:
        return (0.0, 1.0)
    p = passes / n
    centre = (p + z * z / (2 * n)) / (1 + z * z / n)
    half = z * ((p * (1 - p) / n + z * z / (4 * n * n)) ** 0.5) / (1 + z * z / n)
    return (max(0.0, centre - half), min(1.0, centre + half))

def pass_rate(samples: list[dict], slice_key: str | None = None) -> dict:
    groups: dict[str, list[bool]] = {}
    for s in samples:
        key = s.get(slice_key, "all") if slice_key else "all"
        groups.setdefault(key, []).append(per_sample_pass(s))
    return {k: {"rate": sum(v) / len(v), "n": len(v), "ci95": wilson_interval(sum(v), len(v))}
            for k, v in groups.items()}


samples = [
    {"answerable": True, "abstained": False, "answer_correctness": 0.9, "faithfulness": 0.95, "type": "policy"},
    {"answerable": True, "abstained": False, "answer_correctness": 0.5, "faithfulness": 1.0, "type": "policy"},
    {"answerable": False, "abstained": True, "type": "out_of_scope"},
]
report = pass_rate(samples, slice_key="type")
assert report["policy"]["rate"] == 0.5 and report["out_of_scope"]["rate"] == 1.0
lo, hi = wilson_interval(160, 200)
assert (round(lo, 2), round(hi, 2)) == (0.74, 0.85)   # 0.80 on n = 200

6. Diagnosing failures with the retrieval × generation 2×2¶

Read retrieval metrics (context recall, context precision, or recall@k) together with generation metrics (faithfulness and answer correctness). Place each failing sample into one of four diagnostic buckets:

flowchart TD
    F["Failing sample: low answer correctness"] --> CR{"Context recall high?"}
    CR -- "No" --> RET["Retrieval problem: chunking, embeddings, query rewrite, hybrid, K, filters, RBAC filter too strict"]
    CR -- "Yes" --> FA{"Faithfulness high?"}
    FA -- "No" --> HAL["Generation hallucinating: tighten prompt, require citations, reduce context noise, change model"]
    FA -- "Yes" --> CP{"Context precision high?"}
    CP -- "No" --> NOISE["Model followed the wrong chunk: rerank, dedupe, resolve conflicting or stale docs"]
    CP -- "Yes" --> REF["Check the reference answer, corpus freshness, and judge reliability"]

Diagnostic workflow:

• Run your evaluation scorecard and sort samples by answer correctness ascending.
• Bucket the bottom 20–50 failing samples into the 2×2 matrix using context recall and faithfulness thresholds.
• Inspect the full traces in the largest failure bucket manually (see production data flywheel).
• Record a concrete failure taxonomy (such as "table data split across chunks" or "outdated policy ranked first").
• Build a targeted component eval for the top failure mode, iterate on that component, and re-run the E2E suite.

7. Practical setup checklist¶

• Build a dataset with q, a*, an answerable label, slice tags, and relevant chunk IDs (dataset strategy).
• Pin the judge model snapshot, judge prompts, and embedding model version. Use the lowest temperature the judge model supports, or a fixed reasoning effort for reasoning models (LLM-as-a-judge §4.1).
• For agentic runs, build C from every tool output and memory item the agent saw, not only the last search.
• Log C with chunk IDs and ranks on every run so you can recompute retrieval metrics offline.
• Validate every LLM-judged metric before gating releases: at least 100 human-labelled items with at least 30–50 failures, and TPR ≥ 0.85, TNR ≥ 0.85, and Cohen's κ ≥ 0.60 on a held-out split (≥ 0.90 for high-risk slices). See LLM-as-a-judge.
• Pin the RAGAS (or other framework) version. Metric prompts change between versions.
• Cache judge responses by (metric_version, q, a, C_hash) to cut evaluation cost and latency.
• Gate releases on per-sample pass rates and slice-level regression checks (running offline experiments).
• Promote reference-free metrics (faithfulness, answer relevancy, context relevance) to online eval pipelines.

Key takeaways¶

• Start with the RAG triad. Measure context relevance, groundedness, and answer relevance without ground truth, then add reference-based answer correctness and context recall.
• Remember that faithful does not mean correct. Reference-free metrics only verify consistency with retrieved chunks, not factual truth.
• Invest in claim decomposition and verification. Judge quality directly determines the reliability of faithfulness, correctness, context recall, and noise sensitivity.
• Include an unanswerable slice. Measure both missed abstentions and false refusals on out-of-scope and RBAC-restricted queries.
• Use the retrieval × generation 2×2 matrix. Combine context recall and faithfulness to decide whether to fix retrieval or generation first.
• Track distributions, slices, latency, and cost. Gate releases on per-sample pass rates rather than aggregate means alone, and report each rate with n and a confidence interval.
• In agentic runs, score faithfulness against everything the agent saw. Include tool outputs and memory in C, with source labels.
• Pin the framework version. RAGAS renamed and removed metrics in v0.4; scores are not comparable across versions.

Further reading¶

• Es et al., RAGAS: Automated Evaluation of Retrieval Augmented Generation, arXiv:2309.15217
• Saad-Falcon et al., ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems, arXiv:2311.09476
• TruLens, The RAG Triad
• RAGAS docs, List of available metrics · Migration from v0.3 to v0.4
• DeepEval, Metrics introduction
• Hamel Husain, Your AI Product Needs Evals
• Within this repo:
  • Retrieval metrics
  • LLM-as-a-judge
  • Instruction & citation adherence
  • Metric cheatsheet