Metric Cheatsheet¶

TL;DR

• Look up any metric fast. Compare every evaluation and guardrail check by ground-truth needs, method, runtime location, and cost.
• Start with the quick picks. Use §3 to choose 1–2 primary metrics for your specific diagnostic question.
• Separate offline gates from online monitors. Use reference-based metrics to gate releases and reference-free metrics to monitor live traffic.
• Prioritize cheap deterministic checks. Run schema, citation, SQL, and tool-call checks on 100% of traffic before sampling LLM judges.

• You will learn:
  • The full metric landscape at a glance
  • Which metrics require reference data
  • Which checks fit 100% production traffic
  • Which metric answers each diagnostic question
• Prerequisites: The evals & guardrails map
• Related: Eval selection decision tree, Guardrail selection decision tree, Glossary

1. Legend¶

Costs are relative per evaluated item. When a Judge metric runs online ("On"), you usually run it on sampled traffic (§5). Before a Judge metric gates a release, validate the judge against human labels (LLM-as-a-judge §7). The detail pages are canonical for every formula. If this table and a detail page disagree, the detail page wins.

2. The master table¶

2.1 Retrieval¶

2.2 Generation and end-to-end RAG¶

2.3 Instruction and citation adherence¶

2.4 Agent, tool use, and trajectory¶

2.5 Text-to-SQL¶

2.6 Safety and responsible-AI evals¶

2.7 Runtime guardrail checks¶

2.8 Measuring the guardrails themselves¶

2.9 Statistics for release gates¶

2.10 Online, feedback, and operational signals¶

3. Quick picks by question¶

Start with the bold metric. Add the secondary metrics only when you need finer diagnostic detail.

3.1 Retrieval¶

3.2 Generation and answers¶

3.3 Citations¶

3.4 Agents and tools¶

3.5 Text-to-SQL¶

3.6 Safety and guardrails¶

3.7 Production¶

4. Ground-truth requirements at a glance¶

flowchart LR
    subgraph NoGT["No ground truth needed"]
        A1["Faithfulness"]
        A2["Answer relevancy"]
        A3["Context relevance"]
        A4["Format and schema validity"]
        A5["Toxicity, PII, safety classifiers"]
        A6["Citation format validity"]
        A7["SQL validity"]
        A8["Feedback and ops signals"]
    end
    subgraph Pol["Policy spec only"]
        P1["Instruction adherence"]
        P2["Tool allowlist and argument policy"]
        P3["Unsafe action execution and attempt rate"]
        P4["SQL statement policy, budgets"]
    end
    subgraph GT["Per-item ground truth needed"]
        B1["Answer correctness"]
        B2["Recall, precision, MRR, nDCG, ERR"]
        B3["Context recall"]
        B4["Execution accuracy"]
        B5["Tool selection and argument match"]
        B6["Trajectory match"]
        B7["RBAC leak tests"]
        B8["Judge validation"]
        B9["Guardrail recall and FPR"]
    end

Run reference-free and policy checks online on live traffic. Run reference-based metrics offline to gate releases. A policy spec (role → tool matrix, allowlist, SQL policy) is ground truth you write once. You do not label each item.

5. Cheap enough for 100% of traffic?¶

Treat these sampling rates as starting heuristics (online evals §4.2). Stratify the sample so that rare, high-risk intents (such as HR/finance) get enough traces. Calculate your exact sampling rate from your traffic volume and judge budget (see eval selection §9).

Key takeaways¶

• Use deterministic checks first because schema, citation validity, SQL execution, tool-call match, policy gates, and step counts are fast and free.
• Run reference-free metrics (faithfulness, relevancy, safety) online, and use reference-based metrics (correctness, recall@k, execution accuracy) to gate releases.
• Start with 1–2 metrics per diagnostic question and skip metrics that nobody will act on.
• Report every rate with a confidence interval. Compare systems with paired tests, and size zero-tolerance gates with the rule of three.
• Validate every LLM judge against human labels before it gates a release, and measure recall, false-positive rate, and latency for every guardrail.
• For agents, measure both capability (pass@k) and reliability (pass^k). Measure unsafe action execution separately from attempts.

Further reading¶

• In this repo:
  • Retrieval metrics
  • E2E RAG evals
  • Agent & trajectory evals
  • Text-to-SQL evals
  • LLM-as-a-judge
  • Running offline experiments
  • Dataset strategy §5: sizing and statistics
  • Guardrails
  • Tooling landscape
• Es et al., RAGAS (arXiv:2309.15217)
• Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv:2306.05685)
• Chen et al., Evaluating Large Language Models Trained on Code (arXiv:2107.03374), the unbiased pass@k estimator
• Yao et al., τ-bench (arXiv:2406.12045), pass^k
• Chapelle et al., Expected Reciprocal Rank for Graded Relevance (CIKM 2009)
• Moffat & Zobel, Rank-Biased Precision for Measurement of Retrieval Effectiveness (TOIS 2008)
• Eugene Yan, Task-Specific LLM Evals that Do & Don't Work
• OWASP, Top 10 for LLM Applications 2026 and Top 10 for Agentic Applications 2026