Eval and Observability Tools: The Landscape (October 2026 snapshot)¶

TL;DR

• Separate your system of record from metric libraries. Pick one platform for traces, datasets, and experiments, and use swappable libraries for metrics.
• Instrument once with OpenTelemetry. Use OpenInference or OTel GenAI semantic conventions so you can switch backends without rewriting application code.
• Calibrate every judge on your own data. Two tools with "faithfulness" metrics use different prompts and scales, so validate against human labels.
• Write domain metrics as deterministic code. Check SQL execution accuracy, RBAC leakage, and tool-argument schemas directly in code.

• You will learn:
  • What each major eval tool does best
  • Which metrics each tool ships
  • Capability matrix across 8 dimensions
  • Other tools to recognize, including dormant and discontinued ones
  • How to run a bake-off
• Prerequisites: The evals & guardrails map, End-to-end RAG evals
• Related: Guardrail tools, Observability & tracing, Running offline experiments, LLM-as-a-judge

Read this first: this is a dated snapshot¶

This document reflects the ecosystem as of October 2026. The LLM tooling market moves faster than any other part of this repository. In the 18 months before this snapshot, vendors acquired, rebranded, re-licensed, rewrote, or sunset several tools (see Ownership and status changes).

• Check current project status: Review the project's README, changelog, latest release date, license, and pricing page before you adopt it.
• Verify capabilities directly: Treat the capability matrix as a starting hypothesis because vendors ship new features every month.
• Calibrate every metric on your data: Never trust a metric by name alone because prompts, judge models, and scales differ (LLM-as-a-judge).

The concepts in this repository (datasets, metrics, judges, and guardrail architecture) are durable. Tool names change quickly, so design your stack to make tools replaceable.

The landscape at a glance¶

flowchart TD
    A["Your agentic RAG app"] -->|"OpenTelemetry / SDK spans"| B["System of record: traces, datasets, experiments, annotations"]
    B --> C["Offline experiments on datasets"]
    B --> D["Online evals on sampled production traces"]
    C --> E["Metric libraries and judges"]
    D --> E
    E --> F["Scores, dashboards, CI gates, alerts"]

    subgraph L1["A. Open-source eval libraries"]
      E1["RAGAS / DeepEval / TruLens / promptfoo / Inspect / Giskard / ARES / OpenAI Evals"]
    end
    subgraph L2["B. Eval + observability platforms"]
      B1["LangSmith / Langfuse / Phoenix + Arize AX / Braintrust / W&B Weave / MLflow / Opik / Confident AI / Galileo"]
    end
    subgraph L3["C. Cloud-native services"]
      C1["Microsoft Foundry / Amazon Bedrock + AgentCore / Google Gen AI evaluation service"]
    end
    E -.-> L1
    B -.-> L2
    B -.-> L3

Heuristic: Most production teams combine one platform (B or C) + one or two libraries (A) + custom code for domain checks like SQL execution accuracy and RBAC leakage.

Ownership and status changes to know about¶

These changes affect procurement, long-term maintenance risk, and data-residency reviews. We verified all entries against official project pages in October 2026.

A. Open-source eval libraries¶

Each card covers six fields: Link · Type/License · Strengths · Notable metrics · Best for · Watch-outs. "Notable metrics" lists documented metrics at snapshot time. Check the official docs because metric names change across versions.

RAGAS¶

DeepEval (and DeepTeam)¶

TruLens¶

promptfoo¶

OpenAI Evals (repo) and the OpenAI Evals API¶

Inspect (UK AI Security Institute)¶

Giskard (open source, v3)¶

lm-evaluation-harness (EleutherAI): for model benchmarking context¶

ARES¶

B. Eval + observability platforms¶

These platforms act as candidate systems of record. They store traces, datasets, experiment runs, human annotations, and scores, and they run evaluators on live traffic. See Observability & tracing for what to record in spans.

LangSmith (LangChain)¶

Langfuse¶

Arize Phoenix and Arize AX¶

Braintrust¶

W&B Weave¶

MLflow (GenAI evaluation & tracing)¶

Comet Opik¶

Galileo (now Splunk Agent Observability)¶

Patronus AI¶

Humanloop (sunset)¶

Confident AI¶

Other observability and eval tools¶

These tools do not have full cards, but you will see them in shortlists. We did not score them in the comparison matrix. Check each one with the same bake-off before you adopt it.

C. Cloud-native evaluation services¶

Choose a cloud-native service when your organization already standardizes on that cloud and wants unified IAM, private networking, billing, and model catalog access. Expect tighter coupling to that cloud's regions and models.

Microsoft Foundry (formerly Azure AI Foundry): evaluations & observability¶

Amazon Bedrock evaluations (+ AgentCore Evaluations)¶

Google Gen AI evaluation service (Vertex AI → Gemini Enterprise Agent Platform docs)¶

Comparison matrix¶

Legend:

• ✅ first-class / native
• 🟡 partial, via integration, preview, or enterprise-only
• ❌ not a primary capability (may exist in limited form)
• ❔ not verified (ask the vendor)

For libraries, "Self-host" means the library runs entirely in your environment, even if it calls an external LLM judge API.

This matrix reflects our assessment of public documentation in October 2026. Verify capabilities before you decide.

We omitted Patronus AI (commercial platform status unclear) and Humanloop (sunset) from the matrix.

What the matrix does not capture¶

• Domain metrics: No tool checks whether your text-to-SQL agent leaked rows forbidden by the user's role. You must write those checks yourself (Text-to-SQL evals, Access control & RBAC).
• Judge alignment: A ✅ for judge customization means you can edit the prompt, not that the default judge agrees with your domain experts.
• Scale and cost: Trace ingestion pricing, judge token costs, and retention limits often drive total cost more than base license fees.
• Enterprise security: Check SSO/SCIM, internal RBAC, audit logs, pre-ingestion PII redaction, and data residency.

Same concept, different names: a metric crosswalk¶

Different tools use different names and implementations for the same core concept. Because their underlying prompts and scoring logic differ, never compare raw scores across tools.

See End-to-end RAG evals, Retrieval metrics, and Agent, tool & trajectory evals for mathematical definitions and formulas.

How to choose¶

Step 1: Decide the system of record first¶

Your system of record stores traces, datasets, experiments, and annotations. Because it is the hardest piece to replace, choose your platform first and select metric libraries second.

flowchart TD
    S["Start"] --> Q1{"Must all prompts, traces and data stay in your network?"}
    Q1 -->|"Yes"| Q1a{"OSI open-source license required?"}
    Q1a -->|"Yes"| R1["Langfuse, Opik or MLflow self-hosted"]
    Q1a -->|"No"| R2["Also consider Phoenix ELv2 or enterprise self-host of LangSmith or Braintrust"]
    Q1 -->|"No"| Q2{"Standardised on one cloud and its agent runtime?"}
    Q2 -->|"Azure"| R3["Microsoft Foundry evaluations and tracing"]
    Q2 -->|"AWS"| R4["Bedrock evaluations and AgentCore"]
    Q2 -->|"GCP"| R5["Gen AI evaluation service"]
    Q2 -->|"No or multi-cloud"| Q3{"Primary framework?"}
    Q3 -->|"LangChain or LangGraph"| R6["LangSmith"]
    Q3 -->|"Databricks or existing MLflow"| R7["MLflow"]
    Q3 -->|"Mixed or none"| R8["Shortlist 2-3 of Langfuse, Phoenix or AX, Braintrust, Opik, Confident AI and run a bake-off"]

Step 2: Add libraries for metrics and CI¶

Step 3: Rules of thumb¶

1. Instrument with OpenTelemetry: Use OpenInference or the OTel GenAI semantic conventions so your traces stay portable across Langfuse, Phoenix, MLflow, Foundry, and AgentCore. The GenAI conventions now live in their own repository and still have Development status, so attribute names can change. Pin the semconv version in your instrumentation and map attributes in your OTel collector if a backend lags.
2. Own your datasets: Store golden datasets as JSONL or Parquet in Git or object storage, and treat your eval tool as a synced cache (Dataset strategy).
3. Own your judge prompts: Version judge prompts in Git and calibrate them against SME labels using TPR, TNR, and Cohen's $\kappa$ (LLM-as-a-judge).
4. Pin library and judge versions: Lock both your eval library version and your judge model snapshot so silent upgrades do not shift your scores.
5. Write domain checks in plain code: Implement SQL execution accuracy, RBAC leakage, citation checks, and tool-schema validation as fast, deterministic code.
6. Run only one system of record: Avoid splitting traces and datasets across multiple platforms. Use one platform and as many metric libraries as you need.
7. Budget judge token costs: Multiply examples, metrics, judge calls per metric, and token price before you schedule nightly runs.

Step 4: Run a bake-off on your own data¶

Vendor feature lists look similar on paper, so test how each tool fits your data and workflow. Time-box your bake-off to 1–2 weeks.

• Select 2–3 candidate platforms using the decision flowchart above.
• Instrument one branch of your app with OpenTelemetry so you can send identical traces to each candidate.
• Load the same golden dataset of 100–200 RAG, SQL, and tool-call examples into each platform.
• Implement 2–3 core metrics (such as faithfulness, SQL execution accuracy, and citation validity) in each tool.
• Compare built-in judges against SME labels for agreement, false-pass rate, cost, and latency. Use the bar in LLM-as-a-judge: ≥ 100 labelled items with ≥ 30–50 FAIL examples, TPR and TNR ≥ 0.85, and Cohen's $\kappa$ ≥ 0.60 (≥ 0.90 for high-risk slices).
• Test the SME annotation workflow: Verify that a reviewer can grade a trace in under 1 minute and promote failures to a dataset in two clicks.
• Test online production evaluations: Sample 5% of live traces, run a groundedness judge, and trigger a test alert.
• Test CI regression gating: Configure a pull request check that fails on newly failing items in critical slices, or on a paired test against a re-run baseline (Running offline experiments).
• Verify security and governance: Check SSO, role-based access, pre-ingestion PII redaction, retention controls, data residency, and audit logs.
• Project 12-month total cost based on your expected production trace volume and judge call frequency.
• Verify your exit path: Export a full trace, dataset, score, and annotation bundle in an open format like JSONL or Parquet.

A reference stack for the running example¶

Our running example is an internal enterprise assistant. It answers questions over company docs using vector search, queries a data warehouse via text-to-SQL, creates support tickets, and enforces per-user RBAC.

flowchart LR
    APP["Assistant: RAG + text-to-SQL + ticket tool"] -->|"OTel spans with user role, retrieved doc IDs, SQL, tool args"| SOR["System of record e.g. Langfuse or Phoenix self-hosted, or a cloud-native equivalent"]
    GOLD["Golden + synthetic datasets in git or object storage"] --> CI["CI job: pytest with DeepEval or RAGAS metrics + custom SQL and RBAC checks; promptfoo red-team suite"]
    CI -->|"scores, experiment runs"| SOR
    SOR -->|"sampled traces"| ONLINE["Online evaluators: groundedness judge, SQL error rate, refusal rate"]
    ONLINE --> ALERT["Dashboards and alerts"]
    SOR -->|"failing traces"| QUEUE["SME annotation queue"]
    QUEUE --> GOLD

Common mistakes¶

Key takeaways¶

• Separate your system of record from your metric libraries: Choose one platform for traces and datasets, and keep metric libraries swappable.
• Prevent vendor lock-in early: Instrument with OpenTelemetry and store your datasets and judge prompts in your own Git repository.
• Never equate metrics by name alone: Calibrate every judge on your domain data and never compare raw scores across tools or versions.
• Filter by data residency first: Self-host Langfuse, Opik, or MLflow (OSI licenses) or Phoenix (ELv2) when data cannot leave your network.
• Write domain checks in code: Implement SQL execution accuracy, RBAC leakage, and citation checks as deterministic code.
• Re-verify tool status yearly: The market consolidates quickly (Humanloop sunset, promptfoo → OpenAI, Langfuse → ClickHouse, Galileo → Cisco/Splunk, WhyLabs discontinued).

Further reading¶

• Guardrail tools: companion landscape for runtime guardrails, policy engines, and red teaming
• Observability & tracing: what to record on every trace and span
• Running offline experiments: experiment design, statistical significance, and CI gates
• Implementation playbook: where tool selection fits in your rollout sequence
• Hamel Husain, Your AI Product Needs Evals: why error analysis must come before tool selection
• Eugene Yan, Evaluating the Effectiveness of LLM-Evaluators: how to validate any tool's LLM judge
• OpenAI, Evaluation best practices
• Anthropic, Demystifying evals for AI agents
• References: full curated reading list