Skip to content

03 · Online Evals & Observability

TL;DR

  • Instrument full traces first. Capture every retrieval, rerank, LLM call, tool call, SQL query, and guardrail decision in production.
  • Evaluate live traffic without ground truth. Run deterministic checks on 100% of requests and reference-free LLM judges on samples.
  • Combine user signals with monitoring. Link explicit and implicit feedback to traces, track segmented KPIs, and detect drift early.
  • Close the production flywheel. Route production failures and incident findings back into your offline evaluation datasets.

Offline evals tell you whether a version is ready to ship. This section covers what happens after you ship. You trace every agent step, evaluate live traffic without ground truth, and collect user feedback. You also monitor KPIs and SLOs, detect drift, and respond to AI quality incidents. Every online signal feeds back into your datasets and offline evals. This loop keeps agentic RAG applications improving instead of decaying.

This section uses an internal enterprise assistant as a running example. The assistant answers questions over company documents with vector search and queries a data warehouse with text-to-SQL. It also calls tools such as create_ticket and serves users across different roles (RBAC).

Files in this section

File What it covers Read when…
01-online-evals.md Offline vs online evals, deterministic checks, sampled async LLM judges (and hardening them against prompt injection), guardrail signals, sample sizing with Wilson intervals and judge-error correction, shadow/canary/A/B/interleaving tests with sequential testing and CUPED, metric mapping, and dataset feedback loops You are launching or operating in production and need to measure live traffic
02-observability-and-tracing.md Span model for agentic RAG (retrieval, rerank, LLM, tools, SQL, guardrails), logging spec, OTel GenAI (attributes, content capture, evaluation events, metrics) and OpenInference conventions, version tagging, PII redaction, linking delayed signals, telemetry cardinality, and example trace You are instrumenting an application or cannot answer "why did the agent say that?"
03-user-feedback-and-implicit-signals.md Explicit feedback (thumbs, reasons, corrections), implicit signals (rephrase, regenerate, escalation, ticket outcomes, copy), biases, feedback UX, trace linking, and error analysis You are designing feedback UX or want to use user signals without bias
04-monitoring-alerting-and-drift.md KPI catalog, dashboards, AI SLOs and error budgets, multi-window burn-rate alerting, query/corpus/model/judge drift detection, drift statistics (PSI, KS, embedding drift), incident runbooks, kill switches, and re-evaluation cadence You are setting up dashboards and on-call alerts, or debugging a silent quality drop

Suggested reading order

  1. Start with Observability & tracing. Every other practice in this section depends on complete traces.
  2. Read Online evals. Learn which deterministic checks and judges to run on your traces.
  3. Read User feedback & implicit signals. Capture explicit and behavioral signals from users and link them to traces.
  4. Read Monitoring, alerting & drift. Turn traces, scores, and feedback into dashboards, alerts, and incident runbooks.

Short on time? Read the "What to do first" and "Getting started" sections in 01 and 04, plus the logging spec tables in 02.

How this section connects to the rest

flowchart LR
    OFF["Offline evals and CI gates"] --> PROD["Production agent"]
    PROD --> TR["Traces"]
    TR --> OE["Online evals"]
    TR --> FB["User feedback"]
    OE --> MON["Monitoring, alerts, drift"]
    FB --> MON
    MON --> IR["Incident response"]
    OE --> EA["Error analysis"]
    FB --> EA
    IR --> EA
    EA --> DS["Datasets"]
    DS --> OFF