Monitoring, Alerting & Drift¶

TL;DR

• Segment KPIs by version, role, and intent. Track quality, retrieval, safety, SQL, latency, and cost metrics with confidence intervals across slices.
• Set a few user-facing SLOs and hard safety invariants. Four to seven SLOs is a common starting point, not a rule. Alert on multi-window budget burn or hard invariant breaches rather than noisy single-point spikes.
• Use real statistics for drift and regressions. Compare rates with a two-proportion or sequential test, read PSI and KS with effect sizes, and correct for many slices.
• Monitor all four drift axes. Detect shifts in user queries, corpus and warehouse schemas, hosted model snapshots, and LLM judge behavior.
• Prepare an AI incident runbook and kill switches. Use runtime config rollbacks, tool kill switches, and mandatory regression tests for every incident.

• You will learn:
  • Segmented KPI catalog and dashboard design
  • AI SLOs and low-fatigue alerting
  • Query, corpus, model, and judge drift detection
  • Drift statistics: PSI, KS, Wasserstein, and embedding drift, with thresholds and caveats
  • Incident response runbooks, kill switches, and re-evaluation cadences
• Prerequisites: Online evals, Observability & tracing
• Related: User feedback, Guardrails architecture, Running offline experiments, Maturity model, Implementation playbook

1. What makes AI monitoring different¶

Traditional service monitoring tracks the four golden signals: latency, traffic, errors, and saturation (Google SRE book). Agentic RAG systems need those metrics too, but an agent can return HTTP 200 in 800 ms while hallucinating financial numbers. AI monitoring differs from classic APM in three ways:

1. Quality is a statistical metric, not an HTTP status code. Quality signals come from online evals, so they are sampled, asynchronous, and noisy.
2. External dependencies shift silently. Prompts, hosted models, document corpora, warehouse schemas, user queries, and LLM judges all change over time.
3. Failures hide inside narrow slices. A regression often hits only one user role, one data source, one language, or one intent cluster.

Always monitor metrics by slice and by version, and treat drift detection as a core operational responsibility.

2. KPIs and dashboards¶

2.1 KPI catalogue¶

2.2 Dashboard layout (suggested)¶

Organize your monitoring workspace into seven focused views:

1. Executive health overview: Total traffic, p95 latency, error rate, daily cost, unsupported-answer rate, 👎 rate, and guardrail block rate with week-over-week deltas.
2. Quality breakdown: Sampled LLM judge rates with 95% confidence intervals, sliced by user role, intent cluster, data source, and config_hash.
3. Retrieval & corpus health: Empty and weak retrieval rates, index ingestion lag, top zero-result queries, and document age distributions.
4. Text-to-SQL health: Parse and execution failure rates by error class and warehouse table, empty-result rate, and judged SQL alignment.
5. Safety & guardrails: Trigger rates per guardrail and action (allow, redact, block), top blocked categories, and human override rates.
6. Cost & efficiency: Cost per session by model, input/output/cached token distributions, agent step counts, and semantic cache hit rates.

7. Release comparison view: All core KPIs split by config_hash, prompt version, and model snapshot to compare canary vs. control arms.

8. Segment every chart by config_hash, user role, intent cluster, data source, and channel so aggregate averages do not hide slice regressions.

9. Plot confidence intervals on sampled metrics (see Online evals §4.1) so engineers do not chase random sampling noise.

3. SLOs for AI features¶

A Service Level Indicator (SLI) measures a specific performance or quality rate, a Service Level Objective (SLO) sets the target threshold over a time window, and the error budget is the share of failures the SLO allows over that window: $1 - \text{SLO}$. For a 95% SLO, the budget is 5% of events. An error-budget policy is a separate decision: it states what the team does when the budget runs out, for example a release freeze (see the Google SRE book on SLOs and Embracing risk, and the workbook's error budget policy).

3.1 Example SLOs for the running example (heuristic starting points)¶

Follow these rules when setting AI SLOs:

• Keep the number of SLOs small. Four to seven user-facing indicators is a common heuristic. Keep component metrics as diagnostic indicators rather than paging SLOs.
• Pin the judge version behind quality SLOs. Upgrading a judge model changes the measurement scale, so re-baseline your SLO whenever you update a judge (Section 5.4).
• Never assign error budgets to security invariants. Treat RBAC leaks or unauthorized write actions as immediate incidents.
• Write an error-budget policy. Agree in advance what happens when the budget runs out. For example, when a rollout burns the weekly grounded-answer budget, freeze feature changes and prioritize quality fixes.

4. Alert design¶

4.1 Principles¶

• Page on user-visible symptoms, such as quality drops, high latency, or safety breaches, and use component metrics for root-cause diagnosis.
• Attach a runbook link and pre-filtered trace query to every alert notification (Section 6).
• Require sustained deviations before firing quality alerts because sampled LLM metrics fluctuate over short windows.
• Route alerts by severity. Page on-call engineers only for sev-1 and sev-2 incidents, and send sev-3 anomalies to tickets or Slack channels.
• Prune noisy alerts every month. Delete or recalibrate any alert that fired without requiring human action.

4.2 Alert types¶

4.3 Avoiding alert fatigue¶

• Enforce minimum sample sizes on rate alerts (for example, require at least 100 feedback events or 200 judged traces in the window before firing).
• Configure hysteresis thresholds so an alert fires at $X$ but resolves only after dropping below $Y$.
• Use seasonality-aware baselines that account for weekday vs. weekend traffic and month-end finance spikes.
• Group correlated alerts into a single incident. When an LLM provider outage spikes latency, errors, fallbacks, and refusals simultaneously, page once.
• Overlay deployment and config-change markers on every dashboard chart to speed up root-cause correlation.
• Link every alert directly to a sliced breakdown view so the responder sees the failing role, intent, or version immediately.

4.4 Example alert rules (illustrative)¶

The burn rate is how fast you spend the error budget. A burn rate of 1 spends exactly the whole budget over the SLO window. The SRE workbook recommends pairing a long window, which shows the problem is significant, with a short window about 1/12 as long, which shows the problem is still happening. Its starting values for a 30-day SLO are: page at 14.4× over 1 hour and 5 minutes (2% of budget spent), page at 6× over 6 hours and 30 minutes (5%), and ticket at 1× over 3 days and 6 hours (10%). These pairs need many events per window. Sampled judge metrics have few events per hour, so use longer windows and minimum sample counts for them, as in the example below.

# illustrative: adapt to your alerting system's syntax
- name: rbac_citation_leak
  expr: sum(increase(app_citations_leaked_total[5m])) > 0
  severity: sev1
  action: page security + AI on-call
  runbook: rbac-leak

# Multi-window burn rate, 30-day latency SLO (95% of turns < 6 s, budget 5%).
# Pairs and factors from the SRE workbook: 14.4x over 1h AND 5m; 6x over 6h AND 30m.
- name: latency_slo_fast_burn
  expr: slow_turn_ratio_1h > 14.4 * 0.05 and slow_turn_ratio_5m > 14.4 * 0.05
  severity: sev2
  runbook: latency-regression
- name: latency_slo_slow_burn
  expr: slow_turn_ratio_6h > 6 * 0.05 and slow_turn_ratio_30m > 6 * 0.05
  severity: sev2
  runbook: latency-regression

# Sampled judge SLO (95% supported over 7 days, budget 5%). Judges see only ~500 turns
# a day, so use longer window pairs and a minimum sample size. not_supported = 1 - supported.
- name: grounded_answer_slo_burn
  expr: >
    not_supported_rate_24h > 3 * 0.05 and judged_samples_24h >= 200
    and not_supported_rate_6h > 3 * 0.05 and judged_samples_6h >= 50
  severity: sev2
  runbook: quality-regression

- name: empty_retrieval_anomaly
  expr: empty_retrieval_rate_30m > 2 * empty_retrieval_rate_same_hour_last_week and requests_30m > 200
  severity: sev3
  runbook: retrieval-degradation

- name: sql_unknown_column_spike
  expr: rate(sql_errors_total{class="unknown_column"}[15m]) > 5 * baseline
  severity: sev3
  runbook: schema-drift

- name: cost_per_session_p95
  expr: cost_per_session_p95_1h > 2 * budget
  severity: sev3
  runbook: runaway-agent

5. Drift¶

Drift occurs when user queries, underlying knowledge, model behavior, or evaluation judges change after offline validation. Agentic RAG systems experience four distinct types of drift:

flowchart LR
    Q["Query drift: users ask new things"] --> SYS["Agentic RAG system"]
    K["Corpus and schema drift: docs and tables change"] --> SYS
    M["Model and provider drift: the LLM changes"] --> SYS
    SYS --> MEAS["Online evals"]
    J["Judge drift: the evaluator changes"] --> MEAS
    MEAS --> DEC["Dashboards, alerts, decisions"]

5.1 Input / query drift¶

What changes: The distribution of user prompts shifts due to new product launches, company reorganizations, newly onboarded departments, seasonal events (such as open enrollment or quarterly close), or new languages.

How to detect it:

• Track embedding cluster shifts. Cluster query embeddings on a baseline reference window and monitor three signals on incoming traffic:
  • Shifts in traffic share across existing clusters.
  • Rising novelty rate (share of queries whose nearest centroid distance exceeds threshold $r$).
  • Dense new clusters formed by previously unseen topics.
• Measure categorical distribution distance. Compute Population Stability Index (PSI), Jensen–Shannon divergence, or chi-square statistics on intent, user role, language, and selected tools using libraries such as Evidently. See Section 5.6 for thresholds and caveats.
• Watch downstream retrieval symptoms: Rising weak-retrieval rates, no_context refusals, and user rephrase rates within a specific slice.

How to respond:

• Sample traces from novel clusters, run error analysis, and add labeled examples to your offline datasets (see Production data flywheel).
• Verify whether the knowledge corpus actually covers the new topic before tuning prompts or retrievers.
• Update intent routers and topic guardrails if the new queries belong in or out of scope.

Simple sketch: novelty rate

import numpy as np

def novelty_rate(query_embs: np.ndarray, centroids: np.ndarray, radius: float) -> float:
    """Share of queries whose nearest reference centroid is farther than `radius` (cosine distance).
    Calibrate `radius` as, e.g., the 95th percentile distance on the reference window (heuristic)."""
    q = query_embs / np.linalg.norm(query_embs, axis=1, keepdims=True)
    c = centroids / np.linalg.norm(centroids, axis=1, keepdims=True)
    nearest_dist = 1 - (q @ c.T).max(axis=1)
    return float((nearest_dist > radius).mean())

5.2 Corpus / knowledge drift (and schema drift)¶

What changes: Your knowledge sources evolve. Authors add, edit, or delete policy documents, permissions change, warehouse tables or columns change, or the ingestion pipeline lags behind the source of truth.

Common failure modes:

• Stale vector index: A policy document changes, but the index still serves the old chunk. The assistant generates a response that is 100% faithful to the retrieved text yet factually outdated; faithfulness judges cannot catch this failure.
• Orphaned or ACL-mismatched chunks: Deleted or permission-restricted documents remain searchable in the vector index (see Retrieval & context guardrails).
• Conflicting document versions: Multiple editions of the same policy coexist in the index, causing the generator to blend old and new rules.
• Warehouse schema drift: Data engineers rename columns, deprecate tables, or change metric definitions (such as revenue switching from gross to net), causing SQL syntax errors or silent calculation errors.

How to detect it:

• Monitor ingestion pipelines: Alert on failed sync jobs, parsing errors, embedding failures, and lag between source edits and index updates.
• Run periodic index-to-source reconciliation: Compare document IDs, modification timestamps, and ACLs in the vector index against the source repository.
• Track retrieved-document age: Monitor the age distribution of cited chunks and flag citations to documents past their review-by date.
• Subscribe to warehouse DDL change feeds: Automatically diff the live database catalog against the schema descriptions embedded in your text-to-SQL prompts.
• Re-verify golden datasets after major corpus updates so outdated reference answers do not fail valid model outputs (Section 7).

How to respond: Re-index updated documents, purge superseded chunks, prefer newer effective_date metadata during reranking, update SQL schema prompts, and refresh invalidated golden reference answers.

5.3 Model / provider drift¶

What changes: Hosted LLM behavior changes even when you deploy zero code changes. Model providers update weights behind unpinned aliases, tweak default safety filters, or deprecate endpoints (see Chen, Zaharia & Zou, "How is ChatGPT's behavior changing over time?", which compares dated snapshots of the same alias). Changes in the serving stack can also change the output of a pinned snapshot. In September 2025, Anthropic reported three infrastructure bugs (a context-window routing error, a TPU server misconfiguration, and an XLA:TPU compiler bug) that lowered Claude's output quality for some requests with no model change (postmortem).

How to detect it:

• Log the exact gen_ai.response.model string on every LLM span (see Observability & tracing) and alert whenever it changes.
• Pin dated model snapshots in production configurations and treat snapshot upgrades as full releases requiring offline evals and canary rollouts. Pinning reduces drift but does not remove it, so keep the sentinel suite running against pinned models too.
• Run a scheduled sentinel prompt suite: Execute 20–50 fixed prompts (covering JSON formatting, refusals, tool calls, and SQL) hourly or daily against production. Hosted models are not deterministic even at temperature 0 (Atil et al., 2024), so do not alert on single-output diffs. Run each prompt several times (for example 5), score each run pass/fail with deterministic checks, and compare the pass rate with the baseline using a two-proportion or binomial test. Alert on a significant drop, not on one changed answer.
• Monitor sudden changes in output token length, refusal rates, tool-call frequency, JSON schema validity, finish_reason distributions, and provider content-filter blocks.
• Track provider deprecation calendars and schedule model migrations weeks before forced cutovers.

How to respond: Roll back to a pinned snapshot or pre-tested fallback model, adjust prompt instructions if needed, and pass the full offline evaluation suite before re-promoting.

5.4 Judge drift¶

What changes: Your evaluation instrument shifts. A provider updates the judge model, an engineer edits the judge prompt or context truncation limit, or user traffic shifts into domains where the judge was never calibrated. Human standards also evolve as reviewers inspect more data, a phenomenon Shankar et al. term criteria drift in "Who Validates the Validators?".

Symptoms: Dashboard quality scores jump or drop while user feedback, deterministic checks, and system versions remain flat, or a step change coincides with a judge update.

How to detect and prevent it:

• Version every judge configuration (model snapshot + prompt template + parser + threshold) and log eval_version on every score record.
• Maintain a pinned judge calibration dataset: Score every candidate judge version against human-labeled traces before deployment. Use the bar from LLM-as-a-judge: at least 100 labeled items with at least 30–50 FAIL cases, TPR and TNR $\ge 0.85$, and Cohen's $\kappa \ge 0.60$ ($\ge 0.90$ for high-risk checks such as RBAC). Use the measured TPR and TNR to correct online rates for judge error (Rogan–Gladen; see Online evals §4.1).
• Run weekly or monthly SME spot checks: Have domain experts label a fresh random sample of judged production traces and alert if human–judge agreement drops.
• Never change the system and the judge simultaneously. When upgrading a judge, re-score a historical baseline window with both judge versions to re-baseline your dashboards and SLOs.
• Run the exact same judge version offline and online (see LLM-as-a-judge).

5.5 Drift summary¶

5.6 Drift statistics: thresholds and caveats¶

Pick the statistic by signal type, and always report an effect size next to any p-value.

Follow these rules when you apply them:

• Treat PSI thresholds as rules of thumb. The 0.10 and 0.25 values come from credit-risk practice, not from a statistical test. Their behavior depends on the number of bins and the sample size (Yurdakul, 2018). Calibrate them on your own stable weeks.
• Fix the bins on the reference window. Use quantile bins of the reference data, keep the same edges for every comparison, and add open-ended outer bins for values outside the reference range. Smooth empty bins with a small epsilon, or PSI becomes infinite.
• Expect tests to fire at large $n$. With 200,000 turns per window, KS and chi-square tests find trivially small shifts "significant". Gate alerts on the effect size and use the p-value only as a second check.
• Use a domain classifier for embeddings. Train a simple classifier to tell reference queries from current queries. An AUC near 0.5 means no detectable drift. Higher values mean drift, and the most confidently classified examples show you what changed. Rabanser et al. found that two-sample tests on low-dimensional representations detected shift best, and that domain classifiers help explain what shifted and whether it is harmful (Failing Loudly, NeurIPS 2019). MMD is a kernel two-sample test for the same job (Gretton et al., 2012). Evidently implements domain-classifier, distance, and MMD methods, and uses a domain classifier by default for text drift (docs).
• Correct for many slices. Testing 20 roles × intents per day yields false alarms by chance. Apply Benjamini–Hochberg across the slices (scipy.stats.false_discovery_control), or require the shift to persist across two windows.
• Drift is a reason to look, not a verdict. A shift in query mix is not a quality drop. Pair every drift alert with the quality metrics for the drifted slice.

The snippet below computes PSI for numeric and categorical signals and a KS test with effect sizes. It runs as shown with numpy and scipy.

# Requires numpy and scipy.
import numpy as np
from scipy import stats

def _psi(ref_pct: np.ndarray, cur_pct: np.ndarray, eps: float = 1e-4) -> float:
    ref_pct = np.clip(ref_pct, eps, None)   # smoothing: an empty bin would give log(0)
    cur_pct = np.clip(cur_pct, eps, None)
    return float(np.sum((cur_pct - ref_pct) * np.log(cur_pct / ref_pct)))

def psi_numeric(reference, current, bins: int = 10) -> float:
    """PSI for a numeric signal. Bin edges are quantiles of the reference window."""
    reference = np.asarray(reference, dtype=float)
    current = np.asarray(current, dtype=float)
    inner = np.unique(np.quantile(reference, np.linspace(0, 1, bins + 1)[1:-1]))
    edges = np.concatenate(([-np.inf], inner, [np.inf]))   # catch values outside the reference range
    ref_pct = np.histogram(reference, edges)[0] / len(reference)
    cur_pct = np.histogram(current, edges)[0] / len(current)
    return _psi(ref_pct, cur_pct)

def psi_categorical(reference: dict, current: dict) -> float:
    """PSI for counts per category (intent, role, tool). New categories count too."""
    cats = sorted(set(reference) | set(current))
    ref = np.array([reference.get(c, 0) for c in cats], dtype=float)
    cur = np.array([current.get(c, 0) for c in cats], dtype=float)
    return _psi(ref / ref.sum(), cur / cur.sum())

def continuous_drift(reference, current) -> dict:
    """KS test plus effect sizes. At large n, read the effect size, not only the p-value."""
    ks = stats.ks_2samp(reference, current)
    return {"ks_d": float(ks.statistic), "ks_p": float(ks.pvalue),
            "wasserstein": float(stats.wasserstein_distance(reference, current)),
            "psi": psi_numeric(reference, current)}

if __name__ == "__main__":
    rng = np.random.default_rng(0)
    ref = rng.normal(400, 120, 5_000)                          # e.g. output tokens last month
    print(continuous_drift(ref, rng.normal(400, 120, 5_000)))  # no change
    print(continuous_drift(ref, rng.normal(520, 120, 5_000)))  # +1 SD shift
    print(psi_categorical({"policy": 600, "sql": 300, "ticket": 100},
                          {"policy": 450, "sql": 300, "ticket": 100, "benefits_enrollment": 150}))

In the demo, a 1 standard-deviation shift in output length gives PSI ≈ 0.95 and KS $D$ ≈ 0.39, and a new intent that takes 15% of traffic gives PSI ≈ 1.1. In a separate test with 200,000 samples per window and a 0.02 standard-deviation shift, KS gives $p < 10^{-6}$ while $D < 0.01$ and PSI < 0.001: the shift is "significant" but too small to matter.

6. Incident response for AI quality incidents¶

Treat AI quality and safety failures as production incidents. A hallucinated HR policy sent to thousands of employees, an RBAC citation leak, or a runaway agent creating 500 tickets requires the same disciplined incident response as a service outage.

6.1 Severity guide (example)¶

6.2 Controls you need before an incident¶

• Put prompts, model snapshots, retriever parameters, and guardrail thresholds in versioned runtime config so rollback takes a config flip rather than a code deployment.
• Build granular kill switches at four levels:
  • Disable individual write tools (such as create_ticket) or force them into human-approval mode.
  • Disable text-to-SQL execution for specific roles or warehouse schemas.
  • Exclude a compromised or stale data source from retrieval.
  • Switch the entire assistant to search-only links or a maintenance banner.
• Pre-configure and test a fallback model on a secondary provider or pinned snapshot.
• Enforce strict rate limits on write tools per user, session, and hour to cap the blast radius of agent loops.
• Retain hot traces long enough to support forensic root-cause analysis (see Observability & tracing §5).
• Assign an AI on-call rotation and maintain a contact list for every upstream data source owner.

6.3 Runbook¶

flowchart TD
    A["Alert or user report"] --> B["Triage: confirm with traces, scope by segment and version"]
    B --> C{"Sev-1 safety or data exposure?"}
    C -->|"yes"| K["Kill switch or disable tool or data source; notify security"]
    C -->|"no"| D{"Correlated with a change?"}
    K --> D
    D -->|"prompt, model, config, guardrail"| R["Roll back to last known good version"]
    D -->|"provider model changed"| P["Pin previous snapshot or switch to fallback model"]
    D -->|"corpus or schema change"| X["Reindex, purge stale docs, fix schema docs"]
    D -->|"no change found: drift or attack"| Y["Mitigate: tighten guardrail, restrict scope, add rate limits"]
    R --> V["Verify recovery on dashboards and sentinel set"]
    P --> V
    X --> V
    Y --> V
    V --> RC["Root cause analysis using traces"]
    RC --> RT["Add regression cases to offline datasets and CI gates"]
    RT --> PM["Blameless postmortem and follow-up actions"]

Execute this step-by-step checklist during an incident:

1. Triage (first 15 minutes)
   • Inspect 10–20 affected traces from the alert link to confirm the failure is real.
   • Scope the blast radius by role, intent, data source, config_hash, and start timestamp.
   • Check recent deployments, prompt/config changes, index ingestion runs, warehouse DDL logs, and gen_ai.response.model strings.
   • Assign an incident severity level and designate an incident commander.
2. Contain (15–60 minutes)
   • Flip the appropriate kill switch immediately for any Sev-1 data leak or runaway write tool before debugging further.
   • Roll back the suspect prompt, model, or config version, or dial canary traffic down to 0%.
   • Reverse or quarantine bad write actions (for example, bulk-close duplicate tickets via API) and notify affected teams.
   • Place exposed traces under legal/security hold and restrict trace viewer permissions if an RBAC leak occurred.
3. Verify recovery
   • Confirm deterministic metrics return to baseline, oversample online judges on post-rollback traffic, and run the sentinel prompt suite.
4. Diagnose root cause
   • Identify the exact failing component in the trace tree (see User feedback §7) and determine why offline CI gates missed it.
5. Prevent recurrence
   • Add redacted failure traces to your offline regression dataset and verify that the CI gate fails on the broken version.
   • Add or tighten the online check or alert rule that would have caught the issue faster.
   • Add a runtime guardrail if the failure mode requires synchronous prevention (see Guardrail selection decision tree).
   • Conduct a blameless postmortem (see SRE workbook: postmortem culture) that documents both the system bug and the evaluation gap.

6.4 Example: runaway ticket creation¶

• Alert signal: create_ticket call volume spikes to $20\times$ baseline within 30 minutes across a small cluster of sessions.
• Triage: Trace inspection reveals an agent retry loop caused by a ticketing API response schema update that made successful ticket creations look like failures.
• Containment: Flip the create_ticket kill switch to require human approval and bulk-close the duplicate tickets via the ticketing API.
• Code fix: Update the tool response parser to accept both payload schemas, add a per-session idempotency key, and enforce a max-calls-per-session rate limit.
• Prevention: Add a multi-step trajectory test to the offline suite, add a loop-detection deterministic check online, and alert on write-tool call volume spikes (see Agent tool & trajectory evals and Tool & action guardrails).

7. Periodic re-evaluation cadence¶

Even when no alerts fire, production systems slowly drift. Run this recurring evaluation schedule:

8. Getting started¶

1. Build one deterministic health dashboard first. Plot latency, error rates, token cost, guardrail trigger rates, empty retrieval, SQL error rates, and refusal rates sliced by version and user role.
2. Enable paging alerts on hard invariants. Alert immediately on RBAC citation leaks, runaway write-tool volume, and session cost spikes.
3. Add one sampled faithfulness judge KPI. Plot daily supported-answer rates with 95% confidence intervals and review flagged traces weekly.
4. Log gen_ai.response.model and schedule a sentinel prompt set. Catch silent provider model updates within hours.
5. Implement runtime config rollback and tool kill switches before your first production incident occurs.
6. Write a one-page AI incident runbook and rehearse it in a 30-minute tabletop exercise with on-call engineers.
7. Layer on automated drift detection (query embedding clusters, index-to-source reconciliation, and warehouse schema diffs) as traffic and corpus size grow.

Key takeaways¶

• Track quality, retrieval, safety, behavior, tool/SQL, latency, cost, and reliability KPIs sliced by version, role, intent, and data source, and display confidence intervals on all sampled metrics.
• Maintain a small set of user-facing SLOs (four to seven is a common heuristic), and enforce zero-tolerance hard invariants for RBAC leaks and unauthorized write actions. The error budget is $1 - \text{SLO}$; what you do when it runs out is a separate error-budget policy.
• Design low-fatigue alerts using minimum sample sizes, multi-window burn rates, seasonal baselines, and direct links to runbooks and filtered traces.
• Compare rates with a two-proportion or sequential test, not with overlapping confidence intervals or a fixed percentage-point gap.
• Read drift statistics as effect sizes: PSI thresholds are rules of thumb, KS p-values shrink with $n$, and many slices need Benjamini–Hochberg correction.
• Monitor all four drift axes (query, corpus/schema, model/provider, and judge), because faithfulness judges cannot detect stale source documents, pinned models can still change through serving bugs, and judges themselves drift.
• Prepare for AI incidents ahead of time with granular kill switches, runtime config rollback, fallback models, write-tool rate limits, and a runbook that mandates an offline regression test for every incident.
• Follow a disciplined re-evaluation cadence: daily sentinel runs, weekly error analysis, monthly full benchmarks, and quarterly golden-dataset refreshes.

Further reading¶

• Google SRE, Service Level Objectives, Embracing Risk, Monitoring Distributed Systems, Alerting on SLOs, Error Budget Policy, Postmortem Culture
• Schenker & Gentleman, On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals, The American Statistician, 2001
• Anthropic, A postmortem of three recent issues (2025; quality loss from serving bugs, with no model change)
• Atil et al., Non-Determinism of "Deterministic" LLM Settings
• Chen, Zaharia & Zou, How is ChatGPT's behavior changing over time?
• Shankar et al., Who Validates the Validators? (criteria drift and evaluator alignment)
• Evidently, open-source data & ML monitoring / drift detection (docs), Embedding drift detection
• Yurdakul, Statistical Properties of Population Stability Index (PhD dissertation, Western Michigan University, 2018)
• Rabanser, Günnemann & Lipton, Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift (NeurIPS 2019)
• Gretton et al., A Kernel Two-Sample Test (JMLR 2012; MMD)
• NIST, AI Risk Management Framework (the Measure and Manage functions cover ongoing monitoring)
• OWASP, GenAI Security Project / Top 10 for LLM Applications
• Chip Huyen, Building A Generative AI Platform
• Applied LLMs, What We've Learned From A Year of Building with LLMs
• Within this repo: Online evals, Observability & tracing, Maturity model, Implementation playbook