Eval & Guardrail Maturity Model¶

TL;DR

• Progress through five observable levels. Advance from Level 0 (ad hoc vibes) to Level 4 (data flywheel and governance).
• Fix your weakest dimension first. Your overall maturity is capped by your lowest-scoring dimension and by the gating practices of each level (for example, no CI gates means Level 1 at most).
• Match your target level to your risk tier. Reach at least Level 2 before a Tier 2 launch and Level 3 (evals and guardrails) before a Tier 3 launch.
• Advance one level at a time. Score yourself in 20 minutes with §4 and pick your next two actions from §5.

• You will learn:
  • What each maturity level looks like in practice
  • Capabilities across five dimensions
  • How to level up step by step
  • How to score your system in 20 minutes
• Prerequisites: Implementation playbook
• Related: Eval selection, Guardrail selection, Why evals and guardrails, Production data flywheel

1. Why a maturity model¶

• Decide what to build next. Teams often buy an observability platform (Level 3) before they create a single labelled example (Level 1).
• Match rigor to risk. Level 1 suffices for a low-risk internal prototype. A Tier 3 agent handling HR or finance data or tool-enabled turns in production requires Level 3 evals and guardrails. Tiers are defined in the risk questionnaire; controls per tier are in risk tiers.
• Communicate status clearly. Stating "We are at Level 2 on evaluations but Level 1 on guardrails" aligns engineering, leadership, and security teams.

The levels are cumulative. Level 3 requires all Level 2 practices to remain active. Each level also has gating practices (§4.6): you are not at a level until all of its gating practices are in place, whatever your total score.

Minimum level by risk tier (heuristic)¶

Tier 3 needs Level 3 evals as well as guardrails: incident regression sets, judge re-validation, and ongoing monitoring are what regulated and high-blast-radius systems are expected to show (see NIST AI 600-1). Before launch, "in place" for online practices means configured, alerting, and exercised on shadow or canary traffic.

2. The five levels¶

flowchart LR
    L0["Level 0: Ad hoc vibes"] --> L1["Level 1: Basic test set"]
    L1 --> L2["Level 2: Systematic offline evals"]
    L2 --> L3["Level 3: Continuous online evals and guardrails"]
    L3 --> L4["Level 4: Data flywheel and governance"]

Level 0: Ad hoc vibes¶

"We tried a bunch of questions and it looked good."

• Capabilities: None systematic.
• Acceptable for: Throwaway spikes only.

Level 1: Basic test set¶

"We have a spreadsheet of questions we run before demos."

Typical gaps: You lack error analysis, component metrics, versioned datasets, and retrieval or tool traces. Your scores cannot be compared over time, and your guardrails rely on prompt instructions.

Level 2: Systematic offline evals¶

"Every PR runs our eval suite, and we know which component broke."

Typical gaps: You have no online quality signals beyond user feedback, and you do not measure guardrail false-positive rates. Production failures do not flow back into datasets automatically, and red teaming happens only once.

Level 3: Continuous online evals + guardrails¶

"We find out about quality drops from our alerts, not from users."

Typical gaps: A few individuals still drive improvements manually, and metrics do not connect to business KPIs. Teams lack shared standards and automated governance artifacts.

Level 4: Data flywheel & governance¶

"Production data continuously improves the system, and we can show an auditor how."

Typical gaps at Level 4: Governance overhead can slow iteration if you do not automate evidence collection.

3. Capability matrix¶

Rows represent dimensions. Cells describe what "done" looks like at each level.

4. Self-assessment scorecard¶

Score each statement 0 = no, 1 = partially, or 2 = yes, consistently. Score a 0 if you only have an unimplemented backlog ticket.

4.1 Datasets (max 10)¶

4.2 Offline evals (max 12)¶

4.3 Online evals & observability (max 10)¶

4.4 Guardrails & security (max 14)¶

4.5 Process & governance (max 10)¶

4.6 Interpreting the score¶

The maximum total score is 56. Your effective level is the lowest of three numbers:

1. The level of your total score (as a percentage of 56).
2. The level of your weakest dimension (its percentage, with the same bands).
3. The highest level whose gating practices are all in place. Score 2 on every gating statement for that level and every level below it.

scores = {
    "datasets": [2, 1, 2, 2, 0],
    "offline": [2, 2, 2, 2, 1, 2],
    "online": [2, 1, 1, 0, 0],
    "guardrails": [2, 2, 1, 2, 1, 1, 1],
    "process": [2, 2, 1, 1, 1],
}
# (dimension, 1-based statement index) that must score 2 for each level
GATES = {
    1: [("online", 1)],
    2: [("datasets", 1), ("offline", 4), ("guardrails", 1), ("process", 2)],
    3: [("online", 2), ("online", 4), ("guardrails", 5), ("guardrails", 6), ("guardrails", 7)],
    4: [("datasets", 5), ("process", 5)],
}
BANDS = [(0.83, 4), (0.62, 3), (0.37, 2), (0.15, 1), (0.0, 0)]


def level(fraction: float) -> int:
    return next(lvl for cut, lvl in BANDS if fraction >= cut)


total = sum(sum(v) for v in scores.values())
pct = {k: sum(v) / (2 * len(v)) for k, v in scores.items()}
weakest = min(pct, key=pct.get)
gated = 0
for lvl in (1, 2, 3, 4):
    if all(scores[dim][i - 1] == 2 for dim, i in GATES[lvl]):
        gated = lvl
    else:
        break
effective = min(level(total / 56), level(pct[weakest]), gated)
print(f"total={total}/56 -> L{level(total / 56)}", {k: f"{p:.0%}" for k, p in pct.items()})
print(f"weakest={weakest} -> L{level(pct[weakest])}; gating practices -> L{gated}; effective=L{effective}")

For the sample scores above, this prints a total of 39/56 (Level 3 band), a weakest dimension of online at 40% (Level 2), gating practices at Level 2 (N2 and N4 are below 2), and an effective level of 2.

Use these bands for orientation, not certification. The band cut-offs and gating statements are starting rules of thumb. If your team scores 38 overall but 20% on guardrails, treat your maturity as Level 1 and fix guardrails first.

5. How to level up¶

5.1 Level 0 → 1 (days)¶

• Enable tracing for inputs, outputs, retrieved chunks, and tool calls.
• Write 20–50 realistic queries with expected answer notes.
• Read the outputs, record one note per failure, and group them into categories.
• Move "never do X" rules out of the prompt and into deterministic code for auth and tool permissions.
• Add deterministic output format checks.

Biggest trap: Buying an evaluation platform before you read 50 traces.

5.2 Level 1 → 2 (weeks)¶

• Version your golden set, grow it to 100+ stratified examples, and collect SME references for top intents.
• Add component evaluations for your dominant failure modes (recall@k, execution accuracy, tool-call correctness).
• Replace Likert or generic judges with binary, criterion-specific judges. Validate each gating judge on ≥ 100 human labels with 30–50 FAILs (TPR/TNR ≥ 0.85, κ ≥ 0.60; see judge validation).
• Configure CI with a Stage 1 smoke suite on pull requests and a Stage 2 full nightly suite (CI stages). Size hard safety gates with the rule of three and gate regressions with paired tests (statistics for gates).
• Add approval or confirmation for write and external tools by tier, and step, tool-call, and cost budgets per run.
• Enforce authorization with end-user identity across retrieval, databases, and tools.
• Complete a threat model and your first red-team pass.

Biggest trap: Adding 20 metrics at once. Keep 3–7 metrics that map directly to your observed failure modes.

5.3 Level 2 → 3 (1–3 months)¶

• Run deterministic checks on 100% of online traffic, run validated LLM judges on 1–5% of traffic plus all flagged traces, capture user feedback, and build dashboards.
• Configure alerts with named owners and runbooks, then test them with a synthetic regression.
• Run a weekly trace review meeting.
• Implement layered guardrails for your risk tier and remove lethal-trifecta legs by design. Where relevant, add taint tracking, MCP tool pinning, and memory-write validation (agentic threats).
• Measure recall and false-positive rate for each guardrail, configure explicit fail modes (fail closed on tool-enabled and Tier 3 turns), and promote each guardrail through shadow -> flag -> block.
• Run red-team tests on every release, including indirect injection and (where relevant) memory and tool poisoning, and turn findings into regression tests.
• Roll out changes through canaries with pre-defined rollback triggers (rollout criteria), and monitor judge drift.
• Add production failures to your golden set on a regular schedule.

Biggest trap: Creating alerts with no owner and running online judges that were never validated on production data.

5.4 Level 3 → 4 (quarters)¶

• Automate sampling and triage of production traces into human labelling queues.
• Implement dataset lineage, PII-safe data pipelines, and retention policies.
• Standardize a shared evaluation and guardrail platform across teams.
• Manage guardrails as policy-as-code and run continuous automated red teaming.
• Link online metrics to business KPIs and automate canary analysis and A/B tests (with sequential statistics) for major changes.
• Maintain governance artifacts (risk register, system cards, audit trail) aligned with the NIST AI RMF.

Biggest trap: Building governance theater with static documents nobody reads. Automate evidence collection directly from your evaluation and tracing systems.

6. Worked example: the running example's assessment¶

Here is the scorecard for our internal enterprise assistant after a 6-week pilot. The HR/finance and tool-enabled paths are Tier 3 (tiering), so the launch target is Level 3 for evals and guardrails.

The gap to the Tier 3 launch target is two levels. Prioritize these next steps in order:

1. Complete the lethal-trifecta and Rule of Two review. Strip external images and links, require approval for send_email and fetch_url, add an inline preview with one-click confirmation for create_ticket (with on-call approval for high priority), add step and cost budgets, and enforce a SQL parser policy (policy table).
2. Build an RBAC leak test set per role (for example 60 queries × 7 roles = 420 pairs, enough for a "<1%" claim by the rule of three) and enforce it as a hard CI gate.
3. Create 60 gold SQL pairs and measure execution accuracy with a confidence interval.
4. Convert LLM judges to binary pass/fail criteria and validate them on ≥ 100 human labels with 30–50 FAILs (TPR/TNR ≥ 0.85, κ ≥ 0.60; ≥ 0.90 for HR/finance).
5. Enable Stage 1 smoke gates on pull requests and a Stage 2 full nightly evaluation suite with paired regression tests.

Key takeaways¶

• Measure maturity by observable practices (versioned datasets, validated judges, measured guardrails, CI gates, weekly reviews), not by purchased tools.
• Remember that your weakest dimension caps your effective maturity level.
• Match your target maturity level to your risk tier, and require Level 3 evals and guardrails before launching Tier 3 agents.
• Count a level only when its gating practices are in place (§4.6). A high total score does not compensate for missing CI gates or unvalidated judges.
• Advance one level at a time, and avoid buying platforms before error analysis or creating unowned alerts.
• Re-score your system every quarter and pick your next two or three improvements from the leveling checklists.

Further reading¶

• In this repo:
  • Implementation playbook
  • Eval selection
  • Guardrail selection
  • Production data flywheel
  • Monitoring & drift
• Hamel Husain, Your AI Product Needs Evals, A Field Guide to Rapidly Improving AI Products
• Anthropic, Demystifying evals for AI agents
• NIST, AI Risk Management Framework, Generative AI Profile (NIST AI 600-1)
• OWASP, Top 10 for LLM Applications 2026
• OWASP, Top 10 for Agentic Applications 2026
• Google, Secure AI Framework (SAIF)