Implementation Playbook: From Zero to Production¶

TL;DR

• Execute in seven sequential phases. Progress from defining success and instrumenting traces to CI gates, online monitoring, and a data flywheel.
• Start with error analysis on real traces. Read 20–50 traces first (then grow toward 100) to identify your top failure modes before you build metric dashboards.
• Gate every release in CI. Enforce zero-failure hard safety gates sized with the rule of three, and paired-test regression gates, across code, prompt, model, and index changes.
• Roll out with pre-defined rollback triggers. Promote through a canary against a concurrent control and roll back automatically on any RBAC leak or significant quality, safety, or cost regression.
• Turn production failures into tests. Sample live traffic, review traces weekly, and feed failure cases back into your golden dataset.

• You will learn:
  • A seven-phase implementation plan with checklists
  • Team roles and RACI matrix
  • Pre-launch readiness criteria and rollout/rollback criteria
  • Common evaluation anti-patterns to avoid
• Prerequisites: Why evals and guardrails, Eval selection, Guardrail selection
• Related: Maturity model, Metric cheatsheet, Running offline experiments, Production data flywheel, Agent archetypes (if your agent is not a RAG assistant)

0. The playbook at a glance¶

flowchart LR
    P0["Phase 0: Define success and risks"] --> P1["Phase 1: Tracing, first golden set, error analysis"]
    P1 --> P2["Phase 2: Component and E2E offline evals, align judges"]
    P2 --> P3["Phase 3: Guardrails and red teaming"]
    P3 --> P4["Phase 4: CI gates and launch readiness"]
    P4 --> P5["Phase 5: Production: online evals, monitoring, feedback"]
    P5 --> P6["Phase 6: Continuous improvement flywheel"]
    P6 -->|"new failure modes, new features"| P1

Treat these effort ranges as heuristics for a 2–5 person team building one agentic RAG application. Regulated domains and multi-agent systems require more time. SME availability is the single biggest predictor of how fast you finish Phases 1 and 2.

1. Roles and RACI¶

Follow the benevolent-dictator rule: Assign one named person (usually the SME or PO) as the final decision-maker for each quality criterion. Design-by-committee produces inconsistent ground-truth labels.

In the table below, R = Responsible, A = Accountable, C = Consulted, and I = Informed. Each row has exactly one A. S marks a required sign-off: the launch cannot proceed without it, but the A still owns the decision.

Phase 0: Define success and risks¶

Goal: Agree on what "good" and "unacceptable" mean before you build any metrics.

0.1 Activities¶

1. Write user stories and top intents. For our running example, include stories for regional revenue SQL queries, parental-leave policy questions, IT onboarding ticket creation, and emailing a summary to a colleague.
2. Draft a failure taxonomy v0 listing hypotheses about what can fail per component. Replace this draft with an empirical taxonomy in Phase 1 (start from agentic RAG failure modes).
3. Classify system risk using the risk questionnaire (the canonical tier definitions) and run the lethal-trifecta and Rule of Two check. Score each path separately. The running example is Tier 3 for HR/finance and tool-enabled turns, and Tier 2 for read-only Q&A.
4. Set binary acceptance criteria per intent and risk slice. You can mark quality targets as "TBD after baseline," but set hard safety gates to zero failures immediately. A zero-failure gate proves only as much as its test set: 0 failures in n independent cases bounds the true rate at about 3/n (the rule of three), so size each hard-gate set for the bound you need (dataset strategy §5.2).
5. Select your evaluation archetypes using eval selection §2.
6. Schedule SME review time on the calendar now so domain grading does not block Phase 1.

0.2 Acceptance criteria template¶

Per-turn approval and confirmation rules come from the role → tool policy table. Do not restate them in other documents; link to the table.

0.3 Checklist¶

• List the top 5–15 user intents with realistic example queries.
• Document all user roles and their permitted data and tool access.
• Draft your initial failure taxonomy v0.
• Assign a risk tier per path and complete the lethal-trifecta and Rule of Two check.
• Draft the acceptance criteria table and mark all hard safety gates.
• Book SME calendar time and name one benevolent dictator per criterion.
• Define data-handling and PII retention rules for production traces.
• If a regulation can apply (for example HR decisions under the EU AI Act), ask legal to confirm which obligations apply and when. The dates change; see EU AI Act status. This is not legal advice.

Phase 1: Instrument tracing, build first golden set, do error analysis¶

Goal: Observe real system failures, group them into categories, and build your first golden dataset.

1.1 Tracing first¶

You cannot evaluate an agent that you cannot inspect. Capture these fields on every run (tracing and OpenTelemetry):

• Record the input query, pseudonymized user ID, user role, and session ID.
• Log every LLM call with prompt version, model name, parameters, token counts, and latency.
• Log retrieval queries, chunk IDs, similarity scores, and applied ACL filters.
• Log tool calls with tool name, arguments, output or error status, and latency.
• Log generated SQL queries, execution status, and returned row counts.
• Record the final response, citations, and all guardrail decisions.
• Attach version metadata for the application, search index, and database snapshot.

Use OpenTelemetry-compatible instrumentation so you can switch observability backends without rewriting code (eval & observability tools).

1.2 First golden set (20–50 examples, growing toward 100)¶

Prioritize data sources in this order: real queries from pilot users or search logs, SME-written queries, builder-written queries, and synthetic queries for coverage gaps. See dataset strategy and SME golden datasets.

Stratify your dataset across these five dimensions:

• Cover every top user intent.
• Include every user role, especially for RBAC-restricted data.
• Balance easy, medium, and hard queries (multi-hop, ambiguous, date math).
• Include 10–20% unanswerable or out-of-scope queries.
• Add known edge cases such as acronyms, typos, and multiple languages. Tag the language so you can report per-language slices: PII and injection classifiers often lose recall outside English.
• Include freshness cases: questions whose answer changed in a recent document version. Faithfulness cannot catch an answer that is grounded in an outdated document.

Grow this set to 100–300 expert-verified examples before you use it as a release gate (dataset strategy §5.1). Include these minimum fields on every example: id, input, user_role, intent, expected, tags, source, created_at, and notes. Store the reference answer, chunk IDs, gold SQL, or expected tool call inside expected.

1.3 Error analysis¶

Error analysis is the highest-leverage activity in this playbook (popularized by Hamel Husain and Shreya Shankar):

1. Run your system across the golden dataset and any early pilot traffic.
2. Perform open coding: Read each trace and write a brief note describing the first failure that occurred.
3. Perform axial coding: Group your notes into failure categories (such as retrieval: outdated doc version or sql: wrong fiscal calendar).
4. Count the occurrences of each category and rank them by frequency × severity.
5. Repeat this process on fresh traces until no new failure categories appear.

from collections import Counter

notes = [
    {"id": "q07", "category": "retrieval: outdated doc version"},
    {"id": "q12", "category": "sql: wrong fiscal calendar"},
    {"id": "q19", "category": "retrieval: outdated doc version"},
    {"id": "q23", "category": "generation: ignored retrieved table"},
]
for cat, n in Counter(n["category"] for n in notes).most_common():
    print(f"{n:>3}  {cat}")

1.4 Checklist / exit criteria¶

• Verify that tracing is live in dev and staging with all required span fields.
• Commit golden set v1 (20–50 stratified examples, growing toward 100) to git or your dataset store.
• Read at least 50 end-to-end traces with your AI engineer and domain SME.
• Finalize failure taxonomy v1 with category counts and identify the top 3 failure modes.
• Classify each top failure mode as either a specification bug (unclear prompt) or a generalization gap (needs an evaluation metric).
• Fix obvious prompt specification bugs immediately before building complex evaluations for them.

Phase 2: Component and E2E offline evals; align LLM judges¶

Goal: Build a repeatable offline evaluation suite that measures your observed failure modes with validated judges.

2.1 Build the suite¶

Select your metrics using the eval selection decision tree and look up formulas in the metric cheatsheet. For tool-using flows, run the agent in a sandboxed environment with state-based graders (agent eval environments). Here is how Phase 1 failures map to metrics for our running example:

2.2 Build and align LLM judges¶

Follow the workflow in LLM-as-a-judge:

• Build one binary pass/fail judge per failure mode with written criteria and few-shot examples.
• Require the judge to output a brief critique before its pass/fail verdict.
• Label at least 100 examples manually with your SME, including 30–50 FAILs. Oversample likely failures; do not force a 50/50 balance.
• Split human labels into a development set (for prompt tuning) and a held-out test set.
• On the held-out test set, require TPR ≥ 0.85, TNR ≥ 0.85, and Cohen's κ ≥ 0.60 before the judge gates releases. Require ≥ 0.90 on high-risk slices (such as HR/finance and SQL equivalence). See the canonical bar.
• Apply the Rogan–Gladen correction (or prediction-powered inference) if you report absolute pass rates from an imperfect judge (LLM-as-a-judge §8).
• Pin the judge model snapshot and prompt version and log them with every evaluation run. Use the lowest temperature that the model supports; for reasoning models, fix the reasoning effort instead.
• Re-validate your judge whenever you change the judge model, judge prompt, or core product behavior (judge lifecycle).

2.3 Experiment hygiene¶

Follow the practices in running offline experiments:

• Log dataset, application, prompt, model, index, and judge versions on every evaluation run.
• Compare candidate variants with paired tests on the exact same dataset examples: McNemar for pass/fail, paired bootstrap or permutation for continuous scores (running experiments §5).
• Report every pass rate with a Wilson 95% interval, not a bare percentage.
• Run the unchanged system 3–5 times (A/A runs) to measure run-to-run noise before you set any regression threshold.
• Run non-deterministic agent scenarios 3–5 times and report pass^k (reliability) next to pass@k (capability) (pass@k vs pass^k).
• When you test many slices, control false discoveries with Benjamini–Hochberg (dataset strategy §5.3).
• Inspect evaluation scores per slice (intent, role, difficulty) rather than relying on overall averages.
• Read the trace diffs for examples that flipped from pass to fail between runs.

2.4 Exit criteria¶

• Record baseline scores with confidence intervals for all must-have evaluations in your archetype stack.
• Verify that every gating LLM judge meets the validation bar (TPR/TNR ≥ 0.85, κ ≥ 0.60; ≥ 0.90 on high-risk slices).
• Confirm that at least one metric tracks each of your top 3 failure modes.
• Run the entire offline evaluation suite end to end with a single command.

Phase 3: Guardrails and red teaming¶

Goal: Block unacceptable behavior at runtime and verify your defenses with adversarial testing. You can run Phase 3 in parallel with Phase 2.

3.1 Implement guardrails by tier¶

Select your controls using the guardrail selection decision tree and implement them in this order:

1. Enforce identity and authorization: Propagate end-user identity and enforce ACL retrieval filters, database RLS, and per-role tool allowlists (RBAC).
2. Enforce tool controls: Validate schemas and enforce read-only SQL rules. Require human approval for T2 (external) and T3 (destructive) tools, and an inline preview with one-click confirmation for shared T1 writes such as create_ticket. Cap steps, tool calls, cost, and wall-clock time to stop loops and denial of wallet (Tool guardrails, policy table).
3. Block data exfiltration: Sanitize outbound links and images and restrict network egress to an allowlist (Output guardrails).
4. Scan outputs: Check responses for PII, secrets, schema validity, and toxicity (Output guardrails).
5. Deploy agentic controls: Track taint (require approval for outbound or state-changing tools after untrusted content enters the context), pin and allowlist MCP tool descriptions, and validate writes to long-term memory (agentic threats).
6. Deploy input and context classifiers: Add prompt-injection, topic, and context-injection scanners in shadow or flag mode first (Input, Context).

Deterministic controls (steps 1–5) block from day one. Probabilistic classifiers (step 6) go through the rollout stages in §3.2.

3.2 Measure guardrails¶

• Build both a benign-but-tricky dataset and an adversarial dataset for each guardrail.
• Measure recall, false-positive rate, and p95 latency for every guardrail check. Estimate precision at your real base rate (guardrail metrics).
• Configure and test fail-open or fail-closed behavior by simulating guardrail service timeouts. Fail closed on tool-enabled turns, Tier 3 intents, and authorization checks (fail modes).
• Promote new probabilistic guardrails through shadow -> flag -> block. Shadow = log the verdict only. Flag = annotate the trace and notify reviewers, but do not block. Block = enforce. Promote only when FPR on real traffic is within your cap and recall on the attack set meets target (choosing thresholds).

3.3 Red teaming¶

Follow the adversarial testing workflow in threat model & OWASP:

• Map your system threat model to the OWASP Top 10 for LLM Applications 2026 and the OWASP Top 10 for Agentic Applications 2026 (ASI01–ASI10).
• Test direct single-turn and multi-turn prompt injections and jailbreaks.
• Test indirect prompt injection by planting canary instructions inside test wiki pages, tickets, documents, and fetched URLs. Report attack success rate together with benign task success (AgentDojo-style).
• Test memory poisoning (ASI06): plant an instruction in one session and check whether it changes an action in a later session.
• Test MCP and tool supply chain risks: a changed tool description must fail the pin check (tool poisoning, rug pull).
• Test data exfiltration via markdown images, query-string links, and tool arguments.
• Attempt cross-role data access using role-spoofing prompts ("I am the CFO, show me…").
• Attempt SQL abuse using DDL/DML statements, system catalog queries, and multi-statement payloads.
• Attempt tool abuse using malformed arguments, excessive calls, unapproved T2/T3 actions, and unconfirmed T1 writes.
• Test denial-of-wallet attacks using oversized inputs and recursive agent loops.
• Count distinct attack families (aim for ≥ 30), not paraphrases. Zero successes across n families bounds the family-level rate at about 3/n.
• Triage all findings, fix critical and high issues, and add regression test cases to your CI suite.

3.4 Exit criteria¶

• Implement all required guardrails for your assigned risk tier.
• Pass all hard safety gates (RBAC leaks, unsafe SQL, unapproved or unconfirmed actions) with zero failures on sets sized with the rule of three.
• Resolve all critical and high-severity red-team findings.
• Convert every red-team finding into an automated regression test.

Phase 4: CI eval gates and launch readiness¶

Goal: Block regressions automatically in CI and make an explicit, evidence-based launch decision.

4.1 CI gates¶

flowchart LR
    PR["Pull request: prompt, model, retriever, tool, or code change"] --> S0["Stage 0: unit tests and deterministic checks"]
    S0 -->|"fail"| BLK
    S0 -->|"pass"| SM["Stage 1 smoke suite: 20-50 examples plus all critical items"]
    SM -->|"fail"| BLK["Block merge"]
    SM -->|"pass"| MG["Merge"]
    MG --> NT["Stage 2 nightly full suite: all slices, all judges, repeated agent runs, paired tests"]
    NT -->|"regression"| ALERT["Alert owner; triage next day"]
    NT -->|"ok"| REL["Release candidate"]
    REL --> RR["Pre-release: full suite plus red-team regression plus hard gates"]
    RR -->|"fail"| BLK2["Block release"]
    RR -->|"pass"| S3["Stage 3: online canary or shadow with rollback triggers"]

Run model version upgrades, embedding updates, and search index rebuilds through the exact same CI gates as application code changes. The stage names match running offline experiments §10.1; gate design details are in §10.2. Binary verdicts per item aggregate to rates, so gates act on rates and their intervals.

4.2 Launch readiness checklist¶

Product & quality

• Verify that your golden set meets acceptance criteria across every intent, role, and difficulty slice.
• Confirm that the system refuses unanswerable queries cleanly with a low over-refusal rate.
• Obtain SME sign-off on a spot check of at least 50 recent outputs.
• Document known limitations and display clear user guidance in the UI.

Evals

• Freeze and record the golden dataset version used for launch sign-off.
• Record judge validation results (TPR, TNR, κ, label count, FAIL count) for all gating LLM judges.
• Enable CI smoke gates on pull requests and verify that the nightly suite is green.
• Measure repeated-run reliability (pass^k) for multi-step agent workflows, especially write-tool flows.

Guardrails & security

• Complete the security review of your risk tier and threat model.
• Verify that lethal-trifecta and Rule of Two risks are removed by design or gated by deterministic controls (approval, allowlists, taint tracking). Gating does not lower the tier.
• Propagate end-user identity and test ACL/RLS enforcement across all user roles.
• Enable tool allowlists, argument validation, SQL parser policies, and approval/confirmation flows per tool tier.
• Check approval prompts for fatigue (ASI09): show the exact action and arguments, keep prompts rare, and track approval rate and time-to-approve.
• Pin MCP tool descriptions and validate writes to long-term memory, if you use them.
• Enable outbound link and markdown image sanitization.
• Record recall, false-positive rate, and fail-open/closed behavior for every guardrail.
• Complete red-team testing with zero open critical or high findings.
• Test per-capability feature flags and emergency kill switches.

Operations

• Enable production tracing with PII scrubbing and retention policies configured.
• Run deterministic checks and cheap classifiers on 100% of live traffic, and LLM judges on 1–5% (or ~500 traces/day) plus 100% of flagged traces.
• Publish dashboards for quality, safety, guardrail triggers, latency, cost, and error rates.
• Configure alerts with named owners and runbooks, and train on-call engineers on the kill switch.
• Enable in-product user feedback capture (thumbs up/down plus optional free-text comment).
• Pin prompt, model, and index versions so you can roll back within minutes.
• Write down the rollout plan and rollback triggers (§4.3) before the launch review.
• Enforce per-run step, tool-call, cost, and wall-clock budgets, per-user rate limits, and provider spend alerts (denial of wallet, LLM06:2026).

Governance (Tier 3 / regulated)

• Obtain compliance and legal sign-off, including LLM provider data-processing agreements.
• Publish an internal model or system card documenting intended use and evaluation results.
• Verify that audit log retention meets regulatory requirements.
• Define the human oversight workflow for high-stakes decisions, including reviewer SLAs and the fail-closed path when approvers are unavailable.

4.3 Rollout and rollback criteria¶

Decide these numbers before the rollout starts. See canary releases and A/B tests for the method.

4.4 Exit criteria¶

• Hold the launch readiness review and record the launch decision with the product owner's sign-off (and the Risk/Compliance sign-off for Tier 3).
• Execute the staged rollout in §4.3 and verify the metrics at each stage before you promote.

Phase 5: Production: online evals, monitoring, feedback¶

Goal: Detect quality or safety regressions within hours rather than weeks. See online eval pipelines, monitoring & drift, and user feedback.

5.1 Online evaluation setup¶

5.2 Monitoring & alerting¶

• Establish baseline ranges during the first 1–2 weeks of production traffic and set alert thresholds.
• Route every alert to a named owner with a diagnostic runbook (check index freshness, recent deploys, and model versions).
• Monitor input topic drift and retrieval similarity score distributions.
• Monitor index freshness (age of retrieved chunks vs their source documents) (corpus drift).
• Alert on cost per run, budget breach rate, and tool-call spikes (denial of wallet, runaway loops; see runaway ticket creation).
• Track unsafe action attempt rate, approval rate, and time-to-approve for every gated tool.
• Track guardrail trigger rates per language and per role so you catch attack spikes, broken guardrails that drop to zero, and slices where a classifier misses.
• Log provider model version updates and correlate them with online metric shifts.

5.3 Weekly quality review (30–60 minutes)¶

Include the product owner, AI engineer, and domain SME weekly (plus Security monthly):

1. Review weekly dashboard trends and triggered alerts.
2. Read 20–50 sampled traces together across low judge scores, negative feedback, and random traffic.
3. Update failure taxonomy counts and log any new failure categories.
4. Decide whether to fix each issue immediately, add examples to the golden set, or add a new evaluation.
5. Assign a single owner to every action item.

5.4 Exit criteria (for "steady state")¶

• Verify alert routing by injecting a synthetic regression in staging or production.
• Complete at least three consecutive weekly quality reviews with tracked action items.
• Promote your first batch of real production failures into the offline golden set.

Phase 6: Continuous improvement (the flywheel)¶

Goal: Use production traces to continuously improve your datasets, evaluations, guardrails, and product. See production data flywheel.

flowchart LR
    A["Production traces and feedback"] --> B["Sample and triage: low scores, negative feedback, guardrail hits, random"]
    B --> C["Label and categorize: SME plus error analysis"]
    C --> D["Add to golden or regression set; update judges"]
    D --> E["Fix: prompt, retrieval, tools, guardrails, data"]
    E --> F["Offline eval and CI gates"]
    F --> G["Deploy with canary and online evals"]
    G --> A

6.1 Cadences (heuristics)¶

6.2 Checklist¶

• Tag golden set examples with provenance metadata and add new production failures regularly.
• Retire or update stale reference answers whenever underlying source documents change.
• Version datasets and judges so metric trends remain comparable over time.
• Add a permanent regression test case for every production incident.
• Prune unused metrics quarterly by removing or demoting any metric nobody acted on.

7. Anti-patterns¶

8. A 30/60/90-day plan (example)¶

Use this 90-day roadmap when starting from a working prototype of our enterprise assistant:

Key takeaways¶

• Define success criteria, failure modes, and risk tiers before you build metrics, and enforce hard safety gates from day one.
• Instrument tracing and run error analysis on 20–50 real traces (then grow toward 100) before building any evaluation dashboard.
• Build evaluations that target your observed failure modes, and validate every LLM judge against human labels (≥ 100 labels, TPR/TNR ≥ 0.85, κ ≥ 0.60).
• Implement deterministic guardrails (identity, tools, SQL, network egress) first, and measure false-positive rates on all classifier guardrails.
• Gate every code, prompt, model, and index change in CI using hard safety gates sized with the rule of three and paired-test regression gates.
• Roll out through canaries against a concurrent control, with hard rollback on any RBAC leak and statistical rollback on quality, safety, or cost regressions.
• Run cheap deterministic checks on 100% of production traffic, sample LLM judges, review traces weekly, and turn failures into regression tests.

Further reading¶

• Hamel Husain, Your AI Product Needs Evals, Using LLM-as-a-Judge For Evaluation, A Field Guide to Rapidly Improving AI Products, AI Evals FAQ
• Shankar et al., Who Validates the Validators? (arXiv:2404.12272)
• Anthropic, Demystifying evals for AI agents, Building effective agents
• Martin Fowler / Thoughtworks, Engineering Practices for LLM Application Development
• Microsoft, Lessons From Red Teaming 100 Generative AI Products (arXiv:2501.07238)
• NIST, AI Risk Management Framework
• OWASP, Top 10 for LLM Applications 2026 and Top 10 for Agentic Applications 2026
• In this repo:
  • Datasets
  • Offline evals
  • Online evals
  • Guardrails