Skip to content

05 · Decision Framework

TL;DR

  • Turn theory into action. Use these decision trees, checklists, and scorecards to select your evaluations and guardrails.
  • Match controls to your architecture and risk. Pick the right metrics and runtime guardrails for your specific system archetype.
  • Follow a phased rollout. Build from initial trace analysis to automated CI gates and production monitoring.
  • Score your maturity. Evaluate your team across five dimensions and prioritize your next two improvements.

This section uses one running example across all guides: an internal enterprise assistant. The assistant searches company documents, runs text-to-SQL queries, calls tools (tickets, email, web fetch), and enforces role-based access control (RBAC). HR/finance intents and tool-enabled turns are Tier 3; read-only policy Q&A is Tier 2 (tiering). You can apply these frameworks to simple RAG chatbots, agentic RAG, text-to-SQL agents, multi-agent workflows, and multi-turn assistants.

Start here if you are…

You are… Go to Then
Building your first RAG prototype Playbook: Phases 0–1 (define success, tracing, 20–50 examples growing to 100, error analysis) Eval selection §8.0 universal core
Choosing metrics for an agent / text-to-SQL / multi-agent system Eval selection decision tree (find your archetype in §8) Metric cheatsheet quick picks
Asking "which metric tells me X?" Metric cheatsheet Follow the table link to the detailed metric card
About to launch Launch readiness checklist and rollout and rollback criteria Guardrail selection to confirm your tier's required controls
Facing a security / risk review Guardrail selection: risk questionnaire + lethal trifecta Threat model & OWASP, Access control & RBAC
Adding a write/action tool to an existing assistant Lethal-trifecta check and A2 guardrail stack A2 eval stack
Already in production and quality dropped Playbook: Phase 5 (online evals, weekly review) Monitoring & drift, Production quick picks
Upgrading the model or embedding provider CI gates (treat it as a code change) Running offline experiments
Unsure whether your LLM judge can be trusted Deterministic vs judge vs human LLM-as-a-judge: how many labels
Unsure whether a score change is real Statistics for release gates Running offline experiments
Short on budget or SME time Budget section and which dataset at which stage Synthetic datasets
A lead planning a roadmap / reporting status Maturity model (run the scorecard) Playbook RACI & 30/60/90 plan

Files in this section

File What it covers Read when…
01-eval-selection-decision-tree.md Five input questions, master decision tree, method choices, dataset mapping, and minimum eval stacks for five archetypes You need to decide which evals to build
02-guardrail-selection-decision-tree.md Canonical risk tiers, lethal-trifecta and Rule of Two check, agentic threats (OWASP 2026), required guardrails by tier, runtime modes, fail modes, latency budgets, and archetype stacks You need to decide which guardrails to build and configure
03-metric-cheatsheet.md One reference table of every metric, eval, and guardrail check with formulas, ground-truth needs, methods, and costs, plus guardrail measurement, statistics for release gates, and quick picks You need to look up a metric fast
04-implementation-playbook.md Phases 0–6 from defining success to the production flywheel, with checklists, RACI, CI gates, rollout and rollback criteria, and anti-patterns You are planning or executing the rollout
05-maturity-model.md Levels 0–4, minimum level by risk tier, capability matrix, self-assessment scorecard with gating practices, and step-by-step leveling checklists You want to score your current state and plan next steps

Suggested reading order

  1. Skim the phase table and anti-patterns in the Implementation playbook to learn the overall workflow.
  2. Select your archetype and minimum viable eval suite in the Eval selection decision tree.
  3. Run the risk questionnaire and lethal-trifecta check in the Guardrail selection decision tree.
  4. Keep the Metric cheatsheet open as a quick reference while you build.
  5. Score your system with the Maturity model now and repeat the assessment every quarter.
flowchart LR
    A["Playbook: the phases"] --> B["Eval selection: what to measure"]
    A --> C["Guardrail selection: what to enforce"]
    B --> D["Metric cheatsheet: lookup"]
    C --> D
    D --> E["Maturity model: where you stand, what next"]

How this section relates to the rest of the repo

Need Section
Concepts and vocabulary 00-foundations
Building the datasets your evals run on 01-datasets
Detailed metric cards (formulas, examples, pitfalls) 02-offline-evals
Production evals, tracing, feedback, monitoring 03-online-evals-and-observability
How each guardrail works 04-guardrails
Which tools implement what 06-tooling-landscape
Papers, posts, standards 07-references