SME Golden Datasets¶

TL;DR

• A golden dataset is a small (50–300 row) expert-verified dataset that defines what "correct" means for your domain. Use 100–300 rows for release gates. LLM-judge validation needs ≥100 human-labelled outputs, including 30–50 FAILs.
• Respect expert time: pre-fill draft answers and show source chunks inline so subject-matter experts (SMEs) only have to verify and edit.
• Use binary Pass/Fail rubrics and atomic key-facts lists instead of vague 1–5 Likert scales.
• Measure reviewer agreement with Cohen's $\kappa$, Fleiss' $\kappa$, or Krippendorff's $\alpha$, with a confidence interval. If two human experts cannot agree on a label, the target is ill-defined, and judge-agreement numbers against those labels mean little.
• Watch the prevalence paradox. When almost all answers pass, $\kappa$ can be low even at 90%+ raw agreement. Report raw agreement, Gwet's AC1, and agreement on FAILs next to $\kappa$.

• What you will learn:
  • How to work with subject-matter experts (SMEs) without wasting their time
  • How to write clear annotation guidelines and binary Pass/Fail rubrics
  • Why storing both a reference answer and a key_facts list gives the most reliable grading
  • How to handle unanswerable, partial, and false-premise questions
  • How to calculate and interpret Cohen's $\kappa$, Fleiss' $\kappa$, and Krippendorff's $\alpha$ (including ordinal scales and confidence intervals)
  • How skewed pass rates distort $\kappa$, and what to report instead
  • How to find label errors in a gold set
• Prerequisites: Dataset Strategy
• Related guides:
  • LLM-as-a-Judge
  • End-to-End RAG Evals
  • Production Data Flywheel

1. What a golden dataset is (and isn't)¶

A golden dataset is a set of eval examples whose expected outputs were written or verified by domain experts (such as HR policy owners, finance analysts, support leads, or data engineers).

Golden data gives you the highest return on investment in five places:

1. High-stakes RAG answers (HR policy, finance rules, legal compliance).
2. Gold SQL queries and expected results (only data owners know that "active customer" means status = 'A' AND churn_date IS NULL).
3. Graded retrieval relevance labels (relevant_docs graded 0–3).
4. Policy and RBAC boundaries (which requests must be refused or escalated).
5. LLM-judge calibration sets (human Pass/Fail grades on actual system outputs, used to test your LLM judge).

Important distinction: You need two kinds of human labels:

1. Golden references: The expected answer, SQL query, or relevant docs for a question.
2. Golden judgments: A human's Pass/Fail grade on a specific system output (used to calibrate your LLM-as-a-Judge).

2. Working with subject-matter experts¶

Expert time is your scarcest resource. Design your labeling workflow so SMEs never waste a minute.

2.1 Roles¶

Assigning a single "benevolent dictator" (one accountable domain expert) prevents endless committee debates over what counts as a good answer.

2.2 Make SME sessions efficient¶

1. Pre-fill draft answers. Never give an SME a blank text box. Show the question, the retrieved chunks, and a draft answer from your system. Editing a draft takes 2 minutes; writing from scratch takes 15 minutes.
2. Show source documents side by side. Display the exact policy excerpt and version date right next to the answer so SMEs do not have to search for files.
3. Batch items by domain. Send all HR questions to the HR expert and all SQL questions to the analytics engineer.
4. Time-box labeling sessions to 45–60 minutes. Start each session by grading 3–5 calibration examples together as a group.
5. Require a short written critique for every failure. A 1-sentence note explaining why an answer failed becomes your next prompt fix or LLM-judge few-shot example.
6. Give SMEs quick issue flags. Let reviewers click ambiguous_question, docs_outdated, or not_my_domain so they never get stuck on a broken item.

2.3 Sourcing the inputs for SMEs to label¶

3. Annotation guidelines¶

Without written guidelines, every reviewer invents their own rules. Write a 1-page guideline document before labeling begins.

3.1 What to include¶

# Annotation Guidelines: Acme Assistant, HR Policy Answers (v1.3)

## Your task
For each question, review the draft answer and the source documents. Provide:
1. A corrected reference answer (2–4 sentences).
2. A list of atomic key facts the answer MUST contain.
3. The doc_ids/chunk_ids that support each fact.
4. Answerability: answerable | partially_answerable | unanswerable.

## Definitions
- "Correct" = matches the CURRENT policy version shown. Do not use unwritten tribal knowledge.
  If the policy document itself is outdated, flag `docs_outdated` and grade against the text shown.
- "Key fact" = a single checkable statement whose absence would make the answer wrong or misleading.
  Do not include greetings or generic disclaimers.

## Decision rules
- If a policy varies by country or role and the user did not specify theirs → either ask a clarifying
  question OR state the general rule and list the variations (if 3 or fewer).
- Numbers: always include units and scope ("16 weeks, full-time employees, Germany").
- Partially answerable questions: mark `partially_answerable`, list key facts for the covered part,
  and require the assistant to state what is missing (`must_acknowledge_gap: true`).

## Worked examples
(Include 3–5 complete examples: 1 clear pass, 1 clear fail, 1 borderline partial, 1 unanswerable)

3.2 Guideline principles¶

• Replace vague adjectives with testable rules. Do not write "the answer should be helpful." Write "the answer states the numeric limit in the first sentence and cites the policy document."
• Grade separate dimensions separately. Do not ask for one overall "quality" score. Ask separate Pass/Fail questions for correctness, completeness, faithfulness, and policy compliance.
• State what reviewers should ignore. For example: "Ignore bullet vs. paragraph formatting unless it changes the meaning."
• Update the guidelines after every adjudication round. Treat guidelines as versioned code.

4. Rubric design¶

4.1 Prefer binary (pass/fail) per criterion¶

Avoid 1–5 Likert scales for release gates. Reviewers rarely agree on the difference between a 3 and a 4, and LLM judges cluster around 3 or 4. Instead, use binary Pass/Fail checks per criterion paired with a 1-sentence critique.

4.2 Example checklist rubric (RAG answer)¶

Define Overall Pass as passing all critical criteria (Correct, Complete, Faithful, and Policy & RBAC).

4.3 Rubrics for other eval types¶

5. Reference answers vs key facts lists¶

You can represent the expected answer as a free-form paragraph (reference_answer) or as a list of atomic claims (key_facts):

Best practice: store both. Write a short 2–3 sentence reference_answer for human readers, and break it into 1–5 atomic key_facts marked critical: true | false:

key_fact_recall      = (critical facts present in output) / (total critical facts)
contradiction_free   = output does not contradict any key fact or source chunk
answer_correct (bin) = (key_fact_recall == 1.0) AND contradiction_free

{
  "id": "rag-fin-0019",
  "question": "What's the hotel nightly cap for business travel in London?",
  "reference_answer": "The nightly hotel cap in London is GBP 250 including taxes. Exceptions require director approval in advance.",
  "key_facts": [
    {"fact": "Nightly cap in London is GBP 250", "critical": true},
    {"fact": "Cap includes taxes", "critical": false},
    {"fact": "Exceptions require advance director approval", "critical": true}
  ],
  "supporting_chunks": {"0": ["fin-travel-emea#c12"], "1": ["fin-travel-emea#c12"], "2": ["fin-travel-emea#c14"]}
}

How to write good key facts:

• Make each fact atomic: Put "GBP 250" and "includes taxes" in separate facts.
• Include scope qualifiers: Keep the country, user role, or effective date attached to the number.
• Link each fact to its chunk ID: This gives you reference_contexts for Context Recall at the same time.

6. Handling "no answer" and unanswerable questions¶

If your dataset only contains answerable questions, you will never know if your assistant hallucinates when documents are missing. Include at least 10–15% unanswerable or partial questions.

6.1 Taxonomy¶

6.2 Labeling rules¶

1. Set "answerable": false (or "partial") and set an explicit "expected_behavior" enum ("abstain", "clarify", "correct_premise", "refuse").
2. Write "acceptable_response_notes" stating what the assistant must not do (for example, "Must NOT guess from the full-time employee policy").
3. Verify against the actual corpus. Search the corpus to confirm no document answers the question, and re-verify whenever new documents are indexed.

7. Inter-annotator agreement (IAA)¶

If two human experts cannot agree on whether an answer passes or fails, your rubric is unclear. The target is then ill-defined: an LLM judge may match one expert, but its agreement score against the "gold" labels means little.

How to measure agreement:

1. Debug the rubric: Have at least 2 reviewers independently grade an overlap batch of 30–50 items. Read every disagreement and fix the guidelines before you label the rest of the dataset.
2. Certify the labels: Before you use the labels for release gates, collect at least 100 overlap items. You can accumulate them across batches. Report the agreement with a 95% confidence interval (§7.6).

Why so many items? Agreement statistics are noisy on small samples. In the §7.2 example ($\kappa = 0.57$ on 100 items), the 95% bootstrap interval is about 0.40 to 0.73. On a 34-item subset, it widens to about 0.33 to 0.87. That range covers "fair" to "almost perfect".

7.1 Percent agreement (and why it's not enough)¶

Raw percent agreement is simply:

$$p_o = \frac{\text{items where reviewers agree}}{\text{total items}}$$

Why is raw agreement misleading? If 90% of system answers pass, two reviewers who blindly click "Pass" 90% of the time will agree 82% of the time by pure chance ($0.90 \times 0.90 + 0.10 \times 0.10 = 0.82$). Always report a chance-corrected metric like Cohen's $\kappa$ or Krippendorff's $\alpha$. But read §7.5 first: when one label dominates, chance correction has its own problem.

7.2 Cohen's kappa (two annotators, nominal labels)¶

Use Cohen's $\kappa$ when two reviewers grade the same items:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Where $p_o$ is observed agreement and $p_e$ is the agreement expected by chance.

Worked example: Two SMEs (A and B) independently grade 100 RAG answers as Pass or Fail:

1. Observed agreement ($p_o$): $(55 + 25) / 100 = 0.80$ (80%).
2. Chance agreement ($p_e$): $(0.70 \times 0.60) + (0.30 \times 0.40) = 0.42 + 0.12 = 0.54$.
3. Cohen's $\kappa$: $(0.80 - 0.54) / (1 - 0.54) = 0.26 / 0.46 \approx \mathbf{0.57}$.

Even though raw agreement looks high (80%), $\kappa = 0.57$ is only moderate. Notice that Reviewer A passed 15 items that Reviewer B failed. Reviewing those 15 disagreements together will immediately reveal which rule in your guideline needs tightening.

def cohens_kappa(a: list[str], b: list[str]) -> float:
    assert len(a) == len(b) and a
    n = len(a)
    labels = set(a) | set(b)
    p_o = sum(x == y for x, y in zip(a, b)) / n
    p_e = sum((a.count(k) / n) * (b.count(k) / n) for k in labels)
    return 1.0 if p_e == 1 else (p_o - p_e) / (1 - p_e)

For graded scales (such as retrieval relevance 0–3), use weighted kappa (sklearn.metrics.cohen_kappa_score(..., weights="quadratic")), which penalizes a 0 vs. 3 disagreement much more heavily than 2 vs. 3.

7.3 Krippendorff's alpha (any number of annotators, missing data, any scale)¶

Use Krippendorff's $\alpha$ when you have 3 or more reviewers or when not every reviewer grades every item:

$$\alpha = 1 - \frac{D_o}{D_e}$$

Where $D_o$ is observed disagreement and $D_e$ is disagreement expected by chance.

from collections import Counter
from itertools import permutations

def krippendorff_alpha_nominal(items: list[list[str | None]]) -> float:
    o = Counter()
    for row in items:
        vals = [v for v in row if v is not None]
        m = len(vals)
        if m < 2:
            continue
        for i, j in permutations(range(m), 2):
            o[(vals[i], vals[j])] += 1 / (m - 1)
    cats = {c for pair in o for c in pair}
    n_c = {c: sum(v for (x, _), v in o.items() if x == c) for c in cats}
    n = sum(n_c.values())
    d_o = sum(v for (x, y), v in o.items() if x != y) / n
    d_e = sum(n_c[x] * n_c[y] for x in cats for y in cats if x != y) / (n * (n - 1))
    return 1.0 if d_e == 0 else 1 - d_o / d_e

This hand-rolled version handles nominal labels only. For other scales, use a library (below).

Ordinal and interval labels¶

For graded labels (such as relevance 0–3), a 0 vs. 3 disagreement is worse than 2 vs. 3. Use the ordinal or interval form of $\alpha$. The krippendorff package takes a raters × items array, with np.nan for items a rater skipped:

import numpy as np
import krippendorff

# rows = raters, columns = items; relevance grades 0-3; np.nan = rater skipped the item
grades = np.array([
    [3, 2, 0, 1, 3, 2, np.nan, 0],
    [3, 3, 0, 1, 2, 2, 1,      0],
    [2, 3, 1, np.nan, 3, 2, 1, 0],
])
print(round(krippendorff.alpha(reliability_data=grades, level_of_measurement="ordinal"), 3))
print(round(krippendorff.alpha(reliability_data=grades, level_of_measurement="nominal"), 3))

Here the ordinal $\alpha$ (0.85) is much higher than the nominal $\alpha$ (0.54), because most disagreements are one grade apart. Nominal $\alpha$ counts those as full disagreements.

Fleiss' kappa (three or more raters, complete data)¶

If every item gets the same number of raters, you can also use Fleiss' $\kappa$ (Fleiss, 1971). It extends chance-corrected agreement to more than two raters. Use $\alpha$ instead when some raters skip items.

import numpy as np
from statsmodels.stats.inter_rater import aggregate_raters, fleiss_kappa

# rows = items, columns = 3 raters; every rater labels every item
labels = np.array([
    ["Pass", "Pass", "Pass"], ["Pass", "Fail", "Pass"], ["Fail", "Fail", "Fail"],
    ["Pass", "Pass", "Fail"], ["Fail", "Fail", "Pass"], ["Pass", "Pass", "Pass"],
    ["Fail", "Fail", "Fail"], ["Pass", "Pass", "Pass"],
])
counts, categories = aggregate_raters(labels)   # items x categories count table
print(categories, round(fleiss_kappa(counts, method="fleiss"), 3))

7.4 Interpreting agreement¶

Many teams describe $\kappa$ with the bands from Landis & Koch (1977). These labels are a convention, not a statistical test. Landis and Koch called them arbitrary.

For $\alpha$, Krippendorff recommends stricter cut-offs: rely on data with $\alpha \ge 0.80$, use data with $0.667 \le \alpha < 0.80$ only for tentative conclusions, and discard data below 0.667 (Krippendorff, Content Analysis, 2004; see also Hayes & Krippendorff, 2007).

Use these starting rules of thumb for $\kappa$ or $\alpha$, and calibrate them for your domain:

Also check the per-class numbers in §7.5. A high $\kappa$ can hide poor agreement on the rare class.

Note: This bar is for human-vs-human agreement on gold labels. The bar for an LLM judge is different: you compare the judge with adjudicated gold labels and require TPR and TNR ≥ 0.85 and $\kappa \ge 0.60$ on ≥100 items (see §7.7).

7.5 The prevalence paradox: when κ is low but agreement is high¶

When one label dominates, $\kappa$ can be low even when raw agreement is very high. This is the prevalence paradox (Feinstein & Cicchetti, 1990). RAG pass rates are often 85–95%, so this happens often.

Report these statistics next to $\kappa$:

• Raw agreement $p_o$.
• PABAK (prevalence-adjusted bias-adjusted kappa) $= 2p_o - 1$ for two categories (Byrt, Bishop & Carlin, 1993).
• Gwet's AC1, which uses a chance term that stays small when one label dominates. For two categories, $p_e = 2\pi(1 - \pi)$, where $\pi$ is the mean share of Pass labels (Gwet, 2008).
• Positive and negative specific agreement: how often the raters agree on Pass and on Fail, separately.

def binary_agreement(a: list[str], b: list[str], pos: str = "Pass") -> dict:
    """Agreement statistics for two raters and two labels (pos / not pos)."""
    n = len(a)
    both_pos = sum(x == pos and y == pos for x, y in zip(a, b))
    both_neg = sum(x != pos and y != pos for x, y in zip(a, b))
    disagree = n - both_pos - both_neg
    p_o = (both_pos + both_neg) / n
    pa, pb = a.count(pos) / n, b.count(pos) / n
    p_e = pa * pb + (1 - pa) * (1 - pb)
    pi = (pa + pb) / 2                        # mean share of `pos` labels
    p_e_ac1 = 2 * pi * (1 - pi)               # Gwet's chance agreement, two categories
    return {
        "raw_agreement": p_o,
        "cohens_kappa": (p_o - p_e) / (1 - p_e) if p_e < 1 else 1.0,
        "pabak": 2 * p_o - 1,
        "gwet_ac1": (p_o - p_e_ac1) / (1 - p_e_ac1),
        "positive_agreement": 2 * both_pos / (2 * both_pos + disagree),
        "negative_agreement": 2 * both_neg / (2 * both_neg + disagree) if both_neg + disagree else float("nan"),
    }

# 100 answers: 88 both Pass, 4 + 4 disagreements, 4 both Fail
a = ["Pass"] * 88 + ["Pass"] * 4 + ["Fail"] * 4 + ["Fail"] * 4
b = ["Pass"] * 88 + ["Fail"] * 4 + ["Pass"] * 4 + ["Fail"] * 4
for k, v in binary_agreement(a, b).items():
    print(f"{k:20s} {v:.2f}")

raw_agreement        0.92
cohens_kappa         0.46
pabak                0.84
gwet_ac1             0.91
positive_agreement   0.96
negative_agreement   0.50

Raw agreement (92%) and AC1 (0.91) look excellent, and $\kappa$ (0.46) looks poor. The specific agreement explains the gap. The reviewers agree on Pass 96% of the time but on Fail only 50% of the time. For an eval, the FAILs are what matter, so here $\kappa$ points to a real problem. Do not just switch to AC1 to get a better number.

Practical fix: Enrich the overlap batch with likely failures (low judge scores, negative feedback) so that both labels have enough examples. Report that you enriched it.

7.6 Confidence intervals for agreement¶

Report every agreement statistic with a 95% confidence interval. A percentile bootstrap over items works for any statistic in this section:

import random

def bootstrap_ci(a: list, b: list, stat, iters: int = 2000, seed: int = 0) -> tuple[float, float]:
    """95% percentile bootstrap CI for an agreement statistic; resample items, not labels."""
    rng = random.Random(seed)
    n, vals = len(a), []
    for _ in range(iters):
        idx = [rng.randrange(n) for _ in range(n)]
        try:
            vals.append(stat([a[i] for i in idx], [b[i] for i in idx]))
        except ZeroDivisionError:   # a resample where both raters used one label only
            continue
    vals.sort()
    return vals[int(0.025 * len(vals))], vals[int(0.975 * len(vals)) - 1]

Call it as bootstrap_ci(a, b, cohens_kappa). For $\alpha$, the Hayes & Krippendorff (2007) macro gives bootstrap intervals. You can also resample items (columns of the raters × items array) with the same loop.

7.7 Human-vs-judge agreement¶

When you validate an LLM judge, compare it with adjudicated gold labels, not with a single reviewer. Report the judge's TPR (share of human PASSes that the judge passes) and TNR (share of human FAILs that the judge fails) separately, plus $\kappa$. A single agreement number hides whether the judge misses failures or raises false alarms. The canonical bar (starting rule of thumb) is ≥100 labelled items with 30–50 FAILs, TPR and TNR ≥ 0.85, and $\kappa \ge 0.60$ on a held-out split (≥ 0.90 for high-risk slices: Tier 3 actions, text-to-SQL correctness, and safety). Re-validate after any change to the judge prompt or model. See LLM-as-a-Judge for the full method, including how to correct pass rates for judge error.

8. Adjudication¶

Adjudication is the step where your principal domain expert resolves reviewer disagreements and updates the annotation guidelines:

flowchart TD
    A["Batch of items"] --> B["Two annotators label independently"]
    B --> C{"Agree?"}
    C -->|yes| D["Accept label"]
    C -->|no| E["Adjudicator reviews both labels and rationales"]
    E --> F{"Root cause"}
    F -->|"annotator error"| G["Correct label"]
    F -->|"guideline gap"| H["Update guideline + changelog"]
    F -->|"item is ambiguous"| I["Rewrite item or mark ambiguous and exclude from gates"]
    F -->|"docs are wrong or stale"| J["Flag to doc owner; label against current docs"]
    G --> D
    H --> D
    I --> D
    J --> D
    D --> K["Recompute agreement on next overlap batch"]

Three rules for adjudication:

1. Keep first-pass labeling blind. Reviewers must not see each other's grades before submitting.
2. Never majority-vote away expert knowledge. If two junior reviewers outvote your senior HR policy owner on an HR rule, trust the policy owner and update the guideline.
3. Save adjudicated items as LLM-judge few-shot examples. Hard boundary cases that required human adjudication make the best teaching examples for your automated judge. Keep them out of the judge's test split.

8.1 Finding label errors in the gold set¶

Gold labels also contain errors. Northcutt et al. (2021) estimated that at least 3.3% of test-set labels, on average, are wrong across 10 widely used ML benchmarks. A wrong gold label penalizes a correct system on every run.

Audit the gold set on a schedule:

• Review rows where the system and a strong judge both disagree with the gold label. If both say "Pass" and the label says "Fail", the label is often wrong. Send these rows back to adjudication. Do not flip them automatically.
• Review rows that fail across many system versions. A row that every prompt and model fails can be a hard case, or a wrong reference.
• Use confident learning for large classified sets. For guardrail or routing labels with a trained classifier, confident learning (implemented in cleanlab) ranks the examples most likely to be mislabeled.
• Record each correction in the dataset changelog with the reason, and bump the dataset version.

9. Labeling tools¶

Pick the simplest tool that displays the question, retrieved chunks, tool calls, and final answer on a single screen:

10. Cost and time planning¶

Use these starting estimates to plan SME sessions (measure your own speed in session 1):

Sample budget for a 200-row Golden RAG dataset:

• 200 pre-filled rows × 3 minutes = 10 SME hours
• 50% double-annotated overlap (100 rows, enough to certify agreement, §7) = +5 SME hours
• Adjudication + guideline calibration = +5 SME hours
• Total: ~20 SME hours spread across a few 1-hour sessions.

11. Pitfalls¶

12. Golden dataset checklist¶

• Row schema matches your target eval types (Dataset Strategy §3).
• Slice coverage plan is agreed upon with your principal domain expert.
• Written annotation guidelines include operational definitions and boundary examples.
• Rubric uses binary Pass/Fail checks per criterion.
• RAG rows store both a reference_answer and source-linked key_facts.
• Unanswerable, partial, and ambiguous rows specify expected_behavior.
• Rubric debugged on a 30–50 item overlap batch; then ≥100 overlap items (accumulated across batches) reached $\kappa$ or $\alpha \ge 0.80$ with a lower 95% bound $\ge 0.667$.
• With skewed labels, raw agreement, Gwet's AC1, and agreement on FAILs are reported next to $\kappa$.
• Gold labels are audited for errors on a schedule (§8.1).
• Every row records doc_versions, created_by, and label_tier: "gold".
• Held-out test rows are blocked from being used in prompts or fine-tuning.

Key takeaways¶

• Small, high-quality golden datasets anchor your entire eval system. 100–300 expert-verified rows are enough to gate releases. Validating an LLM judge needs ≥100 human-labelled outputs, including 30–50 FAILs.
• Pre-fill drafts and show sources inline to respect SME time.
• Use binary Pass/Fail criteria and atomic key_facts lists instead of 1–5 scales.
• Measure reviewer agreement ($\kappa$ or $\alpha$) early, with a confidence interval. Low agreement means your rubric is ambiguous, not that your reviewers are careless. Certify gate labels on ≥100 overlap items.
• Treat Landis & Koch bands as a convention. When most answers pass, also report raw agreement, Gwet's AC1, and agreement on FAILs.
• Audit gold labels for errors. Rows where the system and a strong judge both disagree with the label are the first place to look.

Further reading¶

• Hamel Husain, Creating a LLM-as-a-Judge That Drives Business Results
• Shankar et al., Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• Krippendorff's alpha guide: k-alpha.org
• Landis & Koch, The Measurement of Observer Agreement for Categorical Data (1977)
• Hayes & Krippendorff, Answering the Call for a Standard Reliability Measure for Coding Data (2007)
• Feinstein & Cicchetti, High Agreement but Low Kappa: I. The Problems of Two Paradoxes (1990)
• Gwet, Computing Inter-Rater Reliability and Its Variance in the Presence of High Agreement (2008)
• Northcutt et al., Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (2021)
• Next in this section:
  • Synthetic Datasets
  • Production Data Flywheel