Golden Dataset for LLM Evals: Rubrics, Labels and Cohen's Kappa

A golden dataset is only as good as its labels, and most teams never check theirs. If two careful people disagree on one case in three, no eval score built on those labels can tell you much. The fix is cheap: write the rubric down, have two people label a sample, and compute Cohen's kappa. This post does that with code you can run, then covers the dev/test split, versioning, judging the judge and drift.

I took a different angle from the existing guide to golden test sets, which covers sourcing, annotation tooling and edge-case coverage. Read that for how to gather examples. This one starts after you have examples and asks whether you can trust the labels. For where a golden set sits in a testing strategy, see the testing pyramid.

What goes into a rubric before anyone labels?

A rubric is a short written definition that two strangers could apply the same way. The failure to avoid is a rubric like "is the answer good?". Use binary criteria where you can. Practitioners such as Hamel Husain, as summarised by Lenny's Newsletter in a page I read through search on 2026-10-23 (the full originals are paywalled), argue that pass/fail judgments are more useful than arbitrary 1 to 5 scores, and that you should look at real data first. There is also a practical reason: fewer label options leave fewer ways for two raters to differ.

A template you can copy:

Task: customer-support answer about refunds.
Label PASS only if ALL are true:
1. States the refund window exactly as in the policy text provided.
2. Does not promise anything the policy does not say.
3. Tells the customer the next step (form, email, or contact).
Label FAIL if ANY is false. If the question is out of scope and the answer
declines politely, label PASS. If you cannot decide in 30 seconds, label
UNSURE and write one line why.
Examples: [3 PASS, 3 FAIL, 2 borderline with the reason for each]

Three details do most of the work: "ALL true" removes vague overall impressions, the UNSURE option stops raters guessing, and borderline examples with reasons settle most arguments in advance. Keep UNSURE cases out of the kappa calculation and discuss them as a group instead.

How do I label a sample and measure agreement?

1. Pull 40 to 60 real cases (a rule of thumb, large enough that one disagreement does not swing the result).
2. Two people label independently, without talking and without seeing each other's labels.
3. Compute raw agreement and kappa.
4. Discuss every disagreement, update the rubric, and relabel a fresh sample if kappa was low.

Cohen's kappa compares observed agreement with the agreement you would expect by chance, given how often each rater says PASS. The formula is (observed minus chance) divided by (1 minus chance). Here it is, with made-up labels for two raters on 20 answers, checked against scikit-learn:

from collections import Counter
from sklearn.metrics import cohen_kappa_score

# Made-up pass/fail labels (1 = acceptable). Not from a real dataset.
a = [1,1,1,0,1,1,0,1,1,1,0,1,1,0,1,1,1,0,1,1]
b = [1,1,0,0,1,1,0,1,1,1,1,1,1,0,1,0,1,0,1,1]

def kappa(x, y):
    n = len(x)
    po = sum(i == j for i, j in zip(x, y)) / n
    cx, cy = Counter(x), Counter(y)
    pe = sum((cx[k] / n) * (cy[k] / n) for k in set(x) | set(y))
    return po, pe, (po - pe) / (1 - pe)

po, pe, k = kappa(a, b)
print(f"raw agreement={po:.2f} chance agreement={pe:.2f} kappa={k:.3f}")
print(f"sklearn kappa={cohen_kappa_score(a, b):.3f}")

I ran this on 2026-10-23 (Python 3.13, scikit-learn 1.9.1). Output:

raw agreement=0.85 chance agreement=0.60 kappa=0.625
sklearn kappa=0.625

The hand-written function matches scikit-learn. Raw agreement of 0.85 sounds great; kappa of 0.625 is more modest, because with 70 percent PASS labels on one side, a lot of agreement happens by luck. Many teams treat about 0.6 as a floor and want 0.8 for labels that gate releases, but those are conventions. Pick a bar before you look at the number.

Why is raw agreement misleading on skewed data?

Because most eval sets are mostly PASS. Same script, a second set where both raters label nearly everything PASS:

x = [1]*18 + [0, 0]
y = [1]*17 + [0, 1, 0]
po2, pe2, k2 = kappa(x, y)
print(f"skewed set: raw agreement={po2:.2f} chance={pe2:.2f} kappa={k2:.3f}")

skewed set: raw agreement=0.90 chance=0.82 kappa=0.444

Ninety percent agreement, kappa 0.444. The raters agree on the easy PASSes and split on the rare FAILs, which are exactly the cases your eval exists to find. If your kappa is low only on the FAIL class, the rubric's failure criteria need sharper examples. If you have more than two raters or more than two labels, look at other agreement statistics such as Krippendorff's alpha rather than forcing Cohen's kappa onto it.

How do I split a golden dataset into dev and test?

You tune prompts against the dev split and report on the test split, and you touch the test split rarely. Otherwise you overfit your prompt to the very cases you use to judge it. The practical problem is stability: if you reshuffle with a random seed every time you add rows, old test cases leak into dev. Hash the case ID instead.

import hashlib
from collections import Counter

def split(case_id, test_pct=30):
    h = int(hashlib.sha256(case_id.encode()).hexdigest(), 16) % 100
    return "test" if h < test_pct else "dev"

ids = [f"case-{i:03d}" for i in range(20)]
print(Counter(split(i) for i in ids))
print([split(i) for i in ids[:5]], "again:", [split(i) for i in ids[:5]])

Real output:

Counter({'dev': 13, 'test': 7})
['dev', 'test', 'test', 'test', 'dev'] again: ['dev', 'test', 'test', 'test', 'dev']

The split is stable across runs and across new rows: a case keeps its side forever. With 20 cases the 30 percent target came out as 7 of 20; hashing gives approximate proportions, which smooths out on larger sets. Keep near-duplicate questions on the same side by hashing a group ID (for example the source document or customer) rather than the row ID, or paraphrases of one test case will leak into dev.

How do I version a golden dataset?

Treat it like code, in the repo, as JSONL with one case per line:

{"id": "case-014", "input": "Can I return a sale item?", "reference": "policy:returns#sale", "label": "PASS_CRITERIA", "labelled_by": ["asha", "ravi"], "rubric_version": "v3", "added": "2026-10-23", "source": "prod-log"}

• Version the rubric and the data separately. A rubric change invalidates old labels; record rubric_version per row and relabel when it changes.
• Never edit a label silently. Change it in a commit with a reason, so a score jump can be traced to a label change and not a model change.
• Tag every eval run with the dataset commit, prompt version and model ID. Without that, a trend line is unreadable.
• Keep provenance. A source field (prod log, hand-written, synthetic) lets you check whether synthetic cases score differently from real ones.

How do I check a judge against my golden labels?

If an LLM judge will grade at scale, treat it as a third rater and compute kappa against your human labels on the dev split. Same function, different input. Made-up judge labels against rater A:

human = a
judge = [1,1,1,0,1,1,1,1,1,1,0,1,1,1,1,1,1,0,1,1]
print(f"judge vs human kappa={cohen_kappa_score(human, judge):.3f}")
tp = sum(h == 0 and j == 0 for h, j in zip(human, judge))
print(f"human-fail cases the judge also failed: {tp} of {human.count(0)}")

judge vs human kappa=0.692
human-fail cases the judge also failed: 3 of 5

Kappa of 0.692 looks acceptable, yet the judge caught only 3 of the 5 cases the human failed. For regression testing, the recall on failures is the number you care about, so report both. This judge is invented to show the calculation; I did not run a real model. For position, verbosity and self-preference checks, use the LLM-as-a-judge pitfalls guide. When you wire a judge into pytest, the DeepEval tutorial shows a custom judge model.

Two cautions. Do not tune the judge prompt on the test split, or its agreement number is inflated. And if you let an LLM draft labels to save time, keep humans on every case that gates a release, since a judge that labels its own training data will agree with itself.

If the judge needs a paid API, a capped key limits the cost of repeated calibration runs. AICredits is one OpenAI-compatible option (INR via UPI from Rs 50, https://api.aicredits.in/v1), and provider spend limits do the same job.

Disclosure: the AICredits link is an affiliate link.

How do I catch dataset drift?

A golden set ages as your users, documents and models change. Signs and responses:

A routine that costs an hour a month: sample 20 recent production cases, label them with two raters, compute kappa and the share your golden set would have covered, and add the failures. That keeps the set tied to reality, which is the whole point.

Checklist

• Written binary rubric with borderline examples and an UNSURE option
• Two independent raters on at least a sample, kappa computed, bar chosen in advance
• Disagreements discussed, rubric updated, and the sample relabelled if kappa was low
• Stable hashed dev/test split, with related cases grouped
• Dataset and rubric versioned, with every label change explained
• Judge kappa and failure recall reported together, on dev only
• Monthly refresh from production

What I did not verify

The labels in every example are made up to demonstrate the arithmetic, so none of the kappa values describe a real dataset. The kappa thresholds in this post are common conventions, not results. The size guidance is a rule of thumb. The Husain summary comes from search results on 2026-10-23, not from reading the originals.