"Show me your golden set." No question organises a conversation about AI in production faster, because the answers always fall into the same three. First: we don't have one. Second, the most common: yes, there's a sheet with examples — and what they show you are the examples living inside the prompt. Third, the rare one: yes, 180 cases, versioned in the repository, with an owner and a last-reviewed date.
The first three weeks of work with a team look very different depending on which of the three it is. And it is not a question of technical maturity: we have seen teams with impeccable distributed tracing, automated canaries and an expensive observability backend who, when you ask which cases they use to decide whether tomorrow's release is better than today's, do not have the file.
In the article on LLMOps we describe the full lifecycle of an LLM system in production, and in the one on the rubric and the baseline we take apart what "works better" actually means. Both assume the same object: a set of labelled cases from which everything else follows. This article is about that object — where the cases come from, how many you need, who labels them, how it is protected from contamination, and why, if you look at it too much, it stops being useful.
What a golden set is, and the four things confused with it
A golden set is a versioned collection of cases with acceptance conditions written before looking at the system's output. Each case carries its input, the initial state it needs, what must hold for it to count as solved, which capability it exercises, where it came from and who answers for it. It lives in the repository, it is reviewed in pull requests, and it changes with a version number. It is, literally, the operational definition of "works" for your domain.
Almost every evaluation problem we have seen starts by confusing it with something that looks like it.
| Not… | Why it gets confused | What breaks if you use it that way |
|---|---|---|
| A public benchmark | It brings ready-made cases and a figure comparable with everyone else's | It measures a task that is not yours, and it is probably already in the model's training data |
| The prompt's examples | They are real cases, they are written down, and someone already reviewed them | The system has seen them: you are evaluating memory, not capability. This is the most frequent mistake |
| A folder of logs | There is volume, it is real, and collecting it costs nothing | Without an acceptance condition there is no evaluation: there is only stored traffic |
| A fine-tuning dataset | Same format, same labelling effort, same people | If the same set trains and evaluates, the number that comes out means nothing |
The four confusions that produce reassuring, false evaluations
Why it is the longest-lived asset in the whole system
One question puts the golden set in its architectural place: if you switch model provider tomorrow, what do you take with you?
Prompts get rewritten, because each model family responds differently to the same instructions. The retrieval strategy gets tuned. Thresholds get recalibrated. The orchestration framework can be replaced wholesale in a quarter. Even the rubric gets adjusted, because new failure modes appear. The only thing that crosses over intact is the set of cases: that is knowledge about your business, not about the provider.
This has a practical consequence for budgeting. The golden set is usually treated as preparation work — something done quickly so measurement can start — when in fact it is the investment with the best amortisation in the project. Over the three or four years an AI system lives in production, the case set will have been used to choose a model, to validate every release, to justify cost per useful task to finance and to answer an audit. Everything else will have been rewritten twice.
And there is a second, less obvious consequence: it is also the asset that cannot be generated externally. You can contract the architecture, you can contract the evaluation, you can contract the deployment. The cases that define what a correct answer is in your domain live in the head of someone in your organisation, and getting them out is work that does not get subcontracted. When a programme requires us to build it, our role is the method and the pace; the criteria come from whoever knows the domain.
Where the cases come from
Three origins, and each case should carry a label saying which, because they are not worth the same.
Real traffic. It is the base and should be the majority. Sampled from production, not from anyone's imagination. If the system is not in production yet, sample from the human process doing that task today: the tickets support already resolved, the invoices finance already processed, the queries that already arrived in the inbox.
Incidents. These are the most valuable cases you will have, because they already proved they hurt. Every failure that reached a user, every answer someone had to fix by hand, every escalation. An incident that does not end up as a case in the golden set is an incident that can happen again without anyone noticing — and that is exactly the loop almost everyone leaves open.
Risk hypotheses. What has not happened yet but would be costly: the customer asking for something the system must not grant, the personal data that cannot appear in the answer, the hostile instruction hidden in one of the documents the system retrieves. Here you do write cases that never occurred, and it pays to mark them for what they are — hypotheses, not evidence. That last type of case also has its own discipline, because what decides the damage is not the text but the permission the tool had open.
Stratify by capability, not by volume
The natural instinct is to build the set proportional to traffic: if 70% of queries are about order status, make 70% of the cases about order status. It is a mistake, and it is what makes a 300-case golden set carry less information than a well-distributed 80-case one.
A set proportional to traffic mostly measures what you already know works. What decides whether a system is deployable is not its performance on the majority case, which also tends to be the easiest: it is its behaviour on the infrequent capabilities that are expensive to fail. The right stratification is by capability — understanding intent, retrieving the right piece of data, respecting a business constraint, chaining two tools, recognising it cannot solve something — with a minimum number of cases in each, even if one of them represents 2% of traffic.
The methodological reference here predates LLMs and is still the best: CheckList, by Ribeiro, Wu, Guestrin and Singh (ACL 2020), which brings behavioural software testing to language models. Their proposal is a matrix of capabilities by test type instead of a single aggregate accuracy figure, and their results are the best argument for adopting it: a team responsible for a commercial sentiment analysis model — already extensively tested — found new, actionable bugs using it, and in a second study practitioners working with CheckList wrote twice as many tests and found almost three times as many bugs as those who did not. Same task, same people, a different way of organising the cases.
Synthetic cases: variation yes, foundation no
Asking a model for 200 evaluation cases is tempting and wrong in one specific sense: a model-generated set mostly measures agreement with the model that generated it. It inherits its blind spots — precisely the ones you need to detect — and tends to produce clean, well-written cases with the ambiguity removed, which is the opposite of what production looks like.
The rule we apply: generation allowed as a variation of a real case already anchored — rephrasing the same query six ways, changing language, adding typos, lengthening the context — and forbidden as a foundation. And each generated case inherits the acceptance conditions of the case it came from, reviewed by the domain person. If the variation changes the correct answer, it stops being a variation and becomes a new case.
How many cases: the question almost always asked wrong
"How many cases do we need?" has an answer that disappoints everyone: rather fewer in total than people fear, and rather more per capability than there usually are. A reasonable starting point is 30-50 cases per evaluable capability. A system with six well-defined capabilities starts saying useful things at 200-300 cases, not 5,000.
But the number alone is misleading, and the arithmetic is worth doing once. With 50 cases and an accuracy around 80%, the standard error of that proportion is about 5.7 points; the 95% confidence interval spans some eleven points either side. On that set, a three-point improvement between two versions is not an improvement: it is noise formatted as data. This is elementary arithmetic, not a study, and it is enough to discard half the comparisons that circulate in sprint reviews.
The interesting part is that the way out is not more cases. As Evan Miller argues in Adding Error Bars to Evals, an evaluation is an experiment and deserves the same treatment as any other: standard error, paired comparison, and power analysis before running. Comparing paired — both systems on exactly the same cases, measuring the difference case by case instead of subtracting two means — and repeating each case three or five times to average out non-determinism give far more precision per euro invested than doubling the size of the set. And the marginal cost of repeating is much lower than the cost of labelling.
It is worth reading alongside Bowman and Dahl's diagnosis in What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021): the fronts on which evaluation needs to progress are not only task design, but the reliability with which tasks are annotated and their size. Five years on, that is still exactly the homework list for any team building its golden set.
Labelling is the ceiling for everything else
Here is the point most often skipped, and the most expensive one: your labelling error rate is the upper limit of what you can measure. If 8% of your cases have the wrong reference answer, chasing a five-point improvement is chasing your own noise. And this is not a hypothetical risk.
In 2021 Northcutt, Athalye and Mueller reviewed the test sets of ten of the most used datasets in vision, language and audio, and found at least 3.3% label errors on average, with at least 6% in the ImageNet validation set. The percentage is not the relevant part: what happens when you correct it is. In their analyses the ranking between models changes — ResNet-18 beats ResNet-50 once the level of wrong labels rises by six points — which means a noisy set does not just measure worse: it can recommend the wrong system to you. These are datasets that had been the discipline's reference standard for a decade.
The most useful example for a product team is SWE-bench Verified. When it became clear that the original set contained under-specified problem statements and unit tests that marked valid solutions as incorrect, the answer was to rebuild it by hand: a subset of 500 instances filtered by 93 professional developers, with three annotators reviewing each sample. It is probably the best-funded cleanup effort in the industry. And even so, Epoch AI — which runs it independently — estimates that between 5% and 10% of cases still have problems. The conclusion is not discouraging, it is calibrating: if the best possible effort lands there, your 200-case set labelled over two afternoons is not in the top 1%.
Disagreement is information, not a failure of the people
When two domain people label the same case differently, the usual reaction is to look for who got it wrong. That is almost always the wrong question. Lora Aroyo and Chris Welty argued as much in Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation (AI Magazine, 2015): human annotation of semantic interpretation tasks leans on an outdated ideal — that of a single correct truth — when interpretation is inherently subjective, and measuring disagreement across several people represents the range of reasonable readings better than forcing a consensus.
Translated to a company golden set: disagreement between two annotators is a defect in the acceptance condition, not in the annotators. If two people from the claims team disagree on whether an answer was acceptable, what you have found is that the criterion was not written precisely enough — and that is exactly what needs to go to the rubric. It is the cheapest signal in the whole process: thirty cases, two people, half a morning.
The ritual that works is simple. Double labelling on a 10-20% sample of cases, a measure of agreement, and a short session where disagreements are resolved by changing the criterion, not by voting. Cases that remain ambiguous after that have their own place: they get marked as ambiguous and drop out of the headline metric. A case the business has no firm opinion on cannot block a deployment.
The three drawers and the looking budget
A golden set used in full for everything is exhausted within months without anyone noticing. The minimum separation is three drawers with different access rules.
- Development. Looked at daily, debugged against, iterated on. It gets contaminated by design and that is fine: its job is to let the team move fast. No number coming out of here is fit for a deployment decision.
- Held out. Touched on rare occasions, and every query is logged. This is the one that produces the numbers for quality gates.
- Sealed. Not opened until the release decision, or until a model change. It is the only measurement that has not influenced any prior design decision.
The reason the second drawer needs discipline is not organisational, it is statistical, and it has been formalised for a decade. Dwork, Feldman, Hardt, Pitassi, Reingold and Roth published it in Science in 2015: the validity guarantees of a held-out set assume the analysis procedure was fixed before looking at the data, whereas real analysis is inherently adaptive — each iteration is chosen based on what was seen in the previous one. Their proposal, the reusable holdout, is a technique for reusing it many times while preserving validity; that is, a method for spending it slowly, not for not spending it.
From which comes a rule we recommend writing into the header of the file itself: the held-out set is a consumable resource, with a looking budget. An explicit number of queries — twenty, thirty — and a rotation date. When it runs out, it is replenished with new production cases and the old one drops down to the development drawer. This is not academic purism: it is the difference between a number that blocks deployments and a number the team has already learned to beat without meaning to.
Contamination: the kind that comes in and the kind that goes out
There are two directions of contamination and usually only one is watched.
Inbound: cases the model has already seen. If you took cases from a public benchmark, assume they are in the training data. Sainz and colleagues framed this as a position in 2023 — classic evaluation on annotated benchmarks is in trouble, and contamination needs measuring benchmark by benchmark, not in general — and the cleanest demonstration came from the GSM1k work: 1,000 new problems built to replicate the style and difficulty of GSM8k, with accuracy drops of up to 8 points and a positive relationship (Spearman r² of 0.36) between the probability of a model generating examples from the original benchmark and its loss of performance on the new one. Several model families showed systematic overfitting; frontier ones, barely any.
Outbound: cases that leave your control. A golden set pasted into a public ticket, uploaded to a third-party tool, or included as an example in a prompt sent to a provider is no longer a clean set. The operational rule: held-out and sealed cases do not leave the repository, and those used as prompt examples are permanently marked as burned.
There is a third, subtler effect worth knowing before drawing epic conclusions from a difference between sets. In 2019 Recht, Roelofs, Schmidt and Shankar rebuilt the CIFAR-10 and ImageNet test sets following the original procedure, and the accuracy of every model dropped: between 3 and 15 points on CIFAR-10, between 11 and 14 on ImageNet. Their conclusion is the interesting part: the drops did not come from overfitting through reuse, but from the new set turning out to be somewhat harder. The lesson for a golden set is more uncomfortable than the overfitting one: two sets built with the same procedure are not interchangeable. Only measurements taken on the same set are comparable, and changing set forces you to re-measure the baseline.
Anatomy of a case
A case with an input and an expected answer is half a case. These are the fields that keep a set usable eighteen months later, when whoever wrote it has left the project.
| Field | Why it is there |
|---|---|
| Stable identifier | Lets you follow the same case across versions and compare paired |
| Input and initial state | An agent acts on a system: without a starting state the case is not reproducible |
| Acceptance conditions | What must hold, not a literal answer — linked to the rubric's criteria |
| Capability exercised | It is the dimension you stratify by and the one you read the result along |
| Origin and date | Traffic, incident or hypothesis: they do not carry the same weight nor age the same |
| Criticality | Separates the case that annoys from the case that must never fail (a blocking criterion) |
| Drawer: development · held out · sealed | Without this marker the set contaminates itself within weeks |
| Labeller and agreement | Who labelled it and whether it was double-reviewed: it is the data's traceability |
| Last verified | A two-year-old case may describe a business policy that no longer exists |
Nine fields. The last four are the ones almost nobody adds, and the ones that make a set age well
A golden set is a process, not a file
A set built in one sprint and then frozen degrades in three ways at once: the business changes its rules, the system learns to beat those particular cases, and the easy cases stop carrying information. Maintenance is not optional; it is half the work.
Discriminating power: the case that no longer measures anything
A case every version always gets right does not distinguish between them. It takes up execution budget and lifts the average, but it does not help decide. It is an old concept, borrowed from test psychometrics, where each item is characterised by its difficulty — what proportion gets it right — and its discrimination — how well it separates high performers from low ones. An item everyone gets right has zero discrimination, and in a well-built exam it is retired.
In a golden set that translates into a very cheap periodic review: for each case, look at how many of the recent versions got it right. Those that have been green for everyone for six months move to a smoke set — they keep running, because a regression there would be serious, but they leave the headline metric. And the freed budget goes where there is signal: the cases where versions disagree, which are the ones actually deciding.
Replenishment: the loop that closes the cycle
The rule that improves a set's quality fastest is also the most boring: every incident becomes a case within five working days, with its acceptance condition written by whoever suffered it. Not "when there is time", not "in the next quality sprint". It is the task that turns observability into learning, and without it the LLMOps cycle stays open: failures get detected, fixed by hand, and nothing stops them coming back.
Forty-five minutes a week, two people, three items on the agenda: what failed in production, which new case goes in, which case has expired. Over a quarter that is thirty to forty new cases, all anchored in real failures. No synthetic generation exercise produces material of that quality.
Validity: making sure the case measures what it claims to
One check remains that almost nobody does, and which the work on best practices for agentic benchmarks — Zhu and others, NeurIPS 2025 — systematises into two questions. Task validity: is the case solvable with the information and tools the system has? An impossible case does not measure capability, it measures frustration, and it counts as an eternal failure that pollutes the average. Outcome validity: does the check accept exactly the correct solutions, no more and no fewer? A verification that is too strict fails valid solutions; one that is too lax passes empty answers.
Both are checked with the same cheap gesture: solve ten randomly chosen cases by hand and run those human solutions through the automated verification. If any of them fail, the problem is in the case. It takes half an hour, and it is the half hour that stops you discovering six months later that 12% of your set was measuring something else.
| Cadence | What happens | Real cost |
|---|---|---|
| Weekly | The week's incidents go in as cases, with their acceptance conditions | 45 min · 2 people |
| Monthly | Double labelling of a sample, agreement measurement, disagreements to the rubric | Half a morning |
| Quarterly | Discriminating power: always-green cases to smoke, budget to the ones that disagree | 2 h · 1 person |
| Half-yearly | Rotation of the held-out set, review of cases expired by business policy changes | 1 day · 2 people |
Under two person-days a quarter. That is what it costs to keep evaluation meaning something
Five symptoms of a golden set that no longer works
None of them announces itself. All of them are recognisable from outside in a twenty-minute review.
The success rate has been above 95% for months. It is not that the system is excellent: it is that the hard cases never went in, or have already been debugged against one by one. A healthy set lives uncomfortably, between 70% and 90%.
Nobody can say where a particular case came from. Without an origin field, you cannot tell the case that cost you a customer from the hypothesis someone wrote one afternoon. And both weigh the same in the average.
The file has not changed in a quarter while the product shipped eight releases. The evaluation is measuring a system that no longer exists.
The cases were written by whoever builds the system, with no domain person in the room. With no bad faith at all, the criteria end up resembling what the solution already does well.
Nobody has ever looked at how wrong the labels are. It is the quietest of the five and the one that invalidates everything else: without measuring your own noise, any small improvement being celebrated may be an artefact of the labelling.
The first 40 cases
If you are starting from scratch, this is what produces the most signal per hour invested. You do not need a project: you need a week with the right people in the room.
1. Write the capabilities before the cases. Between five and eight, in business language: "understand what the customer is asking", "find the right clause in the contract", "promise nothing outside policy", "recognise when it has to escalate to a human". This list is what will organise the entire set, and it is a business conversation, not a technical one.
2. Pull five real cases per capability. From traffic or from the current human process. Real ones, with their mess: badly written queries, incomplete context, two questions in one. Forty cases.
3. Have the acceptance conditions written by whoever knows the domain, answering two questions per case: what must appear no matter what, and what must never appear. Those answers are also the raw material of the rubric.
4. Double-label ten cases with a second person and count the disagreements. That number is your ceiling, and it is better known on day one than in month six.
5. Split into drawers and give it an owner. Twenty-five to development, fifteen to held out. A name in the file header, a date, a version number. And the weekly forty-five-minute slot in the calendar.
At the end of that week you will not have a better system. You will have what is needed for the rubric and baseline from the previous piece to be built on something, and for the quality gate in the LLMOps cycle to have something to block with. Forty well-made cases already tell a good release from a bad one. Five thousand cases without acceptance conditions tell you nothing.
What you buy and what you build
There is an asymmetry worth keeping in mind when budgeting an AI project. Architecture can be bought. Deployment can be bought. Observability can be bought, and there is good product out there. The cases that define what "correct" means in your business are not for sale anywhere, and they are precisely the ones that decide whether all of the above is worth anything.
It is the same logic we apply to code with Spec-Driven Development: the criterion is written before the artefact, and the criterion is the asset. In onext AI-Accelerated Development programmes the golden set is not a final-phase deliverable, it is one of the first things we build with the team — because without it there is no way to prove the rest of the programme worked. The results we sign with clients in 2026 — ×7 delivery speed, 0 sprints lost, −50% time-to-production — are measured against cases the client wrote, not against a promise of ours.
If your team already has traces and monitoring but no case file, you have the thermometer without having decided what counts as a fever. And that is a one-week conversation, not a one-quarter one.
Frequently asked questions
What is a golden set in an AI project?
It is the versioned collection of cases your team uses to decide whether an AI system works: each case carries its input, the initial state it needs, its acceptance conditions, the capability it exercises, where it came from and who maintains it. It is not a public benchmark, nor a folder of logs, nor the examples living inside the prompt. It is the operational definition of "works" for your domain, written before looking at the system's output and maintained like code.
How many cases does a golden set need?
Fewer than people imagine in total, and more per capability than there usually are. A good starting point is 30-50 cases per evaluable capability, not several thousand undifferentiated ones. The reason is statistical: with 50 cases and an accuracy near 80%, the standard error is around 5-6 points, so a 3-point improvement is indistinguishable from noise. And the cheap way to gain precision is not multiplying cases, but comparing paired on the same cases and repeating each run several times.
Where should the cases in a golden set come from?
From three origins, and it pays to label which is which: real traffic (the base), production incidents (the most valuable cases, because they already proved they hurt) and risk hypotheses (what has not happened yet but would be expensive). Model-generated cases work as variations of a real case already anchored, never as a foundation: a synthetic set mostly measures agreement with the model that generated it.
Why can't the same held-out set be reused forever?
Because every decision you make by looking at it spends it. Dwork and colleagues' work in Science on adaptive data analysis formalises the problem: the statistical guarantees of a held-out set assume the analysis was fixed before looking at it, and in practice each iteration is chosen based on what was seen in the previous one. Their answer — the reusable holdout — is a technique for spending it slowly, not for not spending it. In practice: an explicit query budget and scheduled rotation.
How much labelling error is normal, and why does it matter?
More than assumed. Northcutt, Athalye and Mueller found at least 3.3% label errors on average in the test sets of ten benchmark datasets, and at least 6% in the ImageNet validation set; correcting them can even change the ranking between models. SWE-bench Verified was filtered by hand with 93 developers and three annotators per sample, and Epoch AI still estimates 5-10% of cases have problems. It matters because your labelling error rate is the ceiling: you cannot reliably detect improvements smaller than your own noise.
What do I do with cases where two people disagree?
Treat them as the cheapest signal you will get, not as a quality problem with the people. Aroyo and Welty argued this in 2015 with the idea of crowd truth: human interpretation is subjective and disagreement measures the real ambiguity of the task. If two domain people disagree on a case, it almost always means the acceptance condition is badly written, not that one of them is wrong. Disagreement goes to the rubric and is resolved there.
Sources cited
- Ribeiro, Wu, Guestrin, Singh — Beyond Accuracy: Behavioral Testing of NLP Models with CheckList (ACL 2020; capability matrix, twice the tests and almost three times the bugs)
- Northcutt, Athalye, Mueller — Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks (≥3.3% on average, ≥6% on ImageNet, ranking changes between models)
- Aroyo, L. and Welty, C. — Truth Is a Lie: Crowd Truth and the Seven Myths of Human Annotation (AI Magazine 36(1), 2015)
- Dwork, Feldman, Hardt, Pitassi, Reingold, Roth — The reusable holdout: Preserving validity in adaptive data analysis (Science 349, 2015; extended open-access version)
- Recht, Roelofs, Schmidt, Shankar — Do ImageNet Classifiers Generalize to ImageNet? (11-14 point drops when the test set is rebuilt)
- Zhang and others (Scale AI) — A Careful Examination of Large Language Model Performance on Grade School Arithmetic (GSM1k; drops of up to 8 points, Spearman r² 0.36)
- Sainz, Campos, García-Ferrero, Etxaniz, Lopez de Lacalle, Agirre — NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark (2023)
- Bowman, S. R. and Dahl, G. E. — What Will it Take to Fix Benchmarking in Natural Language Understanding? (2021; annotation size and reliability)
- Miller, E. — Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (paired comparison and power analysis)
- Zhu and others — Establishing Best Practices for Building Rigorous Agentic Benchmarks (task validity and outcome validity, NeurIPS 2025)
- OpenAI — Introducing SWE-bench Verified and the benchmark's page at Epoch AI (500 instances, 93 developers, 5-10% estimated residual error)
Jordi García is Tech Lead at onext. He works on bringing AI into governed production across development and product teams —with Spec-Driven Development, context engineering and human verification at every step— and authors onext's technical insights on the method, quality and cost of applied AI.
LinkedIn →