LLMOps: the real path to production for an LLM system, and how to keep it alive

The demo worked. The system did not. It is the pattern we run into most often when we join a team that already has something LLM-based in front of real users: the prototype convinced the committee in fifteen minutes and six months later nobody can say whether it answers better or worse than it did in March. There is no metric that says so. There is a feeling, and feelings do not hold up a system in production.

LLMOps is the discipline that closes that gap: the set of practices for building, deploying, measuring and maintaining systems whose behaviour is not written in your code. It is not a platform you buy or a role you hire. It is a cycle with stages, quality gates and owners — and most teams have it open somewhere without knowing it.

This article is the operational version of that cycle: what you actually version, what the flow looks like stage by stage, where the two gates go, which metrics decide, and which failures only show up once there is real traffic. If what you are after is the business conversation — what a Head of Product measures to decide whether an AI feature stays alive — that is in how to know if your product's AI works. This here is the engineering layer underneath.

LLMOps is not MLOps with a different model

The most expensive starting mistake is treating an LLM system as a classic machine learning project. They share the intent — operating software whose behaviour is statistical, not deterministic — but not the object being operated on. In MLOps the central artefact is the weights, the features and the training dataset; the model changes when you decide to retrain it. In LLMOps someone else trains the model, it changes when that someone else wants, and what you control is everything else.

That difference is not philosophical. It reorders the entire cycle:

The practical consequence is uncomfortable: you can have the system frozen, without a single commit in three weeks, and be serving worse answers than a month ago. In traditional software that does not happen. In LLMOps it is the base scenario you design against.

What you actually version: the context bundle

Before talking about flow, the unit of deployment has to be fixed. If you ask a team what they version in their LLM system, the answer is usually "the prompt" — and the prompt is between 10% and 20% of what determines behaviour. These are the eight elements that make up the context bundle, and all of them have to live in the repository:

• Prompt templates: system prompt, few-shot examples, task templates. With their own version and changelog, not edited live from a panel.
• Tool definitions: name, description and JSON Schema for each parameter. A tool's description is not documentation: it is prompt sent on every call, and it decides whether the model uses the tool well or ignores it.
• Retrieval configuration: embedding model, chunking and overlap strategy, top-k, reranker, metadata filters, similarity threshold.
• The index: version, ingestion date and corpus hash. A rebuilt index is a deployment, even if not a single line of code changed.
• Routing policy and model pin: which model serves which task, with an explicit version identifier. claude-sonnet-5 is a pin; "the latest Claude" is not.
• Sampling parameters: temperature, top_p, max_tokens, stop sequences.
• Guardrails: input and output filters, PII policy, refusal lists, step and per-task cost limits.
• The evaluation set and its thresholds: the cases, the rubrics, and the bar that must be cleared for promotion.

The operational rule is simple and fairly strict: a release is the complete tuple. If you raise top-k from 5 to 8, you are not tuning a parameter — you are deploying a new version of the system, and it deserves the same evaluation cycle as a prompt change. Teams that treat retrieval as configuration and the prompt as code end up with half of their behaviour outside version control.

The flow: nine stages, two gates, one loop that closes

This is the cycle we install when we join a team with LLM systems already in production. There is nothing exotic about it: most of the pieces exist in any serious engineering practice. What almost never exists is the order and, above all, the closing of the loop.

Stage 0 · The task contract comes first, not the prompt

Almost every LLM system we audit starts with the prompt. That is the wrong order. Before writing a line you have to fix in writing what counts as a correct answer, what format is mandatory, which sources are admissible and — this is what almost nobody writes down — what the system must do when it does not know. A system with no defined abstention policy always makes things up, because generating is exactly what it knows how to do.

The contract is signed by whoever knows the domain: the lawyer if it is procurement, the support manager if it is customer service, the analyst if it is reporting. Not the AI team. If the contract only exists in conversations, later evaluation will measure what the technical team thinks is correct, which rarely matches what the business needs.

Stages 1 and 2 · The golden set and the honest baseline

The golden set is the most underestimated asset in the whole cycle. Anyone can replicate a good prompt in an afternoon; a set of 200 cases labelled with domain judgement is months of accumulated work and cannot be copied. Start small — 30 to 50 cases per capability — stratify by difficulty and include three kinds of adversarial case: out-of-domain inputs, legitimately ambiguous questions where the correct answer is to acknowledge the ambiguity, and attempts to inject hidden instructions into the documents the system retrieves.

The honest baseline is the countermeasure against optimism. Measure the simplest system that solves the task first — often a single call with good context — and publish the number. Without that reference point, any later architecture looks like an improvement. With it, many teams discover their seven-step pipeline performs the same as the single call and costs five times more.

This is the stage where most teams get stuck, which is why it has an article of its own: how a golden set is built — where the cases come from, why you stratify by capability rather than volume, how much labelling error you are carrying without knowing, and why the held-out set is spent every time you look at it.

Stage 4 · Gate 1: offline evaluation decides, it does not opine

The first gate runs in CI, on every pull request touching any element of the bundle. And it evaluates four things at once, not one:

• Quality per capability, not aggregate. An average that rises while hiding an eight-point drop in the critical capability is worse than not measuring.
• Zero regression in the critical subset: the cases that must never fail — the ones that cost you an incident — are a separate block with an absolute threshold.
• Cost per useful task compared with the baseline. Cost is an acceptance criterion, not a consequence discovered on the invoice.
• p95 latency end-to-end, including tools and retrieval.

For this to be sustainable it helps to split the suite in two: a fast suite of a few minutes that runs on every PR, and a nightly full suite with the complete set. A single 400-case suite that takes forty minutes ends up disabled within three weeks, and a disabled gate is worse than no gate because it produces false confidence.

Stages 5 and 6 · Gate 2: real traffic is the only definitive judge

No evaluation set captures the real distribution of inputs. That is why promotion to production is progressive and not a switch: first shadow (the new system processes real traffic but its output is not served, only compared), then canary at 5%, 25% and 100%, with a minimum window at each step that depends on volume — not on impatience.

The non-negotiable condition is that the rollback criterion is written before traffic is opened: which metric, which threshold, how long, and who decides. A rollback debated in the heat of the moment at eleven at night is always delayed, because at that point everyone has a hypothesis for why the numbers are noise.

Stage 7 · Observability: the trace is the unit, not the log

A log of requests and responses is no use for debugging an agentic system. What you need is the complete trace of the task: every step as a nested span, with the rendered prompt, the retrieved documents and their scores, the tools invoked with their arguments, input and output tokens, latency and cost. OpenTelemetry's gen_ai.* semantic conventions cover most of those attributes and save you inventing a schema of your own that nobody else understands.

The quality criterion for instrumentation is concrete: faced with a user complaint, can you reconstruct exactly what context the model saw when it produced that answer? If the answer is no, observability is incomplete however pretty the token dashboards are. It is the same gap we describe in the quality gap almost nobody measures in agents in production: consumption gets monitored, the outcome does not.

Stage 8 · Failure mining is what turns the cycle into a loop

This is where most implementations we have seen break. Traces get instrumented, dashboards get built, thumbs-down get collected… and nobody has, in their week, the task of turning those failures into evaluation cases. The result is a system that accumulates telemetry and does not learn: the same errors reappear release after release because they never entered the suite that would have caught them.

The minimum ritual that works: a weekly 45-minute session where two people review a sample of failed traces, classify the root cause — insufficient context, poor retrieval, ambiguous instruction, broken tool, model limitation — and promote to the golden set the cases that represent a pattern. It is the cheapest stage of the cycle and the one that decides whether in six months the system is better or simply older.

Metrics: what decides and what is just pretty noise

Any provider's default telemetry gives you tokens, latency and errors. All three are necessary and none of them tells you whether the system works. These are the four families that do decide:

• Quality: success rate per capability; groundedness (percentage of claims backed by a citation that actually resolves); validity of tool calls (correct schema and plausible arguments); rate of correct abstention when there is not enough information.
• Operational reliability: p50 and p95 end-to-end latency; steps per task; loop and timeout rates; tool error rate.
• Cost: cost per useful task — credits consumed divided by correct results that were actually used — input-to-output token ratio, and the marginal cost of carried context. Cost per token is a provider metric; cost per useful task is a business metric.
• Adoption: containment (tasks closed without human intervention), escalation rate, and the percentage of outputs a human edits before accepting — the most honest indicator of perceived quality there is.

Of all of them, the one that most often changes an architectural decision is cost per useful task. A system with 92% accuracy at €0.40 per task can be a worse business than one with 86% at €0.04 if the remaining 8% is absorbed by a human queue that already exists. That comparison is impossible to make if you measure tokens. On why that figure explodes between pilot and scale, we go deeper in the real cost of putting AI into production.

Evaluating with an LLM: useful, but the judge needs calibrating

Using a model as evaluator is what makes measuring quality at scale viable, and also the entry point for most self-deception. An LLM-based judge is just another classifier: before trusting it you have to measure its agreement with human labels on a sample. As a working rule, below substantial agreement — a kappa around 0.7 — the judge is measuring its own style, not your quality.

Four rules we always apply: a rubric written with a single criterion per call (do not ask for "score overall quality from 1 to 10"); a judge model different from the one evaluated, so as not to reward its own style; pairwise comparison when the criterion is preference and absolute scoring only when there is an objective rubric; and a permanent human sampling of around 10% to detect when the judge drifts. The judge drifts too, and it needs its own golden set as well.

How that rubric is written — and how you choose the baseline to compare against, which is the other half of the problem — is covered in what it actually means for an agent to "work better", with the research behind it.

The four failures that only show up in production

None of these appear in development. All four appear when there is real traffic and time.

1. Provider drift. The model is updated and your system changes without anyone making a commit. The countermeasure is twofold: an explicit version pin in the bundle and a scheduled regression suite that runs against production weekly and publishes the comparison with the baseline. It is the cheapest control in the whole cycle and the one fewest teams have.

2. Silent retrieval degradation. The corpus grows, semantic density increases, the top-k that worked with 2,000 documents stops bringing back the right chunk at 50,000. The symptom is treacherous: the model keeps answering with complete confidence, only now over incomplete context. It is detected by measuring retrieval separately — context precision and recall — and not just the final answer. It is where a RAG system is most at stake, and we cover it in depth in RAG for enterprise applications.

3. Regression from a prompt edit. Someone fixes the case a customer reported and breaks three nobody was watching. It is exactly the failure Gate 1 exists to block; without it, every fix is a bet. And with prompts editable live from an admin panel, with no release and no evaluation, it is a bet placed several times a week.

4. Context cost explosion. The whole conversation history carried along, the complete file when a section would have done, twenty tool descriptions replicated on every call. Input cost usually exceeds output cost and almost nobody separates it in their analysis. The countermeasure is a periodic audit of context composition, which is context engineering applied to the budget.

Governance: who signs off on each gate

A cycle with no assigned owners degrades within two months. The minimum split that works in teams of 10 to 100 people assigns the task contract and the golden set to product or domain; the bundle, the gates and observability to engineering; and the decision to promote and to roll back to a single owner — usually the Tech Lead. Three roles, not a committee.

On top of that comes the regulatory layer, which in Europe is no longer theoretical: for systems classified as high risk, traceability and event logging stop being good engineering practice and become a requirement. The good news is that the cycle just described produces almost all that evidence as a by-product — retained traces, signed versions, archived evaluations — provided it is designed that way from the start. Doing it afterwards, as a retrofit, is where cost multiplies: we detail it in Compliance-First AI Design.

Where your team is: four levels

A maturity model is good for one thing only: deciding what the next move is, not for hanging a medal on yourself.

• Level 0 · Demo. It works, it is not measured, the prompt lives outside the repository. Next move: the task contract.
• Level 1 · Reproducible. Full bundle versioned, model pinned, index snapshotted. You can rebuild any release. Next move: the golden set and Gate 1.
• Level 2 · Evaluated. Offline evaluation in CI, progressive rollout, complete traces. You know whether a change improves or worsens things before serving it. Next move: closing the loop with failure mining.
• Level 3 · Governed. Loop closed, cost per task as an acceptance criterion, named owners, auditable evidence. Here the system improves with use instead of ageing with it.

Most teams we work with come in at level 0 or 1 and believe they are at 2, because they have dashboards. Dashboards are level 1; level 2 begins when a number can block a deployment. If you want to place this layer inside the company's full maturity map — and not just the engineering team's — the piece on AI engineering: from pilot to production positions it against the other six.

The first two weeks

If you have an LLM system in production and none of the above installed, this is the order that produces the most signal per euro invested. It requires no new platform and no rewriting.

Week 1. Write the task contract with the domain person — two pages, including the abstention policy. Instrument complete traces with gen_ai.* attributes on the main flow. Freeze the bundle: put everything on the eight-element list into git and pin the model version.

Week 2. Build 30 golden cases from real traffic, including the five failures that hurt most. Measure the baseline and publish it. Set up the fast suite in CI with two thresholds: quality on the critical subset and cost per task. From then on, every change goes through the gate.

In two weeks you will not have mature LLMOps, but you will have the one thing genuinely needed to start: the ability to answer "yes" or "no" to whether yesterday's change improved the system. Everything else is built on top of that answer.

The same discipline you already apply to code

None of the above is new as a principle. Contract before implementation, automated evaluation before promotion, progressive rollout, telemetry that closes the loop: it is ordinary software engineering, applied to a component that is not deterministic. That is why teams already working with Spec-Driven Development adopt LLMOps in weeks rather than quarters — they already have the habit of writing the specification before the artefact, and the context bundle is just another versioned specification.

In onext AI-Accelerated Development programmes this cycle is part of the harness we leave installed, not a later add-on. The results we sign with clients in 2026 — ×7 delivery speed, 0 sprints lost, −50% time-to-production — do not come from writing better prompts: they come from teams being able to deploy changes to AI systems without fear, because there is a gate that says whether the change is good before a user sees it.

Frequently asked questions

Are LLMOps and MLOps the same thing?

They share the intent — operating systems whose behaviour is not written in your code — but not the object. In MLOps you version weights, features and training datasets, and the model only changes when you retrain it. In LLMOps you version a context bundle (prompts, tools, retrieval, routing policies, guardrails and thresholds) on top of a model you do not control and that can change without you touching anything. That shortens the change cycle from weeks to hours, makes evaluation the critical artefact, and shifts cost from training to inference.

How many cases does a golden set need to start evaluating?

Between 30 and 50 per capability is enough to catch large regressions, which is 80% of the value. Do not start with 400: a large set nobody runs is worth less than a small one that runs on every pull request. Stratify by difficulty and include adversarial cases — out-of-domain inputs, legitimate ambiguity where the right answer is to acknowledge it, and instruction injection in retrieved documents. The golden set grows later, fed by real production failures.

Can I use the same model as judge of the system I am evaluating?

You can, but it has to be calibrated against human labels on a sample first. As a working rule, below substantial agreement (kappa around 0.7) the judge measures its own style, not your quality. It is also advisable to use a model different from the one evaluated, a rubric written with a single criterion per call, and a permanent human sampling of around 10% to detect when the judge drifts.

How often should it be re-evaluated if nothing changes?

Weekly at minimum, and whenever the provider announces a model update. An LLM system can degrade without a single commit: the model changes, the indexed corpus grows and recall drops, or real usage drifts away from what you tested. A cron job that runs the regression suite against production and publishes the comparison with the baseline is the cheapest control in the cycle.

Do I need a dedicated LLMOps platform or is my current stack enough?

In most cases the current stack goes further than people think: git for the bundle, the CI you already have for Gate 1, feature flags for progressive rollout, and your tracing backend for observability with gen_ai.* conventions. A dedicated platform mostly adds ergonomics for annotation and comparison of evaluations. Buying it before having a task contract and a golden set accelerates nothing: it adds one more surface to maintain.