AI engineering: from pilot to production

What this layer actually measures

It measures repeatability.

A demo proves something can happen. Production requires it to happen every time, on inputs nobody anticipated, at a predictable cost, and without quality degrading silently when a model changes underneath.

What it does not measure: model sophistication. Model choice is one of the least decisive decisions in the set, and usually the one that consumes most committee time.

Why this is the fourth layer and not the first

Almost everyone enters here, because it is the only layer with a salesperson knocking at the door. And it is not an absurd mistake: it is where something visible happens.

The problem is that engineering solves very well the problem it is given. If the problem is badly chosen — because nobody did the value layer — it will produce an impeccable solution to something that did not matter. If the material does not exist — because nobody did the data layer — it will produce confident, foreign-sounding answers. In both cases the system works and the project fails, which is the most expensive way to fail.

Placing it fourth does not demote it. It protects it: it arrives once the problem is bounded and measured, which is when engineering work actually pays.

The four decisions that matter

1 · Build or buy. Our criterion is not initial cost but exit cost. Buying is right when the process is common across the sector and does not differentiate you. Building is right when the process encodes how your company works, because there the asset is context and you do not want it living at the vendor. The question that settles it: if in two years we wanted to change vendor, what would we take with us? If the answer is "nothing", you are renting your differentiation.

2 · Evaluation. It is what separates teams that reach production from those that do not, and it is the least practised. Without a set of test cases with known correct answers, there is no way to tell whether a change improves or degrades the system. Decisions end up made on impressions, and impressions do not detect an 8% regression.

3 · Cost per task. Not monthly cost: cost per unit of work. It is the metric that tells you whether a case scales, and the one almost nobody instruments until an unpleasant invoice arrives.

4 · Observability. What was logged, with what context, what came back, and who reviewed it. Without this, a production failure is not diagnosable: it is an anecdote.

Diagnosis: where you are

Stage 1 · Demo. It works in the hands of whoever built it. No evaluation, no cost per task, no logging. Observable signal: when it fails, the explanation is "AI gets it wrong sometimes".

Stage 2 · Operable pilot. There is an evaluation set, a known cost per task and basic logging. Observable signal: there is a number for how often it is right, and it gets re-measured whenever something changes.

Stage 3 · Service. It has an owner, thresholds, alerts when quality drops and a procedure for when it does. Observable signal: degradation is detected before a user reports it.

The first 90 days

Weeks 1-2 · Evaluation set. Thirty to fifty real cases with known correct answers, drawn from history. It is the most profitable engineering asset you can build and it takes days, not months.

Weeks 3-4 · Documented build/buy decision, with the exit-cost criterion written down, not assumed.

Weeks 5-8 · Bounded build or integration, measured against the evaluation set on every iteration.

Weeks 9-10 · Cost per task and thresholds. What each unit costs and above what figure it stops making sense.

Weeks 11-12 · Logging and a human review gate wherever output has external impact, connecting to the governance layer.

What NOT to do yet

• Do not build an AI platform. A platform for one case is infrastructure looking for a problem. With three cases it justifies itself, and you will also know what it has to do.
• Do not optimise the model before you have evaluation. Without measurement, "better" is an opinion.
• Do not chase every new model release. The cost of keeping current is high and the return, in most enterprise cases, marginal against improving context.
• Do not start with autonomous agents. Gartner expects more than 40% of agent-based projects to be cancelled before 2027. Autonomy is earned in stages.

What skipping it costs

It costs the technical team's credibility. A system that reaches production without evaluation or observability fails randomly in the eyes of the business, and the team cannot explain why. From there every incident reinforces the idea that "this is not mature", when what was not mature was the engineering practice around it.

And it costs money quietly: without cost per task, a case that does not scale can take months to reveal itself, and by then it is embedded in operations.

Fourth layer of the series on the seven layers of AI maturity.

Frequently asked questions

What does it take to get AI genuinely into production?

Four things: an evaluation set of real cases with known answers, instrumented cost per task, logging of what went in and what came out, and a human review gate wherever output has external impact. Without evaluation you cannot know whether a change improves the system.

Should we build or buy an AI solution?

The decisive criterion is exit cost, not initial price. Buying suits processes common across your sector that do not differentiate you. Building makes sense when the process encodes how your company works: there the asset is context, and it should stay in-house.

What does it cost to run an AI system in production?

It depends on volume, but the metric to demand is cost per task, not monthly cost. It is the only one that tells you whether the case scales. A system without instrumented cost per task can take months to reveal that it is not viable.

Is it worth starting with autonomous agents?

As a rule, not at the outset. Gartner expects more than 40% of agent-based projects to be cancelled before 2027. Autonomy is granted in stages, after the system demonstrates measured accuracy under human supervision.