How to measure whether your product's AI works: evals and cost per useful task

Every software company that puts AI into its product reaches, sooner or later, the same uncomfortable question: does this actually work, or does it just look like it? And a second, even more uncomfortable one when the bill arrives: does it pay off? Most teams can't answer either with data, because they measured the AI the way you measure a normal feature —"it passes the tests, it ships"— when generative AI demands measuring differently. This is the practical guide to what to measure, how to instrument it and what decision to make with the result. It isn't theory: it's what separates an AI feature that holds up in production from one that races toward the wall.

Why "it works" isn't a metric

Traditional software is deterministic: the same input gives the same output, and a test that passes today passes tomorrow. Generative AI doesn't work like that. The same question can give different answers; an answer can sound impeccable and be false; and the model can degrade when the provider updates it without warning. That's why the "it seems to work" of the demo is treacherous: you make the demo, with the happy case and the tuned prompt, and production is used by thousands of customers with data you didn't anticipate.

The classic symptom: an AI feature that impressed in the internal review and, three weeks after shipping, piles up support complaints over invented answers, while the token bill has tripled and nobody can explain why. It's not that the AI is useless. It's that it shipped without instrumentation, and "works" was a feeling, not a number. Measuring well isn't a maturity luxury: it's the requirement for your product's AI to be operable, governable and profitable.

The only two metrics that matter

You can instrument an AI feature with fifty dashboards and understand nothing. The discipline is reducing to the essential. There are two metrics that govern the decision; the rest are diagnostic.

Notice that the two are coupled by the word "useful": a cheap output that's wrong doesn't count, nor does a correct one nobody used. The intersection —correct and used— is where the value lives. Everything else (latency, tokens, error rate by type) is diagnostic: it helps you improve, but it isn't the decision criterion.

How to build evals that actually help

An eval is an automatic, reproducible way to answer "is this output correct?". Without evals, quality is judged by whoever looks at the demo that day, with their own criteria and their own mood. With evals, the criterion is explicit, versioned and executable on every change. Not all evals are equal; they're built by levels, from cheap to expensive:

The rule that avoids self-deception: the eval is written before the feature. If you don't know how you're going to decide that an output is correct, you're not ready to build it — you're ready to make another demo. And an honest metric above average accuracy: pass@k (in how many attempts does the system produce at least one correct output?) versus the first-attempt success rate; it tells you whether the problem is capability or consistency, which are fixed differently.

How to measure cost per useful task (with numbers)

The most common cost mistake is measuring cost per call or cost per user. Neither tells you whether the feature is profitable. The right metric is cost per useful task: what it costs to produce a result that passed the eval and was used. The formula:

A concrete example to see it:

• An assistant feature in your SaaS consumes €500 a month in tokens.
• It generates 20,000 responses a month. But 70% pass the eval (14,000 correct) and, of those, the user uses 60% (8,400 useful).
• Cost per useful task = €500 ÷ 8,400 = €0.06 per useful result.

Now the decision is possible: if each useful result saves your customer €2 of manual work or improves retention, €0.06 is a bargain. If the feature is cosmetic, maybe not. And notice the leverage: raising the success rate from 70% to 85% lowers the cost per useful task by 18% without touching the model — it's method (better context, better eval), not a more expensive model. That's exactly the work of context engineering.

The minimum dashboard: what to check every week

You don't need a command center. You need five lines a Head of Product and a CTO can read in a minute and act on:

1. % of outputs that pass the eval (verified quality). Trend: is it going up, down, holding?
2. Cost per useful task, with its trend. It's the profitability metric.
3. % of useful outputs used (real adoption of the feature). If it's low, maybe the problem isn't the AI: it's that the feature doesn't matter.
4. Quality drift after provider changes: a drop in the pass rate without you touching anything = the model changed. Without this panel, you find out through support.
5. Queue of labeled failures: the cases that failed, grouped by type. It's the improvement backlog — and it feeds the golden dataset.

When an AI feature deserves to stay alive

Measuring is for deciding. With the two metrics and the dashboard, each AI feature falls into one of four quadrants, and each one has a clear action:

This is also where build vs. buy is resolved: for the pieces in the "high quality, high cost" quadrant that you can't optimize inside your product, the honest alternative is to delegate execution to a provider who takes on the model and pricing risk in the contract, not on your bill. You build what differentiates you; you take out of your variable cost what doesn't.

The mistakes that break your measurement

• Measuring average accuracy instead of per case. A 90% accuracy with the 10% of failures concentrated in your most important use case is a broken product that looks healthy.
• Evals that self-approve. An LLM-as-judge without human calibration ends up rewarding the style of the model it judges. Always sample with humans.
• Measuring cost per call, not per useful task. You optimize for making cheap calls that don't help instead of results that do.
• Not watching provider drift. The model changes under your feet; without the drift panel, your quality drops and you blame the users.
• Instrumenting at the end. Cost and evals are built with the first prototype, not when the first surprise bill arrives.

Frequently asked questions

How do I know if my SaaS product's AI actually works?

With reproducible evals that measure whether the output is correct according to an explicit criterion (not "sounds good"), combined with the real user signal in production. "It seems to work" is an impression; the % of outputs that pass the eval, with its trend, is a data point. Without evals, every release is a bet and failures are discovered late, in production.

What is cost per useful task and why is it the metric that matters?

It's what it costs to produce a result that passed the eval and was actually used: credits consumed ÷ correct and used outputs. Unlike cost per call or per user, it tells you whether the feature is profitable and lets you decide build vs. buy. Raising quality lowers the cost per useful task without touching the model: it's method, not a more expensive model.

Do I need to build evals even if my AI feature is simple?

Yes, even if they're of the cheapest level (deterministic rules + a small golden dataset). Without an automatic way to know whether an output is correct, you can't put the feature in production with guarantees or detect when the model degrades. The eval is written before the feature: if you don't know how to decide that it works, you're not ready to build it yet.

How often do I review these metrics?

Weekly for the decision dashboard (pass rate, cost per useful task, adoption, provider drift, failure queue) and on every release/model change for the golden dataset in CI. Provider drift forces continuous vigilance: a model can change without warning and your quality drop without you touching anything.

Conclusion

Your product's AI isn't governed with impressions: it's governed with two numbers. Verified quality tells you whether it works; cost per useful task tells you whether it pays off. With those two and a five-line dashboard, the decision to scale, optimize, fix or kill an AI feature stops being a boardroom opinion and becomes a reading of data. And almost always, the lever to improve both numbers isn't a more expensive model: it's method —context of your product and verification at every step—, which is exactly what takes AI from the demo to production and keeps it profitable.

If you have AI features in production without instrumentation —or you want to build them well from the start— start with a diagnosis: in a few weeks you'll have the evals, the cost per useful task and the decision dashboard running, and you'll know which features to scale and which to redesign. And if you're a software or SaaS company, here's our full approach: AI for software and SaaS companies.