Something strange happens to enterprise AI projects on the journey from pilot to production: the bill doesn't grow linearly. It grows 30 times, 60 times, in some cases 200 times, according to the FinOps Foundation itself. And it's not because the system does 30 times more work. It's because the system starts doing things that weren't measured in the pilot — and by the time someone looks at them, they're already in the annual contract.
There's a pattern we keep seeing in client after client: the pilot passes all the technical KPIs, someone signs off on production, the first quarterly bill arrives and the conversation changes tone. The CFO doesn't ask about accuracy. They ask why a process that cost €200 a month now costs €12,000.
This post is the version a CTO shows their team the following Monday. It's not a manifesto about FinOps. It's an inventory of the 7 concrete anti-patterns that multiply your bill, a metric that matters more than cost/token, and a set of circuit breakers any team can instrument before signing off on the next roadmap.
The cost of enterprise AI in figures
Public data from the FinOps Foundation, Gartner and State of FinOps 2026
1. Why AI cost doesn't look like traditional cloud cost
Traditional cloud has a property its operators took ten years to internalize but that everyone knows today: cost is proportional to usage, and usage is proportional to useful work. If an API receives 10x more requests, it bills 10x more. If a cluster serves 10x more users, it pays 10x more in compute. There are gaps (data egress, hidden costs of managed databases) but the proportionality principle stays reasonable.
Enterprise AI doesn't behave that way. Useful usage and billed usage decouple as soon as the system leaves the pilot, for three structural reasons:
- The input isn't the billable unit. When a user asks a 12-word question to a RAG assistant, what gets billed isn't 12 words: it's the 8,000 tokens of retrieved context plus the 1,200-token system prompt plus the 600-token output. A small request can bill 20,000 tokens. What's invisible to the user is the only thing the provider charges for.
- The retries aren't the user's, they're the system's. A call to an agent with tool use can generate 5-10 internal calls to the model before returning an answer. If one of those calls fails, there are retries. If the agent is poorly calibrated, there are loops. The bill grows exponentially, not proportionally.
- "Context bloat" is the most expensive technical debt that exists today. Every time a team adds a new source to the RAG, a new tool to the agent or a new instruction to the system prompt, the cost per request rises permanently. There's nothing in standard observability that warns of that.
Result: the cost curve has a discontinuity at the move to production that traditional FinOps tools don't detect because they're designed for VMs, not for tokens.
2. The 7 anti-patterns that multiply your bill
An inventory, in approximate order of impact observed in mid/large enterprise. No anti-pattern is exotic. Each one is defensible the first time it's introduced. The problem isn't any of them individually; it's the silent accumulation.
The system prompt starts at 200 tokens. Every time something goes wrong in production, someone adds a defensive instruction ("never return X", "always ask before Y"). In six months it's 4,500 tokens. Every request pays that tax. An assistant with 1M requests/month pays 45 million additional tokens per month just on the system prompt — without changing a line of the product.
The hypothesis that "more context is always better" leads teams to retrieve 10-20 chunks per request when 3-5 would be enough. Accuracy improves marginally; the bill grows linearly with the chunks retrieved. The chunks that aren't used are pure payment with no value.
Classic pattern: a "supervisor" agent calls a "researcher" agent that calls a "writer" agent that returns to the supervisor for validation. Three calls to the model where one well-designed call would suffice. In poorly written agentic workloads, 60-80% of the cost is internal calls the user never sees.
A call fails, the code retries up to three times. The post-call validation fails, the model is called again. The user doesn't notice the latency because there's intermediate response caching, but the bill counts every attempt. Without explicit retry caps, a request can bill 5-15x what it should.
Enabling "thinking" or "extended reasoning" in models that support it drives up output tokens. In models with extended reasoning (Claude Opus, GPT with o1-style reasoning, Gemini Deep Think), a request with CoT can bill 8-15x the tokens of a direct answer. Sometimes it's worth it; many times it isn't, and no one audits it.
The team deploys a new version of the RAG service. The pipeline rebuilds the embeddings of the corpus's 200,000 documents. That month's embeddings bill is multiplied by the number of deployments. Teams that deploy 4-5 times a month pay 4-5 embeddings bills when they should pay one.
Anthropic, OpenAI and Google have offered 50-90% discounts on cached tokens for over a year. Teams that send the same system prompt in every request without setting cache_control are paying full rate for static information. The discount doesn't apply on its own: you have to turn it on.
The good news: all seven are avoidable with basic instrumentation. The bad news: none of them are detected with standard provider dashboards — you have to measure them at the application level, not the billing level. It's the same difference as between architecting RAG well or badly: the errors aren't technically impossible to avoid, they're the ones no one is watching.
3. The metric that matters: cost per useful task completed
Cost/token is a provider metric, not a business one. Looking at it is like looking at the cost per milligram of fuel when talking about the cost of a trip.
The metric that matters for a CTO or CFO is cost per useful task completed. A "useful task completed" is defined at the product level: a correct answer from the assistant, a document extracted without errors, a scoring decision delivered with confidence above the threshold, generated code that passed the tests.
This metric has two properties that cost/token doesn't:
- It captures the retries and loops problem. If your agent needs 8 calls to the model to deliver a useful task, the cost per task is 8x the cost per call. The KPI reflects it; the token KPI doesn't.
- It makes quality explicit as an economic variable. If you drop accuracy from 95% to 70%, the cost per useful task rises — because non-useful tasks are pure payment with no business value. A system that's cheap per request but with 30% non-useful tasks is an expensive system at the business-unit level.
Two illustrative cases: the before-and-after math
To make cost per useful task concrete instead of abstract, two composite scenarios based on patterns we see repeatedly in mid-large enterprise. The numbers are illustrative but the arithmetic squares with the multipliers cited above — they're built so any CTO can reproduce the calculation for their own case.
Before (no instrumentation)
- 200,000 requests/month on Claude Sonnet
- Accumulated system prompt: 4,500 tokens (context bloat)
- RAG with
top_k=15(~9,000 tokens retrieved) - Extended reasoning active on 30% of requests
- No
cache_controlconfigured - Average input: ~14,000 tokens · average output: ~1,330 tokens
- Cost/request: ≈ $0.062
≈ €12,000/month
After (5 fixes in 2-3 sprints)
- System prompt audit → 1,500 tokens (CB1)
- Explicit token cap (CB2)
- CoT gating to 5% of requests (CB5)
- Prompt caching enabled (anti-pattern 7)
top_k=5with reranker (anti-pattern 2)- Average input: ~5,000 tokens (4,500 cached at 25%) · output: ~555 tokens
- Cost/request: ≈ $0.013
≈ €2,800/month
Reduction: −77% on the monthly bill without touching accuracy. ROI in the first month.
Before (no instrumentation)
- 50,000 tasks/month with tool use and no iteration cap
- Average: 8 calls to the model per task (loops between sub-agents)
- Aggressive retry with no distinction between transient vs. semantic error
- No caching
- Rate of tasks with useful output: ~60%
- Cost/model call: ≈ $0.030
- Cost/gross task: ≈ $0.24
≈ €0.38/useful task
After (1-2 sprints + alignment)
task_outcomeinstrumentation in observability- Iteration cap at 5 (CB3)
- Retry policy differentiated by error class (CB4)
- Caching over system + tool definitions
- Rate of useful tasks rises to ~85% by reducing failed loops
- Cost/call: ≈ $0.020
- Cost/gross task: ≈ $0.10
≈ €0.11/useful task
Reduction: −71% on the cost per useful economic unit. Accuracy rises — not falls — because the infinite loops were the symptom, not the cure.
These two cases aren't the most extreme we've seen. They're the most illustrative of the pattern.
4. Cost circuit breakers: 5 controls that block the damage
Circuit breakers are to AI cost what rate limits are to APIs: pre-incident operational protection, not post-mortem analysis.
Budget per feature
Each product feature that uses AI has a monthly budget assigned in euros. The instrumentation tracks it in real time. When consumption crosses 80%, it alerts the owning team. When it crosses 100%, the feature degrades to a deterministic fallback until the next period. There are no surprises at month-end because it's structurally impossible for there to be any.
Token cap per request
Each call to the model has an explicit max_tokens for both input and output. If a request is about to exceed it, it's rejected before reaching the provider. This blocks the most expensive scenario: a pathological request with 200K tokens of context and an unlimited response. Input cap + output cap, non-negotiable.
Call cap per agentic session
Any agent with tool use has a maximum of N iterations per session (typically 8-15 depending on the case). On reaching the cap, the agent returns the best available partial answer and records the session as partial. This blocks infinite loops before they become infinite bills.
Aggressive retry sampling
If a call fails, a maximum of 1 retry — and only if the error is transient (timeout, 429, 5xx). Semantic errors (post-call validation fails) are not retried automatically: they're recorded and traced for observability. Retrying semantic errors is the most expensive way not to fix a bug.
"Cost audit" mode in CI
Before merging any change that touches prompts, RAG, agents or tool use, a test runs a representative set of requests and compares cost and latency with the baseline. If cost rises >15% without the PR justifying it, the merge is blocked. Cost as a quality variable — just like regression tests, just like security tests.
All five are cheap to implement (1-3 weeks for a team that already has basic observability). What's expensive is not having them when the first unexpected quarterly bill arrives.
5. When to choose a small, large or on-prem model: threshold table
The large-model vs. small-model vs. local-inference decision isn't ideological. It's economic, and it depends on three variables: request volume/month, accuracy criticality and data sensitivity.
| Scenario | Volume | Critical accuracy | Sensitive data | Practical recommendation |
|---|---|---|---|---|
| End-customer conversational assistant | >5M/month | medium | low | Small model + fine-tuning (Haiku, GPT-4o-mini, Gemini Flash). Cost/request 10-30x lower than the flagship at the same accuracy in a bounded domain. |
| Legal / compliance analysis | <100K/month | very high | high | Large managed model with BAA + zero retention (Claude Sonnet/Opus on Bedrock or Vertex). Cost/request doesn't dominate; the risk of error does. |
| Autonomous operations agent | 50K-1M/month | high + multi-step | medium | Router pattern: large model for reasoning + small model for tool calling. Reduces cost 40-60% vs. using the large one for everything. |
| Batch document processing | >10M docs/month | medium | variable | Small model + aggressive embeddings caching + open-source alternative (Llama, Mistral) on-prem if sensitivity justifies it. |
| Internal development copilot | 200-500 devs | medium | high (code) | Managed + local open-source mix according to repo sensitivity. Reasonable: €30-80/dev/month. Above €150/dev/month you have to audit usage. |
No row is definitive. The point of the table is that the model decision is a cost decision as much as a capability one — and almost no one treats it that way. The complete selection framework, with TCO and an exit-cost axis, we develop in how to choose an LLM in 2026.
6. The invisible debt
State of FinOps 2026 places AI cost as the CFO's #1 priority. Gartner predicts that >40% of agentic AI projects will be canceled before the end of 2027 — and the main reason isn't that the technology doesn't work, but that the economics don't add up.
They don't add up because there's an invisible debt that accumulates from day 1 of the pilot: every unaudited token, every over-included RAG chunk, every agent that calls another agent with no budget, every deployment that regenerates embeddings unnecessarily. None of those decisions is individually suspicious. All together they're the difference between €200 and €12,000 a month for the same work.
The CTO who instruments token economics, defines cost per useful task as a KPI and deploys circuit breakers from the pilot phase builds on auditable foundations. The one who doesn't will be explaining to their CFO in six months why the project's estimated ROI doesn't appear in the P&L — and why it can't be fixed without redesigning parts of the system.
The Monday test: open the latest LLM provider bill. Do you know which feature consumed each line item? Do you know what percentage are internal calls not visible to the user? Do you know your cost per useful task? If the answer to any of them is "no", you already have your entry point.
At onext we help engineering teams build that instrumentation from day one, not from the first unexpected bill. If you want to compare your current setup against the 7 anti-patterns in this post, let's talk.
Cited sources
- FinOps Foundation — FinOps for AI Overview
- FinOps Foundation — State of FinOps 2026
- Aretove — The 2026 FinOps Frontier: governing LLM costs
- Finout — FinOps in the Age of AI: CPO's guide
- arXiv — Beyond Accuracy: CLEAR framework enterprise agentic AI
- Gartner via Joget — >40% of agentic AI projects will be canceled before 2027
Jordi García is Tech Lead at onext. He works on bringing AI into governed production across development and product teams —with Spec-Driven Development, context engineering and human verification at every step— and authors onext's technical insights on the method, quality and cost of applied AI.
LinkedIn →