AI agents in production: the quality gap no one measures (and how to close it)

Your team has deployed an AI agent to production. The dashboards are green: latency is low, tokens are within budget, traces are stored correctly, uptime is 99.9%. Everything seems to work. Until a customer writes in the support channel: "the answer your assistant gave me was completely wrong". And that's when you discover you've been serving mediocre answers for weeks without a single alert going off.

This is the quality gap in enterprise AI agents. And it's far more common than most CTOs believe.

Observability is not quality

Recent data paints an uncomfortable picture. 89% of teams that deploy AI agents to production have some form of observability: they see traces, measure latency, count tokens, control costs. It's the equivalent of monitoring that a server responds — necessary, but insufficient.

Only 52% of those teams actively evaluate the quality of the answers their agents generate. And quality remains the number-one barrier in production for 32% of teams — ahead of latency, cost or technical integration.

The reason for this gap is structural: classic observability tools (Langfuse, LangSmith, Helicone, Datadog) are designed to answer "did the agent run?". None answers, natively and automatically, the question that really matters: "was the answer good?".

The uncomfortable data point: in most enterprise deployments, quality problems are discovered when a user complains, not when an alert goes off. A customer's negative feedback arrives days or weeks after the agent started to degrade. By then, hundreds or thousands of conversations have already served poor answers.

The 6 quality dimensions that matter in enterprise

Evaluating quality in an AI agent isn't giving each answer a grade from 1 to 10. It's measuring multiple dimensions that, together, determine whether the answer generates real value for the user or generates risk for the organization. These are the six a CTO should demand before going to production:

1. Relevance. Does the answer address the question the user asked? An agent can generate perfect text that has nothing to do with what it was asked. Relevance measures that alignment.
2. Usefulness. Does the answer provide concrete value? An answer can be relevant and at the same time empty: "it depends on the context" is relevant but not useful. Usefulness measures whether the user can act on the answer.
3. Completeness. Are important details missing? An answer that's right but omits a critical nuance can be worse than not answering. Completeness detects those omissions.
4. Coherence. Is the logic easy to follow? In long or multi-step answers, coherence measures whether the reasoning makes sense from start to finish, without internal contradictions.
5. Safety. Is there legal or reputational risk? This is the dimension that most worries legal and compliance teams. A single answer that recommends something illegal, discriminatory or factually dangerous can have disproportionate consequences. In the context of the EU AI Act, which applies from August 2025, this dimension moves from "nice to have" to "mandatory".
6. Overall score. A consolidated score (0-10) that weights the previous five dimensions and lets you answer at a glance: is the agent working well or not?

These six dimensions aren't theoretical. They're the ones teams in production are measuring today to make operational decisions: do I deploy this new version of the prompt? Do I switch models? Do I need to intervene manually? Without them, each of those decisions is made blind.

The LLM-as-judge pattern: automatic evaluation without datasets

The most scalable way to evaluate quality in production is a pattern that has consolidated over the last 12 months: LLM-as-judge. The idea is simple: a second LLM, independent of the one that generates the answers, evaluates each conversation against the defined quality dimensions. It doesn't need manual datasets, it doesn't need human evaluators for each answer, and it scales to thousands of daily conversations with no operational effort.

The evaluating LLM receives the full conversation (user question + agent answer + context) and produces a score per dimension. If the safety score drops below a threshold (for example, 3 out of 10), an alert fires automatically. If the overall score degrades consistently over 24 hours, the team receives a notification before any user complains.

What's critical is that the evaluation happens asynchronously: the user receives their answer with no additional delay. The LLM-as-judge works in the background, and alerts reach the team in real time. Zero impact on latency, full visibility into quality.

Sentygent: automatic quality monitoring for agents in production

Sentygent is a quality-monitoring platform for AI agents that implements exactly this pattern. An independent LLM-as-judge automatically evaluates each conversation across the six quality dimensions — with no manual configuration, no datasets, no human review by default.

Integration requires five lines of code (a wrapper over the LLM provider client you already use) and evaluation happens asynchronously — the user never experiences additional latency. That makes it a tool you can add to an already-deployed agent without touching the system's logic.

What it brings to an enterprise team

• Automatic safety alerts. If the safety score of any conversation drops below a threshold (3/10 by default), Sentygent fires an alert via webhook immediately. For a CTO, this means safety problems are detected in minutes, not days.
• Real-time quality dashboards. Scores per dimension, hourly trends, filtering by tags (model, prompt version, intent type). The team can see at a glance whether a new version of the agent improved or worsened quality.
• Multi-agent traceability. In architectures with several orchestrated agents (increasingly common in enterprise), Sentygent shows cost and quality broken down per individual agent. That lets you know exactly which of the pipeline's agents is degrading the final answer.
• Native RAG. Retrieval operations are captured in detail: retrieved chunks, individual relevance scores, sources. If quality drops because the retriever is bringing back irrelevant documents, it's visible directly in the trace.
• Zero integration friction. Compatible with Anthropic, AWS Bedrock, OpenAI, Cohere, Mistral, Groq, Ollama and the Vercel AI SDK. Instrumentation doesn't require refactoring existing code — you wrap the LLM provider client and the SDK captures everything automatically.

For a team that already has observability but not quality monitoring, Sentygent closes exactly that gap: it goes from "I know the agent responded" to "I know the agent responded well, and if it didn't, I find out before the user does".

How to connect quality in production with quality in development

This is where the conversation gets interesting for a CTO who thinks in systems, not isolated tools.

Quality monitoring in production detects problems. But problems originate in development — in a poorly designed prompt, in insufficient context, in an ambiguous specification. If the development team works with Spec-Driven Development, the specifications define the acceptance criteria for each feature. Those same criteria can feed quality monitoring in production.

The full flow looks like this:

1. In development, the team specifies what the agent must do: which questions it must answer, with what level of completeness, which safety constraints apply. That's a spec.
2. At deploy, the agent's code is instrumented with Sentygent (5 lines, no refactor).
3. In production, Sentygent automatically evaluates each answer against the 6 dimensions. If quality degrades, the alert reaches the team.
4. In iteration, the team compares the quality scores with the original specs and decides whether the problem is the prompt, the model, the context or the spec itself.

This cycle — spec → deploy → monitor → iterate — is what turns an AI agent into a real enterprise system. Without quality monitoring, the cycle breaks at the third step and the team flies blind.

Practical connection. At onext, teams that use onext AI-Accelerated Development define quality rules in the project's constitution (what the agent must and must not do). Tools like Sentygent make it possible to close the loop by verifying in production that those rules are met conversation by conversation. The spec says "the agent must never recommend medication"; Sentygent detects the first time it does, before any user escalates it.

What a CTO should demand before going to production

Based on what we see in teams that go from pilot to enterprise production successfully, these are the five minimum conditions a CTO should verify before giving the green light:

1. Active quality monitoring across the 6 dimensions. Not just observability. If you can't answer "what was the average quality of the agent's answers this week?", you're not ready for production.
2. Automatic safety alerts. If the safety score drops below a threshold, the team finds out in minutes. Not in days, nor when a customer complains.
3. Cost traceability per conversation and per agent. In multi-agent architectures, knowing how much each answer costs is fundamental to controlling the budget. If one agent in the pipeline is consuming 80% of the cost but contributing 10% of the value, you need to know.
4. A defined iteration flow. When quality degrades, who acts? What gets reviewed first? Do you change the prompt, the model, the context or the spec? Without a clear protocol, quality alerts turn into noise.
5. Written specifications of what the agent must and must not do. If the only source of truth about the agent's expected behavior is the memory of the developer who built it, quality evaluation has no reference to compare against.

Conclusion: quality is the new uptime

Over the last 20 years, the software industry built a healthy obsession with uptime. Today nobody deploys a service without availability alerts, latency dashboards and incident runbooks. That infrastructure took a decade to mature, but today it's standard.

With AI agents, quality is the new uptime. An agent that's available but answers badly is worse than a downed agent — because the damage is silent, cumulative and hard to reverse once the user loses trust. Quality-monitoring infrastructure is where uptime infrastructure was in 2010: it exists, it works, but most teams haven't yet adopted it.

The teams that adopt it today — with tools like Sentygent for automatic monitoring and with SDD for specifications in development — will have the advantage of detecting problems before their users, iterating with real data and building the trust that separates a pilot from an enterprise product.

Observing tokens is necessary.
Evaluating quality is what separates agents that work from agents that work well.