The exception path: what decides whether an AI automation works

We've written here about automating sales proposals, RFP responses, accounts payable, B2B support and insurance claims with AI. All five pieces settle the human part with a similar line: "validating the sensitive parts with a person", "human control on the exceptions".

The line is correct and says almost nothing. It doesn't say who that person is, what they receive, how long they have to resolve it, what happens if the queue grows, or — above all — what happens to the case the agent got wrong without flagging it as doubtful, which is the one that never reaches that person.

This piece is what was missing underneath that line. The thesis fits in one sentence: an automated process isn't judged by its accuracy rate, but by its exception path. The accuracy rate describes the happy path. The exception path decides whether the process survives its first month in production, whether people adopt it, and whether anyone can explain what it did when a customer asks.

Four boxes, and the dashboard shows two

Every case that enters an automated process ends up in one of four boxes. The system either resolves it on its own or sends it to a person; and either way, that decision was the right one or it wasn't.

The dangerous box is invisible by construction. An error the system doesn't flag generates no event: there is nothing to count, so it doesn't show up in any metric that counts events. The only thing that makes it visible is going to look for it, and we'll come back to that further down.

The two boxes in the top row pull in opposite directions, and that is the whole design problem. Escalating more reduces silent errors, but it fills the queue with noise. And noise isn't just a cost in hours: a person who approves forty cases in a row that were fine learns to approve the forty-first without reading it. Noise ends up manufacturing silent errors inside the exception path itself.

Accuracy depends on how much you let through

There is an idea in machine learning, more than half a century old, that almost never reaches sales proposals: the reject option, or selective classification. A system that can abstain chooses, case by case, whether to answer or not, and its error is measured only over what it answers.

Yonatan Geifman and Ran El-Yaniv formulated it for deep neural networks at NeurIPS 2017: "the risk of a selective classifier can be traded-off for coverage", and its performance is described by a risk-coverage curve, not by a number. Their example: a 2% top-5 error on ImageNet guaranteed with 99.9% probability… while answering only almost 60% of the cases.

The consequence is direct: any accuracy rate is one point on a curve. An accuracy percentage that doesn't say what share of the cases it is calculated over is half a number.

An example with made-up figures (the arithmetic is ours). Two proposals for the same process, handling 2,000 cases a month:

• Proposal A: resolves 95% on its own and gets 4% of what it resolves wrong. It leaves 100 exceptions and 76 silent errors a month.
• Proposal B: resolves 80% on its own and gets 1% wrong. It leaves 400 exceptions and 16 silent errors a month.

In the presentation, A wins: it automates more. Which one is actually better depends on two things that aren't in either figure: what a silent error costs in that process — a mislabelled field is not a duplicate payment — and whether you have the capacity to handle 400 exceptions a month properly. Those two questions are the exception path. Without answering them, you pick the point on the curve that looks best on a slide.

The model's confidence is a poor trigger

The intuitive answer to "which cases do we escalate?" is "the ones the model isn't sure about". It has two problems, and the second matters more than the first.

The first is calibration. In the GPT-4 technical report, the pre-trained model was highly calibrated — its confidence matched the probability of being right — and after post-training, calibration got worse: on a subset of MMLU the calibration error went from 0.007 to 0.074, and the figure caption says it plainly: "The post-training hurts calibration significantly." It is one specific model, from 2023, measured with internal probabilities on multiple-choice questions, not with the confidence a model verbalises when you ask for "a score from 1 to 10". What it does teach is that calibration has to be measured per model and per task, not assumed. Since September 2026 there is a model trained precisely for that; we analyse it in the piece on Jev.

The second problem isn't fixed by a better-calibrated model: confidence measures the model's doubt, not the consequence of the case. 97% confidence on a €200 invoice and on a €200,000 one are not the same case, and the model won't know that unless the process tells it.

The triggers that work best are the ones the model doesn't control:

In accounts payable the first trigger has been solved for decades, which is why it's one of the most rewarding processes to automate. The third is the fourth question of the ERP boundary seen from the other side: when the system is wrong, the only thing that gives it away is another source saying something different.

Putting a person in the loop doesn't fix anything on its own

So far, the design seems to be about choosing well what gets escalated. The uncomfortable part is still missing: what happens when the case reaches the person.

In October 2024, Michelle Vaccaro, Abdullah Almaatouq and Thomas Malone published in Nature Human Behaviour a preregistered meta-analysis of 106 experiments and 370 effect sizes, all published between January 2020 and June 2023, each measuring three things at once: the performance of the person alone, of the AI alone, and of the combination. The central finding is that, on average, "human–AI combinations performed significantly worse than the best of humans or AI alone" (Hedges' g = −0.23).

It has nuances that matter. The combination did improve on the person alone; what it didn't do, on average, was beat the AI alone when the AI was better. The losses were concentrated in decision tasks — choosing between a closed set of options — and the gains in content-creation tasks. And the finding most useful for designing an exception path: when the person outperformed the AI on that task, the combination gained; when the AI outperformed the person, it lost.

More than 95% of the systems studied followed the same pattern: the AI proposes and the person decides. It's the "human on the exceptions" design from the beginning of this piece, the norm in the industry and in our own pieces too. Showing confidence levels or explanations didn't change performance significantly. The authors suggest it would have gone better to split the task according to who is better at each part, but only three experiments tested that and the result wasn't significant: it's a reasonable hypothesis, not a conclusion.

The practical reading: the exception path isn't about putting a person behind the model, but about sending that person the cases where they're better than the model. What the model gets right more often than the reviewer shouldn't reach the queue; what does reach it should go to someone who knows more than the system about that type of case.

Lisanne Bainbridge saw it in 1983, studying process plants and flight decks, in five pages titled Ironies of Automation. Her second irony fits AI exactly: "the designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate" — an arbitrary collection of tasks for which "little thought may have been given to providing support". The others still hold: "efficient retrieval of knowledge from long-term memory depends on frequency of use"; nobody can keep watching something where almost nothing happens for more than about half an hour, so rare abnormalities have to be watched by an alarm; and, the last one, "it is the most successful automated systems, with rare need for manual intervention, which may need the greatest investment in human operator training".

Translated to an exception queue: the better the automation works, the rarer and harder the exceptions, and the less practice the person resolving them has. An exception isn't an average case the agent didn't feel like doing: it is, by construction, one of the hardest cases in the process. The prudent assumption is that it takes longer than the old manual average and needs more judgement, not less.

What the person receives, and in what order

The package an exception arrives with decides the quality of the answer, and there are two studies worth knowing before you design it.

The first, by Gagan Bansal and seven co-authors from the University of Washington and Microsoft Research (CHI 2021), tested whether AI explanations improved human-AI teams on three datasets, with an AI whose accuracy was comparable to the people's. They didn't. Instead, "explanations increased the chance that humans will accept the AI's recommendation, regardless of its correctness".

The second, by Zana Buçinca, Maja Barbara Malaya and Krzysztof Gajos (CSCW 2021), tested three ways of forcing people to think: showing the AI's suggestion only when the person asks for it, having them decide first and see the suggestion afterwards, or making them wait thirty seconds. All three reduced overreliance compared with showing the suggestion alongside its explanation, although they didn't eliminate it or significantly improve the team's result. And the designs that reduced it most were the ones people liked least. It was 199 people, a nutrition task and a simulated AI: not a corporate protocol, but it points in a clear direction.

Five design decisions for the exception package follow from that:

1. Why it arrived. The specific rule that fired — "the amount doesn't match purchase order 4471" — not "low confidence". Without a reason, the person has to rediscover the problem, and in a hurry they won't.
2. The source data, not just the summary. The document, the record, the thread. An agent's summary is a conclusion, and someone reviewing a conclusion accepts it more easily than someone reviewing the facts.
3. The agent's proposal, in the right order. For exceptions where the person is the expert, the evidence first and then, on request, what the agent proposes. For high-volume, low-consequence ones, the proposal in plain view. It isn't dogma: it's choosing according to which of Vaccaro's cases that type of exception falls into.
4. What can be done with it. Approve, correct, send back with an instruction or escalate to another level. And the correction captured in a structured way, because it's the most valuable data the process produces.
5. The deadline. How long that case can wait before the process jams or the customer notices, visible to whoever receives it.

Who and when: the queue is a system too

"Exceptions go to the finance team" isn't a path, it's an email address. The path has an owner per exception type, named and with authority to change the decision, because the new-supplier exception and the disputed-amount exception aren't resolved by the same person or with the same judgement.

The deadline comes from the process, not from another team's service-level agreement: if an invoice has to be posted before month-end close or a claim answered within a legal time limit, the exception inherits that clock.

And capacity is calculated, not assumed. Continuing with proposal B from before (400 exceptions a month, our arithmetic): at 15 minutes per exception that's 100 hours a month, a little over half a full-time person. If exceptions are, as Bainbridge said, the hard cases and each takes 25 minutes, it's about 167 hours: a whole person. The business case that promises to free up one role and creates another one for exceptions isn't wrong, but it is incomplete, and it's better for the project to say so before the team finds out.

When the queue grows faster than capacity, the same thing always happens: the reviewer speeds up, and reviewing fast is approving. That's why the metric that best describes the health of the path isn't time in queue but the amendment rate: the share of exceptions that end with the person changing what the agent proposed. If it stays close to zero for months, there are only two explanations, and both are bad: either you're escalating what didn't need escalating, or people are signing without looking.

The case that never reaches the queue

That leaves the bottom-left box. One of the most cited public cases is small in money and large in lessons.

In November 2022, after their grandmother died, Jake Moffatt asked the chatbot on Air Canada's website about bereavement fares. The chatbot told them they could fly now and apply for the reduced fare afterwards, within 90 days of the ticket being issued. The airline's own page said the opposite: the policy didn't apply to requests made after travel. Moffatt flew, asked for the refund and spent three months exchanging emails, until in February 2023 they sent a screenshot of the conversation. Air Canada then admitted the chatbot had provided "misleading words" and said it had "noted the issue so it could update the chatbot".

A year later, British Columbia's Civil Resolution Tribunal found in their favour. According to the decision, Air Canada was in effect suggesting that the chatbot was "a separate legal entity that is responsible for its own actions"; the tribunal called this "a remarkable submission" and wrote that it should be obvious to the airline that "it is responsible for all the information on its website", whether it comes from a static page or a chatbot. The award was CAD 812.02.

The amount is anecdotal. What isn't: the company didn't find the error. The customer brought it to light, and the airline acknowledged it three months later, when they pressed the point with the screenshot. The decision doesn't go into what technology sat behind the chatbot, and for the lesson it doesn't matter. That is the default exception path of any system that doesn't look for its own silent errors: the customer, the auditor or the month-end close. A well-built automated support agent knows how to say "I don't know"; this case shows what happens with what the agent didn't know it didn't know.

There are three tools for finding those cases before the customer does, and none of them is expensive:

• Random sampling of what the system resolved on its own. A fixed sample every week, reviewed by someone who decides before seeing what the agent did: it's the "decide first" design from the Buçinca study, applied to auditing.
• Signals that arrive later. Reopened cases, credit notes, complaints, manual corrections in the ERP to records the agent wrote. Each one is a silent error that has already become visible; you just have to cross-reference them.
• Every error found, turned into a case in the evaluation set. It's the rule we developed in the piece on the golden set: an incident that doesn't end up as a test case is an incident that can happen again without anyone noticing.

On sample size there's a calculation worth doing before celebrating anything (our arithmetic, using a well-known statistical approximation). If you review n cases and find no errors, the true error rate still compatible with that result, at 95% confidence, is roughly 3 divided by n. Fifty clean cases are still compatible with a 6% silent error rate; to rule out 1% you need about 300. Sample size isn't a question of effort: it's deciding which error rate you want to be able to rule out.

The law already distinguishes between reviewing and signing

None of the above is just good practice. Two European texts say it in words worth reading in full.

Article 14 of the EU AI Act requires that high-risk systems can be effectively overseen, and spells out what that means: among other things, that the people overseeing them are able "to remain aware of the possible tendency of automatically relying or over-relying on the output produced by a high-risk AI system (automation bias)" and "to decide, in any particular situation, not to use the high-risk AI system or to otherwise disregard, override or reverse the output of the high-risk AI system". It only binds high-risk systems, and its timetable has moved; we went through it when writing about insurance claims.

The other text is older and more direct. The Article 29 Working Party guidelines — the predecessor of the European Data Protection Board — interpreting Article 22 of the GDPR state that "the controller cannot avoid the Article 22 provisions by fabricating human involvement", and that to count as such, oversight has to be "meaningful, rather than just a token gesture", carried out "by someone who has the authority and competence to change the decision", who should "consider all the relevant data". It concerns decisions about people with legal or similarly significant effects — a claim, a loan, a hire — not a supplier's invoice.

Even if your process falls into neither case, those sentences are the best available definition of an exception path that works: someone with authority to change the decision, with all the data in front of them, aware of the tendency to accept what the machine proposes and with a real ability to discard it. A queue where one hundred per cent gets approved in twenty seconds meets none of those four conditions, and the amendment rate is precisely the evidence that oversight isn't a token gesture. How to build that in from the design stage is covered in compliance-first: agents that pass an audit.

Two reports for the same committee

All of the above ends up changing one very concrete thing: what gets shown when someone asks whether the automation works.

The report on the right is uglier and longer, and it has an advantage the one on the left will never have: it survives the first awkward question. When a customer complains, when an auditor asks or when someone on the committee wants to widen the scope, the honest answer isn't a percentage. It's: we know what it resolves on its own and with what error, what reaches us, who receives it, how long it takes, how often the person changes what the system proposed, and how many errors slip past us — and how we know.

If the last two are missing, the accuracy rate is an opinion with decimals.

Frequently asked questions

What is the exception path of an automated process?

It's what happens to the cases the system shouldn't resolve on its own: what detects them, who they reach, with what information, within what deadline, and what that person can do with them. It includes the part almost nobody designs: how you find the cases the agent got wrong without flagging them. A process without a designed exception path still has one, just by default: the customer, the auditor or the month-end close.

What share of cases should go to a person?

There's no right number: the more you escalate, the less the system gets wrong in what it resolves on its own, and the literature on selective classification describes this as a risk-coverage curve. The point on the curve is chosen with two figures that aren't in any accuracy rate: what a silent error costs in that process and how many exceptions a month your team can handle properly.

Can I use the model's own confidence to decide what gets escalated?

As one signal among several, not as the only one. In the GPT-4 technical report, post-training significantly worsened the model's calibration: the calibration error went from 0.007 to 0.074 on a subset of MMLU. And even when well calibrated, confidence measures the model's doubt, not the consequence of the case. The more reliable triggers are the ones the model doesn't control: business rules, amount thresholds, disagreement between sources and new kinds of case.

Doesn't having a person review what the AI proposes guarantee quality?

Not on its own. A meta-analysis of 106 experiments published in Nature Human Behaviour in 2024 found that, on average, human-AI combinations perform worse than the better of the two alone, and that they gain when the person outperforms the model on that task. On top of that, AI explanations increase acceptance of its proposal, whether it's right or not. Review works when it reaches someone who knows more than the model, with the source data and with time to think.

How do you measure the error nobody sees?

With random sampling of what the system resolved on its own, reviewed by someone who decides before seeing the agent's answer; with the signals that arrive later, such as reopened cases, credit notes or complaints; and by turning every error found into an evaluation case. Size matters: 50 reviewed cases with no errors are still compatible with a true rate of up to 6%, and ruling out 1% takes about 300.

Is human oversight mandatory?

Article 14 of the EU AI Act requires it for high-risk systems, and specifically asks that the people overseeing them remain aware of automation bias and be able to disregard or reverse the output. When a fully automated decision has legal or similarly significant effects on a person, Article 22 of the GDPR also applies, and its guidelines make clear that a token gesture doesn't count as human involvement. Even if your process falls into neither case, it's the best available definition of real oversight.

Sources cited

• Vaccaro, M., Almaatouq, A. and Malone, T. — When combinations of humans and AI are useful: A systematic review and meta-analysis (Nature Human Behaviour 8, 2293–2303, published 28 October 2024; preregistered meta-analysis of 106 experiments and 370 effect sizes; Hedges' g = −0.23 against the better of the two; losses in decision tasks; gains when the person outperforms the AI)
• Bainbridge, L. — Ironies of Automation (Automatica 19(6), 775–779, 1983; the tasks the designer cannot automate, loss of skill, the limits of vigilance and the final irony about training)
• Bansal, G., Wu, T., Zhou, J., Fok, R., Nushi, B., Kamar, E., Ribeiro, M. T. and Weld, D. S. — Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance (CHI 2021; explanations increase acceptance of the AI's recommendation, regardless of its correctness)
• Buçinca, Z., Malaya, M. B. and Gajos, K. Z. — To Trust or to Think: Cognitive Forcing Functions Can Reduce Overreliance on AI in AI-assisted Decision-making (CSCW 2021; 199 participants; three designs that reduce overreliance without eliminating it, and the least preferred by users)
• Geifman, Y. and El-Yaniv, R. — Selective Classification for Deep Neural Networks (NeurIPS 2017; the risk-coverage curve and the ImageNet example)
• OpenAI — GPT-4 Technical Report (2023; figure 8: calibration of the pre-trained versus the post-trained model on a subset of MMLU)
• Civil Resolution Tribunal of British Columbia — Moffatt v. Air Canada, 2024 BCCRT 149 (14 February 2024; full text of the decision)
• European Union — AI Act, Article 14 · Human oversight (automation bias and the ability to disregard, override or reverse the output of high-risk systems)
• Article 29 Data Protection Working Party — Guidelines on Automated individual decision-making and Profiling for the purposes of Regulation 2016/679 (WP251rev.01) (adopted in 2017 and last revised on 6 February 2018; meaningful human involvement versus a token gesture)