Decision support
Where should a human approve what your AI agent does?
Gate on the action, not on the model’s confidence. Reversibility and blast radius decide, and confidence scores are not calibrated.
The question
Which of the agent’s actions should a person approve first?
The short answer
Classify by two properties of the action, not by how sure the model claims to be: can it be undone, and how far does it reach. Anything irreversible and wide gets a human gate. Anything reversible and narrow does not need one and is made worse by adding one, because approval fatigue is real and a reviewer who approves five hundred trivial items will approve the dangerous one too. Do not threshold on model confidence — LLM self-reported confidence is not calibrated, so a 0.9 does not mean ninety percent right, and a gate built on it fails exactly when the model is confidently wrong. For a gate to count as oversight rather than ceremony, the reviewer must see the evidence the agent used, be able to tell when the agent was uncertain by some measure other than its own assertion, and have enough time per item for the review to be real.
Written for
- Leaders deciding how much autonomy to grant an agent
- Teams designing an approval step that will not be rubber-stamped
- Anyone who has to defend an automated decision afterwards
Why the answer is that
The instinct is to gate on confidence: let the agent act when it is sure, ask a human when it is not. It is intuitive and it does not work, because the number is not a probability. Language models are poorly calibrated about their own correctness, and the failure mode that costs you money is precisely the confident wrong answer — which sails through a confidence gate by construction. If you build one thing differently after reading this, make it that.
What works is boring: classify actions. Reversibility first — can you undo this with a compensating action, and how expensive is that. Sending an email is irreversible in the sense that matters. Updating a database row is reversible if you kept the old value and nobody has read it yet. Blast radius second — does this touch one record or ten thousand, one customer or all of them. Those two properties are known at design time, they do not vary per request, and they are properties of your system rather than guesses about the model.
Then be disciplined about not gating the cheap stuff. Approval fatigue is the failure mode that quietly defeats most of these designs: a reviewer facing a queue of mostly-fine items becomes an approve button with a person attached, and by the time the genuinely dangerous item arrives, the habit is already formed. Every unnecessary gate makes the necessary ones less effective. If you find yourself approving nearly everything, that gate is not protecting you and should be removed or batched.
And design what the reviewer sees, which is the part that gets skipped. A screen showing the agent’s proposed action and two buttons produces rubber-stamping. A screen showing the proposal, the evidence it was drawn from, the specific fields that will change, and a signal of genuine uncertainty — a retrieval that returned nothing relevant, a tool that errored and was retried, an argument that failed validation once — produces review. The difference is not the reviewer’s diligence, it is the information architecture.
Action classes and the gate each one needs
Place each of your agent’s actions in a row. The gate follows from the row, not from a confidence score.
| Action class | Example | Gate |
|---|---|---|
| Reversible, narrow | Draft a reply, tag a ticket, add an internal note | None. Log it and move on |
| Reversible, wide | Re-tag a thousand records | No per-item gate; a dry run plus a bulk undo |
| Irreversible, narrow, low value | Post an internal comment | None, but keep it in the audit trail |
| Irreversible, narrow, customer-visible | Send an email, publish a message | Human approves the exact text, or a hold-and-cancel window |
| Irreversible, moves money | Issue a refund, apply a credit | Human approval, always, with the amount and reason shown |
| Irreversible, wide | Bulk delete, mass notification, schema change | Human approval plus a second pair of eyes |
| Affects a person’s rights or opportunities | Screen a candidate, decline an application | Meaningful human review, and a bias assessment on the pipeline |
| Externally observable and permanent | Public post, filing, contractual commitment | Human approval and a named accountable person |
Two rows deliberately say “no gate”. That is not laxity — it is what keeps the reviewer’s attention available for the rows that matter. A gate on everything is functionally a gate on nothing.
Does your approval step actually count as oversight?
Six checks. If you cannot answer yes to the first four, you have a rubber stamp, and in a regulated context that is worse than no gate — it is evidence you claimed a control you did not have.
- 1The reviewer sees the evidence the agent used, not only its conclusion.
- 2The reviewer sees exactly which fields or values will change.
- 3Genuine uncertainty is surfaced from system signals — empty retrieval, tool errors, failed validation — not from the model’s own confidence claim.
- 4Median review time is long enough for the review to be real, and you measure it.
- 5The reviewer can edit rather than only approve or reject, so a near-miss does not force a rejection.
- 6What was shown to the reviewer is logged alongside their decision, so the oversight can be demonstrated later.
The fourth item is the one that catches teams out. If your logs show a median of four seconds per approval, you have measured the absence of review — and you have written it down.
Get the answer for your own system in four minutes
The free scorecard produces a score out of 100, a breakdown across reliability, cost and observability, and your gaps ranked by production risk with a fix and effort estimate for each. It prints to a PDF you can take into a planning meeting — which is usually more persuasive than an argument.
Run the scorecardIf you are the engineer who found this page: Written to be forwarded. If you are the engineer who found this, send the relevant one up — it is framed for the person who has to approve the work.
If you decide to do the work
The engagements this decision leads to, each with the full method and effort estimates — whether your team runs it or I do.
Questions
Where should a human approve an AI agent’s actions?
Gate on the action’s properties, not the model’s confidence. Two questions decide it: can the action be undone, and how far does it reach.
Irreversible and wide always needs a human. Irreversible and customer-visible — sending an email, issuing a refund — needs a human. Reversible and narrow does not, and adding a gate there makes things worse by consuming the reviewer’s attention.
Should we use the model’s confidence score to decide when to ask a human?
No. Self-reported LLM confidence is not calibrated, so a 0.9 does not mean ninety percent likely correct.
Worse, the failure that costs you money is the confident wrong answer, which passes a confidence gate by definition. Use system signals instead — retrieval returned nothing relevant, a tool errored, an argument failed validation — and gate on the action class regardless.
Why do human approval steps stop working over time?
Approval fatigue. A reviewer facing a queue of mostly-fine items becomes an approve button with a person attached, and the habit is already formed by the time a dangerous item arrives.
The fix is counter-intuitive: remove gates. Every unnecessary approval makes the necessary ones less effective. If you approve nearly everything at a given gate, it is not protecting you.
What does the reviewer need to see for approval to be meaningful?
The proposed action, the evidence it was drawn from, the exact fields that will change, and a signal of real uncertainty. Plus the ability to edit rather than only approve or reject.
And log what was shown alongside the decision. Without that you cannot later demonstrate that oversight happened — and “a human approved it” is a claim you may need to substantiate to an auditor or a regulator.
Want a second opinion on the decision?
Describe your situation and I will tell you what I would do — including when that is "nothing" or "handle it internally". No charge for that, and it is genuinely how a lot of these conversations end.
Other decisions
- What an agent audit should cover
- Questions to ask before launch
- Build or buy: observability
- Metrics worth reporting upward
- Do you need outside help?
- Can an agent touch our internal data?
- Is an agent different from a chatbot?
- Governance, audit logs and compliance
- Do we need our own model?
- Agent cost versus hiring
- Working with the systems you already have