Designing approval that a human will still read in week three

Every team building an email agent adds an approval step, and most of them stop reading it within a fortnight. The queue fills, the reviewer starts approving in batches, and the control that was supposed to catch the one bad message becomes a formality that catches nothing while slowing everything down.

5 min read

A review queue with a small number of held messages and a clear next action
A queue worth reading is a short one. Length is the failure mode, not the feature.

The failure is volume, not design

An approval gate works when the reviewer can hold the whole queue in their head and knows what they are looking for. It fails when either stops being true, and both stop at roughly the same point: somewhere around a dozen items a day, or the moment the reviewer cannot say in one sentence what would make them reject something.

So the first design question is not what the queue looks like. It is what should never enter it.

What deserves a person

Hold for reviewWhy
First contact with a domain you have never emailedThe one genuinely irreversible category
Anything involving money above a thresholdUnbounded downside, and a limit is easy to state
Legal, press, security, or data deletionProcess requirements a reply can prejudice
A message the agent itself flagged as unusualCheap signal, and it costs nothing to honour
The first N messages of a newly deployed agentTime-boxed, and it replaces guesswork with evidence

Everything else should flow. Replies inside existing threads, in particular, are where the agent earns its keep, and holding them removes the speed that made the project worth doing.

The last row deserves emphasis because it is the one people skip: approval should usually be a phase, not a permanent state. Hold the first fifty messages, read them, fix what is wrong, then release the gate for that category and keep it only where the table above still applies.

Design the review, not just the queue

A reviewer needs three things on screen, and giving them fewer is why reviews get slow and then get skipped.

The thread, not the message. Judging a reply without the conversation above it is guesswork, and it is exactly the mistake you are trying to catch the agent making.

What the agent believed. The action it chose, the source it grounded the answer in, and its own flag if it raised one. A reviewer who has to reconstruct the reasoning will not.

One-click outcomes. Approve, edit and send, reject with a reason. The edit path matters more than it sounds: most held messages are nearly right, and forcing a reviewer to choose between shipping a flawed message and writing a new one from scratch produces bad approvals.

Capture the reject reason as structured data rather than free text where you can. Reject reasons are the single best input to the next prompt revision, and they are the thing that turns a queue into evaluation data.

A review card showing the thread, the agent's reasoning, and three actions
Thread, reasoning, three buttons. Anything more and the review takes longer than writing the reply.

Decide what happens when nobody looks

Every queue eventually goes unattended: a holiday, a weekend, a reviewer who left. Decide in advance which way each category fails.

  • Fail closed for anything irreversible. An unreviewed cold send expires unsent, and the agent tells the requester it could not proceed.
  • Fail open for things where silence is worse than an imperfect message, which in practice means acknowledgements. Even here, prefer sending a short "a colleague will pick this up" over sending the full held reply.
  • Escalate on age. Anything older than your promised response time should page someone rather than sitting.

An unattended queue that silently holds a customer's question for four days does more damage than a slightly imperfect reply would have.

Tell the sender something

The person waiting does not know a review is happening. If a message is held for more than a few minutes and someone is waiting on an answer, send a short acknowledgement immediately and the real reply after review. Two messages beat one silent gap, and the acknowledgement is exactly the sort of low-risk send that should never enter the queue itself.

Where approval is not the right tool

Approval is a review of an action. Some risks are better handled by removing the action.

If the agent should never issue refunds above a limit, do not review those sends: make the refund tool refuse. If it should never contact people outside a known set, run the identity in whitelist mode rather than approving each stranger. If it should never touch bank detail changes, make that an escalation with no send path at all. Structural controls do not get tired, and the reasoning is worked through in keeping an autonomous agent from becoming a spam problem and prompt injection by email.

Approval is for judgement calls that genuinely need a person's judgement. Everything you can specify, specify.

Removing the gate

Removing approval should feel like a decision with evidence behind it, not a resignation. The signals that it is time, per category:

  1. Fifty or more reviewed messages in that category.
  2. An approval rate above roughly ninety-five percent, with the rejections being taste rather than error.
  3. No correction needed after sending on any approved message for two weeks.
  4. A rollback that takes one config change, so re-adding the gate is cheap.

Then remove it for that category only, keep watching the correction rate, and leave the weekly read of fifteen threads in place. That last habit is what replaces the gate, and it is the one worth keeping permanently.

Questions

Should every AI agent email be approved by a human?
No. Approval on everything becomes rubber stamping within a fortnight and removes the speed that justified the agent. Hold first contact with new domains, money above a threshold, sensitive categories, and the first batch from a new agent.
How big can an approval queue get before it stops working?
Roughly a dozen items a day, or the point at which the reviewer can no longer say in one sentence what they are checking for. Past that, approvals become batch clicks.
What should the reviewer see?
The whole thread, what the agent believed and grounded its answer in, and three actions: approve, edit and send, reject with a reason. Reject reasons are the best input to the next prompt revision.
What happens to messages nobody reviews?
Decide per category. Irreversible things should expire unsent and tell the requester. Anything where silence is worse should fall back to a short acknowledgement, and anything past your promised response time should page someone.
When can I remove the approval step?
Per category, after about fifty reviewed messages with a high approval rate and no post-send corrections for two weeks, provided re-adding the gate is one config change.
Is approval better than a hard limit?
No. Prefer a structural control wherever the rule can be stated: a refund tool that refuses above a limit, or whitelist mode instead of approving each stranger. Reserve approval for genuine judgement calls.

Give your agent an address it can answer from.

Create an inbox