How to tell whether your email agent is any good
Every email agent looks good in the demo, because the demo is one message with an obvious answer. The interesting question is what happens on message four hundred, in a thread with a confused customer and a half-remembered order. This is how to measure that, and how to keep measuring it when someone changes the prompt on a Friday.
6 min read
Deflection rate is a bad primary metric
It counts the threads a human did not touch, which rewards exactly the failure you most want to avoid: an agent that answers confidently and wrongly, and closes the thread. Deflection goes up, satisfaction goes down, and the number tells you nothing until the complaints arrive.
Use these instead. Each is cheap to compute from the audit trail, and each maps to a specific failure.
| Metric | Definition | The failure it catches |
|---|---|---|
| Resolution without reopen | Threads closed by the agent that the sender did not reopen within seven days | Confident wrong answers |
| Correction rate | Threads where a human had to follow up and fix an agent reply | Grounding that is too loose |
| Escalation precision | Of escalated threads, the share that genuinely needed a person | Rules too tight, or an agent that dumps |
| Escalation recall | Of threads that needed a person, the share the agent escalated | Rules too loose, the expensive direction |
| Turns to resolution | Median messages exchanged before the thread closes | Replies that answer beside the question |
| Time to first response | Split by agent-handled and human-handled | Routing problems masquerading as model problems |
| Repeat-question rate | Threads where the sender had to restate something already provided | Missing contact memory |
The two that most teams do not track and should are correction rate and escalation recall. They are the only ones that measure the damage rather than the throughput.
Golden threads: a test suite for conversations
Metrics tell you something regressed. A test suite tells you what, before it ships. Build one out of real conversations, anonymised, from your own inbox.
Twenty to fifty threads is plenty, chosen for coverage rather than volume:
- Ten straightforward cases the agent must get right every time.
- Five that require thread context, where the last message alone is ambiguous.
- Five that require contact memory, where the answer depends on something recorded earlier.
- Five that must escalate: money above the limit, legal, an angry sender, a request the docs do not cover.
- Five adversarial: instructions in the body, a request to send a summary elsewhere, an authority claim, invisible text, a hostile link. See prompt injection by email.
- Five ugly ones: a forwarded chain, a top-posted reply, an out of office, a one-word reply, a message with an attachment and no text.
Store each as the thread plus the expected outcome, where the outcome is an action rather than a string: reply citing source X, escalate, wait, do nothing. Comparing actions is stable; comparing exact wording is not.
{
"id": "order-status-after-two-turns",
"thread": ["...", "...", "..."],
"contact": { "known_facts": ["order 4012 shipped 12 July"] },
"expect": { "action": "reply", "must_cite": "order_lookup", "must_not_ask": ["order number"] }
}Grading without pretending to be objective
Three grading methods, used together, cost less than they sound.
Assertions. Cheap, deterministic, and where most of your value is: did it escalate, did it call the lookup tool, did it avoid asking for something already known, did it stay under a length, did it avoid a forbidden phrase. Write these first.
A model as judge, with a rubric. For the parts assertions cannot reach, such as whether the answer is responsive and correctly grounded. Give the judge the thread, the reply, and the source material, and ask for a verdict per criterion rather than a score out of ten. Judges are consistent about criteria and vague about numbers.
A human sample. Fifteen threads read end to end, weekly at first. This is the only method that finds the failure nobody encoded, and it is where "technically correct and completely unhelpful" gets caught.
Run it against a test identity, not against customers
The mechanics matter here. Create a separate agent identity for evaluation, put it in whitelist mode so nothing external can reach it, and replay threads into it. Every send is scoped to that identity and shows up in its own audit trail, so a bad run is contained and readable. Identity scoping and mail rules are covered in identities, and the reasons for one identity per agent are in agent inbox or shared mailbox.
Regression before deploy, then, is: replay the suite against the candidate prompt, diff the actions against the previous run, and read the diffs. Not "does it pass" but "what changed and do I agree". A prompt change that alters three unrelated behaviours is telling you something.
What to watch in the first month of live traffic
- Week one: read every thread. Not a sample, all of them. This is the highest-value week of the project and it is over quickly.
- Week two: fix the systematic failures in the prompt and the expensive ones in tooling, and add each to the golden set.
- Week three: start tracking correction rate and escalation precision as numbers rather than impressions, and wire up the alerts in what to monitor in production.
- Week four: decide what else to hand over, based on where escalation recall is strong.
The prompt-level fixes usually cluster in a few places, and the sections that produce them are annotated in the system prompt for an email agent.
The signals that mean stop
Some results are not a tuning problem.
- Correction rate above a few percent that does not fall after two rounds of fixes. The agent is answering questions it cannot ground.
- Escalation recall dropping as volume grows. The rules are being outrun by variety.
- Complaint or bounce rates moving at all. Stop sending and read deliverability for agent senders.
- Turns to resolution rising. The agent is producing replies that are responsive in form and useless in substance.
How the team around the agent changes as the easy half disappears is in what your team does once the agent handles the easy half.
Rolling back to human handling for a category is a normal, reversible decision, and treating it that way is what makes the whole project safe to attempt.
Questions
- What metrics matter for an AI email agent?
- Resolution without reopen, correction rate, escalation precision and recall, turns to resolution, time to first response, and repeat-question rate. Deflection rate flatters an agent that answers wrongly and closes the thread.
- How do I test an email agent before customers see it?
- Build twenty to fifty golden threads from real, anonymised conversations, including adversarial and messy ones, and replay them against a whitelisted test identity. Grade with assertions first, a rubric-driven judge second, and a human sample third.
- How many threads should a human read?
- Every thread in week one, then about fifteen a week. Human review is the only method that finds the failures nobody thought to encode.
- Should I compare exact reply wording between runs?
- No. Compare actions and assertions: did it escalate, did it cite the right source, did it avoid asking for known information. Wording changes constantly without meaning anything.
- What does a good correction rate look like?
- Low single digits at worst, falling over the first month. Persistently higher means the agent is producing answers it cannot ground, and grounding rules or tooling need to change rather than tone.
- Can I run evaluations against production mail?
- Run measurement against production and experiments against a test identity. Replaying customer threads into a live agent risks sending a real message to a real person, and the isolation setup is in test inboxes for agent development.
Give your agent an address it can answer from.
Create an inbox