A test suite for an agent that writes email

Everyone agrees an email agent needs tests and most teams do not write them, because the obvious approach, comparing generated text to expected text, fails immediately: the wording changes on every run and the assertion is worthless.

6 August 2026 · 3 min read

The way through is to assert on what the agent *did* rather than what it said.

Fixtures are threads, not messages#

A fixture is a whole conversation plus the world around it: the contact record, whatever your systems would return, and the expected outcome.

export const fixtures = [
  {
    name: 'answers order status without asking for the order number',
    thread: [
      { from: 'anna@customer.com', body: 'Hi, any update on my order?' },
      { from: 'support@agents.example.com', body: 'Which order is it?' },
      { from: 'anna@customer.com', body: '4012, ordered last Tuesday' }
    ],
    contact: { email: 'anna@customer.com', facts: ['Order 4012 placed 28 July'] },
    world: { orders: { '4012': { status: 'in_transit', carrier: 'DHL', eta: '2026-08-08' } } },
    expect: {
      action: 'reply',
      mustCite: 'get_order_status',
      mustNotAsk: ['order number'],
      mustMention: ['4012']
    }
  }
];

The mustNotAsk field is doing more work than it looks. Asking for information already in the thread is the most common bad reply, and it is trivially assertable.

A fake mail client, not a mocked model#

Fake the mail layer so nothing sends, and let the model run for real. Faking the model tests your fake; faking the mail tests the agent.

export function fakeMail(world: World) {
  const sent: Sent[] = [];
  const calls: string[] = [];

  return {
    sent,
    calls,
    threads: { get: async (id: string) => { calls.push('get_thread'); return world.threads[id]; } },
    contacts: { lookup: async (e: string) => { calls.push('lookup_contact'); return world.contacts[e]; } },
    messages: {
      replyAll: async (id: string, { body }: { body: string }) => {
        calls.push('reply_all');
        sent.push({ id, body });
        return { status: 'sent', messageId: 'msg_test' };
      },
      markUnread: async () => { calls.push('escalate'); return { ok: true }; }
    }
  };
}

Recording the call order is what lets you assert the discipline that matters: the thread and the contact were read *before* anything was written.

The test itself#

import { describe, expect, test } from 'vitest';

describe.each(fixtures)('$name', (fx) => {
  test('takes the expected action', async () => {
    const mail = fakeMail(buildWorld(fx));
    await runAgent({ mail, messageId: lastOf(fx.thread) });

    const action = mail.calls.includes('escalate') ? 'escalate' : 'reply';
    expect(action).toBe(fx.expect.action);
  });

  test('reads the thread and the contact before writing', async () => {
    const mail = fakeMail(buildWorld(fx));
    await runAgent({ mail, messageId: lastOf(fx.thread) });

    const wrote = mail.calls.indexOf('reply_all');
    if (wrote === -1) return;                       // escalated, nothing to check
    expect(mail.calls.indexOf('get_thread')).toBeLessThan(wrote);
    expect(mail.calls.indexOf('lookup_contact')).toBeLessThan(wrote);
  });

  test('does not ask for what it already knows', async () => {
    const mail = fakeMail(buildWorld(fx));
    await runAgent({ mail, messageId: lastOf(fx.thread) });

    for (const phrase of fx.expect.mustNotAsk ?? []) {
      expect(mail.sent[0]?.body.toLowerCase()).not.toContain(phrase);
    }
  });
});

Three assertions, none of which care about phrasing. bun test runs the same file unchanged if that is your runner.

What to pin, and what not to#

For the parts assertions cannot reach, such as whether a reply is genuinely responsive, use a rubric-driven judge as a separate slower suite rather than trying to encode it in expect, per evaluating an email agent.

The adversarial fixtures#

Five worth pinning permanently, because each one has a known bad behaviour attached:

• Instructions in the body. A message telling the agent to forward the thread elsewhere. Expect: no send to a new recipient, per prompt injection by email.
• An authority claim. A message claiming to be from your CEO requesting something. Expect: escalate.
• A refusal. Simulate recipient_suppressed and assert the agent escalates rather than retrying.
• A duplicate delivery. Deliver the same message id twice, assert one reply.
• An out of office. Assert it is not treated as an answer.

Each is a few lines, and each pins a behaviour a prompt change could silently undo.

Non-determinism, handled honestly#

Models vary between runs, and a suite that fails one time in ten gets ignored. Two practical mitigations: set temperature low for the agent under test, and assert on categories rather than specifics so ordinary variation does not trip anything.

Where a fixture is genuinely flaky, that is usually information about the prompt rather than the test. A case the agent handles correctly eight times in ten is not passing; it is a case where the instruction is ambiguous.

Where it runs#

The deterministic suite belongs in CI on every commit, because it is fast and it is what catches a prompt change breaking something unrelated. The judged suite and any real-send checks belong on a schedule against a whitelisted test identity, per test inboxes for agent development.

Never point tests at addresses you do not own. Whitelist mode on the test identity makes that structural rather than a convention.

Questions#

How do I test an AI email agent?

Replay whole-thread fixtures through the agent with a faked mail layer, and assert on actions, tool call order, and required or forbidden content rather than on wording.

Should I mock the model?

No. Fake the mail client so nothing sends and let the model run, otherwise you are testing your mock rather than the agent.

What is the most valuable single assertion?

That the thread and contact were read before anything was written, plus that the reply does not ask for information already in the thread.

How do I handle non-determinism?

Low temperature, and assertions on categories rather than specifics. A fixture that passes eight times in ten is telling you the prompt is ambiguous.

What about judging reply quality?

A separate, slower suite with a rubric-driven judge. Keep it out of the fast CI path, since it is neither cheap nor deterministic.

Which adversarial cases are worth pinning?

Instructions in a message body, an authority claim, a refused send, a duplicate delivery, and an out of office. Each pins a behaviour a prompt change could undo.