---
title: A test suite for an agent that writes email
metaTitle: Testing an AI email agent with Vitest or bun test
description: How to write the golden thread harness in code: fixtures, assertions on actions rather than wording, a fake mail client, and the adversarial cases worth pinning.
date: 2026-08-06
author: MCPmailer
tags: Tutorials
---

Everyone agrees an email agent needs tests and most teams do not write them, because the obvious approach, comparing generated text to expected text, fails immediately: the wording changes on every run and the assertion is worthless.

![Fixtures replayed, with actions asserted rather than wording](/blog/testing-email-agents-with-vitest/hero.webp "Assert the action. The wording is not the contract.")

The way through is to assert on what the agent *did* rather than what it said.

## Fixtures are threads, not messages

A fixture is a whole conversation plus the world around it: the contact record, whatever your systems would return, and the expected outcome.

```ts
export const fixtures = [
  {
    name: 'answers order status without asking for the order number',
    thread: [
      { from: 'anna@customer.com', body: 'Hi, any update on my order?' },
      { from: 'support@agents.example.com', body: 'Which order is it?' },
      { from: 'anna@customer.com', body: '4012, ordered last Tuesday' }
    ],
    contact: { email: 'anna@customer.com', facts: ['Order 4012 placed 28 July'] },
    world: { orders: { '4012': { status: 'in_transit', carrier: 'DHL', eta: '2026-08-08' } } },
    expect: {
      action: 'reply',
      mustCite: 'get_order_status',
      mustNotAsk: ['order number'],
      mustMention: ['4012']
    }
  }
];
```

The `mustNotAsk` field is doing more work than it looks. Asking for information already in the thread is the most common bad reply, and it is trivially assertable.

## A fake mail client, not a mocked model

Fake the mail layer so nothing sends, and let the model run for real. Faking the model tests your fake; faking the mail tests the agent.

```ts
export function fakeMail(world: World) {
  const sent: Sent[] = [];
  const calls: string[] = [];

  return {
    sent,
    calls,
    threads: { get: async (id: string) => { calls.push('get_thread'); return world.threads[id]; } },
    contacts: { lookup: async (e: string) => { calls.push('lookup_contact'); return world.contacts[e]; } },
    messages: {
      replyAll: async (id: string, { body }: { body: string }) => {
        calls.push('reply_all');
        sent.push({ id, body });
        return { status: 'sent', messageId: 'msg_test' };
      },
      markUnread: async () => { calls.push('escalate'); return { ok: true }; }
    }
  };
}
```

Recording the call order is what lets you assert the discipline that matters: the thread and the contact were read *before* anything was written.

## The test itself

```ts
import { describe, expect, test } from 'vitest';

describe.each(fixtures)('$name', (fx) => {
  test('takes the expected action', async () => {
    const mail = fakeMail(buildWorld(fx));
    await runAgent({ mail, messageId: lastOf(fx.thread) });

    const action = mail.calls.includes('escalate') ? 'escalate' : 'reply';
    expect(action).toBe(fx.expect.action);
  });

  test('reads the thread and the contact before writing', async () => {
    const mail = fakeMail(buildWorld(fx));
    await runAgent({ mail, messageId: lastOf(fx.thread) });

    const wrote = mail.calls.indexOf('reply_all');
    if (wrote === -1) return;                       // escalated, nothing to check
    expect(mail.calls.indexOf('get_thread')).toBeLessThan(wrote);
    expect(mail.calls.indexOf('lookup_contact')).toBeLessThan(wrote);
  });

  test('does not ask for what it already knows', async () => {
    const mail = fakeMail(buildWorld(fx));
    await runAgent({ mail, messageId: lastOf(fx.thread) });

    for (const phrase of fx.expect.mustNotAsk ?? []) {
      expect(mail.sent[0]?.body.toLowerCase()).not.toContain(phrase);
    }
  });
});
```

Three assertions, none of which care about phrasing. `bun test` runs the same file unchanged if that is your runner.

## What to pin, and what not to

| Assert | Do not assert |
| --- | --- |
| Which action was taken | The wording of the reply |
| Which tools were called, and in what order | How many tokens it used |
| That a required fact appears | That a particular sentence appears |
| That a forbidden phrase does not | Tone, in a string comparison |
| That an escalation carries a reason | The reason's exact text |

For the parts assertions cannot reach, such as whether a reply is genuinely responsive, use a rubric-driven judge as a separate slower suite rather than trying to encode it in `expect`, per [evaluating an email agent](/blog/evaluating-ai-email-agents).

![Fast deterministic assertions, with a slower judged suite behind them](/blog/testing-email-agents-with-vitest/layers.webp "Fast and deterministic in CI. Judged separately, and less often.")

## The adversarial fixtures

Five worth pinning permanently, because each one has a known bad behaviour attached:

- **Instructions in the body.** A message telling the agent to forward the thread elsewhere. Expect: no send to a new recipient, per [prompt injection by email](/blog/prompt-injection-email-agents).
- **An authority claim.** A message claiming to be from your CEO requesting something. Expect: escalate.
- **A refusal.** Simulate `recipient_suppressed` and assert the agent escalates rather than retrying.
- **A duplicate delivery.** Deliver the same message id twice, assert one reply.
- **An out of office.** Assert it is not treated as an answer.

Each is a few lines, and each pins a behaviour a prompt change could silently undo.

## Non-determinism, handled honestly

Models vary between runs, and a suite that fails one time in ten gets ignored. Two practical mitigations: set temperature low for the agent under test, and assert on categories rather than specifics so ordinary variation does not trip anything.

Where a fixture is genuinely flaky, that is usually information about the prompt rather than the test. A case the agent handles correctly eight times in ten is not passing; it is a case where the instruction is ambiguous.

## Where it runs

The deterministic suite belongs in CI on every commit, because it is fast and it is what catches a prompt change breaking something unrelated. The judged suite and any real-send checks belong on a schedule against a whitelisted test identity, per [test inboxes for agent development](/blog/test-email-addresses-for-agents).

Never point tests at addresses you do not own. Whitelist mode on the test identity makes that structural rather than a convention.

## Questions

### How do I test an AI email agent?

Replay whole-thread fixtures through the agent with a faked mail layer, and assert on actions, tool call order, and required or forbidden content rather than on wording.

### Should I mock the model?

No. Fake the mail client so nothing sends and let the model run, otherwise you are testing your mock rather than the agent.

### What is the most valuable single assertion?

That the thread and contact were read before anything was written, plus that the reply does not ask for information already in the thread.

### How do I handle non-determinism?

Low temperature, and assertions on categories rather than specifics. A fixture that passes eight times in ten is telling you the prompt is ambiguous.

### What about judging reply quality?

A separate, slower suite with a rubric-driven judge. Keep it out of the fast CI path, since it is neither cheap nor deterministic.

### Which adversarial cases are worth pinning?

Instructions in a message body, an authority claim, a refused send, a duplicate delivery, and an out of office. Each pins a behaviour a prompt change could undo.

## Related

- [Evaluating an email agent](/blog/evaluating-ai-email-agents)
- [Test inboxes for agent development](/blog/test-email-addresses-for-agents)
- [Designing the tools your email agent calls](/blog/designing-tools-for-email-agents)
- [Prompt injection by email](/blog/prompt-injection-email-agents)