Rotating an agent's key without breaking it

Key rotation is the security task everyone agrees with and nobody schedules, because the honest reason for avoiding it is that rotating a live credential can stop an agent mid-conversation and nobody wants to be the person who did that on a Thursday afternoon.

4 min read

Two keys valid at once, and one of them retiring
Two valid keys for a week turns an outage into a non-event.

The fix is an overlap window, which makes the whole thing boring.

Overlap, then revoke

Never swap a key. Add one, migrate, then remove the old one.

  1. Create a second key for the same identity, scoped identically.
  2. Deploy it wherever the agent reads its configuration.
  3. Watch which key is being used. The old one going quiet is the signal that migration is complete, and it is the step people skip.
  4. Revoke the old key once it has been unused for long enough to cover anything running on a slow schedule.

The window is a judgement call. A week is comfortable for most deployments, and longer is sensible when something runs weekly or monthly and would otherwise be discovered broken at the worst moment.

What makes step three possible is per-key visibility. If you cannot tell which key made a call, rotation becomes guesswork and everyone stops doing it.

Where keys actually leak from

Not from your database. From the ordinary places credentials end up while people work.

SourceWhy it happens
A .env committed once and force-pushed awayGit keeps it, and so do forks and CI caches
CI logsAn echoed variable, printed by a debug step
A message to a colleagueThe fastest way to share it, and permanent
A screenshotTerminal window with the key visible
A model promptConfig pasted into a chat while debugging
A stale environmentA staging deploy nobody has touched in a year

The last one deserves attention because it is invisible: a long-lived key in an environment nobody owns is the most likely thing in your estate to still be valid in three years.

That is also the argument for narrow scopes generally. A key that can send from one identity and read one mailbox has a bounded blast radius, per deciding what each agent is allowed to see.

When one is exposed

Order matters more than speed here, but not by much.

Revoke first. Before investigating, before telling anyone, before working out how it happened. An exposed key is worth nothing revoked, and the investigation is easier without the clock running.

Then look at what it did. Sends made, mailboxes read, and whether anything happened outside your own systems' patterns. This is what turns "a key leaked" into "a key leaked and nothing used it", which is a materially different conversation.

Then remove the copy. Rewriting history, purging the log, deleting the message. Necessary, but it is cleanup rather than containment, and doing it first is the common mistake.

If anything was sent, that is an incident with its own process, per when the agent sends something it should not have.

Revoke, investigate, clean up, in that order
Revoking first makes everything after it easier.

One key per thing

The rule that makes rotation cheap: a distinct key per agent, per environment, and per deployment target.

Shared keys make every rotation a coordination exercise across teams, which is why shared keys never get rotated. Distinct keys mean rotating one affects one thing, and revoking one during an incident does not take the rest of your estate down with it.

The same logic applies to running several agents in one workspace: separate identities and separate keys, so a compromise is bounded to one agent's scope, per running several email agents in one workspace.

On a schedule, or on an event

Two defensible policies, and the choice is less important than picking one.

Scheduled, quarterly or half-yearly, has the advantage that the procedure stays exercised. A rotation you have done four times is one you can do calmly during an incident.

Event-driven, on staff changes, on suspicion, and on any environment being decommissioned, catches the cases that actually matter but leaves the procedure unpractised.

Most teams should do both, with a lightweight schedule mainly to keep the muscle memory. The measurable outcome is the age of your oldest live key, and if you do not know that number, that is the first thing to find out.

The agent should never hold the only copy

A small design point with large consequences. Credentials belong in your configuration or secret store, injected at runtime, so revocation is something you do centrally rather than something requiring the agent to cooperate.

An agent that stores its own key somewhere you cannot see is one you cannot cut off, and being unable to cut something off is the definition of not being in control of it.

Questions

How do I rotate an agent's API key without downtime?
Create a second key, deploy it, confirm the old one has gone quiet, then revoke it. A week of overlap covers most deployments; longer if anything runs on a slow schedule.
How often should keys be rotated?
Quarterly or half-yearly is a reasonable schedule, plus immediately on staff changes, suspicion of exposure, or decommissioning an environment. The number to track is the age of your oldest live key.
What do I do first if a key leaks?
Revoke it. Then investigate what it did, then remove the exposed copy. Cleaning up before revoking is the common and costly mistake.
Should agents share a key?
No. One key per agent, per environment, per deployment target, so rotation and revocation affect exactly one thing.
Where do keys usually leak from?
Committed env files, CI logs, messages to colleagues, screenshots, config pasted into a chat, and forgotten environments still holding a valid key.
Should the agent store its own credentials?
No. Inject them from your configuration or secret store at runtime, so revocation is central and does not depend on the agent cooperating.

Give your agent an address it can answer from.

Create an inbox