The key-rotation playbook: provider by provider, downtime-free

For anyone rotating credentials on purpose rather than during an incident: one invariant, five ordered steps, provider-specific procedures cited to current documentation, and the verification commands that prove a rotation finished.

Rotation has a public-relations problem. It gets described as risky surgery on live systems, so teams postpone it until an incident forces it under the worst conditions possible — pressure, partial information, and an audience. The reality underneath nearly every major platform is friendlier by design: providers issue multiple simultaneous credentials precisely so a successor can run alongside its predecessor while consumers migrate gradually. Done in that order, rotation touches nothing user-visible. Done in any other order, it becomes self-inflicted downtime or, worse, theater — a new key created, old key left valid, ticket closed.

This article is built to be the tab you keep open mid-rotation. It states the ordering invariant once, walks the five steps that apply everywhere, then goes provider-by-provider through the majors with citations to current official documentation, because dashboard paths drift and stale procedure guides cause exactly the outages they promise to prevent. It pairs with the incident-response runbook for confirmed-compromise cases and the fix guides for repairing whatever leak path made rotation necessary. Every example credential below is a synthetic placeholder (FAKE… markers included).

The invariant, and its one exception

Default — overlap: consumers finish migrating before the old credential dies.

create successor → migrate consumers gradually → verify old-key silence → retire predecessor → record

Exception — suspected active compromise: when there is evidence, or honest inability to rule out, that someone else holds the credential, overlap becomes indefensible. A known-or-suspected-compromised key stays fully functional throughout every day of migration. Revoke first, absorb the brief breakage, restore service under pressure:

revoke immediately → deploy replacement hot → audit the exposure window afterward

The decision rule compresses cleanly: any suspicion of third-party use means revoke-first. Suspicion includes "we can't tell." Signals that trigger it: the credential appeared anywhere publicly reachable (a client bundle counts — published output is collected continuously by automated scanners), provider logs show unfamiliar sources, exposure duration exceeds hours, or your inventory can't establish who saw the value. Everything below assumes the default overlap path except where marked.

Step zero: inventory consumers before touching anything

The most common rotation failure isn't ordering — it's discovering missed consumers after revocation, when they announce themselves as outages. Before creating anything, enumerate where the old value lives and what reads it:

# Source references (value itself should NOT be here — names are enough):
rg -n "STRIPE_SECRET_KEY|STRIPE_KEY" --hidden -g '!.git'

# Configuration and infrastructure definitions:
rg -n -i "secret_key|api_key|token" .github/ docker-compose* k8s/ infra/ 2>/dev/null

# Local stores worth remembering (run on every machine that ever touched it):
ls ~/.env* 2>/dev/null; grep -rn "sk_live_FAKE" ~/projects/*/\.env* 2>/dev/null

Then classify consumers by propagation model, because that classification determines how long overlap must last:

Two rows deserve emphasis from incident history patterns generally documented across postmortem culture: scheduled jobs are the canonical silent straggler (weekly reports failing quietly for days), and compiled-in values mean a "server-side" rotation still isn't done until the artifact layer rotates too (why bundles freeze values at build time).

The five steps

1. Create the successor alongside the incumbent. Same scope, same permissions — resist the temptation to fix scoping mid-rotation unless downscoping deliberately (see the closing section; rotation is actually the cheapest moment to narrow permissions, just do it consciously). Nothing breaks; the old key continues serving normally.

2. Canary the successor. One authenticated request with the new key against a harmless endpoint proves validity before anything depends on it:

# Stripe example (synthetic key):
curl -sS -o /dev/null -w "%{http_code}\n" \
  -u "rk_live_FAKE000000000000000000000:" \
  https://api.stripe.com/v1/balance
# Expect 200 before proceeding.

3. Migrate consumers gradually, stragglers-first if ordering matters. Flip scheduled jobs early (their next run may be days away), long-tail services early, user-facing services when comfortable. Watch error rates per flip; a spike identifies the misconfigured consumer while the old key still catches it.

4. Verify genuine silence on the predecessor. Every provider exposes per-credential usage ground truth (table below). Zero traffic over a window longer than your slowest consumer cadence — daily jobs need several quiet days, weekly reports need weeks — is the pass condition. Retiring early converts the next cron run into an outage.

5. Retire, confirm dead, record. Expire/revoke at the provider, verify authenticated requests now fail with the old value, and write it down: what rotated, why, when, which consumers migrated, who approved. Future audits and incidents will read this record; write it for them.

Provider by provider

Stripe

Stripe documents the downtime-free path explicitly: Rotate key generates a replacement immediately while the original remains valid for up to seven days, with official guidance to roll the new key out to a subset of servers first and watch logs before completing (Rotate an API key). Dashboard path: Developers → API keys → ⋯ on the key → Rotate key; choose expiration timing in the dialog (Now deletes immediately — the compromise path). Per-key request logs provide step-four verification via the same menu (view request logs). Two upgrades worth attaching to any rotation here: migrate legacy sk_ usage toward restricted keys scoped to what each integration needs (tier analysis), and rotate whsec_ webhook signing secrets separately, per endpoint (webhooks).

Supabase

Supabase supports multiple secret keys simultaneously: create the replacement in Dashboard → Settings → API Keys, migrate consumers, then delete the compromised predecessor — with the documented caveat that deleting a secret key is irreversible, so step four carries extra weight (API keys). Legacy service_role JWTs should convert to modern sb_secret_ keys during rotation rather than being replaced in kind (migration guide); tier semantics covered in the Supabase deep-dive.

OpenAI and Anthropic

Both platforms allow multiple active keys within a project/workspace, making overlap free: create the successor in the same scope, redeploy consumers, watch the usage dashboards for old-key consumption reaching zero, then delete (OpenAI authentication, Anthropic getting started). Scope separation is the leverage point: rotating a project-scoped LLM key compromises nothing outside that project's budget and files — provided projects were separated in the first place (blast-radius mechanics).

AWS

AWS bakes overlap into the platform: each IAM user holds at most two active access keys, and the documented rotation dance uses both slots (Managing access keys):

# 1. Create the successor (second slot):
aws iam create-access-key --user-name ci-deployer

# 2. Migrate consumers to it (deploys, laptops, CI stores)...

# 3. Deactivate the predecessor — reversible safety position:
aws iam update-access-key --user-name ci-deployer \
  --access-key-id AKIAFAKEEXAMPLE000000 --status Inactive

# 4. After the observation window, delete:
aws iam delete-access-key --user-name ci-deployer \
  --access-key-id AKIAFAKEEXAMPLE000000

The deactivate-then-delete gap is AWS's recovery window; use it. And let every IAM-key rotation double as pressure toward eliminating the class: roles, STS, and OIDC federation remove long-lived pairs entirely (best practices, full analysis).

GitHub

Fine-grained PATs coexist until expiry: mint the replacement with identical repository selection and permissions, migrate consumers (including any machine accounts and credential managers), then let the old token expire or delete it (Managing PATs). Check git remote -v across workspaces for tokens embedded in clone URLs — the classic forgotten consumer (token model analysis).

Slack

Slack documents token rotation endpoints and flow for app credentials (token rotation); workspace apps installed across many teams add coordination lead time — start vendor-facing rotations early since third parties control their own migration clocks.

Verification: trust logs over feelings

Step four's ground truth, per provider:

The AWS command deserves its place in muscle memory — it answers "did we miss a consumer?" directly:

aws iam get-access-key-last-used --access-key-id AKIAFAKEEXAMPLE000000
# {"UserName":"ci-deployer","LastUsedDate":"2026-05-02T11:14:00Z", ...}

A LastUsedDate inside your observation window means someone still holds it: find them before deletion, not after.

Two verification blind spots deserve standing checks, because both hide from provider dashboards:

Scheduled-job enumeration. Cron entries, CI schedules, and workflow triggers reference keys from config files that deploys never touch:

# Where stragglers hide:
crontab -l 2>/dev/null | grep -v '^#'
rg -n "schedule:" .github/workflows/
rg -n "STRIPE_SECRET|OPENAI_API|AWS_ACCESS_KEY" k8s/ cron/ scripts/ --hidden -g '!.git'

Every hit is a consumer that must migrate on the playbook's timeline — and the reason step four's silence window must exceed your slowest schedule rather than your fastest deploy.

Webhook receivers and callbacks. Outbound integrations holding your credentials (vendor dashboards, partner endpoints) verify through their own configuration surfaces: after rotating anything a third party stores, confirm their side actually cut over. Provider logs showing zero traffic from your infrastructure but nonzero from unknown sources reads very differently than zero everywhere — the suspicious version of that finding belongs to the incident runbook; routine rotations just need the confirmation habit.

Making rotation boring

Boring rotations come from scheduling and habituation, not heroics:

• Cadence: quarterly for high-value production keys is a defensible default; tighter where blast radius is large (AWS administrator-equivalents) or monetization is instant (LLM keys).
• Event triggers layered on schedule: team departures, provider advisories, any scanner finding, role changes. Schedule handles hygiene; events handle risk.
• Downscope during every rotation: swapping AdministratorAccess for task-scoped roles, unrestricted sk_ keys for RAKs — each rotation either narrows the standing blast radius or wastes the cheapest opportunity to.
• Rehearse annually: rotate one staging key as a game-day exercise and time the steps. Teams discover during rehearsals that their inventory is stale, their dashboards lack permissions, or their "documented" process has drifted — all cheap discoveries in staging.
• Automate where platforms allow: some providers support scheduling key expiry ahead of rotation (Stripe's scheduled rotation among them); wire what exists, calendar-entry the rest.

The endgame worth naming: rotations get rarer as architectures mature past static credentials — federation, short-lived STS sessions, brokered database access (maturity model). Until then, this playbook executed quarterly beats any document that lives in a wiki untouched.

Rotation runbook template

Copy once per credential; fill during the rotation:

## Rotation — <provider> <key-name> — <date>

Trigger: [scheduled | event: ___]
Mode: [overlap | revoke-first] — reason: ___

Inventory complete:
- [ ] Code references searched (names only)
- [ ] CI/CD stores checked: ___
- [ ] Platform dashboards checked: ___
- [ ] Scheduled jobs enumerated: ___
- [ ] Third parties holding it: ___

Successor created: <date/time> · canary request OK: <time>
Consumers migrated: <list + timestamps>
Predecessor silent from: <date/time> (verified via: ___)
Predecessor retired: <date/time> · dead-key check OK: <time>

Notes / follow-ups: ___

Five minutes of filling prevents the archaeology expedition that otherwise follows every incident touching this key.

Credentials other parties hold

Rotation inventories habitually miss one consumer class: third parties you gave keys to. Vendor integrations, agency contractors, partner platforms — each holds a copy whose migration runs on their schedule, not yours. The playbook additions:

• Inventory the outward shares alongside inward usage: who holds what, since when, under what agreement. This list lives beside the consumer inventory from step zero and ages the same way.
• Rotate vendor-held keys on your cadence anyway, with overlap long enough for their integration cycle — then confirm cutover in writing before retiring predecessors. "They said they updated it" is not step-four verification.
• Prefer platform-managed credentials where offered: hosting-platform managed Stripe keys, marketplace OAuth grants, and similar arrangements move rotation into the platform's contract rather than your ticketing queue (Stripe's managed-key note).
• Watch for the vendor-breach notification path in reverse: when a processor or SaaS discloses their compromise, your keys among their holdings are affected regardless of anything you did — event-driven rotation triggers apply to vendors' incidents, not just yours.

The pattern underneath: every credential has a holder inventory, and holders include organizations as well as machines. Rotation discipline that covers only infrastructure misses the copies most likely to be forgotten precisely because someone else is minding them.

Incident rotation differs from maintenance rotation in tempo and ordering, and conflating them produces both slow incidents and painful maintenance:

Keeping the two procedures separate — different runbooks, different triggers — prevents the common failure of applying maintenance caution during active compromise, or incident urgency to a Tuesday-morning scheduled swap. The incident runbook owns the left column of exceptions; this playbook owns the default world.

FAQ

How long should overlap last? Long enough to cover your slowest consumer plus a verification margin: days for web-only stacks, weeks wherever mobile binaries hold values. Providers cap the free version (Stripe's seven-day grace); beyond caps, manually-created successors extend overlap indefinitely.

We lost track of where a key is used. Rotate anyway? Yes — rotate with imperfect inventory and let failures reveal missed consumers loudly, while overlap keeps them functioning. The reverse order discovers the same consumers via outage tickets instead.

Is scheduled rotation really necessary without a breach? Its value isn't responding to known incidents — it's bounding unknown ones and keeping the procedure rehearsed. Teams that never rotate also never test half their failover assumptions; the first real rotation shouldn't be anyone's first rotation.

What about rotating webhook signing secrets? Same overlap logic where you control signature verification: during migration, accept signatures from both secrets, then drop the old one. Per-endpoint scoping keeps each rotation small (the whsec threat model).

Does rotation replace fixing the leak path? Never — they're different closures. Rotation invalidates the exposed value; path repair stops the next exposure. Incidents that rotate without repair recur on schedule (the runbook's closure section).

---

Rotation cleans up what escapes. Find out what already has: KeyDrift's free scan — no signup — checks what your deployed app serves in minutes.