Secret-leak incident response: the first hour and the first day

For whoever discovers a leaked key — likely you, alone, possibly at night: a runbook with explicit decision points, grounded in each provider's documented tooling, and honest about what response cannot undo.

Incident guides usually fail practitioners in two directions: too abstract ("assess impact") to act on at 2 AM, or too theatrical to take seriously. This runbook takes a middle path — concrete commands and decision rules, ordered by what actually limits damage, with the irreversibility question addressed head-on because pretending otherwise causes worse decisions.

Scope: a credential has been found somewhere it shouldn't be — a public repository, a client bundle, a log, a chat message. The companion pieces supply context this article assumes: why git deletion doesn't remediate, the rotation procedures themselves, and provider-specific fix guides. Here it's sequence and judgment.

All example keys are synthetic placeholders.

The clock, stated correctly

Two timing facts shape everything:

1. Exposure began at leak-time, not discovery-time. A key found today in a bundle may have served every visitor for months. Response urgency derives from the minimum plausible exposure duration, not from how recently you noticed.
2. Different losses run at different speeds. Metered spend (LLM keys, compute keys) accrues continuously until revocation. Data retrieval completes in minutes once begun. Money movement varies by provider's controls. The runbook front-loads actions that stop ongoing losses, then works backward through assessment.

Minute zero through ten: confirm and preserve

Before changing anything, capture state:

• Screenshot or export the finding — the location, the value's masked prefix, surrounding context. Once you rotate and redeploy, the evidence evaporates; incident documentation starts now.
• Note discovery time and earliest-plausible leak time. For bundles: deploy history gives the upper bound. For repos: commit dates. You'll need this window repeatedly.
• Resist deleting the leak source yet. Removing the bundle file or repo file feels productive and destroys information needed for scoping (what else shipped alongside it? which build?). Remediation order comes later in the runbook; containment comes first.

One more preservation rule: don't paste the full secret anywhere new — tickets, chats, notes. Masked prefixes identify the credential fine; full values multiply the incident's cleanup surface.

The central decision: revoke-first or overlap?

Single highest-judgment moment in the runbook, reducible to one rule:

Any possibility that a third party has used or holds the credential → revoke-first. Certainty the exposure was passive only (e.g., key sat in a private-but-wrong place, no public path ever existed) → overlap rotation is defensible.

Operational signals pushing toward revoke-first: the key appeared anywhere publicly reachable (bundle = public, always); provider logs show unfamiliar IPs or request patterns; exposure duration exceeds hours; you cannot rule out automated collection. Given bundles specifically — where scrapers and researchers sweep continuously — treat bundle findings as confirmed-compromise-grade regardless of how quiet the logs look.

Revoke-first consequences, accepted deliberately: legitimate consumers break within seconds-to-minutes. That's survivable for nearly every architecture; the rotation playbook's pressure-deploy sequence restores service quickly. The reverse mistake — granting a live attacker continued validity during leisurely migration — compounds every downstream loss.

Provider-side immediate actions differ slightly (details); the fastest containment moves per class:

The AWS quarantine pattern deserves its snippet since it preserves investigative access better than deletion:

{
  "Version": "2012-10-17",
  "Statement": [
    { "Sid": "Quarantine", "Effect": "Deny", "Action": "*", "Resource": "*" }
  ]
}

Applied via aws iam put-user-policy, this freezes the principal's capabilities while CloudTrail history remains fully queryable — containment without evidence destruction.

Hour one: assess usage from provider logs

Every provider keeps the forensic surface you need; assessment means querying it systematically for the exposure window:

• Stripe: per-key request logs (via Dashboard) — look for unfamiliar IPs/user agents, customer listing pagination unlike your app's patterns, refund/payout operations, newly created webhook endpoints.
• AWS: CloudTrail filtered by accessKeyId across the window — enumerate every event, flagging IAM mutations (CreateUser, CreateAccessKey, policy attachments), STS activity, and access from unexpected geographies/networks.
• GitHub: security log for the token's identity — new PATs, added collaborators, modified workflow files, repo creations.
• LLM providers: usage dashboards reconciled against your own traffic baseline; excess spend quantifies the direct loss; file-API activity flags data retrieval attempts.
• Databases (incl. Supabase): connection/query logs where retained — bulk selects, auth-table access, schema modifications.

Write the timeline as you go: timestamp, event, assessment (legitimate/unfamiliar/undetermined). Undetermined items become follow-up tasks, not blockers — assessment converges over days, containment already happened.

Where each provider's evidence lives, for quick reference during triage:

Two assessment outputs matter disproportionately later: the set of data types reachable (customer PII? files? payment instruments?) drives notification analysis; evidence of actual retrieval versus mere capability shapes severity honestly — capability-without-use is a materially different event, but proving either requires the logs you're capturing now.

Downstream and derived secrets

Leaks rarely stop at one credential. Audit adjacent material sharing roots or channels with the exposed key:

• Webhook signing secrets on endpoints the compromised integration touches — rotate alongside.
• Connection strings and database URLs stored in the same .env, CI store, or bundle — if one value leaked through a channel, assume siblings did; scan the same surfaces for the rest of the inventory.
• JWT/session signing secrets: if the leaked material could sign or derive them, session invalidation (force logout everywhere) joins the checklist.
• **Keys created by the exposed credential** — the AWS persistence pattern above, but generalizable: any provider permitting key creation via API warrants an explicit "did the attacker mint helpers?" pass.

This stage catches the common partial-recovery failure: rotating the discovered key while its siblings sail on untouched.

The irreversibility conversation

Honesty here prevents the worst incident behaviors:

• Data read is data disclosed. Rotating stops further access; it unreads nothing. Customer-data exposure triggers notification-analysis obligations regardless of subsequent cleanup, scoped by what logs show about retrieval.
• Funds movement depends on provider mechanisms and speed. Engage provider fraud/support channels immediately where money moved (Stripe disputes/payout reversal paths, bank rails for payroll-type accounts) — while planning internally as if recovery fails. Optimism isn't a control.
• Abuse under your identity has reputational half-life. Spam sent via your SES, fraudulent charges under your merchant descriptor, ToS-violating model usage billed to your org — cleanup proceeds, but external parties saw what happened. Communication plans belong in day-one work for externally-visible classes.

The counterweight, equally true: teams that respond fast and document thoroughly routinely convert potential catastrophes into footnotes. Speed plus honesty beats perfection plus delay.

Day one: close the loop

By end of day, three closures separate incidents from recurring conditions:

1. Kill the leak path. Root-cause the exposure mechanism — prefix rename, pasted literal, log echo, artifact inclusion — and change the structure that permitted it (server-route refactor, import boundary, masking fix). Fix guides per provider/tooling live at /fix. Rotation without path repair schedules a repeat.
2. Document the incident minimally and durably: timeline, credentials affected and their replacements, data-types-reachable determination, actions taken with timestamps, follow-ups with owners. One page suffices; its future readers include auditors, insurers, and the next responder.
3. Verify externally. Rescan the leaking surface (redeployed bundle clean?), confirm retired credentials reject requests, confirm monitoring now covers the class of leak — deploy-triggered and scheduled scanning turns "we'll watch more closely" into an enforced process, and the free scan provides the immediate post-fix verification.

Verification deserves explicit commands rather than confidence:

# Retired credential rejects:
curl -s -o /dev/null -w "%{http_code}\n" \
  -H "Authorization: Bearer sk_live_FAKE000000000000000000000" \
  https://api.stripe.com/v1/balance        # expect 401

# Served surface clean (any hit = unfinished):
curl -s https://your-app.example | grep -rEo 'sk_live_|sb_secret_|AKIA[0-9A-Z]{16}' -

Optionally, schedule the 30-day review: notification decisions finalized, provider support outcomes recorded, hardening items closed or consciously deferred. The review agenda fits one page:

• Rotation verification — every rotated credential shows zero usage since retirement (playbook step four)
• Path-repair verification — rescan of the leaking surface clean; structural fix present in current code, not just the incident branch
• Follow-up closure — each assessment "undetermined" item resolved or explicitly accepted with rationale
• Control delta — the one process change this incident bought (a gate added, a cap configured, a runbook edit); incidents that produce no control change get asked why
• Record finalization — timeline completed, evidence archived per retention policy, postmortem circulated to whoever will respond next time

Thirty days out, the pressure is gone and the facts are settled — the only window where closure work is both honest and calm. Teams skipping it rediscover the same incident twelve months later wearing new credentials.

Notification analysis: scoping what was reachable

Assessment feeds a second analysis that runs in parallel: determining whether disclosure obligations exist, which means classifying what the credential could reach — not what was necessarily taken. Capability versus evidence stays the organizing distinction:

Two disciplines keep this honest. First, bound claims by evidence: "the credential could read customer records" and "query patterns consistent with bulk export appear on these timestamps" are different sentences with different follow-ups — write both kinds separately and never blend them. Second, preserve the raw exports underlying every conclusion; notification decisions get revisited by counsel, auditors, and future-you, and re-derivable data makes those conversations factual.

Jurisdiction determines whether any of this becomes mandatory disclosure — regulations vary by data type, volume, and geography, and that determination belongs to qualified counsel rather than to engineering judgment. Engineering's job is producing the clean inputs: timelines, capability maps, evidence files. Teams that maintain these as standing practice find the legal conversation takes hours instead of weeks.

Communication skeletons

Incident communication fails through improvisation; skeletons prevent that. Three artifacts cover most cases:

Internal record (start in minute one, update continuously):

INCIDENT — <date> — <credential class>
Discovery: <how, when, by whom>
Exposure window: <first-plausible → containment>
Containment: <action, timestamp>
Assessment status: <in progress | complete> — owner: <name>
Data classes reachable: <from table above>
Evidence preserved: <links to logs/exports>
Follow-ups: <list with owners>

Provider support ticket: account identifiers, credential fingerprint (masked), exposure window, actions already taken (revoked/rotated at specific times), specific requests (fraud review, log retention extension, refund path where applicable). Providers respond materially faster to complete tickets — the skeleton is also a speed optimization.

External notice (only where counsel determines obligations exist): what happened in one sentence without speculation, what data classes were potentially involved per the assessment table, what you did and when, what recipients should do, contact channel. Draft it during day one even when sending isn't yet determined — incidents that end without sending still benefit from having written it, and ones that do send lose nothing to preparation.

The skeletons share one property worth naming: each converts judgment calls made under pressure into fill-in-the-blank work. That conversion is the entire value of runbooks — not predicting every incident, but ensuring the first hour's decisions get documented well enough to defend later (and to improve the process).

Preservation commands, for the common surfaces:

# CloudTrail events for the exposed key (AWS):
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAFAKEEXAMPLE000000 \
  --start-time <window-start> --output json > evidence-cloudtrail.json

# Served bundle snapshot at discovery time:
curl -sL https://your-app.example -o evidence-page.html
grep -oE 'src="[^"]+\.js[^"]*"' evidence-page.html \
  | sed -E 's/src="//; s/"//' | while read p; do
      curl -sL "https://your-app.example$p" -o "evidence-$(basename $p)"
    done

Evidence directories get the same treatment as finding screenshots: timestamps, access limited to responders, retention aligned to whatever legal review might follow. Incidents occasionally become disputes, and the party with orderly artifacts tells a materially better story.

Closure authority deserves explicit assignment: someone specific declares the incident over, against stated criteria — retired credentials verified dead, path repair confirmed by rescan, assessment items dispositioned, monitoring active on the leaking surface. Ambiguous closure produces two familiar failure shapes: incidents that linger forever, consuming attention past usefulness, and incidents closed prematurely that reopen at worse moments. Written criteria plus a named closer resolve both, and the reopen criterion ("new evidence of unauthorized use") belongs in writing too.

FAQ

It's a test/sandbox key. Full runbook? Abbreviated: rotate it, remove the leak path, done — sandbox isolation caps the blast radius. But run the abbreviated version genuinely: sandbox databases increasingly mirror production schemas, and "test" keys guarding realistic fixtures aren't nothing.

We found the leak ourselves, no evidence of use. Notify anyone? Depends on data types reachable and jurisdiction — legal/notification counsel belongs in the loop whenever customer data could have been read, even absent usage evidence. Capability-without-evidence is exactly the case where documented assessment (logs queried, window established) protects you later.

Solo developer, no incident process. Minimum viable version? Everything above scales down cleanly: mask-and-preserve, revoke-first by default, provider-log review for the window, fix the path, write five sentences documenting it. The five sentences feel optional and prove invaluable at the next incident — yours or your successor's.

When can we stand down? When four statements hold: exposed credentials provably inert, leak path structurally fixed and verified by rescan, usage window assessed with undetermined items ticketed, and monitoring active on the surface that leaked. Standing down earlier just means discovering later.

---

Prevention beats response: know what ships before attackers do. KeyDrift's free scan — minutes, no signup.