4 · The first hour
The Secret Leak Field Manual · 5 min read
Question this chapter answers: In what order do you act so the leak stops getting worse?
The first hour has one goal: make the leaked credential worthless before it is used again, without breaking your own product on the way. Order matters more than speed. The runbook below assumes you have already done a fast classification (Chapter 2) and a fast rating (Chapter 3); if you have not, steps 2–3 do it inline.
The runbook
- Start the record. One file or ticket, timestamped. Capture: where the key was found, which surface (Chapter 1 row numbers), masked value (first 8 + last 4), fingerprint or hash if your tooling produces one. Never paste the full value into tickets, chat, or the record itself — response tools are themselves surfaces.
- Classify. Family and scope model from Chapter 2. If attribution is ambiguous (Stripe-shaped vs. Clerk), note both candidates; rotation proceeds regardless.
- Choose the containment path.
- If the provider supports creating a replacement alongside the old credential → step 4.
- If the credential is single-slot (one secret key per environment) → still step 4; most modern providers issue replacements immediately (Stripe's rotation flow does).
- If it is a temporary cloud credential (
ASIA) → step 4 as well; waiting for expiry is not a plan, because session tokens can be refreshed by whoever holds the originals. - If it is a database URL → jump to step 5 with a maintenance-window decision: password change plus config redeploy plus connection-pool restart.
- Stage the replacement. Create the new credential in the provider console now, before touching code. This converts the incident into a deploy task. Dual-credential mechanics per provider are Chapter 6.
- Rotate. Deploy consumers onto the new credential. Then revoke or expire the old one — in that order for planned rotations; under active abuse, invert: revoke immediately, accept the outage, restore service with the replacement. Decision point: if usage logs show active unfamiliar use right now → revoke immediately, then redeploy. A bounded outage beats an open drain.
- Redeploy every artifact that embedded the old value. Rotation kills the credential; it does not clean surfaces. Rebuild bundles, purge CDN caches, redeploy previews. Until this happens, every visitor still downloads the old value — dead, but public.
- Sweep the remaining copies. CI environment variables and contexts, vault entries,
.envfiles, password-manager items, error-tracker payloads, chat attachments from triage. Each copy is tomorrow's re-leak. - Verify the old credential is dead. From your own infrastructure, make one authenticated call with the old value and confirm the provider rejects it. This is verification, not exploitation — you are testing your key.
- Watch the gap. Check provider-side logs for use of the old key between last-known-good time and revocation. Anything found widens the audit in Chapter 5.
- Rescan the deployed surface. Confirm the rebuilt bundle no longer contains the value and that nothing else surfaced during the fix.
- Close the hour with the record updated: timeline entries, actions, owners, evidence locations. Appendix C holds the template.
The five mistakes that make leaks worse
| Mistake | What it feels like | What it actually does |
|---|---|---|
| Rewriting git history instead of rotating | "Remove the evidence" | History rewrites cannot un-share what was cloned, cached, or crawled; they also break every collaborator's checkout. Rotation removes access; history cleanup is optional hygiene afterward |
| Deleting the alert, report, or finding | "Fix the dashboard" | Destroys the record you need for audit and disclosure decisions |
| Rotating without redeploying | "Done" | Two failure modes: production keeps serving a dead key (outage), or the bundle keeps shipping the old value (the leak looks alive to scanners and strangers alike) |
| Sharing the full key while responding | "Context for the team" | Every paste multiplies surfaces; masks exist so responders never need the literal |
| Treating test-mode keys as harmless | "It's just sandbox" | Usually true at the funds level and false at the habit level; medium severity still gets rotated, just not at the cost of an outage |
A sixth failure deserves its own paragraph because it is the most common of all: stopping after containment. The key is dead, the bundle is rebuilt, and everyone goes home — while the same pattern sits in three other services and the CI configuration that produced it remains unchanged. The first day (Chapter 5) exists to prevent exactly that.
Who does what
Small teams run this solo. Larger teams split it:
| Role | Owns | Steps |
|---|---|---|
| Incident lead | Record, decisions, comms | 1, 3, 11 |
| Platform owner | Provider consoles | 4–5, 8–9 |
| Application owner | Deploys and caches | 6–7, 10 |
Write these names down before an incident. Role assignment during an active leak is where hours go to die.
Do this now
- [ ] Put this runbook somewhere reachable without your laptop (printed or wiki page all responders know).
- [ ] Pre-decide the inversion rule: which conditions trigger immediate revocation over orderly rotation, and who can call it alone.
- [ ] Verify today that you can rebuild and redeploy every public-facing app on short notice; rotation depends on it.
- [ ] Check CDN cache behavior once: how long does a purged asset take to disappear globally?
- [ ] Adopt the masking convention (first 8 + last 4, never more than half) in every tool your team uses to discuss findings.