Build your own secret scanner: regexes, entropy, and their failure modes

For engineers who want to understand what secret scanners actually do: a complete working implementation you can run today, followed by the failure modes that every production scanner exists to fight.

The fastest way to evaluate secret-scanning claims is to build the naive version yourself. It takes seventy lines of Python, catches real leaks immediately, and fails in instructive ways — because the hard problems aren't matching strings, they're everything around matching: placeholders that look exactly like secrets, formats without structure, values whose meaning depends entirely on context, and encodings that scatter single credentials across multiple lines. Every commercial capability traces back to one of these failure modes.

This article builds the naive scanner first, runs it against a synthetic fixture, then walks each failure mode with examples and the mitigation strategies professional tools apply. By the end you'll know both how detectors work and why detector lists under-sell the engineering. Everything here pairs naturally with the open-source landscape, which shows where mature projects landed after fighting these same battles.

The minimal scanner that works

Standard library only — no dependencies beyond Python itself:

#!/usr/bin/env python3
"""mini-scanner.py — a teaching-grade secret scanner."""
import argparse, json, math, re
from pathlib import Path

PATTERNS = {
    "stripe_secret":     re.compile(r"sk_live_[0-9a-zA-Z]{24,}"),
    "stripe_restricted": re.compile(r"rk_live_[0-9a-zA-Z]{24,}"),
    "aws_access_key":    re.compile(r"\bAKIA[0-9A-Z]{16}\b"),
    "github_pat":        re.compile(r"\bghp_[0-9a-zA-Z]{36}\b"),
    "openai_project":    re.compile(r"sk-proj-[0-9a-zA-Z\-_]{40,}"),
    "slack_token":       re.compile(r"\bxox[baprs]-[0-9A-Za-z\-]{10,}"),
    "supabase_secret":   re.compile(r"\bsb_secret_[0-9a-zA-Z_]{20,}"),
    "private_key":       re.compile(r"-----BEGIN [A-Z ]*PRIVATE KEY-----"),
}

SKIP_DIRS   = {".git", "node_modules", ".venv"}
SKIP_FILES  = {"package-lock.json", "yarn.lock"}

def shannon_entropy(s: str) -> float:
    """Bits of information per character — randomness proxy."""
    if not s:
        return 0.0
    freq: dict[str, int] = {}
    for ch in s:
        freq[ch] = freq.get(ch, 0) + 1
    n = len(s)
    return -sum((c / n) * math.log2(c / n) for c in freq.values())

def scan(root: Path) -> list[dict]:
    findings = []
    for path in root.rglob("*"):
        if not path.is_file():
            continue
        if any(part in SKIP_DIRS for part in path.parts):
            continue
        if path.name in SKIP_FILES:
            continue
        try:
            text = path.read_text(errors="ignore")
        except OSError:
            continue
        for rule_name, pattern in PATTERNS.items():
            for match in pattern.finditer(text):
                value = match.group(0)
                findings.append({
                    "rule":    rule_name,
                    "file":    str(path),
                    "line":    text.count("\n", 0, match.start()) + 1,
                    # Never emit the live value — mask like a production tool:
                    "masked":  value[:8] + "…" + value[-4:],
                    "entropy": round(shannon_entropy(value), 2),
                })
    return findings

if __name__ == "__main__":
    parser = argparse.ArgumentParser(description="Teaching secret scanner")
    parser.add_argument("root", help="Directory to scan")
    args = parser.parse_args()
    print(json.dumps(scan(Path(args.root)), indent=2))

Design choices worth noticing even at this scale: values are masked before leaving memory (a scanner that prints full secrets creates new exposure surfaces); line numbers accompany matches (triage needs location); entropy accompanies every finding (the seed of later filtering); and the skip lists exclude dependency trees where third-party fixtures would drown signal.

Run it against a fixture

Build a tiny tree mixing one fake credential with realistic noise:

mkdir -p fixture/src fixture/tests
cat > fixture/src/config.js <<'EOF'
// Synthetic placeholder in Stripe format — safe to commit.
const STRIPE_KEY = process.env.STRIPE_SECRET_KEY ?? "sk_live_FAKE000000000000000000000";
EOF
cat > fixture/tests/helpers.ts <<'EOF'
export const TEST_TOKEN = "ghp_FAKE0000000000000000000000000000000";
EOF
python3 mini-scanner.py ./fixture

Output (abbreviated):

[
  {
    "rule": "stripe_secret",
    "file": "fixture/src/config.js",
    "line": 2,
    "masked": "sk_live_…00000",
    "entropy": 4.32
  },
  {
    "rule": "github_pat",
    "file": "fixture/tests/helpers.ts",
    "line": 1,
    "masked": "ghp_FAKE…0000",
    "entropy": 4.12
  }
]

It works — both planted credentials found, masked, located, measured. This scanner would have caught plenty of real incidents. Now watch it fail.

Failure mode one: the placeholder paradox

Add documentation to the fixture:

cat > fixture/README.md <<'EOF'
## Setup

Set STRIPE_KEY in your environment. Format looks like:
sk_live_FAKE000000000000000000000  (example only)
EOF
python3 mini-scanner.py ./fixture | grep '"file"'

The README now generates an identical finding to config.js — same rule, same severity, same everything. Multiply across a real repository's docs, tutorials, test fixtures, and example configs, and placeholder findings dominate volume while carrying near-zero information. This is the structural seed of alert fatigue: operators cannot distinguish "documentation example" from "live key" because the naive scanner doesn't try.

Production mitigations, in ascending sophistication:

1. Known-sample exclusion: documented demo keys get rejected by name — Stripe publishes its example key in its own documentation, AWS ships AKIAIOSFODNN7EXAMPLE, and flagging either is among the most expensive mistakes a scanner can make, because it burns trust on every developer who recognizes them.
2. Placeholder heuristics: literal substrings (FAKE, EXAMPLE, YOUR_, XXXX) and low-entropy bodies (000…) demote or drop findings.
3. Structural analysis: decode candidate JWTs and check payload roles rather than trusting shape; attribute shared shapes honestly ("Stripe-format or Clerk-format") instead of guessing.

Failure mode two: formats without anchors

The curated regexes above work because major providers publish stable prefixes. Now consider what they never touch:

eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.<payload>.<signature>
postgres://app:S3cretP@ss@db.internal:5432/prod
-----BEGIN OPENSSH PRIVATE KEY-----

JWTs carry no provider prefix — their identity lives in decodable payloads. Connection strings hide passwords inside URLs whose schemes are generic. Private keys announce themselves only at boundaries. The generic fallback everyone reaches for is entropy scoring:

# Hex strings top out around 4.0 bits/char; base64-ish bodies approach 5.8.
# Real credentials cluster high; English text sits near 3.
if shannon_entropy(candidate) > 4.5 and KEYISH_NAME.match(variable_name):
    flag(...)

And entropy alone drowns: minified JavaScript is made of high-entropy tokens — bundle hashes, chunk names, encoded assets. Without anchoring heuristics (variable names, assignment contexts, adjacent keywords), entropy scanning over built artifacts produces thousands of candidates per file. This is precisely why scanner rules combine format knowledge, naming context, and statistical measures rather than trusting any single signal — a layered-confidence design rather than a pattern dump.

Failure mode three: context is the decision

The fixture already contains the lesson. These two lines produce identical findings from the naive scanner:

const STRIPE_KEY = process.env.STRIPE_KEY ?? "sk_live_FAKE000000000000000000000"; // src/config.js
const STRIPE_KEY = "sk_live_FAKE000000000000000000000";                            // src/client/pricing.ts

Yet they mean opposite things. The first reads environment configuration with a synthetic fallback — hygiene holding. The second hardcodes into a module destined for a browser bundle (where substitution mechanics deliver it) — an incident. File path, import graph, and framework context decide which finding matters; no amount of regex sophistication resolves it, because the discriminating information isn't in the string.

Context-aware detection therefore means: path weighting (client directories elevate severity), public-credential recognition (publishable keys belong in bundles — flagging them trains operators to dismiss alerts), and import-graph awareness for frameworks with explicit server/client boundaries. Each addition moves the scanner from string-matcher toward the artifact-aware model, and none of it appears in feature-count comparisons.

Failure mode four: encodings, splits, and assembly

Credentials rarely stay pristine between storage and delivery:

// Base64-wrapped:
const k = Buffer.from("c2tfbGl2ZV9GQUtFMDAwMDAwMDAwMDAwMDAwMDAw", "base64");
// Assembled at runtime:
const parts = ["sk_live_", "FAKE", "0000"]; const key = parts.join("");
// Template-split for "readability":
const url = `${API}?key=${SECRET_PART_A}${SECRET_PART_B}`;

Each transformation defeats naive matching completely — the credential never appears contiguous in scanned text. Mitigations exist (decoding passes for common encodings, de-minification for bundles, constant-folding approximations) but each expands compute cost and false-positive surface, which is why runtime-assembled keys remain honest limitations across the industry, stated plainly by serious tools rather than claimed away. The boundary statement belongs in every scanner's documentation: static analysis sees static strings.

The ceiling, and what lives above it

Assemble the failure modes and the naive scanner's ceiling comes into focus:

Two rows deserve final emphasis because they define categories rather than features. Confidence scoring — combining format match strength, entropy, context, and exclusions into graded certainty, dropping sub-threshold results entirely — separates tools teams keep from tools teams mute. And jurisdiction: everything above still inspects sources; the served-artifact surface needs observers placed there specifically, which is the niche KeyDrift occupies, designed around exactly these lessons about signal quality.

Building your own remains worthwhile — for learning, for custom internal formats, for one-off audits. Shipping it as an organization's defense usually stops at the ceiling, and recognizing the ceiling is the real lesson the seventy lines teach. If yours earns a permanent role, three maintenance habits keep it honest: a fixture suite of known samples (real formats, placeholders, noise) run on every rule change so regressions surface immediately; a calendar entry for provider-format updates, since prefixes and shapes drift; and a written boundary statement — what it cannot see — so nobody mistakes jurisdiction for coverage (the artifact-side gap being the one that matters most). Hand off to dedicated tooling when any of those habits starts slipping; that's the ceiling announcing itself again.

Extending the scanner: three upgrades worth building

Between the teaching version and production tools lie three extensions that each teach another layer of the problem — and each is genuinely useful in its own right.

One: baseline diffing. Persist findings to a committed JSON, then report only deltas on subsequent runs:

def diff(old: list[dict], new: list[dict]) -> dict:
    def key(f): return (f["rule"], f["masked"])
    old_set, new_set = {key(f) for f in old}, {key(f) for f in new}
    return {
        "created":   [f for f in new if key(f) not in old_set],
        "resolved":  [f for f in old if key(f) not in new_set],
    }

Fingerprinting on rule-plus-masked-value keeps retries stable, and resolved findings exit the queue instead of re-alerting forever — the transitions-only principle that keeps operators listening (fatigue mechanics). Note what the mask enables: baselines can be committed because they contain no live values, making scanner state reviewable like any other code artifact.

Two: confidence scoring. Replace boolean flagging with graded certainty from combined signals:

def confidence(rule: str, value: str, path: Path) -> float:
    score = 0.4                                    # format match baseline
    score += 0.2 * (shannon_entropy(value) > 4.5)  # randomness corroborates
    if SAMPLE_KEYS.match(value):                   # documented demos
        return 0.0
    if PUBLIC_TIERS.match(rule):                   # publishable/anon tiers
        return 0.0
    if CLIENT_DIR in path.parts:                   # context elevation
        score += 0.3
    return round(score, 2)

Then enforce the threshold honestly: findings below 0.5 drop entirely rather than emitting caveats — the design choice that separates tools teams keep from tools teams mute (the threshold argument). Tuning the weights against your own repositories is where scanner craft actually lives.

Three: a history walker. Sources change; history persists. Walking commits turns the engine into an archaeology tool:

git rev-list --all | while read sha; do
  git archive "$sha" | tar -x -C "$(mktemp -d)" 2>/dev/null && echo "$sha"
done | xargs -I{} python3 mini-scanner.py ./extracted-{}

(A sketch rather than a prescription — production walkers stream blobs directly via git cat-file --batch for speed.) The historical mode surfaces the git-persistence problem concretely: deleted-file keys remain present in every ancestor commit, and only history-aware scanning finds them.

Four: declarative context policy. Hard-coding client directories and sample keys works until it doesn't; a small YAML policy keeps adjudication maintainable:

# scan-policy.yaml
sample_keys:
  - "sk_live_FAKE000000000000000000000"
  - "AKIAIOSFODNN7EXAMPLE"
client_paths: ["src/client", "app", "components"]
server_paths: ["server", "api"]
weights:
  format_match: 0.4
  entropy_over_4_5: 0.2
  in_client_path: 0.3
drop_below: 0.5

The file becomes both configuration and documentation — reviewers can see exactly why a finding surfaced, and tuning happens in diffs rather than code edits. It also encodes the honest boundary of what static analysis knows: paths and shapes, never intent.

Each extension mirrors a professional capability — baselining, confidence, history coverage — and building them once explains why commercial scanners look the way they do. The remaining gap after all three (live verification, artifact jurisdiction, maintained format catalogs) is exactly the territory where dedicated tooling earns its keep: the landscape comparison for source-side choices, KeyDrift's free scan for the served-artifact side.

Two practical notes for running the teaching version against real trees. Performance: pure-Python walking handles thousands of files acceptably; for larger estates, swap the walker for ripgrep feeding candidate lines through the same pattern set (rg -oN --no-filename -e 'sk_live_[0-9a-zA-Z]{24,}' -e 'AKIA[0-9A-Z]{16}' | sort -u) and reserve Python for scoring and masking. And scope honesty applies to output as much as input — findings feeds, tickets, and chat notifications should carry the masked form only, because a scanner that distributes full values has become a new disclosure surface with your name on it.

Canary testing closes the loop between building a scanner and trusting one. Plant a file of synthetic-but-format-valid markers — sk_live_FAKECANARY000000000000000000, AKIAFAKECANARY00000AA — somewhere representative (a fixture directory in CI, a staging config), then verify the full chain fires: hook catches staged canary, CI scan flags the tree scan, built-output scan finds anything prefixed into dist, and the alert channel delivers a well-formed finding to a human who acts. Every break in that chain discovered by canary costs minutes; the same break discovered by an incident costs the usual. Re-run canaries whenever scanner versions, rule packs, or pipeline stages change — they are, functionally, unit tests for your detection stack, asserting not that detectors exist but that detectors reach people (the difference that matters).

Finally, version your rules like code: a rules file under version control, changes landing through review, and a changelog noting why each rule exists. Scanner archaeology — "why does this flag connection strings?" — answers itself when the rule's commit message carries its rationale, and retirement decisions get made against documented intent instead of vibes. The discipline costs minutes per change and compounds exactly like test suites do.

FAQ

Is this scanner safe to point at production code? Yes with one habit enforced: masked output everywhere, including logs and tickets — the code above masks by design, and extending it should preserve that property. A leak-response tool that reproduces leaked values undermines itself.

Why not just use very long regexes covering every case? Regex expresses string shape, not meaning. The context failures above persist regardless of pattern sophistication because the decisive signals (file role, import graph, deployment destination) aren't properties of the matched text.

What about scanning git history specifically? The same engine applies — walk commits, extract blobs, match per revision. Tools like gitleaks wrap exactly this; performance becomes the challenge at history depth, which is why incremental designs (scan-on-push deltas) dominate practice.

Could I add live verification like trufflehog? Mechanically yes — attempt harmless authenticated calls per candidate. Operationally you inherit network dependencies, rate limits, and provider terms; most self-built scanners stop at pattern-plus-context and hand verification to humans or dedicated tools.

---

Now that you know what detection costs up close: run KeyDrift's free scan against your deployed app — no signup — and see production-tuned detection applied to what you actually serve.