Why Regex PII Redaction Always Leaks (And When to Use It Anyway)

You’re about to paste a production log into a bug report, or a support ticket into an LLM prompt. Running it through a PII scrubber first is a good instinct. But a scrubber that reports “0 findings” and a document that contains no personal data are different claims, and pattern-based tools can only ever make the first one.

This is the third post in what’s turning into a series about redaction that doesn’t redact — after pixelation is not redaction and black boxes don’t redact PDFs. Those two are about tools that look like they removed something and didn’t. This one is different: the removal works fine. It’s the finding that’s incomplete.

What pattern matching is genuinely good at

Some personal data has structure, and where there’s structure a regex is close to ideal — fast, deterministic, offline, and auditable.

Type	Structure	Reliability
Email addresses	`[email protected]`, well-specified shape	High
Credit card numbers	13–19 digits, Luhn checksum	High
IBANs	Country code + check digits + mod-97	High
IPv4 / IPv6	Rigid numeric/hex format	High
US SSNs	`NNN-NN-NNNN` plus range rules	Moderate
Phone numbers	Loose, varies enormously by country	Moderate
API keys / tokens	Often prefixed (`sk_live_`, `ghp_`), high entropy	Moderate–high

The two with real checksums — cards and IBANs — are the strongest, because the checksum lets you reject a match that has the right shape but can’t be a real value. That’s a meaningful improvement over shape alone.

The value proposition is genuine: for the specific, common, high-consequence case of “there’s a customer email and a card number in this log,” a regex pass takes milliseconds, runs entirely locally, and catches it. That’s worth having.

The hard limit: unstructured PII has no pattern

Here’s the part that can’t be engineered around, only acknowledged.

A name is not a pattern. “James Wright” and “james wright” and “J. Wright” are names. So are “Ada Lovelace,” “李伟,” and “O’Sullivan-Nakamura.” But “Bay Area,” “Wells Fargo,” and “Chrome Canary” have the identical shape — two capitalized words — and are not people. There is no regular expression that separates these, because the distinction is semantic, not syntactic.

The same applies to:

Street addresses. Formats vary by country, and “42 Oak Street” is structurally indistinguishable from any other number-word-word sequence.
Dates of birth. A DOB looks exactly like a deployment date, a subscription renewal, or a log timestamp.
Free-text disclosure. “The customer said her daughter is the one with the account” contains no matchable token and is real PII.
Job titles plus employer. “the VP of Engineering at Contoso” identifies exactly one person without containing a single pattern.
Internal identifiers. A user_id that maps to a person in your database is PII in effect, and looks like every other integer.

Finding these requires named entity recognition — a trained model that classifies tokens by meaning and context. That’s a fundamentally different technology with a fundamentally different cost profile: it needs a model (large, or a network call), it’s probabilistic rather than deterministic, and it has its own well-documented failure modes, notably degraded accuracy on names outside its training distribution. Which is its own quiet fairness problem: an NER-based scrubber that reliably catches Anglo names and misses others is worse than useless, because it produces confidence unevenly.

So the honest statement of what a regex scrubber does is: it removes the structured identifiers, and leaves everything else exactly where it was. Our PII scrubber says this on the page itself, and it is the reason its instructions end with “review the output” rather than “you’re done.”

The other direction: checksums create false positives too

Reducing false positives with a checksum is the right move, but it’s worth understanding what it does and doesn’t buy.

(There’s a second reason to keep detection patterns tight: a scanner’s regexes run over whatever text a user pastes, which is exactly the setup for catastrophic backtracking.)

The Luhn algorithm is a single check digit. Roughly one in ten random 16-digit numbers passes Luhn by chance. That’s not a flaw in anyone’s implementation; it’s the arithmetic. A checksum with one decimal digit of redundancy rejects 90% of invalid inputs and accepts 10%. (The same mod-10 idea, with different weights, guards every retail barcode — worked through digit by digit in the UPC/EAN check digit post.)

Practically, that means a long numeric identifier — an order number, a transaction reference, an internal ID — has a real chance of being flagged as a credit card and redacted. If you’re scrubbing a log where order numbers are how you correlate events, a scrubber can quietly destroy the thing you needed the log for.

This cuts both ways and neither error is free:

False positives damage the document. Redacting an order ID or an internal hostname can make a log useless for debugging, and you may not notice until you’re deep in an incident.
False negatives leak data. That’s the failure everyone worries about.

Tuning toward one always means accepting more of the other. Anyone selling you a scrubber with neither is selling you a claim they can’t support.

Some specific tradeoffs worth knowing, using ours as the concrete example:

Phone matching requires separators, so a bare run of digits isn’t flagged. That means 5551234567 written with no formatting is missed — a deliberate trade to avoid redacting every long number in a log.
SSNs must be dashed and pass range rules, for the same reason. 123456789 is not treated as an SSN.
Grouped IBANs separated by exactly one space can be matched as a single finding rather than two. Any punctuation or newline between them avoids it. It’s a narrow edge in the matching loop, and it’s documented rather than pretended away.

A scrubber’s tuning decisions are its actual product. If a tool won’t tell you what they are, you can’t reason about what it missed.

The compliance question

No pattern-matching tool makes you GDPR or HIPAA compliant, and any that claims to should be treated as a red flag rather than a feature.

Those regimes are about process: knowing what data you hold, why you hold it, who can reach it, how long you keep it, and what happens when someone asks for it back. Redacting a string is one control inside that, not a substitute for it. HIPAA’s Safe Harbor de-identification standard, for instance, enumerates 18 specific identifier categories — including names, geographic subdivisions smaller than a state, all date elements more granular than a year, and a catch-all for “any other unique identifying number, characteristic, or code.” A regex can address perhaps half of those, and the catch-all by construction can’t be automated.

Use the tool as a first pass, then read the output. That’s not a limitation of one implementation; it’s the shape of the problem.

Where it’s the right tool anyway

Given all that, the cases where regex scrubbing is genuinely correct:

Before pasting into an LLM. You’re about to send a support ticket to a third party. Stripping the emails and card numbers in one keystroke is a large, cheap reduction in exposure, even if it isn’t complete.
Bug reports and public issues. A log excerpt headed for a public tracker. The structured identifiers are the ones that get scraped.
Screenshots and demos. Cleaning sample data before it goes into a slide deck or a recording.
As a CI tripwire. Failing a build when a Luhn-valid number appears in a committed fixture catches real mistakes, and false positives are cheap to allow-list.
In a pipeline, in front of something smarter. Regex handles the structured 80% deterministically and cheaply; whatever you use for names handles the rest. You don’t need one tool to do everything.

What it isn’t: a compliance control, a guarantee, or a reason to skip reading the document.

Before that log or ticket leaves your machine:

Run the pattern scrubber. Cheap, catches the structured identifiers.
Read the output. Specifically hunt for names, company names, addresses, and dates of birth — the categories no pattern will find.
Check the identifiers you kept. Does that user_id resolve to a person? Does that internal hostname reveal your infrastructure layout?
Check what’s around the data, not just in it. File names, directory paths, and commit messages leak more than people expect.
Consider synthetic data instead. For demos and test fixtures, generating fake data sidesteps the whole problem — you can’t leak what was never real.

Tools

PII Scrubber — emails, phones, SSNs, Luhn-checked cards, IPv4/IPv6, and IBANs, with per-category toggles and labeled or length-preserving replacement. Runs entirely in your browser; nothing is uploaded, which is the point.
Fake Data Generator — realistic-but-invented replacements.
EXIF Remover and PDF Redactor — the same problem for photos and documents.
Browser Fingerprint Test — what you leak without pasting anything at all.