Invisible Unicode Characters: Why Two Identical Strings Don't Match

Two strings that print identically fail ===. grep can’t find a word you’re looking straight at. An API key “looks right” but authentication fails, a YAML config parses into the wrong value, and a shell script dies on line 1 with command not found even though line 1 is just a shebang. When text that looks correct behaves incorrectly, the cause is almost always a character that renders as nothing — or as something indistinguishable from a space.

Unicode contains a whole family of these: format and control characters with zero visual width, and lookalike whitespace that isn’t an ASCII space. Your terminal, editor, and browser all hide them by design. The bytes are there; the pixels aren’t. Once you know the dozen usual suspects and how to dump the actual code points, this entire class of bug goes from maddening to a two-minute fix.

The usual suspects

Here are the characters that cause the vast majority of “identical strings don’t match” bugs, with their code points and UTF-8 byte sequences (what you’ll see in a hex dump):

Character	Code point	UTF-8 bytes	What it does
Zero-width space (ZWSP)	U+200B	E2 80 8B	Invisible; marks a legal line-break point. The classic clipboard stowaway.
Zero-width non-joiner (ZWNJ)	U+200C	E2 80 8C	Prevents adjacent characters from forming a ligature. Legitimate and required in Persian and some Indic text.
Zero-width joiner (ZWJ)	U+200D	E2 80 8D	Requests joining. Load-bearing in emoji: 👨‍👩‍👧 is five code points — man, ZWJ, woman, ZWJ, girl.
Word joiner	U+2060	E2 81 A0	Invisible; forbids a line break. The modern “zero-width no-break space.”
BOM / ZWNBSP	U+FEFF	EF BB BF	Byte order mark at the start of a file; a zero-width no-break space anywhere else. Breaks shebangs and strict JSON parsers.
No-break space (NBSP)	U+00A0	C2 A0	Renders exactly like a space, is not a space. HTML’s   — survives copy-paste.
Narrow no-break space (NNBSP)	U+202F	E2 80 AF	A thinner NBSP; standard in French typography before ? ! : ;.
Soft hyphen	U+00AD	C2 AD	Invisible unless a word wraps there. Common in text copied from justified layouts and PDFs.
Bidi controls	U+202A–U+202E, U+2066–U+2069	E2 80 AA…	Override or embed text direction. Invisible — and the mechanism behind the Trojan Source attack (below).

Two things worth internalizing from this table. First, not all of these are noise: ZWNJ and ZWJ have legitimate, spec-mandated roles in real scripts and in emoji sequences, so “strip everything invisible” is not a safe default. Second, an NBSP is arguably worse than a zero-width character, because it renders as exactly one space — a diff, a code review, and your own eyes all say the strings match, while "api key" and "api key" are different byte sequences.

Where they sneak in

These characters almost never get typed deliberately into a config file. They arrive by transfer:

• Word processors and Google Docs. The same autocorrect that turns " into “ also manages non-breaking spaces around numbers, units, and punctuation. Copy a command or a key out of a doc and the NBSPs come with it.
• Websites and PDFs. Sites insert ZWSPs into long tokens (URLs, hashes) so they can wrap, and soft hyphens as hyphenation hints. Select-and-copy grabs them all. PDF text extraction is especially rich in soft hyphens and ligature leftovers.
• Chat apps and email. HTML email is full of  . Messages that pass through rich-text pipelines can pick up formatting characters that survive into the clipboard.
• Keyboards. On macOS, Option+Space types an NBSP — one modifier-slip away from the space bar, which is exactly how an invisible character ends up in the middle of a hand-typed command. French and other layouts produce NNBSPs by rule.
• Templating and CMS pipelines. A template that was itself authored in a word processor propagates its invisible characters into every rendered output.
• On purpose. Zero-width characters are how people make “blank” usernames and empty-looking messages on platforms that reject truly empty input — our Invisible Character tool exists precisely for that, generating ZWSP U+200B, word joiner U+2060, or Hangul filler U+3164 on demand. Useful when you want it; the same trick is why a username can look empty yet be impossible to search for.

One tell worth knowing: a line-based diff will flag a line as changed while both sides render pixel-identical. If the Text Diff says two lines differ and you can’t see how, stop squinting — you have an invisible character, and it’s time to inspect bytes.

The BOM that breaks shebangs and JSON

U+FEFF deserves its own section because it fails differently: at the start of a file rather than inside a string. Some Windows editors (Notepad, historically) prepend the UTF-8 byte order mark EF BB BF when saving. Three bytes, zero pixels, two classic failures:

Shebangs. The kernel identifies a script by its first two bytes being exactly #!. With a BOM, the first two bytes are EF BB, the shebang is not recognized, and you get ./deploy.sh: line 1: #!/bin/bash: command not found or the script running under the wrong interpreter.

Strict JSON parsers. RFC 8259 says implementations must not add a BOM and may (not must) ignore one. Many don’t ignore it. Node, for example:

JSON.parse('\uFEFF' + '{}')  // same bytes a BOM'd file starts with
// SyntaxError: Unexpected token '', "{}" is not valid JSON

Python’s json.loads rejects it too. So a config file that “is valid JSON” in your editor’s eyes can still refuse to parse. The fix is one sed away (see the removal section below).

Trojan Source: when invisible characters become an attack

The bidi control characters (U+202A–U+202E, U+2066–U+2069) exist so mixed Hebrew/Arabic and Latin text can specify display direction. In 2021, Nicholas Boucher and Ross Anderson at Cambridge showed they also break a core assumption of code review: that the code you see is the code the compiler compiles. The attack was named Trojan Source, tracked as CVE-2021-42574.

The mechanism: compilers read source in logical (byte) order, but editors and code hosts display it in bidi-reordered order. Place bidi override characters inside a comment or string literal — where compilers accept arbitrary characters — and you can make tokens display in a different order than they parse. Code that displays as an inert comment can actually terminate the comment early and execute; a string that displays as ending in one place actually ends in another. The reviewer approves what they see; the compiler builds what’s actually there.

The same paper covered a sibling attack, CVE-2021-42694: homoglyphs. Cyrillic а (U+0430) and Latin a (U+0061) are pixel-identical in most fonts, so an attacker can define a function whose name looks like a trusted one — sсan() with a Cyrillic с is a different identifier from scan(). Same trick, applied to identifiers instead of ordering, and it’s also the engine behind lookalike phishing domains.

The disclosure had real teeth: GitHub now warns on bidi characters in diffs, VS Code highlights them, Rust’s compiler rejects the code points outright, and GCC grew -Wbidi-chars. But plenty of pipelines — code review tools, CI logs, terminal output — still render them silently, which is why scanning source trees for [\u202A-\u202E\u2066-\u2069] is a cheap, high-value lint.

Are invisible characters an “AI watermark”?

You’ll see the claim that invisible Unicode characters are proof text came from a chatbot. Here’s what’s actually documented, versus folklore.

Documented: in April 2025, users found that output from some newer OpenAI models contained narrow no-break spaces (U+202F) where a plain space would be expected, and the story spread as “ChatGPT now watermarks its output.” The characters were real; the watermark interpretation was not established. The mundane explanation — models trained on professionally typeset, multilingual text reproduce the typographically “correct” spacing that text contains (NNBSP is standard French punctuation spacing) — fits the evidence, and OpenAI characterized the characters as a training artifact rather than a watermark. Separately, OpenAI has publicly discussed having built a statistical text-watermarking method it chose not to deploy — and that method works on word-choice patterns, not hidden characters.

Folklore: “zero-width characters in this text prove it’s AI-generated.” No major vendor is documented to watermark chatbot output with zero-width characters. And as this whole article shows, NBSPs and zero-width characters flow into human-written text constantly via Word, the web, and keyboards. An invisible character tells you the text passed through software — which all digital text did. It is not an AI detector, in either direction: their presence doesn’t prove machine authorship, and stripping them doesn’t hide it.

How to detect them

The universal move is: stop looking at rendered text, look at bytes or code points.

Hex dump a suspicious value:

printf '%s' "paste-value-here" | hexdump -C
# ZWSP shows as e2 80 8b, NBSP as c2 a0, BOM as ef bb bf

# Check a file for a leading BOM
head -c 3 script.sh | od -A n -t x1     # "ef bb bf" = BOM present

Grep a codebase for the offenders (GNU grep -P, or ripgrep which supports the same syntax; macOS’s BSD grep lacks -P):

rg '[\x{200B}\x{200C}\x{200D}\x{2060}\x{FEFF}\x{00A0}\x{00AD}\x{202A}-\x{202E}\x{2066}-\x{2069}]' -n .

Inspect character by character. Paste the string into our Unicode Inspector and it lists every character with its code point, decimal value, UTF-8 bytes, and UTF-16 code units — a ZWSP hiding between two letters shows up as its own U+200B row, immediately. For seeing how a string with escapes round-trips between representations, String Escape covers the encode/decode side.

Turn on your editor’s rendering. VS Code highlights invisible and ambiguous Unicode by default (editor.unicodeHighlight.*); most editors have an equivalent. If you regularly paste from docs or the web, leave it on.

How to remove them — carefully

The blunt instrument is a regex strip, but the character list matters, because some invisible characters are meaning-bearing:

// Safe-ish default: strip the pure troublemakers
s = s.replace(/[\u200B\u2060\uFEFF\u00AD\u202A-\u202E\u2066-\u2069]/g, '');

// Normalize lookalike spaces to real spaces
s = s.replace(/[\u00A0\u202F]/g, ' ');

Deliberately not in that first list: U+200D (ZWJ) and U+200C (ZWNJ). Stripping ZWJ shatters emoji — the family 👨‍👩‍👧 (5 code points) becomes three separate people (3 code points) — and both characters are orthographically required in Persian, Hindi, and other scripts. Strip them only from fields you know are ASCII-ish identifiers, never from user-facing prose.

Unicode normalization helps with a different slice of the problem. NFC canonically composes text (e + combining accent → é) and is the right invariant for stored text — but it does not touch any character in the table above. NFKC additionally applies compatibility mappings: it converts NBSP and NNBSP to a regular space (and ² → 2, fi → fi). But — commonly assumed, verifiably false — NFKC does not remove zero-width characters: ZWSP, ZWJ, word joiner, U+FEFF, and the soft hyphen all pass through NFKC unchanged. So normalization complements the regex; it doesn’t replace it. And NFKC is too aggressive for content where those compatibility distinctions matter (it will happily rewrite x² as x2), so reserve it for identifiers, search keys, and comparisons rather than stored documents.

For a BOM at the start of a file:

sed -i '1s/^\xEF\xBB\xBF//' file.json     # GNU sed

The general lesson mirrors the one from magic bytes and file signatures: what software renders and what the bytes say are separate layers, and every confusing behavior in this family lives in the gap between them. When text misbehaves, don’t trust your eyes — dump the code points. It’s a thirty-second check, and it’s the difference between an evening of disbelief and a one-line fix.