Why Your Text Shows "é" and "’": Mojibake, Decoded

A customer named José shows up in your CRM as José. Every apostrophe in a WordPress post has turned into ’. A CSV export from one system lands in another with ü where the umlauts used to be. This is called mojibake, and it looks random. It isn’t. Garbled text is a fingerprint: each pattern maps to one exact mistake, and if no byte was lost along the way, you can reverse it. Nearly every case on the web is the same one: text saved as UTF-8, then read back as Windows-1252.

Every byte sequence and garble below was produced by running it in Python 3’s cp1252 codec and checked against Microsoft’s cp1252 mapping table published by Unicode. This post is about characters that turned into the wrong visible characters. If your problem is characters you can’t see at all (zero-width spaces, stray BOMs breaking a string match), that’s a different bug, covered in invisible Unicode characters.

The lookup table

You see	It should be	UTF-8 bytes	What happened	Fix
é	é	C3 A9	UTF-8 read as Windows-1252, once	s.encode('cp1252').decode('utf-8')
’	’ (curly apostrophe)	E2 80 99	Same mistake. The 3 bytes became 3 characters	Same one-liner
—	— (em dash)	E2 80 94	Same mistake	Same one-liner
ü	ü	C3 BC	Same mistake	Same one-liner
€	€	E2 82 AC	Same mistake	Same one-liner
’	’	E2 80 99	Mangled twice: the garble was saved as UTF-8 and misread again	Run the one-liner twice

The pattern is easy to see once you look at the bytes. Every accented letter used in Western European languages (é, ü, ñ, ö) is two bytes in UTF-8, and the first one is C3. Windows-1252 reads C3 as Ã. So a stray Ã before a symbol almost always means a single accented letter. Curly quotes, dashes, the euro sign and the ellipsis are three bytes starting with E2, which Windows-1252 shows as â. So â€ followed by one more character is almost always punctuation.

A few more fingerprints worth recognizing:

You see	It was	Note
“ and †+ nothing visible	“ and ”	The closing quote’s last byte is 9D, one of the five holes in cp1252 (see below)
–	– (en dash)
…	…
â„¢	™
°, ©, £	°, ©, £	Two-byte symbols start with C2, which shows as Â
à + invisible	Á, Í, Ï, Ð or Ý	Second byte falls in a cp1252 hole
 at the very start	A UTF-8 BOM	The bytes EF BB BF read as Windows-1252
caf�	café	The opposite mistake: Windows-1252 bytes read as UTF-8. Lossy

That last row is the one to worry about. Going from cp1252 to UTF-8, the lone byte E9 isn’t valid UTF-8, so the decoder swaps it for the replacement character � (U+FFFD). The original byte is gone. There is no one-liner that brings it back.

Why it’s Windows-1252 and not Latin-1

People often say the text was “read as Latin-1”. The garble itself shows otherwise. Look at ’. The € comes from byte 80. In Windows-1252, 80 is the euro sign. In real ISO-8859-1 (Latin-1), bytes 80 through 9F are C1 control characters, which are invisible. A true Latin-1 decoder turns E2 80 99 into â followed by two invisible characters, not ’.

So if you can see a € or a ™ in the garble, the decoder was using Windows-1252 rules. That is often true even when the label said ISO-8859-1. The WHATWG Encoding Standard, which browsers follow, makes latin1, iso-8859-1 and even ascii all labels for windows-1252. MySQL does the same: its manual states that “MySQL’s latin1 is the same as the Windows cp1252 character set.” A page served as Latin-1, or a MySQL column declared latin1, decodes with cp1252 rules either way.

Is the data broken, or just the display?

Before you fix anything, find out where the damage is. There are two cases:

1. Only the display is wrong. The stored bytes are correct UTF-8, and something read them with the wrong encoding. Fix the reader (a header, an import setting) and leave the data alone.
2. The garble was saved. Something read the text wrongly, then wrote the garbled characters back out as UTF-8. Now the stored bytes really are é, and every correct reader shows the garble faithfully.

Paste a sample into our text to hex converter, which shows the UTF-8 bytes of whatever you type. A correct é is c3 a9. A stored é is c3 83 c2 a9: four bytes, because Ã and © are each two-byte characters in their own right. If you see the longer form, the data needs repair. For a character-by-character view, the Unicode inspector lists every code point with its UTF-8 bytes and UTF-16 code units. It also gives invisible characters their own row, which is how you spot the U+009D hiding inside a broken ”.

The one-line repair (and where it fails)

To undo a single round of mangling, reverse the steps. Turn the garbled characters back into the bytes Windows-1252 made them from, then decode those bytes properly:

>>> 'été'.encode('cp1252').decode('utf-8')
'été'
>>> '’'.encode('cp1252').decode('utf-8')
'’'
>>> '’'.encode('cp1252').decode('utf-8')
'’'

The double-encoded row needs two passes, which is the whole point of that row. Stop as soon as the text looks right. One pass too many on clean text usually raises a UnicodeDecodeError rather than silently corrupting it. Plain ASCII passes through every round unchanged.

The repair is reversible only if no byte was lost, and Windows-1252 has five places where bytes get lost. The Microsoft mapping table lists 81, 8D, 8F, 90 and 9D as #UNDEFINED. Different software handles those five bytes differently:

• Browsers (per the WHATWG index) and MySQL pass them through as the invisible control characters U+0081, U+008D, U+008F, U+0090 and U+009D.
• Python’s strict cp1252 codec refuses them and raises an error.
• Plenty of tools strip the invisible character, or swap it for �, on copy, paste or save.

That’s why ” is the character that most often survives repair badly. Its bytes are E2 80 9D, and that last byte is a hole. If the garble came from a browser or MySQL, it contains â€ plus an invisible U+009D, and the strict one-liner fails on it:

>>> 'â€\x9d'.encode('cp1252')
UnicodeEncodeError: 'charmap' codec can't encode character '\x9d' ...

A tolerant version maps those five control characters straight back to their byte values:

def unmangle(s):
    out = bytearray()
    for ch in s:
        try:
            out += ch.encode('cp1252')
        except UnicodeEncodeError:
            out.append(ord(ch))  # U+009D -> byte 0x9D, etc.
    return out.decode('utf-8')

unmangle('été “quotedâ€\x9d')  # 'été “quoted”'

If the invisible character was already stripped, though, the byte is simply gone and â€ can’t be decoded. Your best option at that point is a targeted find-and-replace. For text with mixed damage, the Python library ftfy (ftfy.fix_text()) handles these cases heuristically and leaves correct text alone.

If you reproduce any of this in Node, check your version. Some Node 20 and 22 releases decode windows-1252 as plain Latin-1, so TextDecoder('windows-1252') shows â plus invisible characters instead of ’. Node 22.18 does this. Browsers and Python’s cp1252 codec get it right.

Find and replace, when you can’t run code

For a document, a CMS field or a spreadsheet column, plain replacement works fine. Put the garble in “find” and the real character in “replace” in our find and replace tool. Fix the longest patterns first (’ before ’) so a partial match doesn’t leave half a garble behind. Tick Case sensitive before you run it. The tool ignores case by default, and Ã and ã are an uppercase/lowercase pair, so a search for é would also hit ã© in correctly spelled Portuguese.

Stop it happening again

• Pick the encoding on import, don’t let the app guess. In Excel, use Data > From Text/CSV rather than double-clicking the file, and set File origin to 65001: Unicode (UTF-8). In code, say it out loud: open(path, encoding='utf-8'), pd.read_csv(path, encoding='utf-8').
• Declare it on the web. Send Content-Type: text/html; charset=utf-8. Without a charset, the browser guesses, as covered in Content-Type vs file extension.
• Know what the BOM does. A UTF-8 BOM (EF BB BF) at the top of a file tells Excel and other readers “this is UTF-8”. Read by a cp1252 tool it becomes , and read by a script that doesn’t expect it, it sticks to your first column name. Python’s utf-8-sig codec reads files with or without one. The trade-offs for CSVs specifically are in why your CSV opens in one column.
• Make the content encoding-proof. When text has to pass through a system you don’t control, the HTML entity encoder turns every non-ASCII character into a pure-ASCII reference: é becomes é and ’ becomes ’. ASCII bytes are identical in UTF-8 and Windows-1252, so nothing can be misread. (Entities are a data-safety tool, not an XSS defense. See HTML entities vs URL encoding vs escaping.)

Next time you see Ã or â€, don’t retype the text. Paste a sample into the text to hex converter, check whether it shows c3 a9 or c3 83 c2 a9, and you’ll know whether to fix the reader or the data.