The letter ü is one character, two UTF-8 bytes, six URL characters (%C3%BC) and four Base64 characters (w7w=) — all the same text. This article explains the character/byte distinction underneath all three, and why most 'encoding bugs' are really one missing UTF-8 step.
Nearly every mysterious text bug — the café that becomes café, the emoji that breaks a token, the name that will not fit the database column — is one confusion wearing different clothes: characters are not bytes. Unicode assigns each character a number (a code point); UTF-8 is the dominant scheme for turning those numbers into bytes; and every transport format — URLs, Base64, JSON, CSV — moves bytes while pretending to move text. When a system skips or double-applies the UTF-8 step, text corrupts in ways that look random until you count bytes.
This article builds the model once — code points, UTF-8's variable-length design, and then the travel diary of a single character through a URL, a Base64 string and a JSON document — because once the model is clear, the bugs diagnose themselves.
Key takeaways
- Unicode = the catalogue of characters and their numbers. UTF-8 = the byte encoding of those numbers. They are different layers; 'unicode encoding' as a phrase hides the split that causes the bugs.
- UTF-8 uses 1 byte for ASCII, 2 for most European and Middle Eastern scripts, 3 for most Asian scripts, 4 for emoji and rare characters.
- In URLs, non-ASCII text becomes UTF-8 bytes, then each byte becomes %XX. In Base64, the bytes are encoded directly. In JSON, text is Unicode natively, with \uXXXX escapes as an ASCII-only escape hatch.
- btoa() and atob() in browsers operate on Latin-1 byte strings, not UTF-8 text — the classic reason emoji 'break' naive Base64 code.
The two layers, with a single character as guide
Take ü. Unicode names it U+00FC: code point 252, LATIN SMALL LETTER U WITH DIAERESIS. That number is an identity, not a storage format. UTF-8 stores it as two bytes, 0xC3 0xBC — the scheme is self-synchronising and backward-compatible with ASCII: any byte below 0x80 is the ASCII character itself, and multi-byte sequences announce their length in their first byte. This design is why UTF-8 conquered the world: files that were pure ASCII were already valid UTF-8, and the rest of the world's scripts paid only a byte or two of tax.
| Character | Code point | UTF-8 bytes | In a URL | In Base64 (of its bytes) |
|---|---|---|---|---|
| a | U+0061 | 61 (1 byte) | a | YQ== |
| ü | U+00FC | C3 BC (2 bytes) | %C3%BC | w7w= |
| € | U+20AC | E2 82 AC (3 bytes) | %E2%82%AC | 4oKs |
| 🎉 | U+1F389 | F0 9F 8E 89 (4 bytes) | %F0%9F%8E%89 | 8J+OiQ== |
Read the table's pattern: everything downstream — percent-encoding, Base64 — is mechanical operations on the UTF-8 bytes. No format 'supports emoji' or 'doesn't support accents' at that level; there are only pipelines that carry the bytes faithfully and pipelines that reinterpret them mid-journey.
Where corruption is born: the Latin-1 detour
The mojibake café is a fossil of a specific mistake: UTF-8 bytes (café's é = C3 A9) interpreted as if each byte were a Latin-1 character, producing à and ©. The reverse mistake — UTF-8-decoding text that was really Latin-1 — either garbles or fails outright on bytes that are not valid UTF-8 sequences. Every layer of a stack has an opinion about encoding: the file's bytes, the HTTP header, the HTML meta tag, the database connection, the terminal. Corruption happens at the seams, where one layer's assumption meets another's, which is why 'set everything to UTF-8, everywhere, and declare it' is the entire strategy — boring, total, effective.
JavaScript adds its own historical wrinkle: strings are sequences of UTF-16 code units, so characters beyond U+FFFF (most emoji) occupy two units (a surrogate pair) — "🎉".length is 2, not 1. Code that counts, slices or truncates by UTF-16 units can split a pair and manufacture replacement characters. Modern APIs (Array.from, spread, Intl.Segmenter) iterate by code point or grapheme; reach for them whenever text leaves your own keyboard's script.
The browser Base64 trap, and its correct solution
The built-in btoa() encodes a 'binary string' whose characters must each fit in one byte (Latin-1 range). Hand it 'café' and it works by luck (é is in Latin-1, and gets encoded as the single byte E9 — not the UTF-8 pair C3 A9 — silently producing Base64 for the wrong bytes). Hand it '🎉' or '日本' and it throws InvalidCharacterError. The correct recipe is explicit: new TextEncoder().encode(text) to get UTF-8 bytes, Base64-encode the bytes, and on the way back, Base64-decode to bytes and TextDecoder them as UTF-8. Any encoder — including the one on this site — that handles your language's text and emoji correctly is doing exactly this, and one that throws or garbles is not.
JSON's place in the picture
JSON text is defined as Unicode, and in practice is transmitted as UTF-8. The \uXXXX escape exists for environments that must stay inside ASCII — inside it, characters beyond U+FFFF need a surrogate pair of escapes, a frequent source of hand-written-JSON bugs. The rules that keep you safe are few: emit JSON as UTF-8 without escapes unless a consumer demands them; never hand-write surrogate pairs; and when a parser rejects a file 'at position 1', suspect a UTF-16 file or a byte-order mark before you suspect the content — JSON's own grammar was never the problem.
Mistakes and pitfalls
Counting bytes as characters (or the reverse) in limits
A '255-character' database column in bytes, a URL length limit in characters, a truncated multi-byte sequence — length bugs are layer bugs. Specify limits in one unit, measure in that unit, and never slice UTF-8 bytes at an arbitrary offset.
Normalising text silently
Unicode allows multiple spellings of the 'same' text (é as one code point, or e + combining accent). Systems that compare, hash or deduplicate text must agree a normalisation form (NFC is the usual choice) or identical-looking strings will mismatch and passwords set on one device will fail on another.
Assuming ASCII in identifiers and slugs
Code that strips 'non-alphanumeric' characters from names will delete most of the world's writing systems wholesale. Modern URL and filename practice keeps Unicode (percent-encoded in transit) or transliterates deliberately — never silently deletes.
Common questions
Why does my Base64 of the same word differ between two tools?
One tool encoded the UTF-8 bytes; the other encoded Latin-1 or UTF-16 bytes. The outputs are all 'valid Base64' — of different byte sequences. For text, UTF-8 is the interoperable choice; check which your tool documents, and test with a character outside Latin-1 to expose the difference instantly.
Is UTF-8 always the right encoding?
For interchange, yes — it is the web's declared default and the only encoding new protocols choose. Legacy systems (old Windows code pages, some mainframe and East Asian legacy encodings) exist at the edges; convert at the boundary, deliberately and once, rather than letting mixed encodings coexist inside a system.
What is the replacement character (�) telling me?
That a decoder met bytes invalid in the encoding it expected, and substituted U+FFFD rather than crash. It is a symptom with an address: find the first � in a document and the layer mismatch is just upstream of it. Data already saved with � in it has lost information permanently — the fix must happen at the source and the data re-acquired.
To wrap up
Characters are ideas; bytes are their shipping containers; UTF-8 is the container standard the whole web agreed on. Keep the layers distinct, declare UTF-8 at every seam, and encode bytes — never vibes — into URLs and Base64, and the entire menagerie of mojibake, broken tokens and split emoji retires at once.
Try it yourself
The Base64 encoder on this site converts text through UTF-8 properly, so ü, € and 🎉 all round-trip exactly — try the characters from the table and compare with the values printed here.