What Unicode Normalization Changes in Mandarin, Spanish, Japanese, and Arabic: A 160-String Field Test
A reproducible 160-string test of NFC, NFKC, search-key equality, width variants, combining marks, compatibility forms, and directional controls.
Unicode normalization can make visually similar strings comparable, but it can also fold width, punctuation, presentation forms, ordinals, and compatibility symbols in ways an editor may not intend. In this 160-string test, NFC changed 40 records and NFKC changed 128; 86 produced different simple search keys. Every second pass was stable. The result proves encoding behaviour—not linguistic correctness, safe rendering, or permission to rewrite multilingual text automatically.
The character you see is not always the string the machine sees
Two words can look identical on a page and still be represented by different sequences of code points. One may contain a precomposed letter; the other may combine a base letter with an accent. Japanese text may use full-width or half-width forms. Arabic presentation forms may resemble ordinary letters while carrying different underlying encodings. Punctuation that appears familiar may be a compatibility variant with different search behaviour.
Most readers should never have to care.
Search engines, publishing systems, note applications, localization pipelines, and accessibility tools do. They compare strings, split tokens, build indexes, measure duplicates, and transport text between systems. A visually acceptable example can fail to match another copy of itself—or be silently changed by a normalization step no editor has inspected.
The tempting response is to normalize everything. That is also unsafe.
Normalization can answer a technical question: which code-point sequences are canonically or compatibly related? It cannot answer an editorial question: which representation is correct for this language, variety, teaching purpose, quotation, or historical context?
This Field Lab measures the first question and refuses to smuggle in the second.
Research question: Across a frozen 160-string boundary-case dataset, what changes under NFC and NFKC, and which technically stable results still require editorial judgment?
The experiment
Baseline: visual inspection without a recorded normalization policy, transformation ledger, or second-pass stability test.
Protocol: The frozen dataset contains 160 explicitly enumerated code-point sequences:
- 40 Mandarin-related fixtures;
- 40 Spanish-related fixtures;
- 40 Japanese-related fixtures;
- 40 Arabic-related fixtures.
The fixtures are boundary cases, not a sample of ordinary prose. They deliberately concentrate the kinds of differences a multilingual publishing system ought to notice:
| Language set | Tested risks | |---|---| | Mandarin | Pinyin tone marks, width variants, punctuation and compatibility ideographs | | Spanish | Composed/decomposed diacritics, combining marks, ordinals and punctuation | | Japanese | Full-/half-width kana, dakuten, handakuten and compatibility symbols | | Arabic | Presentation forms, combining marks, Arabic-Indic digits and directional controls |
For every fixture, the runner recorded:
- the original sequence in
U+notation; - NFC output;
- NFKC output;
- whether a second normalization pass changed either result;
- whether lowercase NFC and lowercase NFKC produced the same simple search key;
- the exact transformation class;
- the presence of replacement characters, prohibited C0 controls, or explicit bidirectional controls.
Raw strings were not serialized into the result file. Potentially invisible characters are displayed through escaped code points so that the audit itself does not become a vehicle for hidden directional state.
Why compare NFC and NFKC?
NFC performs canonical composition.unicode-uax15, w3c-string-matching In simplified terms, it tries to give canonically equivalent sequences a stable composed representation. A Spanish á written as one precomposed character and the same visible letter written as a plus a combining acute accent can converge under NFC.
NFKC goes further.unicode-uax15, w3c-string-matching It applies compatibility mappings before composition. This can be useful for search because width variants and some presentation forms may converge. It can also erase distinctions that matter to typography, notation, historical fidelity, or a language-learning exercise.
The distinction is not “safe” versus “unsafe.” It is conservative canonical equivalence versus more aggressive compatibility folding.
The correct choice depends on the operation. A search index may want a compatibility-folded key while the displayed lesson preserves the authoritative original. An archival quotation may prohibit even a harmless-looking rewrite. A learner exercise about character width or code-point composition would be destroyed by normalizing away the very contrast it teaches.
Results: stability was easy; editorial meaning was not
The runner produced five central findings from the frozen raw record.unicode-160-datasetunicode-160-datasetunicode-160-dataset, w3c-string-matching
| Measure | Result | |---|---:| | Total fixtures | 160 | | Changed by NFC | 40 | | Changed by NFKC | 128 | | NFC and NFKC produced different simple search keys | 86 | | Failed second-pass idempotence | 0 |
There were no replacement characters and no prohibited C0 controls.unicode-160-dataset
That clean result should not be exaggerated. It means the frozen inputs were stable under this implementation and lacked two specified corruption markers. It does not mean the strings were grammatical, idiomatic, correctly translated, well rendered, or pedagogically useful.
The local runner deterministically reproduces the reported counts and idempotence results. Unicode and W3C sources define the normalization and string-matching boundary. These sources do not establish linguistic correctness, representative prevalence, safe rendering, or permission to overwrite source text.
Claim sources: unicode-160-dataset, unicode-uax15, w3c-string-matching
The transformation classes show where the real editorial decisions begin:
- 31 canonical compositions;
- 9 canonical reorderings;
- 20 width folds;
- 13 punctuation or space folds;
- 10 compatibility-ideograph folds;
- 10 ordinal folds;
- 15 kana width-and-voice folds;
- 10 compatibility-symbol expansions;
- 10 Arabic presentation-form folds;
- 32 records unchanged.
The totals are a map of this constructed test set, not estimates of prevalence in real multilingual corpora.
Four language-specific lessons
Mandarin: Pinyin and Han text fail differently
Pinyin tone marks create canonical-composition questions familiar from other Latin-script text: a visible marked vowel may arrive precomposed or as a base letter plus combining mark. NFC is designed to reconcile canonical equivalents.
Han compatibility ideographs raise a different issue. NFKC may map a compatibility character to a unified form. That can improve matching, yet “the strings now compare equally” does not establish that replacing the source character is appropriate in a quotation, historical document, name, or pedagogical explanation.
The publishing rule should therefore separate display from retrieval: preserve reviewed source text, and derive search keys in a traceable layer.
Spanish: an accent is linguistic information, not decorative noise
Canonical normalization can reconcile two encodings of the same accented letter without stripping the accent. That is valuable. A search pipeline that removes diacritics entirely is performing a different operation—one that Unicode normalization alone does not justify.
In Spanish, accent marks can carry lexical or grammatical information. The test therefore asks whether canonically equivalent encodings converge, not whether marked and unmarked words should be treated as equivalent for every purpose.
Compatibility ordinals and punctuation create another boundary. Folding a typographic ordinal indicator may help a broad search, while preserving it may matter to faithful display. Search convenience and editorial fidelity need not use the same representation.
Japanese: width is not merely a visual preference
NFKC changed full-/half-width kana and compatibility symbols in the fixture set, including sequences involving dakuten and handakuten. The resulting strings may be more useful as comparison keys, but the transformation can alter code-point length and representation.
That matters for cursor movement, annotation offsets, text alignment, educational highlighting, and any system that stores positions before normalization. A pipeline can “fix” matching while breaking references to the original text.
The safe sequence is to preserve the source, normalize a derived comparison key, and never apply offsets from one representation blindly to the other.
Arabic: normalization is not a bidirectional-security policy
Ten Arabic fixtures contained explicit bidirectional controls.unicode-160-dataset, w3c-string-matching NFC and NFKC preserved those controls.
This is the most important negative finding in the experiment. Normalization is not sanitization. A system that assumes normalized text is safe from directional-state problems has confused two different layers of Unicode processing.
Arabic-Indic digits also did not become European digits under NFC or NFKC. A product that wants cross-digit search equivalence needs an explicit policy; it cannot attribute that decision to Unicode normalization.
Presentation forms did fold under compatibility normalization in the constructed fixtures. Whether the displayed text should be rewritten remains an editorial and language-review question, not a mechanical consequence.
The search-key paradox
Eighty-six records produced different simple search keys depending on whether the pipeline began from NFC or NFKC.
That sounds like evidence for applying NFKC everywhere. It is not.
The experiment’s search key was deliberately simple: lowercase the normalized result and collapse Unicode whitespace. It did not remove accents, transliterate scripts, perform language-aware tokenization, resolve Arabic shaping, segment Chinese or Japanese, or test whether a reader would regard two results as equivalent.
The finding is narrower and more useful: normalization choice is part of search behaviour and must be made explicitly.
A credible multilingual system should retain at least three objects:
- the reviewed display string;
- a normalized comparison representation;
- the policy and software version that created it.
Collapsing all three into one field makes later correction almost impossible.
What would have counted as failure?
The protocol defined blocking technical findings before the run:
- a second normalization pass changing the first output;
- a replacement character indicating prior damage;
- a prohibited C0 control;
- a hidden directional control that the pipeline failed to expose;
- a transformed value being written back into source content without review.
No second-pass, replacement-character, or C0 failure occurred. Directional controls were found and remained visible in the escaped audit record. The runner did not write normalized text back into any article.
An uneventful test is still informative when the failure criteria were fixed in advance.
What the experiment cannot tell us
The dataset was deliberately engineered. It is not representative of Mandarin, Spanish, Japanese, Arabic, or multilingual publishing on the open web.
The experiment did not evaluate:
- grammar or translation;
- pronunciation or phonetic transcription;
- regional variety or register;
- font shaping and visual layout;
- grapheme-aware cursor behaviour;
- screen-reader output;
- collation or language-aware tokenization;
- whether a transformation is culturally or pedagogically appropriate.
Model review checked the frozen strings, scripts, labels, and claimed transformation boundaries. It does not certify grammatical, phonetic, cultural, or pedagogical correctness, none of which this normalization experiment attempted to measure.unicode-uax15, w3c-string-matching
The deepest lesson is not “use NFC” or “use NFKC.” It is that multilingual text has several legitimate identities at once: what the source contains, what the reader sees, what the search system compares, and what the editor is entitled to change.
A serious publishing system keeps those identities connected without pretending they are identical.
Raw data: Download the sanitized 160-string record. The negative finding is as important as the changes: the second normalization pass did not alter any fixture, and no replacement-character or prohibited-C0 failure appeared.
For the wider language boundary, read Why Native Speech Sounds So Fast and Can You Learn Two Languages at Once?. For the structural interpretation problem, compare The Multilingual Intelligence Shift.
Normalization is infrastructure: the craft lies in knowing which equivalences belong to machines and which distinctions belong to readers.
Named sources
Evidence and further reading
Published July 29, 2026. No substantive revision has been recorded. Evidence last verified July 28, 2026.