Commit Graph

4 Commits

Author SHA1 Message Date
Arnold Wender 854276c159 fix(encoding): require local evidence before repairing the ambiguous [ÂÃ] window
repair_mojibake rewrote clean text. The [ÂÃ][continuation] alternative of
_HIGH_CONFIDENCE_RUN is only a two-character window, and Portuguese, Vietnamese
and Turkish end all-caps words in Ã/Â — IRMÃ, MAÇÃ, MANHÃ, BÃO, NHÃ, HÂLÂ, IMÂ.
Prose then follows with a closing quote, guillemet, ellipsis or dash, and all of
those are in the continuation class, so "«MAÇû." and "“IRMÔ" matched a shape
that is also perfectly clean text. Each hit collapsed two characters into one
and was reported as a successful repair.

Three failures came out of the same root cause:

* clean prose destroyed — "«MAÇû." became "«MAÇû.", "“IRMÔ" became "“IRMÔ";
* visible text swapped for invisible C1 controls, because  plus 0x80-0x9F
  decodes to exactly that block, so "İMÂ… edildi." lost its ellipsis to U+0085;
* the max_passes loop was not idempotent: genuinely mojibaked Portuguese was
  repaired correctly on pass 1 and then destroyed on pass 2.

The fix follows the precedent already set in this module, where Ä/Å were
excluded as leads because "Ų can be legitimate scientific text" — the same
argument, applied to the continuation side. A lone [ÂÃ] window ending in
typographic punctuation is no longer high-confidence, so it is repaired only
when a lowercase letter runs directly into the lead ("coûte", "Noël"), which
is where mojibake actually occurs — inside a word. Chained units ("ação") and
the 3-4 character â/ð/ï windows are unaffected. A decode product containing a
C0/C1 control is now always refused.

Corroboration is deliberately local. Inferring "this drawer is mojibake" from a
run elsewhere in the text destroys clean prose in a mixed drawer, and drawers
are mixed by construction since the miner concatenates several sources.

The trade is asymmetric and chosen on purpose: an unrepaired string is unchanged
and can be repaired later, while a wrong repair is silent and irrecoverable.
2026-08-10 09:36:18 +02:00
Sandro da Silva 67dcaf3164 fix(encoding): recover undefined CP1252 continuation bytes 2026-08-03 20:05:40 +00:00
Sandro da Silva 44977537e5 fix(encoding): make repair conservative and reversible 2026-08-03 15:47:59 +00:00
Sandro da Silva c56f8b30bb fix(windows): add legacy encoding repair tool 2026-08-03 09:56:02 +00:00