hex/ios/HEXKeyboard/TypingModel/fr.hextm
Ryan Vogel 73094b9bcb
iOS keyboard: French reads Hex's own word list and capitals, not the phone's dictionary (#2404)
* iOS keyboard: French (France) autocorrects with Hex's own model

Add Android's French typing model (fr.hextm, byte-identical to
android/app/src/main/assets/typing/fr.hextm and its SHA256SUMS hash) to the
keyboard's bundle. The existing model path (FullKeyboardSettings.typingModel ->
TypingDecoderModel.bundled) already maps any bundled <code>.hextm, so French
now reads taps with Spatial autocorrect's decoder, the likelier-next-letter
odds and the model's word pairs instead of UITextChecker's one-edit guesses.
No decoder logic changes.

One guard changed: with a bilingual partner, a word the partner's dictionary
knows as typed is no longer respelled with only accents
(AutocorrectEngine.cautiousReplacement, partner branch). French's model
otherwise turned English "these" into "thèse" on a French + English keyboard
(testABilingualKeyboardKeepsBothLanguagesWords). Slips the decoder reads as
another word still apply ("tkm" -> "Tom").

Held-out gate (2,000 Tatoeba sentences on AZERTY, scored as
android/benchmark/typing/quality-gate.py): fewer word errors than raw taps
37.8% -> 77.9%, correctly typed words changed 0.56% -> 0.10%, accents
restored where the plain spelling is no word 70.2% -> 86.9%, critical cases
1/2 -> 2/2. Real touches on the AZERTY keyboard in an iOS 27.2 simulator,
apostrophe key included, typed five sentences right. The live keyboard's
footprint with French active is unchanged (about 19 MB shown, 29.6 MB after
typing, against English's 30.4 MB); the model maps in under 0.4 ms.

Known gap: Canada's and Switzerland's French (fr-CA, fr-CH) have no model of
their own and keep UITextChecker. Settings shows "English, Spanish and French
only" under Spatial autocorrect.

Verified: xcodebuild test on the Studio (857 tests, 0 failures), new cases in
AutocorrectEngineTests and FullKeyboardEngineTests, code-shape.py with no
hard problems.

* iOS keyboard: French model's gate numbers from the replay on today's layout

The replay's taps now land on the nearest letter or space bar when the
synthetic typist's taller rows put them outside Hex's letter keys (on 123,
shift or delete with today's row metrics): 78.7% fewer word errors than raw
taps, against 37.4% with UITextChecker alone (was 77.9% / 37.8% before #2379's
row change). README only.

* iOS keyboard: French model's numbers as the gate on dev scores them

quality-gate.py on dev counts an accent word as unambiguous only when its
plain spelling is in neither the model nor the .valid list; by that rule
French restores 81.1% (78.7% without the model), short of the 85% bar. The
replay now types French's apostrophe on the AZERTY key (FrenchTyping.key),
not from the 123 page: 76.4% fewer word errors than raw taps (35.3% before).
README only.

* iOS keyboard: French reads Hex's own word list, not the phone's dictionary

Add HexWordList (ios/HEXShared/HexWordList.swift): a language's list of real
words outside its typing model (HEXKeyboard/TypingModel/<code>.valid, the
file Android ships, built by android/benchmark/typing/build-valid-words.py),
memory-mapped read-only and searched in place by bisection over its sorted
UTF-8 lines. No copy, no Set; lookups touch only clean pages.

Where the decoder's language has a bundled list, SystemTextChecker answers
isWordValid from the model and that list alone (HexOwnWords, set by
AutocorrectEngine.useOwnWords whenever the decoder changes), and its guesses
and completions count only while UITextChecker is in that language. This
covers both cases where the phone's dictionary is no use: none for the
language, and one that takes any letters for a word.

One guard changed with it (AutocorrectEngine.hasWords): a missing system
dictionary no longer switches off space corrections, the strip, the revision
of the word before and the merge when Hex's model is there. AutocorrectEngine
.swift keeps 1,498 lines (five lines changed in place).

Only French ships a list (fr.valid, 1.45 MB, Google Books v3, CC BY 3.0).
Held-out gate (ios/scripts/typing-gate.py, as quality-gate.py scores it):
French 76.3% -> 76.6% fewer errors than raw taps, correct words changed
0.10% -> 0.23%, unambiguous accents 81.1% -> 86.1% (passes 85%), critical
2/2. With their own lists English (0.27% -> 0.71% changed, fails), Spanish
(0.17% -> 0.36% changed) and German (0.12% -> 0.15%) got worse, so they keep
UITextChecker and ship no list.

Behaviour change: with a list, a name typed in lowercase that UITextChecker
knew only capitalised no longer takes its capital ("mary"), until a capitals
list reader exists.

Verified: xcodebuild test on the Studio (866 tests, 0 failures), new case in
AutocorrectEngineTests; live keyboard in an iOS 27.2 simulator types French
sentences right, footprint 29.8 MB after typing (fr.valid 272 KB resident,
0 dirty); code-shape.py no hard problems.

* iOS keyboard: names take their capital from French's own capitals list

Reading Hex's word list instead of UITextChecker (previous commit) cost
French its names: a name typed in lowercase ("paris", "tom") no longer got
its capital. The gate ignores case, so it could not see that.

HexWordList now also reads <code>.capitals (Android's build-capitals.py
output, the same sorted format, mapped and searched in place) and gives a
line's number (value(of:)). With a list, HexOwnWords takes a word the books
capitalise at least 95% of the time as no word when typed all lowercase
("paris" -> "Paris", as UITextChecker's "only capitalised" rule does), and a
listed name outside the model as a word. A word the books capitalise 50-94%
of the time ("tom", "mary": also ordinary words, capped by the builder) is
a name where the phone's own dictionary for the language knows it only
capitalised, and only when that dictionary tells words from other letters
(one "qzxjvkq" probe per language): a UITextChecker call for those words
alone.

GermanCapitals reads de.capitals through the same reader instead of a
55,000-entry dictionary in memory; its rules are unchanged (German gate
83.6% / 0.12% / 2/2).

ios/scripts/typing-gate.py reports capitals, which its three numbers can't
see: every reference typed all lowercase, then, after a sentence's first
word, the reference words with a capital that keep it and the words
without that gain one. French: names kept 45.4% on UITextChecker, 0.0%
with the list alone, 47.5% with the list and fr.capitals; capitals added
0.01% / 0.00% / 0.00%. The gate's numbers hold: 76.6% fewer errors than
raw taps, 0.23% correct words changed, 86.1% unambiguous accents, 2/2.

Ships fr.capitals (272 KB, Google Books v3 counts, CC BY 3.0).

Verified: xcodebuild test on the Studio (884 tests, 0 failures),
code-shape.py no hard problems; AutocorrectEngine.swift unchanged.
2026-11-22 13:34:40 -05:00

3.4 MiB