hex/ios/HEXKeyboard/TypingModel/en.hextm
Ryan Vogel 3f35348581
English typing model: contractions as common as their words, and Hex's chat and tech words (#2408)
What changed: en.hextm (iOS and Android, one file) is rebuilt by build-typing-model.py from
the same Google Books shards and SCOWL, with two additions:
- Each contraction in the builder's CONTRACTIONS counts as often as its expansion's word pair
  ("do not") when that is commoner, and "i'm", "i'll", "i've", "i'd", "here's", "where's",
  "mustn't" and "needn't" are added in lowercase.
- en-extra-words.tsv (new, beside both copies of the builder): 166 chat and tech words written
  for Hex ("app", "lol", "csv"), each at its tier's cost (10, 12 or 13.5 nats); a word the
  books already have only gets commoner.
Both join the vocabulary before pairs are counted, so their pairs are counted too. en.valid and
en.capitals (Android) are rebuilt from the new model, and build-capitals.py holds Hex's own
words at 94% like SCOWL's lowercase words ("plz" and "okie" are capitalised in books, never by
the keys). SHA256SUMS and the benchmark and iOS READMEs follow.

Why: Google Books splits "don't" into "do" + "n't", so the model priced "don't" at 14.4 nats
and "it's" at 16.2, and lacked "i'm"; a rarer reading beat them. Recorded typing logs had
everyday words the model lacked ("app", "csv", "gpus"), and the next-letter odds moved
correct taps off them.

Reproduced: with the earlier rules the same counts rebuild the earlier en.hextm byte for byte
(14370569...4028), and the valid and capitals builders rebuild the earlier en.valid and
en.capitals from it. New en.hextm c23e9c54...df30: 86,247 words, 417,182 pairs, 3,599,995
bytes (inside the 3.6 MB budget).

Verified:
- iOS gate (scripts/typing-gate.py, SWIFT_DETERMINISTIC_HASHING=1): English 773 -> 734 word
  errors (5.57% -> 5.29%, 83.5% -> 84.4% fewer than raw taps), correct words changed 0.27% ->
  0.26%, critical 2/2, PASS. Spanish and German outputs identical.
- Android gate (quality-gate.py): English 718 -> 680 (5.18% -> 4.90%, 84.7% -> 85.5%), correct
  words changed 0.32% both, critical 2/2, PASS; es, de, fr, it, nb identical.
- Per letter on the iOS engine (held-out inputs, p50/p95): 0.189/2.18 -> 0.181/2.08 ms.
- Swipes, every word of the English held-out sentences through the key plane and SwipeTyping:
  92.38% -> 92.46% clean, 89.83% -> 89.89% with finger noise; one word lost in each ("weed"
  now "we'd"). Existing swipe tests unchanged and passing.
- A tester's recorded taps, single words: 37 -> 34 errors, correctly tapped words changed 12 -> 8.
- xcodebuild test (Studio, own simulator): 882 tests, 0 failures, 4 opt-in skips. Android
  testDebugUnitTest (66, incl. NeverSuggestTest on every shipped model): pass. code-shape.py: pass.
2026-11-22 13:40:28 -05:00

3.4 MiB