hex/android/benchmark/typing/held-out
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Ryan Vogel 34d0a739c8
iOS keyboard: reproducible Polish typing model build, Polish held-out set, Polish credits (no model shipped) (#2401)
* Typing gate: a Polish held-out set beside the other six

pl.tsv.gz holds 2,000 Tatoeba sentences (10,735 words, 4,071 with a Polish letter) typed
with the same touch model as German's and Italian's on Hex's iOS Polish rows (QWERTY at
39.5 pt), Polish's letters typed as their base key (each is held on it; ł as l), in
held-out/'s format with each sentence's Tatoeba ID and contributor. polish_set.py makes it
(gzip without a name or time, so a rerun on the same export gives the same bytes; checked:
SHA-256 a2b14929... twice). quality-gate.py gates only languages with a model and a set, so
Android's gate is unchanged (de, en, es, fr, it, nb); Polish's accent keys and critical
cases are in the README until a model joins them.

Measured on the iOS engine (experiment, not in this repository's tests): 84.3% fewer errors
than raw taps with Hex's own Polish model and word lists, -1.8% without a model.

* iOS: Polish data credits, shown once the keyboard bundles pl.hextm

Settings › Acknowledgements gains NKJP's n-grams (CC BY, its page names no version),
ParlaMint 5.0 (CC BY 4.0) and PoliMorf, with PoliMorf's 2-clause BSD notice in full as
its licence asks of binary distribution. They show only when HEXKeyboard.appex holds
pl.hextm (AcknowledgementsView.shown), which no build does yet, so today's screen is
unchanged; ios/README.md lists them the same way.

* iOS keyboard: build Polish's typing model with the shared builder, from pinned NKJP, ParlaMint and PoliMorf inputs

build-typing-model.py pl <work> pl.hextm [nkjp+parlamint|nkjp|parlamint] builds the Polish
model that passed the typing gate on the iOS engine. Polish has no Google Books counts, so:
- words and pairs come from NKJP's n-grams (300M-word balanced subcorpus), kept where PoliMorf
  lists the word;
- a plain spelling goes when NKJP or ParlaMint 5.0 (which writes every mark) has its marked twin
  at least 20 times as often ('ide' is 3.6 times rarer than 'idę' in NKJP, 97 times in
  ParlaMint); each corpus alone failed the gate (ParlaMint changed 0.84% of correct words, NKJP
  kept 'ide');
- a pair may span a comma ('wiem, że' is typed 'wiem że'); pairs need 10 sightings in NKJP;
- ł's base letter is l (BASES), as é's is e;
- it also writes <output>.valid, PoliMorf's other words NKJP uses at least twice, in Android's
  validity-list format, for the dictionary reader Polish needs on iOS.
POLISH_INPUTS names each input's URL, licence and SHA-256; the build refuses a changed file.
English and Spanish take the same code path as before; write_model's pair threshold became a
parameter whose default is the old MIN_PAIR.

Rebased onto dev 06ac30af with #2408's English contractions and extra words; both kept.

Verified: Polish rebuilds the gated model byte for byte (SHA-256 10da84ee..., 150,000 words,
184,327 pairs, the same 525,084 validity words). English and Spanish: see the pull request.
2026-11-22 14:28:54 -05:00
..
accent-de.tsv
accent-nb.tsv
de.tsv.gz
en.tsv.gz
es.tsv.gz
fr.tsv.gz
it.tsv.gz
nb.tsv.gz
pl.tsv.gz