mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* Android typing: the likelier next letter claims a touch on its neighbour's edge
Ports iOS #2091's next-letter odds (FullKeyboardEngine.key(at:)): a touch
within 5 dp of a neighbouring letter's touch area presses that letter instead
when the word list makes it enough likelier to come next ("th" then a touch
on the e/d border types e; "an" then the same touch types d). Each dp claimed
costs 0.8 nats of odds; nothing changes inside a number or other non-word,
and an accented letter without a key counts as its base letter's.
- TypingModel.continuations(prefix): each letter's share of the word list's
frequency after a prefix, read from the mapped model (the word start's and
the last prefix's kept).
- TypingWord.likelierKey(before, hit, near, keys): the key to press, given the
hit key, the neighbouring letters within reach with their distances in dp,
and the keyboard's letters. The keyboard decides when it applies (autocorrect
on, an ordinary text field, no caps lock), as iOS does.
- The replay applies it to every letter tap, with touch areas tiling the
rows of the benchmark layout.
Verified on the Studio (API 35 emulator, installed debug APK, 2,000 held-out
sentences each), word errors before (dev be78a0a4) -> after:
- English 770 (5.55%) -> 721 (5.20%); today's iOS 798 (5.75%)
- Spanish 1,133 (8.52%) -> 1,090 (8.19%); today's iOS 1,205 (9.06%)
- German 774 (5.71%) -> 729 (5.38%)
- Bokmål 654 (5.24%) -> 647 (5.18%)
Sentences better/worse: 93/45, 72/34, 87/45, 47/40 (the typed letters
change, so sentences move both ways; every set is better on balance).
Strip hits after two letters rise in all four (English 4,390 -> 4,686).
testDebugUnitTest: 36 tests, 0 failures, with iOS's next-letter case
re-expressed.
* Android typing: French and Italian word models
Adds French and Italian keyboard models from Google Books Ngram v3 (CC BY
3.0), built like German's: every untagged shard of the language (fre 6/115,
ita 2/60), 1980-2019, spellings from the corpus alone (Hunspell fr is MPL,
it GPL, so no dictionary is imported), 100,000 words a model in the same 3.6
MB HXTM v3 format, and each with a capitals list (fr.capitals, it.capitals)
from the same 1-grams.
Elisions:
- The books keep many as one token ("c'est", "qu'il", "dell'anno"; French
"l'" and "d'" are split off), so those stay words with their apostrophe,
read without it being typed. The same letters run together that are ten
times rarer than the apostrophe form ("cest", "quil", "jai"; 26 French, 4
Italian forms) are left out of the vocabulary, so space puts it back.
- TypingWord.startsWord(before): a word typed right after an elided article
or pronoun ("l'", "qu'", "dell'") is one the keyboard reads, so "l'hpmme"
becomes "l'homme". previousWord takes an elided word without its
apostrophe.
- In French and Italian a word whose accented spelling is a commoner model
word is never capitalised from the list ("ecole" isn't "Ecole": books drop
accents on capitals).
Safety and credits:
- never-suggest.tsv gains curated French (31 slur, 107 swear) and Italian (30
slur, 112 swear) entries from LDNOOBW 5faf2ba (CC BY 4.0), as the other
languages were: identity slurs always suppressed, profanity opt-in, ordinary
senses kept (pédale, finocchio, scopare, chatte stay available), common
inflections and missing slurs added.
- credits.txt and build.gradle.kts's verified asset list name the new files.
Benchmark: synthetic_de_nb.py is now synthetic_sets.py and also writes
French (AZERTY, the apostrophe typed on its key) and Italian (QWERTY, the
apostrophe an exact tap) sets from held-out Tatoeba sentences; the replay
follows the keyboard (an apostrophe ends the word as typed; a letter after
one starts a read word only after an elision).
Verified on the Studio (API 35 emulator, installed debug APK, 2,000 held-out
sentences each): French raw taps 5,316 (38.91%) -> 1,177 (8.62%), Italian
4,778 (39.73%) -> 895 (7.44%). English 721, Spanish 1,090, German 729,
Bokmål 647 unchanged. The keyboard has no French or Italian layout yet, so
none of this reaches a phone until it does. testDebugUnitTest: 37 tests, 0
failures; the never-suggest cases now run on all six models, plus an
elision case.
* Android typing: French and Italian accents the words around decide
French and Italian have pairs of words spelled with the same letters, with
and without an accent, that are both words ("a"/"à", "ou"/"où", "la"/"là",
"ca"/"ça", "du"/"dû", "sur"/"sûr"; Italian "e"/"è", "la"/"là", "li"/"lì",
"da"/"dà", "ne"/"né", "se"/"sé"). Typed without the accent, the decoder
keeps the word as typed, as iOS does with Apple's dictionary, which never
changes one word into another.
ContextWords now has those groups beside English's and chooses between them
with the word pairs, exactly as for English apostrophes (#2144): the word
before decides at the word's own space when it says so clearly, otherwise
the next word revises it ("je vais a paris" -> "je vais à Paris", "il a
faim" stays; "non e esagerato" -> "non è esagerato", "pane e burro" stays).
The accent counts as same-letter change (3 / 1.5 nat margins). English is
unchanged.
Verified on the Studio (API 35 emulator, installed debug APK, 2,000 held-out
sentences each): French 1,177 (8.62%) -> 947 (6.93%) word errors, 231
sentences better and 13 worse; Italian 895 (7.44%) -> 789 (6.56%), 105
better and none worse. English 721, Spanish 1,090, German 729 and Bokmål
647 unchanged. testDebugUnitTest: 38 tests, 0 failures, with French and
Italian cases beside the English ones.
149 lines
7.1 KiB
Python
149 lines
7.1 KiB
Python
"""Synthetic German, Bokmål, French and Italian typing sets for the Android replay, made the way
|
|
2026-11-20-hex-typing-logs/bench_make.py made the English and Spanish ones: held-out Tatoeba
|
|
sentences (CC BY 2.0 FR), each tap sampled from the measured per-key offset and per-row spread
|
|
in that benchmark's input.json meta (no log rows), with 2% of spaces missed and space taps that
|
|
land on a letter typed as that letter. A letter AZERTY moves takes its row's average offset. Keys, with the English rows' heights and space bar on a 402 pt layout:
|
|
- German and Bokmål: Android's rows (qwertzuiopü / asdfghjklöä / yxcvbnm, qwertyuiopå /
|
|
asdfghjkløæ / zxcvbnm), eleven keys a row (36 pt pitch);
|
|
- French: AZERTY (azertyuiop / qsdfghjklm / wxcvbn and an apostrophe key, never typed), and
|
|
Italian: QWERTY, at English's 39.5 pt pitch.
|
|
Letters without a key are typed as their base key (é with e), hyphens as spaces. Elisions keep
|
|
their apostrophe: French types it on AZERTY's apostrophe key like a letter; Italian, which has
|
|
none on its letters, as an exact tap from the 123 page (written as [0, 0]; the replay only reads
|
|
that an apostrophe was typed). Sentences with letters no key gives (German ß, French œ, æ)
|
|
are left out.
|
|
|
|
Writes OUTDIR/<lang>.jsonl (id, intended, typed, taps, want, ends, ios_shared="") and
|
|
<lang>-dev.jsonl for de, nb, fr and it. TATOEBA holds deu.tsv, nob.tsv, fra.tsv and ita.tsv
|
|
from https://downloads.tatoeba.org/exports/per_language/.
|
|
Usage: python3 synthetic_sets.py INPUT.json TATOEBA OUTDIR"""
|
|
import json, os, random, re, sys, unicodedata
|
|
|
|
meta = json.load(open(sys.argv[1]))["meta"]
|
|
tatoeba, out_dir = sys.argv[2], sys.argv[3]
|
|
OFFSET = {k: tuple(v) for k, v in meta["offsets"].items()}
|
|
ROW_SD = {int(k): tuple(v) for k, v in meta["row_sd"].items()}
|
|
SPACE_MU, SPACE_SD = meta["space"][:2], meta["space"][2:]
|
|
SPACE_OMIT = meta["space_omit"]
|
|
ROW_Y, TOP = [66.5, 122.5, 178.5], 44.5
|
|
SPACE = (64.76, 212.0, 238.34, 45.0)
|
|
ROWS = {"de": ["qwertzuiopü", "asdfghjklöä", "yxcvbnm"], "nb": ["qwertyuiopå", "asdfghjkløæ", "zxcvbnm"],
|
|
"fr": ["azertyuiop", "qsdfghjklm", "wxcvbn'"], "it": ["qwertyuiop", "asdfghjkl", "zxcvbnm"]}
|
|
ALPHABET = {"de": "abcdefghijklmnopqrstuvwxyzäöü", "nb": "abcdefghijklmnopqrstuvwxyzæøåé",
|
|
"fr": "abcdefghijklmnopqrstuvwxyzàâçéèêëîïôùûüÿ", "it": "abcdefghijklmnopqrstuvwxyzàèéìíîòóùú"}
|
|
PITCHES = {"de": 36.0, "nb": 36.0, "fr": 39.5, "it": 39.5}
|
|
CODES = {"de": "deu", "nb": "nob", "fr": "fra", "it": "ita"}
|
|
SEEDS = {"de": (11, 20261121), "nb": (13, 20261122), "fr": (17, 20261123), "it": (19, 20261124)}
|
|
# Elisions keep their apostrophe in the sentence, which isn't typed.
|
|
ELIDES = {"fr", "it"}
|
|
EN_ROWS = ["qwertyuiop", "asdfghjkl", "zxcvbnm"]
|
|
|
|
|
|
def geometry(lang):
|
|
pitch = PITCHES[lang]
|
|
keys = {}
|
|
for r, letters in enumerate(ROWS[lang]):
|
|
for i, c in enumerate(letters):
|
|
x = 21 + pitch * i if len(letters) == 11 else 201 + (i - (len(letters) - 1) / 2) * pitch
|
|
keys[c] = (x, ROW_Y[r], r)
|
|
return keys
|
|
|
|
|
|
def same_place(lang, c):
|
|
"""Whether `c` sits where QWERTY has it, so its measured offset applies."""
|
|
for r, letters in enumerate(ROWS[lang]):
|
|
if c in letters:
|
|
return r < len(EN_ROWS) and c in EN_ROWS[r] and EN_ROWS[r].index(c) == letters.index(c)
|
|
return False
|
|
|
|
|
|
def row_mean(row):
|
|
letters = [c for c in EN_ROWS[row] if c in OFFSET]
|
|
return (sum(OFFSET[c][0] for c in letters) / len(letters), sum(OFFSET[c][1] for c in letters) / len(letters))
|
|
|
|
|
|
def hit(keys, lang, x, y):
|
|
if y >= ROW_Y[2] + 28:
|
|
return "space" if SPACE[0] <= x < SPACE[0] + SPACE[2] else None
|
|
row = 0 if y < ROW_Y[0] + 28 else 1 if y < ROW_Y[1] + 28 else 2
|
|
letters = ROWS[lang][row]
|
|
xs = [keys[c][0] for c in letters]
|
|
if row == 2 and (x < xs[0] - PITCHES[lang] / 2 or x >= xs[-1] + PITCHES[lang] / 2):
|
|
return None
|
|
return min(letters, key=lambda c: abs(keys[c][0] - x))
|
|
|
|
|
|
def fold(lang, c):
|
|
if c in "".join(ROWS[lang]):
|
|
return c
|
|
return unicodedata.normalize("NFD", c)[0]
|
|
|
|
|
|
def sample_letter(rng, keys, lang, ch):
|
|
cx, cy, row = keys[ch]
|
|
moved = lang == "fr" and not same_place(lang, ch)
|
|
ox, oy = row_mean(row) if moved or ch not in OFFSET else OFFSET[ch]
|
|
sx, sy = ROW_SD[row]
|
|
for _ in range(20):
|
|
x, y = rng.gauss(cx + ox, sx), max(rng.gauss(cy + oy, sy), TOP)
|
|
k = hit(keys, lang, x, y)
|
|
if k is not None:
|
|
return x, y, k
|
|
return cx, cy, ch
|
|
|
|
|
|
def sample_space(rng, keys, lang, force):
|
|
for _ in range(20):
|
|
x, y = rng.gauss(SPACE_MU[0], SPACE_SD[0]), rng.gauss(SPACE_MU[1], SPACE_SD[1])
|
|
k = hit(keys, lang, x, y)
|
|
if k == "space" or (not force and k is not None):
|
|
return x, y, k
|
|
return SPACE[0] + SPACE[2] / 2, SPACE[1] + SPACE[3] / 2, "space"
|
|
|
|
|
|
def sentences(lang):
|
|
code = CODES[lang]
|
|
word = re.compile(f"^[{ALPHABET[lang]}]+('[{ALPHABET[lang]}]+)?$" if lang in ELIDES else f"^[{ALPHABET[lang]}]+$")
|
|
rows = []
|
|
for line in open(os.path.join(tatoeba, f"{code}.tsv"), encoding="utf-8"):
|
|
parts = line.rstrip("\n").split("\t")
|
|
if len(parts) < 3:
|
|
continue
|
|
text = unicodedata.normalize("NFC", parts[2])
|
|
if lang in ELIDES:
|
|
text = text.replace("\u2019", "'")
|
|
elif "'" in text or "\u2019" in text:
|
|
continue
|
|
ws = re.sub(r"[.,!?;:\"()\u201e\u201c\u201d\u201a\u2018\u00bb\u00ab\u2013\u2014\-]", " ", text).lower().split()
|
|
if 3 <= len(ws) <= 14 and all(word.match(w) for w in ws):
|
|
rows.append(ws)
|
|
rng = random.Random(SEEDS[lang][0])
|
|
rng.shuffle(rows)
|
|
return rows[:2300]
|
|
|
|
|
|
for lang in ("de", "nb", "fr", "it"):
|
|
keys = geometry(lang)
|
|
rng = random.Random(SEEDS[lang][1])
|
|
rows = sentences(lang)
|
|
for name, part in (("dev", rows[:300]), ("test", rows[300:2300])):
|
|
path = os.path.join(out_dir, lang + (".jsonl" if name == "test" else "-dev.jsonl"))
|
|
with open(path, "w") as f:
|
|
for j, ws in enumerate(part):
|
|
taps, typed, want, ends = [], [], [], []
|
|
for i, w in enumerate(ws):
|
|
for ch in w:
|
|
if ch == "'" and lang == "it":
|
|
taps.append([0, 0]); typed.append("'"); want.append("'")
|
|
continue
|
|
x, y, k = sample_letter(rng, keys, lang, fold(lang, ch))
|
|
taps.append([round(x, 2), round(y, 2)]); typed.append(" " if k == "space" else k); want.append(ch)
|
|
ends.append(len(taps))
|
|
last = i == len(ws) - 1
|
|
if not last and rng.random() < SPACE_OMIT:
|
|
continue
|
|
x, y, k = sample_space(rng, keys, lang, last)
|
|
taps.append([round(x, 2), round(y, 2)]); typed.append(" " if k == "space" else k); want.append(" ")
|
|
f.write(json.dumps({"id": f"{lang}-{name}-{j}", "intended": " ".join(ws), "typed": "".join(typed),
|
|
"taps": taps, "want": "".join(want), "ends": ends, "ios_shared": ""}, ensure_ascii=False, separators=(",", ":")) + "\n")
|
|
print(lang, len(rows), "sentences")
|