mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* Android typing: the likelier next letter claims a touch on its neighbour's edge
Ports iOS #2091's next-letter odds (FullKeyboardEngine.key(at:)): a touch
within 5 dp of a neighbouring letter's touch area presses that letter instead
when the word list makes it enough likelier to come next ("th" then a touch
on the e/d border types e; "an" then the same touch types d). Each dp claimed
costs 0.8 nats of odds; nothing changes inside a number or other non-word,
and an accented letter without a key counts as its base letter's.
- TypingModel.continuations(prefix): each letter's share of the word list's
frequency after a prefix, read from the mapped model (the word start's and
the last prefix's kept).
- TypingWord.likelierKey(before, hit, near, keys): the key to press, given the
hit key, the neighbouring letters within reach with their distances in dp,
and the keyboard's letters. The keyboard decides when it applies (autocorrect
on, an ordinary text field, no caps lock), as iOS does.
- The replay applies it to every letter tap, with touch areas tiling the
rows of the benchmark layout.
Verified on the Studio (API 35 emulator, installed debug APK, 2,000 held-out
sentences each), word errors before (dev be78a0a4) -> after:
- English 770 (5.55%) -> 721 (5.20%); today's iOS 798 (5.75%)
- Spanish 1,133 (8.52%) -> 1,090 (8.19%); today's iOS 1,205 (9.06%)
- German 774 (5.71%) -> 729 (5.38%)
- Bokmål 654 (5.24%) -> 647 (5.18%)
Sentences better/worse: 93/45, 72/34, 87/45, 47/40 (the typed letters
change, so sentences move both ways; every set is better on balance).
Strip hits after two letters rise in all four (English 4,390 -> 4,686).
testDebugUnitTest: 36 tests, 0 failures, with iOS's next-letter case
re-expressed.
* Android typing: French and Italian word models
Adds French and Italian keyboard models from Google Books Ngram v3 (CC BY
3.0), built like German's: every untagged shard of the language (fre 6/115,
ita 2/60), 1980-2019, spellings from the corpus alone (Hunspell fr is MPL,
it GPL, so no dictionary is imported), 100,000 words a model in the same 3.6
MB HXTM v3 format, and each with a capitals list (fr.capitals, it.capitals)
from the same 1-grams.
Elisions:
- The books keep many as one token ("c'est", "qu'il", "dell'anno"; French
"l'" and "d'" are split off), so those stay words with their apostrophe,
read without it being typed. The same letters run together that are ten
times rarer than the apostrophe form ("cest", "quil", "jai"; 26 French, 4
Italian forms) are left out of the vocabulary, so space puts it back.
- TypingWord.startsWord(before): a word typed right after an elided article
or pronoun ("l'", "qu'", "dell'") is one the keyboard reads, so "l'hpmme"
becomes "l'homme". previousWord takes an elided word without its
apostrophe.
- In French and Italian a word whose accented spelling is a commoner model
word is never capitalised from the list ("ecole" isn't "Ecole": books drop
accents on capitals).
Safety and credits:
- never-suggest.tsv gains curated French (31 slur, 107 swear) and Italian (30
slur, 112 swear) entries from LDNOOBW 5faf2ba (CC BY 4.0), as the other
languages were: identity slurs always suppressed, profanity opt-in, ordinary
senses kept (pédale, finocchio, scopare, chatte stay available), common
inflections and missing slurs added.
- credits.txt and build.gradle.kts's verified asset list name the new files.
Benchmark: synthetic_de_nb.py is now synthetic_sets.py and also writes
French (AZERTY, the apostrophe typed on its key) and Italian (QWERTY, the
apostrophe an exact tap) sets from held-out Tatoeba sentences; the replay
follows the keyboard (an apostrophe ends the word as typed; a letter after
one starts a read word only after an elision).
Verified on the Studio (API 35 emulator, installed debug APK, 2,000 held-out
sentences each): French raw taps 5,316 (38.91%) -> 1,177 (8.62%), Italian
4,778 (39.73%) -> 895 (7.44%). English 721, Spanish 1,090, German 729,
Bokmål 647 unchanged. The keyboard has no French or Italian layout yet, so
none of this reaches a phone until it does. testDebugUnitTest: 37 tests, 0
failures; the never-suggest cases now run on all six models, plus an
elision case.
* Android typing: French and Italian accents the words around decide
French and Italian have pairs of words spelled with the same letters, with
and without an accent, that are both words ("a"/"à", "ou"/"où", "la"/"là",
"ca"/"ça", "du"/"dû", "sur"/"sûr"; Italian "e"/"è", "la"/"là", "li"/"lì",
"da"/"dà", "ne"/"né", "se"/"sé"). Typed without the accent, the decoder
keeps the word as typed, as iOS does with Apple's dictionary, which never
changes one word into another.
ContextWords now has those groups beside English's and chooses between them
with the word pairs, exactly as for English apostrophes (#2144): the word
before decides at the word's own space when it says so clearly, otherwise
the next word revises it ("je vais a paris" -> "je vais à Paris", "il a
faim" stays; "non e esagerato" -> "non è esagerato", "pane e burro" stays).
The accent counts as same-letter change (3 / 1.5 nat margins). English is
unchanged.
Verified on the Studio (API 35 emulator, installed debug APK, 2,000 held-out
sentences each): French 1,177 (8.62%) -> 947 (6.93%) word errors, 231
sentences better and 13 worse; Italian 895 (7.44%) -> 789 (6.56%), 105
better and none worse. English 721, Spanish 1,090, German 729 and Bokmål
647 unchanged. testDebugUnitTest: 38 tests, 0 failures, with French and
Italian cases beside the English ones.
204 lines
9.3 KiB
Python
204 lines
9.3 KiB
Python
#!/usr/bin/env python3
|
||
"""Build German, French, Italian and Bokmål HXTM v3 files with the shared iOS count reader / model writer.
|
||
python3 build-layout-model.py {de|fr|it|nb} SOURCE_CACHE OUTPUT.hextm
|
||
German, French and Italian: every untagged Google Books v3 shard of the language, 1980–2019,
|
||
spellings from the corpus alone (no second dictionary). French and Italian keep elided forms
|
||
as the books' tokens have them ("c'est", "l'homme", "dell'anno"), the apostrophe only between
|
||
letters. Bokmål: NB N-gram 2026 digibok, language nob, same years, filtered by Norsk ordbank
|
||
full forms.
|
||
"""
|
||
import collections
|
||
import concurrent.futures
|
||
import csv
|
||
import importlib
|
||
import hashlib
|
||
import http.client
|
||
import io
|
||
import json
|
||
import re
|
||
import subprocess
|
||
import sys
|
||
import unicodedata
|
||
import urllib.request
|
||
import shutil
|
||
import time
|
||
import tarfile
|
||
from pathlib import Path
|
||
|
||
builder = importlib.import_module("build-typing-model")
|
||
builder.FILES["de"] = ("ger", 8, 181)
|
||
builder.LETTERS["de"] = "abcdefghijklmnopqrstuvwxyzäöüß"
|
||
builder.MAX_WORDS["de"] = 95_000
|
||
builder.FILES["fr"] = ("fre", 6, 115)
|
||
builder.LETTERS["fr"] = "abcdefghijklmnopqrstuvwxyzàâæçéèêëîïôœùûüÿ'"
|
||
builder.MAX_WORDS["fr"] = 100_000
|
||
builder.FILES["it"] = ("ita", 2, 60)
|
||
builder.LETTERS["it"] = "abcdefghijklmnopqrstuvwxyzàèéìíîòóùú'"
|
||
builder.MAX_WORDS["it"] = 100_000
|
||
# German, French and Italian use only the corpus, without importing another spelling
|
||
# dictionary. The iOS reader lowercases before adding counts; keep the same alphabet rule
|
||
# on Unicode scalars before the shared writer assigns their compact codes. An apostrophe
|
||
# stands only between letters (an elision), never alone or at a word's edge.
|
||
CORPUS_WORDS = {
|
||
"de": r"[a-zäöüß]{1,32}\Z",
|
||
"fr": r"(?=.{1,32}\Z)[a-zàâæçéèêëîïôœùûüÿ]+(?:'[a-zàâæçéèêëîïôœùûüÿ]+)?\Z",
|
||
"it": r"(?=.{1,32}\Z)[a-zàèéìíîòóùú]+(?:'[a-zàèéìíîòóùú]+)?\Z",
|
||
}
|
||
def corpus_words(lang, work):
|
||
return re.compile(CORPUS_WORDS[lang]).fullmatch
|
||
builder.word_list = corpus_words
|
||
|
||
# The articles and pronouns French and Italian elide before a vowel ("c'est", "dell'anno").
|
||
ELISIONS = {
|
||
"fr": ["c", "d", "j", "l", "m", "n", "s", "t", "qu", "jusqu", "lorsqu", "puisqu", "quoiqu", "presqu"],
|
||
"it": ["c", "d", "l", "m", "n", "s", "t", "v", "un", "all", "dall", "dell", "nell", "sull", "coll",
|
||
"quest", "quell", "bell", "sant", "tutt", "nessun", "buon"],
|
||
}
|
||
|
||
def run_together(word, vocab, elisions):
|
||
"""Whether `word` is an elision typed without its apostrophe ("cest", "quil", "dellanno"): the
|
||
same letters with the apostrophe are a model word at least ten times as common. Scanned books
|
||
and typists both drop it; as a word of its own it would keep the keyboard from putting it back."""
|
||
if "'" in word:
|
||
return False
|
||
return any(word.startswith(p) and len(word) > len(p) and vocab.get(p + "'" + word[len(p):], 0) >= 10 * vocab[word]
|
||
for p in elisions)
|
||
|
||
def refine(lang, work, out):
|
||
"""Rewrites the model without run-together elisions, from the counts the shared builder cached."""
|
||
work = Path(work)
|
||
total = 0
|
||
for path in sorted((work / "uni").glob("*.tsv")):
|
||
for line in open(path, "rb"):
|
||
total += int(line.rstrip(b"\n").split(b"\t")[1])
|
||
vocab = {}
|
||
for line in open(work / "vocab.tsv", encoding="utf-8"):
|
||
w, c = line.rstrip("\n").split("\t")
|
||
vocab[w] = int(c)
|
||
dropped = sorted(w for w in vocab if run_together(w, vocab, ELISIONS[lang]))
|
||
for w in dropped:
|
||
del vocab[w]
|
||
pairs = {}
|
||
for path in sorted((work / "bi").glob("*.tsv")):
|
||
for line in open(path, "rb"):
|
||
k, c = line.rstrip(b"\n").split(b"\t")
|
||
a, b = k.decode("utf-8").split(" ")
|
||
if a in vocab and b in vocab:
|
||
pairs[k] = pairs.get(k, 0) + int(c)
|
||
print(f"{lang}: {len(dropped)} run-together elisions left out: {' '.join(dropped[:40])}", flush=True)
|
||
builder.write_model(lang, vocab, total, pairs, str(out))
|
||
|
||
NB = "https://www.nb.no/sbfil/ngram/ngram_2026/digibok-"
|
||
ORDBANK = "https://www.nb.no/sbfil/leksikalske_databaser/ordbank/20220201_norsk_ordbank_nob_2005.tar.gz"
|
||
WORD = re.compile(r"[a-zæøåéèêóôüäö]{1,32}\Z")
|
||
|
||
def normalize(s):
|
||
return unicodedata.normalize("NFC", s).lower()
|
||
|
||
def nb_counts(path, is_pair, accept):
|
||
counts = collections.Counter()
|
||
with subprocess.Popen(["zstd", "-dc", path], stdout=subprocess.PIPE) as process:
|
||
for line in process.stdout:
|
||
if b'"nob"' not in line:
|
||
continue
|
||
row = json.loads(line)
|
||
if row["lang"] != "nob":
|
||
continue
|
||
first = normalize(row["first"])
|
||
if not accept(first):
|
||
continue
|
||
if is_pair:
|
||
second = normalize(row["second"])
|
||
if not accept(second):
|
||
continue
|
||
key = (first + " " + second).encode("utf-8")
|
||
else:
|
||
key = first
|
||
n = sum(v for y, v in row["counts"].items() if "1980" <= y <= "2019")
|
||
if n:
|
||
counts[key] += n
|
||
if process.wait():
|
||
raise ValueError("Incomplete zstd stream: " + str(path))
|
||
return counts
|
||
|
||
def download_nb(url, path, size):
|
||
# The NB server may close long transfers. Resume independently verified byte ranges.
|
||
if path.is_file() and path.stat().st_size == size:
|
||
return
|
||
parts = path.with_suffix(".ranges")
|
||
parts.mkdir(exist_ok=True)
|
||
block = 4 << 20
|
||
def read(i):
|
||
start, end = i * block, min((i+1)*block, size)
|
||
part = parts / str(i)
|
||
for attempt in range(9):
|
||
pos = start + (part.stat().st_size if part.exists() else 0)
|
||
if pos == end:
|
||
return part
|
||
if pos > end:
|
||
raise ValueError("Oversized NB range")
|
||
req = urllib.request.Request(url, headers={"Range": f"bytes={pos}-{end-1}", "Accept-Encoding": "identity"})
|
||
try:
|
||
with urllib.request.urlopen(req, timeout=60) as res:
|
||
if res.status != 206 or res.headers.get("Content-Range") != f"bytes {pos}-{end-1}/{size}":
|
||
raise ValueError("NB returned an unexpected range")
|
||
with part.open("ab") as f:
|
||
shutil.copyfileobj(res, f, length=262144)
|
||
if part.stat().st_size == end-start:
|
||
return part
|
||
except (OSError, http.client.IncompleteRead, EOFError) as e:
|
||
if attempt == 8:
|
||
raise RuntimeError(f"Could not fetch {url}, range {i}") from e
|
||
time.sleep(min(2**attempt, 10))
|
||
raise IOError("Incomplete NB range " + str(i))
|
||
with concurrent.futures.ThreadPoolExecutor(max_workers=16) as pool:
|
||
complete = list(pool.map(read, range((size + block - 1)//block)))
|
||
assembled = path.with_suffix(".assembling")
|
||
with assembled.open("wb") as out:
|
||
for part in complete:
|
||
with part.open("rb") as f:
|
||
shutil.copyfileobj(f, out)
|
||
if assembled.stat().st_size != size:
|
||
raise IOError("Incomplete NB archive")
|
||
assembled.replace(path)
|
||
shutil.rmtree(parts)
|
||
|
||
def norwegian(work, out):
|
||
work.mkdir(parents=True, exist_ok=True)
|
||
bank = work / "ordbank-2022.tar.gz"
|
||
builder.fetch(ORDBANK, str(bank))
|
||
if hashlib.sha256(bank.read_bytes()).hexdigest() != "57d91cb3f17b85befa50a3b56fcaed9800ca665b39486fe97b668add914d5f60":
|
||
raise ValueError("Unexpected Norsk ordbank 2022 release")
|
||
# The 2026 inflection-only TSV omits uninflected words such as på and og.
|
||
with tarfile.open(bank, "r:gz") as archive, archive.extractfile("fullformsliste.txt") as entry:
|
||
rows = csv.DictReader(io.TextIOWrapper(entry, encoding="latin-1"), delimiter="\t", quoting=csv.QUOTE_NONE)
|
||
forms = {normalize(r["OPPSLAG"]) for r in rows if r["NORMERING"] == "normert" and r["TILDATO"] == "4000"}
|
||
paths = [work / ("digibok-" + n + ".jsonl.zst") for n in ("unigram", "bigram")]
|
||
archives = [(1_269_473_951, "b430ef4e19caf1a12bcf47e1ee4f6866f27943ae133e141c57d1a3db4dd8f0c9"),
|
||
(3_832_769_124, "b6d2f3ca5492288d5f1c66fc0a3976a7239502cb690f77d493a1cdd97c5ebe1e")]
|
||
for path, (size, expected) in zip(paths, archives):
|
||
download_nb(NB + path.name.removeprefix("digibok-"), path, size)
|
||
digest = hashlib.sha256()
|
||
with path.open("rb") as src:
|
||
for block in iter(lambda: src.read(1 << 20), b""):
|
||
digest.update(block)
|
||
if digest.hexdigest() != expected:
|
||
raise ValueError("Unexpected NB archive: " + str(path))
|
||
uni = nb_counts(paths[0], False, WORD.fullmatch)
|
||
total = sum(uni.values())
|
||
vocab = dict(sorted(((w, n) for w, n in uni.items() if w in forms),
|
||
key=lambda item: (-item[1], item[0]))[:95_000])
|
||
pairs = nb_counts(paths[1], True, vocab.__contains__)
|
||
print(f"Bokmål: {total:,} word occurrences; {len(forms):,} Ordbank forms; {len(pairs):,} filtered pairs", flush=True)
|
||
builder.write_model("nb", vocab, total, pairs, str(out))
|
||
|
||
if __name__ == "__main__":
|
||
if len(sys.argv) != 4 or sys.argv[1] not in ("de", "fr", "it", "nb"):
|
||
sys.exit(__doc__)
|
||
lang, work, out = sys.argv[1:]
|
||
if lang in ("de", "fr", "it"):
|
||
builder.main(lang, work, out)
|
||
if lang in ELISIONS:
|
||
refine(lang, work, out)
|
||
else:
|
||
norwegian(Path(work), Path(out))
|