mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS keyboard: Polish autocorrects with Hex's own model and word list
Polish autocorrect today is only as good as iOS's Polish dictionary. Without Apple's word data,
that dictionary takes any letters for a word and guesses in English ("moze" -> "more"), and its
guesses rarely carry a word's marks. A TestFlight tester called it unusable. Now:
- HEXKeyboard/TypingModel/pl.hextm (150,000 words, 184,327 pairs, 3.60 MB) and pl.valid (525,084
words, 8.29 MB), byte for byte what scripts/build-typing-model.py pl builds (#2401: NKJP
n-grams, ParlaMint's check, PoliMorf). Bundling them is all it takes: the decoder reads Polish
taps with the model (FullKeyboardSettings.typingModel), and HexWordList/HexOwnWords (#2404)
answer what is a word from the model and pl.valid instead of the phone's dictionary.
- Settings › Spatial autocorrect says "English, Spanish, French and Polish only".
- The gate: typing-gate.py types Polish (QWERTY rows); accent-keys.json and critical-cases.json
gain pl, and quality-gate.py now accepts a configured language without an Android model
(it required equality). TypingGateReplayTests picks ł from l's long-press menu (ł has no
decomposition to find its key).
- ios/README.md and the benchmark README give the numbers.
Gate (ios/scripts/typing-gate.py run ... pl, iPhone 17 simulator, iOS 27.2): 85.98% fewer word
errors than the raw taps (gate 70%), 0.22% of correctly typed words changed (gate 0.5%), 95.46%
of unambiguous marks restored (gate 85%), 5/5 critical cases: PASS.
Known gaps: no Polish capitals list, so a name typed in lowercase stays lowercase ("warszawa");
English + Polish bilingual typing and a real iPhone's timing are not measured.
* iOS keyboard: Polish names take their capital from Polish's own capitals list
Without a list, Hex's Polish own words (pl.hextm and pl.valid) left every name typed in
lowercase as typed: the gate's names-kept line read 0.00%, as with the phone's dictionary on the
simulator. pl.capitals (2,800 words, 36,472 bytes) is build-capitals.py's output in the format
fr.capitals and de.capitals use, read by HexWordList/HexOwnWords as theirs are.
build-capitals.py pl counts case from ParlaMint 5.0's Polish transcripts (CC BY 4.0, already one
of Polish's credited inputs; checked against its SHA-256), leaving out each sentence's first word,
which any word can start. A word PoliMorf (BSD 2-Clause) lists in lowercase is an ordinary word and
is held at 94% (listed, never capitalised on its own), as SCOWL's lowercase words are in English:
"tom" is a volume, "marszałek" a title, "polska" an adjective. Otherwise the rules are the other
languages': model words from 50%, names outside the model from 90% and once per million words.
Gate (typing-gate.py run ... pl, iPhone 17 simulator, iOS 27.2), names kept / capitals added on
the all-lowercase pass:
- phone's dictionary (no Polish files): 0.00% / 0.00% (and -1.36% errors, FAIL);
- pl.hextm + pl.valid: 0.00% / 0.00%;
- pl.hextm + pl.valid + pl.capitals: 10.65% / 0.00%, with 86.01% fewer errors, 0.22% changed,
95.46% unambiguous marks, 5/5 critical cases (PASS).
Rebuilt on exos from the pinned inputs: SHA-256 34bfa18b..., the committed file.
267 lines
13 KiB
Python
267 lines
13 KiB
Python
#!/usr/bin/env python3
|
|
"""Build a keyboard language's capitals sidecar (<lang>.capitals) from Google Books Ngram v3
|
|
1-grams (2020-02-17, CC BY 3.0), the corpus its .hextm model was counted from.
|
|
|
|
python3 build-capitals.py en MODEL.hextm WORKDIR OUTPUT.capitals scowl-2020.12.07.tar.gz
|
|
python3 build-capitals.py {es|fr|it} MODEL.hextm WORKDIR OUTPUT.capitals
|
|
python3 build-capitals.py pl MODEL.hextm WORKDIR OUTPUT.capitals ParlaMint-PL.tgz polimorf.tab.gz
|
|
|
|
Polish has no Google Books counts: its counts come from ParlaMint 5.0's Polish transcripts (CC BY
|
|
4.0, the corpus build-typing-model.py pl checks marks against), which keep case, leaving out each
|
|
sentence's first word, which any word can start. A word PoliMorf (BSD 2-Clause) lists in lowercase
|
|
is an ordinary word, held at 94% as SCOWL's are in English ("tom" is a volume, "Tom" a name).
|
|
|
|
Every shard is streamed (curl | gzip -dc, never stored) and its 1980-2019 counts of all-lowercase
|
|
tokens and of tokens with only the first letter capitalised are cached in WORKDIR, so a rerun
|
|
resumes and a rule change renders from WORKDIR/counts.tsv in seconds. Tagged tokens ("_") and
|
|
other casings are ignored.
|
|
|
|
Each line is word<TAB>percent (the capitalised share, rounded half up), lowercase, sorted by its
|
|
UTF-8 bytes, with a trailing newline, as German's de.capitals:
|
|
- a model word whose share is at least 50% over at least 40 uses (the keyboard capitalises
|
|
it when typed in lowercase from 95%);
|
|
- a word of three letters or more outside the model that the books capitalise at least 90% of
|
|
the time and use at least once per million words: names and places ("tom", "olivia",
|
|
"tokio"), which the keyboard then takes for real words, as iOS's spelling dictionary does, and
|
|
capitalises from 95%. Rarer ones ("goyo", "eby") are more often a typo of a common word. Two model
|
|
words run together whose pair is ten times as common ("iwant", "apartir") are left out.
|
|
|
|
The keyboard capitalises a word typed in lowercase only where iOS's dictionary would know it only
|
|
with a capital, so a word that is also an ordinary lowercase word is held at 94% (listed, never
|
|
capitalised on its own): in English, one SCOWL (sizes through 70, the lists en.valid comes from)
|
|
spells in lowercase ("united", "jersey", "china"); in Spanish, French and Italian, a model word
|
|
under 98% ("lima", "granada", "san"). In French and Italian, a word whose accented spelling is a
|
|
commoner model word is held, so "ecole" isn't written "Ecole" (books drop accents on capitals:
|
|
"Ecole" is "école").
|
|
"""
|
|
import hashlib, math, multiprocessing, os, re, struct, subprocess, sys, tarfile, unicodedata
|
|
|
|
BASE = "https://storage.googleapis.com/books/ngrams/books/20200217"
|
|
FILES = {"en": ("eng", 24), "es": ("spa", 3), "de": ("ger", 8), "fr": ("fre", 6), "it": ("ita", 2)}
|
|
LETTERS = {"en": "a-z", "es": "a-zñáéíóúü", "de": "a-zäöüß", "fr": "a-zàâçéèêëîïôùûüÿœæ", "it": "a-zàèéìíîòóùú"}
|
|
FIRST_YEAR, LAST_YEAR = 1980, 2019
|
|
MIN_TOTAL = 40
|
|
SOMETIMES = 50
|
|
NAME_SHARE = 90
|
|
NAME_PER_MILLION = 1
|
|
ALWAYS, HELD = 95, 94
|
|
JOINED = 10
|
|
MODEL_ALWAYS = {"en": 95, "es": 98, "fr": 98, "it": 98}
|
|
SCOWL_SHA256 = "5587667caa20c4891390c2d42dbb4d5c4c3f41bee77af1457ece3ba23fb859cc"
|
|
# build-typing-model.py's POLISH_INPUTS.
|
|
PARLAMINT_PL_SHA256 = "66739847c4454e21d524755cea17e41dd923c78cdefe03b6775e9f10720ba957"
|
|
POLIMORF_SHA256 = "2b1f07224c434c8710def382d497cf8221d5764e8d683d2ad34242810ab72746"
|
|
LETTERS["pl"] = "a-ząćęłńóśźż"
|
|
MODEL_ALWAYS["pl"] = 98
|
|
|
|
|
|
def scowl_lowercase(path):
|
|
"""SCOWL's lowercase spellings, sizes through 70."""
|
|
assert hashlib.sha256(open(path, "rb").read()).hexdigest() == SCOWL_SHA256
|
|
words = set()
|
|
with tarfile.open(path) as archive:
|
|
for entry in archive:
|
|
m = re.fullmatch(r".*/final/(english|british|american)-(words|contractions|proper-names|upper|abbreviations)\.(\d+)", entry.name)
|
|
if m and int(m[3]) <= 70:
|
|
words.update(w for w in archive.extractfile(entry).read().decode("latin-1").split() if w == w.lower())
|
|
return words
|
|
|
|
|
|
def model_words(path):
|
|
"""The model's words with -ln P(word), and its stored pairs (first, second) with -ln P(second | first)."""
|
|
data = open(path, "rb").read()
|
|
version, words, pairs, blob, alphabet = struct.unpack_from("<IIIII", data, 4)
|
|
assert data[:4] == b"HXTM" and version == 3
|
|
letters = {}
|
|
for i in range(alphabet):
|
|
scalar, code, _, _ = struct.unpack_from("<IBBH", data, 28 + 8 * i)
|
|
letters[code] = chr(scalar)
|
|
offsets_at = 28 + 8 * alphabet
|
|
rows_at = offsets_at + 4 * (words + 1) + words
|
|
rows_at += -rows_at % 4
|
|
blob_at = rows_at + 4 * (words + 1) + 4 * pairs + pairs
|
|
assert blob_at + blob == len(data)
|
|
offsets = struct.unpack_from(f"<{words + 1}I", data, offsets_at)
|
|
spelled = ["".join(letters[b] for b in data[blob_at + offsets[i]:blob_at + offsets[i + 1]]) for i in range(words)]
|
|
unigram = {spelled[i]: data[offsets_at + 4 * (words + 1) + i] / 8 for i in range(words)}
|
|
rows = struct.unpack_from(f"<{words + 1}I", data, rows_at)
|
|
seconds_at = rows_at + 4 * (words + 1)
|
|
seconds = struct.unpack_from(f"<{pairs}I", data, seconds_at)
|
|
costs_at = seconds_at + 4 * pairs
|
|
pair = {(spelled[a], spelled[seconds[k]]): data[costs_at + k] / 8 for a in range(words) for k in range(rows[a], rows[a + 1])}
|
|
return unigram, pair
|
|
|
|
|
|
def joined(word, share, unigram, pair):
|
|
"""Whether `word` is two model words run together, as scanned books (and typists) do ("Iwant",
|
|
"apartir"): the stored pair is at least JOINED times as common as the word itself (`share` of
|
|
the corpus). Those are no names."""
|
|
best = 0.0
|
|
for k in range(1, len(word)):
|
|
a, b = word[:k], word[k:]
|
|
if (a, b) in pair:
|
|
best = max(best, math.exp(-unigram[a] - pair[(a, b)]))
|
|
return best >= JOINED * share
|
|
|
|
|
|
def count_shard(job):
|
|
lang, index, url, work = job
|
|
out = os.path.join(work, os.path.basename(url) + ".tsv")
|
|
if os.path.exists(out):
|
|
return out
|
|
letters = LETTERS[lang]
|
|
token_re = re.compile(f"^[{letters.upper()}{letters}]?[{letters}]+$".encode("utf-8"))
|
|
download = subprocess.Popen(["curl", "-sSfL", "--http1.1", "--retry", "8", "--retry-all-errors", url], stdout=subprocess.PIPE)
|
|
unzip = subprocess.Popen(["gzip", "-dc"], stdin=download.stdout, stdout=subprocess.PIPE, bufsize=1 << 20)
|
|
download.stdout.close()
|
|
counts = {}
|
|
for line in unzip.stdout:
|
|
tab = line.find(b"\t")
|
|
token = line[:tab]
|
|
if b"_" in token or not token_re.match(token):
|
|
continue
|
|
total = 0
|
|
for field in line[tab + 1:].split(b"\t"):
|
|
year, count, _ = field.split(b",", 2)
|
|
if FIRST_YEAR <= int(year) <= LAST_YEAR:
|
|
total += int(count)
|
|
if total >= 4:
|
|
counts[token] = counts.get(token, 0) + total
|
|
unzip.wait(); download.wait()
|
|
if unzip.returncode or download.returncode:
|
|
raise RuntimeError(f"{url}: curl {download.returncode}, gzip {unzip.returncode}")
|
|
with open(out + ".part", "wb") as f:
|
|
for token, count in counts.items():
|
|
f.write(token + b"\t" + str(count).encode() + b"\n")
|
|
os.replace(out + ".part", out)
|
|
print(f"[{index}] {os.path.basename(url)}: {len(counts):,} tokens", flush=True)
|
|
return out
|
|
|
|
|
|
def sha256(path):
|
|
h = hashlib.sha256()
|
|
with open(path, "rb") as f:
|
|
for block in iter(lambda: f.read(1 << 20), b""):
|
|
h.update(block)
|
|
return h.hexdigest()
|
|
|
|
|
|
def polimorf_lowercase(path):
|
|
"""The forms PoliMorf lists in lowercase."""
|
|
import gzip
|
|
assert sha256(path) == POLIMORF_SHA256, path
|
|
with gzip.open(path, "rt", encoding="utf-8") as f:
|
|
return {form for form in (line.split("\t", 1)[0] for line in f if "\t" in line)
|
|
if form == form.lower()}
|
|
|
|
|
|
def parlamint_counts(path):
|
|
"""ParlaMint-PL's lowercase and capitalised counts per lowercase spelling, each sentence's first
|
|
word left out."""
|
|
import tarfile as tf
|
|
assert sha256(path) == PARLAMINT_PL_SHA256, path
|
|
seg = re.compile(r"<seg\b[^>]*>(.*?)</seg>", re.S)
|
|
sentence = re.compile(r"(?<=[.!?…])\s+")
|
|
word = re.compile(f"[{LETTERS['pl'].upper()}{LETTERS['pl']}]+")
|
|
low, cap = {}, {}
|
|
with tf.open(path) as t:
|
|
for m in t:
|
|
if not (m.isfile() and re.search(r"/\d{4}/ParlaMint-PL_[^/]+\.xml$", m.name)):
|
|
continue
|
|
text = t.extractfile(m).read().decode("utf-8")
|
|
for body in seg.findall(text):
|
|
body = unicodedata.normalize("NFC", re.sub(r"<[^>]+>", " ", body).replace("&", "&"))
|
|
for s in sentence.split(body):
|
|
for token in word.findall(s)[1:]:
|
|
key = token.lower()
|
|
if token == key:
|
|
low[key] = low.get(key, 0) + 1
|
|
elif token[0].isupper() and token[1:] == key[1:]:
|
|
cap[key] = cap.get(key, 0) + 1
|
|
return low, cap
|
|
|
|
|
|
def merged_counts(lang, work):
|
|
"""WORK/counts.tsv: each lowercase spelling with its 1980-2019 lowercase and capitalised counts,
|
|
counted once from the language's 1-gram shards. build-valid-words.py reads it too."""
|
|
os.makedirs(work, exist_ok=True)
|
|
merged = os.path.join(work, "counts.tsv")
|
|
if not os.path.exists(merged) and lang == "pl":
|
|
low, cap = parlamint_counts(sys.argv[5])
|
|
with open(merged + ".part", "w", encoding="utf-8") as f:
|
|
for key in sorted(set(low) | set(cap)):
|
|
f.write(f"{key}\t{low.get(key, 0)}\t{cap.get(key, 0)}\n")
|
|
os.replace(merged + ".part", merged)
|
|
if not os.path.exists(merged):
|
|
code, n = FILES[lang]
|
|
jobs = [(lang, i, f"{BASE}/{code}/1-{i:05d}-of-{n:05d}.gz", work) for i in range(n)]
|
|
with multiprocessing.Pool(4) as pool:
|
|
paths = pool.map(count_shard, jobs, chunksize=1)
|
|
low, cap = {}, {}
|
|
for path in paths:
|
|
for line in open(path, "rb"):
|
|
token, count = line.rstrip(b"\n").split(b"\t")
|
|
word = token.decode("utf-8")
|
|
key = word.lower()
|
|
target = low if word == key else cap
|
|
target[key] = target.get(key, 0) + int(count)
|
|
with open(merged + ".part", "w", encoding="utf-8") as f:
|
|
for key in set(low) | set(cap):
|
|
f.write(f"{key}\t{low.get(key, 0)}\t{cap.get(key, 0)}\n")
|
|
os.replace(merged + ".part", merged)
|
|
return merged
|
|
|
|
|
|
def main():
|
|
lang, model, work, output = sys.argv[1:5]
|
|
lowercase = scowl_lowercase(sys.argv[5]) if lang == "en" else polimorf_lowercase(sys.argv[6]) if lang == "pl" else set()
|
|
if lang == "en":
|
|
# Hex's own chat and tech words (en-extra-words.tsv, which build-typing-model.py adds to the
|
|
# model) are lowercase words: "plz" and "okie", capitalised in books, are held too.
|
|
extra = os.path.join(os.path.dirname(os.path.abspath(__file__)), "en-extra-words.tsv")
|
|
lowercase |= {line.split("\t")[0] for line in open(extra, encoding="utf-8") if line.strip() and not line.startswith("#")}
|
|
merged = merged_counts(lang, work)
|
|
unigram, pair = model_words(model)
|
|
words = set(unigram)
|
|
def folded(w):
|
|
return "".join(c for c in unicodedata.normalize("NFD", w) if not unicodedata.combining(c))
|
|
accented_twin = set()
|
|
if lang in ("fr", "it"):
|
|
twins = {}
|
|
for w in words:
|
|
twins.setdefault(folded(w), []).append(w)
|
|
accented_twin = {w for group in twins.values() if len(group) > 1 for w in group
|
|
if w == folded(w) and any(o != w and unigram[o] < unigram[w] for o in group)}
|
|
corpus = 0
|
|
for line in open(merged, encoding="utf-8"):
|
|
_, lo, ca = line.rstrip("\n").split("\t")
|
|
corpus += int(lo) + int(ca)
|
|
rows = []
|
|
for line in open(merged, encoding="utf-8"):
|
|
key, lo, ca = line.rstrip("\n").split("\t")
|
|
lo, ca = int(lo), int(ca)
|
|
total = lo + ca
|
|
if len(key) < 2 or total < MIN_TOTAL:
|
|
continue
|
|
share = (200 * ca + total) // (2 * total)
|
|
if key in words:
|
|
if share < SOMETIMES:
|
|
continue
|
|
if share < MODEL_ALWAYS[lang]:
|
|
share = min(share, HELD)
|
|
elif (len(key) < 3 or share < NAME_SHARE or total * 1_000_000 < NAME_PER_MILLION * corpus
|
|
or joined(key, total / corpus, unigram, pair)):
|
|
continue
|
|
if key in lowercase or key in accented_twin:
|
|
share = min(share, HELD)
|
|
rows.append((key, share))
|
|
rows.sort(key=lambda r: r[0].encode("utf-8"))
|
|
data = "".join(f"{k}\t{p}\n" for k, p in rows).encode("utf-8")
|
|
open(output, "wb").write(data)
|
|
inside = sum(1 for k, _ in rows if k in words)
|
|
print(f"{len(rows):,} words ({inside:,} in the model, {len(rows) - inside:,} names outside it), "
|
|
f"{len(data):,} bytes; {hashlib.sha256(data).hexdigest()}")
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|