mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* Android typing: names stay as typed, and words take the capitals the books give them
Android's word models are lowercase and know only ~100k words, so a name
typed right was a typo to them: Spanish "tom" became "toma", "mary" became
"mar y", English "sami" became "swami", and nothing was ever capitalised
("mary", "monday", German "haus" stayed lowercase). iOS gets both from its
spelling dictionary, which knows names only with a capital (#2091), and
German's nouns from de.capitals (#2169).
Adds a capitals sidecar per model (<lang>.capitals: word<TAB>percent, sorted
UTF-8, memory-mapped and searched in place by Capitals.kt):
- de.capitals: iOS's file, byte for byte.
- en.capitals, es.capitals: new, from the same Google Books v3 1-grams
(1980-2019, CC BY 3.0) as the models (build-capitals.py). Model words
written with a capital at least half the time, plus names outside the model
of three letters or more, capitalised at least 90% of the time and used at
least once per million words. Two model words run together whose pair is
ten times as common ("iwant", "apartir") are left out. A word that is also
an ordinary lowercase word is held at 94%: in English one SCOWL spells in
lowercase ("united", "china"), in Spanish a model word under 98% ("lima").
The engine uses it in two places:
- A listed name outside the model is a real word (TypingModel.known): the
literal costs 15 instead of 22 and needs the 7-nat margin, as an English
SCOWL word already did. Replaces "unknown = typo" for those names.
- On space, a word typed in lowercase that the books write with a capital at
least 95% of the time takes it ("mary" -> "Mary"); in German also at 50-94%
after an article or determiner ("das leben" -> "das Leben", "zu leben"
stays), and a German word the decoder corrects to takes its capital
("hsus" -> "Haus"), as iOS's nounCapitalised.
Also: build.gradle.kts verifies the three new files' SHA-256 and stores
.capitals uncompressed so they map from the APK; credits.txt names them.
Verified on the Studio (API 35 emulator, installed debug APK; replay of
2,000 held-out sentences each), word errors before (#2245 head) -> after:
- English 818 (5.90%) -> 770 (5.55%); today's iOS 798 (5.75%)
- Spanish 1,342 (10.09%) -> 1,133 (8.52%); today's iOS 1,205 (9.06%)
- German 774 -> 774, Bokmål 654 -> 654 (capitals don't change words)
Sentences better/worse: English 47/2, Spanish 154/1. The installed APK loads
en/es/de.capitals from its assets (checked with app_process).
testDebugUnitTest: 21 tests, 0 failures, with iOS's name-capital and German
noun cases re-expressed against the shipped lists.
* Android typing: the strip says what space will do, and offers words before and between them
Ports iOS's strip and next-word rules (#2125 and the strip composition in
AutocorrectEngine.suggestions) into the engine, as two calls the keyboard can
show:
- TypingWord.strip(autocorrect) returns typed slots (Suggestion with a Kind):
what was typed, then what space will put in (Kind.Fix, to highlight) or the
likeliest candidate, then the runner-up; an unfinished word space leaves
alone ("th", "beca") gets three candidates instead, so completions keep three
slots. Candidates in iOS's order: suggest-only English contractions
("ill" -> "I'll", "well" -> "we'll"), the likeliest completion of a 1-3
letter word after the word before (TypingModel.completionsAfter, by the word
pairs), the decoder's readings (for three letters or more), then completions
by frequency. Names show with their capital ("Monday"), English "i" as "I".
Never what was typed again. Autocorrect off: no fix, no readings.
- predictions(before, capitalize) now gives three common first words at a
sentence's start ("I", "The", "I'm"; "Yo", "El", "La"), clause starters
after a comma ("I", "but", "and"), the word pairs' next words with the
language's commonest words filling empty slots, and nothing after a handle,
path, number, a second space or before a space is typed. Capitalised as
shift would be (by default at a sentence's start). Replaces the old fill of
the three commonest words, lowercase, everywhere.
- suggestions() and literalChip() keep their old behaviour until the keyboard
moves to strip().
Benchmark: the replay counts strip hits as iOS's harness does (word at its
end, next word after the space, word after two letters), from the synthetic
sets' word ends (export_synthetic.py keeps them; synthetic_de_nb.py writes
them).
Verified on the Studio (API 35 emulator, installed debug APK, 2,000 held-out
sentences each). Word errors unchanged (English 770, Spanish 1,133, German
774, Bokmål 654). Strip hits before -> after (word / next / two letters):
- English 12,070 / 2,338 / 3,789 -> 13,180 / 2,349 / 4,390
(iOS 13,253 / 2,287 / 3,235)
- Spanish 11,603 / 2,116 / 2,606 -> 12,407 / 2,110 / 3,236
(iOS 12,364 / 2,073 / 2,901)
- German 12,204 / 1,679 / 5,026 -> 12,918 / 1,680 / 5,624
- Bokmål 11,172 / 2,160 / 4,380 -> 11,988 / 2,159 / 4,875
Next-word hits move by -6 (Spanish) and -1 (Bokmål) of ~11k: the common
words now fill as on iOS. Letter latency median 0.09 ms, p95 0.3 ms (the
readings are read on each letter for words of three letters or more).
testDebugUnitTest: 23 tests, 0 failures, with iOS's sentence-starter,
boundary, short-completion, strip and suggest-only cases re-expressed.
* Android typing: your personal dictionary's words and shortcuts
Reads Android's personal dictionary (UserDictionary.Words), the counterpart of
the UILexicon iOS reads (text replacements #2152-#2173, Contacts names), into
a new PersonalWords lexicon the engine uses:
- Text replacements: textReplacement(before) finds the longest shortcut that
ends the text at a word boundary, digits and dots allowed ("ad2", "s.o"),
after a bracket or comma ("(myem", "hi,myem"), never the end of a longer
word ("xmyem"); a shortcut typed with a leading capital (not all caps)
capitalises the phrase ("Brb" -> "Be right back"), stored capitals stay.
replacement() and strip() apply a letters-only shortcut too, ahead of chat
shorthand ("brb" stays only without a shortcut).
- Your words are read by the decoder beside the model's (Lexicon interface;
SpatialDecoder.search takes either list): a slip in one is read as it
("zorbiz" -> "Zorbix"), and one typed right is a known word scored as a
moderately common one (11 nats), as iOS's UserWords.
- Your word is never corrected into another, and typed in lowercase takes
the capitals you gave it ("github" -> "GitHub", "kate" -> "Kate"), unless
it is also an ordinary word ("will", "world" stay lowercase), as iOS's
vocabulary (#2205) and withoutStrayCapital (#2173, #2218).
- The strip completes your words before the rest of the dictionary's, and
your own words are offered even when the never-suggest list has them, as
iOS offers UILexicon terms.
Android lets an enabled input method read the provider with no permission
(denied callers get an empty answer); PersonalWords.read filters entries to
the keyboard's language (and "no" for Bokmål) and reads at most 5,000.
Nothing is written. Gboard and Samsung keep their own lists, so many users
will have none.
Verified on the Studio (API 35 emulator): the replay is unchanged (no
personal words in it): English 770, Spanish 1,133, German 774, Bokmål 654
word errors, same strip hits. testDebugUnitTest: 25 tests, 0 failures, with
iOS's text-replacement, leading-capital, shorthand, own-word, stray-capital
and still-offered cases re-expressed against PersonalWords. Not verified:
reading the real provider (a shell process can't, only the enabled
keyboard), which needs the keyboard wiring.
31 lines
1.6 KiB
Python
31 lines
1.6 KiB
Python
"""Export only generated held-out Tatoeba rows and their same-model iOS outputs."""
|
|
import json, math, re, sys
|
|
from pathlib import Path
|
|
|
|
def export(source, result, lang):
|
|
if lang not in ("en", "es"):
|
|
raise ValueError("No synthetic set for this language")
|
|
group = "test" if lang == "en" else "es-test"
|
|
items = json.loads(Path(source).read_text())["items"]
|
|
outputs = json.loads(Path(result).read_text())["outputs"]
|
|
selected = [r for r in items if r["set"] == group and re.fullmatch(group + r"-[0-9]+", r["id"])
|
|
and "logged" not in r and r["context"] == "" and r.get("language", "en") == lang]
|
|
rows = {r["id"]: r for r in selected}
|
|
if len(selected) != 2000 or set(rows) != {group + "-" + str(i) for i in range(2000)}:
|
|
raise ValueError("Incomplete or duplicated synthetic set")
|
|
for i in range(2000):
|
|
r = rows[group + "-" + str(i)]
|
|
allowed = r"[a-zñ ]+" if lang == "es" else r"[a-z ]+"
|
|
if (not re.fullmatch(allowed, r["typed"]) or len(r["typed"]) != len(r["taps"]) or
|
|
any(not isinstance(t, list) or len(t) != 2 or
|
|
not all(type(v) in (int, float) and math.isfinite(v) for v in t) for t in r["taps"])):
|
|
raise ValueError("Invalid synthetic key geometry")
|
|
out = {f: r[f] for f in ("id", "intended", "typed", "taps")}
|
|
out.update({f: r[f] for f in ("want", "ends") if f in r})
|
|
out["ios_shared"] = outputs[r["id"]]
|
|
print(json.dumps(out, separators=(",", ":")))
|
|
|
|
if __name__ == "__main__":
|
|
if len(sys.argv) != 4:
|
|
sys.exit("export_synthetic.py INPUT.json R1-OUTPUT.json {en|es}")
|
|
export(*sys.argv[1:])
|