hex/android/benchmark/typing/GateLexicon.java
Ryan Vogel afd7da2243
Gate Android typing quality per language on the held-out replay (#2374)
* Gate Android typing quality on held-out sentences per language

Run the existing touch replay against the app classes on the host JVM. Score noisy word-error reduction, changes to correct words, omitted accents and named critical cases against one shared policy so a noisy-WER win cannot mask bad corrections. The legacy APK replay and historical metrics remain available.

Verified on the Studio: all six noisy counts match prior Android results; the full gate returned 1 and English alone returned 0. Compared all 12,000 references, hit keys and coordinates with their existing held-out exports. code-shape reported no hard problems.

* Gate only unambiguous restorations of hold-only letters

Replace the all-accent score with mappings verified against the Android key rows and long-press menus. Use model and validity spellings to report ambiguous restoration and clean-word harm separately, so literal dedicated keys and legitimate competing spellings cannot fail the 85% gate. Add attributed German and Bokmål evaluation sentences and report frequent Bokmål clean changes.

Verified all six gates on the Studio in 26.5s: English, Spanish and French pass; German, Italian and Bokmål fail. Each of the 18 raw, decoded and clean-change counts matched the preceding commit. All 12 layout-appropriate critical cases passed, French alone returned 0, the full run returned 1, and code-shape passed.

* Preserve reference Shift in replay and score realistic accent omissions

Replace always-lowercase input with source casing by Tatoeba ID. For noisy input apply Shift to the original hit at each intended letter tap, using the generator boundaries so omitted spaces and apostrophes cannot shift later capitals. Keep the comparisons case-insensitive. Replace the single s probe with the conventional German ss substitute and skip the 85% check for fewer than 50 unambiguous occurrences.

Verified all 12,000 references and hit strings lowercase to the previous inputs with identical coordinates and attribution. All six ran on the Studio in 40.2s: German restored 0/52 unambiguous ss forms, and Bokmål skipped its two-word accent sample but still changed 74 clean words. All 12 critical cases and code-shape passed.
2026-11-22 09:15:22 -05:00

26 lines
1.1 KiB
Java

package local.hex.bench;
import java.io.File;
import java.util.*;
import ly.anoma.hex.keyboard.KeyboardLetters;
import ly.anoma.hex.keyboard.LetterFeedback;
import ly.anoma.hex.typing.TypingModel;
/** Export the production model vocabulary and its hold-only letters on the actual layout. */
public final class GateLexicon {
public static void main(String[] args) throws Exception {
String lang = args[1];
TypingModel model = TypingModel.Companion.file(new File(args[0]), lang);
String keys = String.join("", KeyboardLetters.INSTANCE.rows(lang));
List<String> holds = new ArrayList<>();
for (char key : keys.toCharArray()) {
for (String option : LetterFeedback.Companion.options(key, lang)) {
char c = option.charAt(0);
if (Character.isLetter(c) && keys.indexOf(c) == -1 && model.code(c) != 0)
holds.add(option + ":" + key);
}
}
System.out.println(String.join(" ", holds));
for (int i = 0; i < model.getWords(); i++) System.out.println(model.text(i));
}
}