hex/ios/HEXMobileTests/SwipeDecoderTests.swift
Ryan Vogel b719d45e53
iOS: swipe reads the letters under the path, and the emoji bar scrubs (#2189)
* iOS: swipe guesses read the letters under the path, from where the finger turned and paused

X tester (Brandon McConnell): "The swipe keyboard guesses don't appear to account
for letters 'under' the swipe."

SwipeDecoder already matched the whole path SHARK2-style (location by point and
by dynamic time warping, normalised shape, start and end keys, language model),
but on the path resampled to 32 evenly spaced points with the touches' times
thrown away. A straight stroke runs over the keys between its ends, so words
that differ only by a key on that line drew the same ideal path: "tired"/"tied"
(i to e over r), "wire"/"write", "tire"/"tie", "greet"/"get", "pit"/"put", and
the language model alone chose between them.

The decoder now also reads Evidence from the raw path: points where it turns
sharply (angle over 0.2 pitch each side) and, from the touches' timestamps,
where the finger slowed (time per pitch against the swipe's median). The best
80 words are scored on an edit distance between their keys and that evidence,
in order: a turn or pause no key of the word explains costs by its strength, a
key the word turns at without a turn or pause near it costs, and a key it goes
straight through without a pause on it costs (only when the path has times).
SwipeTyping keeps each touch's UITouch.timestamp (KeyboardKeyPlaneView passes
it) and hands it to decode and the preview; paths without times, or whose
times are all equal (tests), read turns only. Parameters were tuned on the dev
sentences and on mined dev sets of common words that share a path, with
fingers that stop at every key, keep 60% or 90% of their speed through a
straight-on key, or hesitate between keys; lmWeight 1.1 -> 1.0 and
locationWeight 40 -> 30 were retuned with them.

Benchmark (experiment 2026-11-20-hex-swipe-typing, held-out test sentences,
390 pt, top-1 before -> after): English typical 94.9 -> 97.1, neat 97.5 ->
98.9, sloppy 76.8 -> 81.2, clean 98.1 -> 99.3, 60 Hz 94.9 -> 97.1, Duo split
94.2 -> 95.1; Spanish typical 90.9 -> 94.2, sloppy 77.3 -> 82.5, split 90.5 ->
91.8. Fingers that barely slow on straight-on keys (flow 0.9): 94.9 -> 95.9 and
90.9 -> 93.0; with a mid-stroke hesitation in 30% of words: 94.7 -> 96.0 and
91.0 -> 93.8; no times at all: 94.9 -> 94.9 and 90.9 -> 91.7. Held-out words
that share a path (mined, 5 swipes each): English 85.1 -> 96.0, Spanish 81.2
-> 94.5; the hand-picked set with Brandon's examples (tired/tried/tire,
great/grant/greet, though/through/tough, where/were/wire, ...) 75.1 -> 89.2.
Decode on an M5 Max: median 0.86 -> 0.91 ms, p99 2.72 -> 2.85 ms, max 3.96 ->
4.34 ms (English); Spanish unchanged.

Live on an iPhone 17 Pro simulator (iOS 27.2, Release, HID touch paths):
tired with a pause on r -> "Tired", the same path without -> "tied", "wire"
with a pause on r and "write" with pauses on r and t read as themselves.

Kept: #2173's withoutStrayCapital casing and #2159's swipe spacing (untouched;
the full suite passes).

Tests: SwipeDecoderTests testAPauseOnALetterTheStrokeCrossesReadsIt and
testAPauseOffTheWordsKeysCountsAgainstIt (SwipePath.timed), and
FullKeyboardEngineTests testASwipeReadsWhereTheFingerPaused (the plane hands
the times through). Full HEXMobileTests: 530 run, 0 failed, 1 skipped (iPhone
17 sim, iOS 27.2).

* iOS: slide along the emoji categories to scrub the grid, and the whole band under the grid takes them

X tester (Brandon McConnell): "The hitbox size for the emoji category buttons
in the emoji picker is really small, making it difficult to quickly navigate the
emoji picker", and "should be able to swipe through these emoji categories,
which is currently unsupported."

Black box, Apple's emoji keyboard (iPhone 17 simulator, iOS 27.2, Safari,
HID touch paths): the grid scrolls sideways, sections left to right, emoji
filling each column top to bottom. A tap on a category jumps the grid to its
start when the finger lifts (a held finger moves nothing). A finger sliding
along the bar scrubs the grid live and continuously: each icon's slot runs
through its own category (its left edge at the category's start, its right
edge at the next one's), the grid staying where the finger lifts. Scrolling
the grid by hand moves the bar's chip. While scrubbing Apple's bar shows a
capsule behind every icon and hides the chip; Hex keeps #2142's chip and moves
it with the finger. The simulator has no haptics, so whether Apple's bar ticks
is unknown; Hex gives the light selection tick it gives the skin tones, per new
category, when haptics are allowed.

EmojiCategoryBar now tracks the touch itself instead of one UIButton per
category: a tap selects on lift, a move past 8 pt starts a scrub that reports
the category under the finger and how far across its slot, and its buttons
only draw and speak (VoiceOver activates them through accessibilityActivate).
EmojiPanelView's band routing (#2142) now hands every touch under the grid
between ABC and delete to the bar, so each category takes the full height to
the panel's bottom edge and the full width of its slot (32.6 pt on a 402 pt
iPhone); it maps a scrub onto the grid (scrub(toSection:fraction:)), and while
a finger scrubs the grid follows the bar rather than the other way round.

Live (iPhone 17 Pro simulator, Release, Hex keyboard in a Safari field): taps
at y 797 (under the icons, at the panel's bottom edge) and y 760 (between the
grid and the icons) open Food and drink and Travel and places; sliding from
Frequently used to Symbols moves the grid through every category with the
chip under the finger. Memory (footprint, one run): tapping every category
peaked at 44.5 MB and scrolling past every emoji by hand at 44.4; one slide
across five categories peaked at 46.0, three slides back and forth across the
bar at 47.5 and ten at 48.3, back to 42-43 MB resident after. Scrubbing
redraws a screen of emoji per move, so its peak sits 1.5-4 MB over browsing;
throttling it to 30 or 15 Hz only took off about 1 MB, so it isn't.

Tests: EmojiSuggesterTests testTheBarsKeysTakeTheWholeBandUnderTheGrid now
checks the bar takes every point of every slot from the grid to the panel's
edge (it checked the buttons), and testSlidingAlongTheBarScrubsTheGrid covers
tap on lift, a small move staying a tap, the scrub through every category with
one tick each, the grid moving within a category, the end of the bar at the end
of the grid, and the chip following a hand scroll. Full HEXMobileTests: 530
run, 0 failed, 1 skipped.

* iOS: a swipe that comes back to its key types its letter, and "this is a test" swiped stays as typed

TestFlight (build 56): "Swipe keyboard doesn't handle next word capitalization
correctly. Also forgets to add a space sometimes. For example try swiping 'this
is a test'. Example: This Isa test."

Reproduced on this branch (rebased on origin/dev 83717ce6) with HID touch paths
in a Safari field (iPhone 17 Pro simulator, iOS 27.2, Release, Hex keyboard):
swipe "this", swipe "is", tap "a", swipe "test" types "This is a test", so
#2159 (the space a swiped word keeps pending before a tapped letter) fixed
"Isa" and #2173 (withoutStrayCapital) the stray "Is"; shift is on only at the
sentence start. A small slide on "a" (8 pt) is still a tap and types "a" too.

Still broken: "a" slid a little past the swipe threshold (a key's pitch) and
back typed "as", because the decoder reads no one-letter word (Apple's
keyboard reads that path as "As" too; it is what the tester's "a" can be).
SwipeTyping now reads a swipe that lifts on the key it went down on, never more
than 1.6 key pitches from it, as a tap that slid: the swipe ends with nothing
typed and the key's letter goes in through the engine's own tap path (the
pending space, shift and autocorrect as a tap's), unless the decoder read a
word that starts and ends on that key ("lol" on l, "pop" on p). A slide from a
that lifts on s still reads "as".

Live after the fix: tap, 8 pt slide, and 45 pt out-and-back slide on "a" all
type "This is a test".

Tests: FullKeyboardEngineTests testThisIsATestSwipedWithASingleLetterBetween
(the sequence with "a" tapped, slid a little, and slid out and back, in a field
that capitalises sentences; then ". " and "it works" swiped starts the next
sentence with a capital). Full HEXMobileTests: 530 run, 0 failed, 1 skipped.
2026-11-21 08:47:45 -05:00

184 lines
10 KiB
Swift

import CoreGraphics
import XCTest
@testable import HEX
/// Swipe typing's decoder (`SwipeDecoder`) on the bundled English and Spanish models, with
/// swipes drawn through Hex's own key geometry. It has its own file because the decoder reads
/// a path without the keys' engine and its fixtures are paths; how the keys type a swipe is
/// in FullKeyboardEngineTests.
@MainActor
final class SwipeDecoderTests: XCTestCase {
private func decoder(_ language: KeyboardLanguage, swearing: Bool = false) throws -> (SwipeDecoder, TypingDecoder.Keys) {
let model = try XCTUnwrap(TypingDecoderModel.bundled(language.rawValue), "the \(language) model is bundled")
var decoder = SwipeDecoder(model: model)
decoder.neverSuggest = TypingDecoder.NeverSuggest(
OffensiveWords.bundled(language.rawValue).blocked(allowSwearing: swearing), model: model
)
if language == .english {
decoder.extraWords = SwipeDecoder.ExtraWords(
AutocorrectEngine.englishContractions, cost: { SwipeDecoder.costOfContraction($0) }, model: model
)
}
let keys = try XCTUnwrap(TypingDecoder.Keys(layout: KeyboardLayout(width: 390, language: language)))
return (decoder, keys)
}
private func read(_ word: String, _ decoder: SwipeDecoder, _ keys: TypingDecoder.Keys, after previous: String? = nil, offset: CGVector = .zero) -> [String] {
decoder.decode(SwipePath.through(word, keys: keys, offset: offset), keys: keys, previous: previous).map(\.text)
}
func testCleanSwipesReadAsTheirWords() throws {
let (en, keys) = try decoder(.english)
for word in ["hello", "the", "keyboard", "world", "typing", "people", "because", "quick", "with", "you", "and"] {
XCTAssertEqual(read(word, en, keys).first, word)
}
}
/// A finger that runs low all the way (as taps land below the keys' centres), or low and
/// to the left.
func testASwipeOffTheCentresStillReads() throws {
let (en, keys) = try decoder(.english)
for off in [CGVector(dx: 0, dy: keys.pitch.height * 0.3), CGVector(dx: -keys.pitch.width / 4, dy: keys.pitch.height / 4)] {
for word in ["hello", "keyboard", "people", "world", "typing", "because", "quick"] {
XCTAssertEqual(read(word, en, keys, offset: off).first, word, "\(off)")
}
}
}
func testSpanishSwipesKeepTheirAccentsAndTheirÑ() throws {
let (es, keys) = try decoder(.spanish)
for word in ["año", "también", "gracias", "mañana", "niño", "teclado"] {
XCTAssertEqual(read(word, es, keys).first, word)
}
XCTAssertEqual(Set(read("esta", es, keys).prefix(2)), ["esta", "está"], "one path, both readings")
}
/// Google Books splits "don't" into "do n't", so the model barely counts it: the decoder
/// reads Hex's English contractions as common words.
func testContractionsReadAsCommonWords() throws {
let (en, keys) = try decoder(.english)
XCTAssertEqual(read("dont", en, keys).first, "don't")
XCTAssertEqual(read("im", en, keys).first, "I'm")
}
func testTheWordBeforePicksBetweenWordsOnOnePath() throws {
let (en, keys) = try decoder(.english)
XCTAssertEqual(Array(read("to", en, keys).prefix(2)), ["to", "too"])
XCTAssertEqual(read("too", en, keys, after: "far").first, "too")
}
func testSlursAreNeverReadAndSwearWordsOnlyWhenAsked() throws {
for language in [KeyboardLanguage.english, .spanish] {
let words = OffensiveWords.bundled(language.rawValue)
for swearing in [false, true] {
let (decoder, keys) = try decoder(language, swearing: swearing)
let probes = language == .english ? ["fuck", "shit", "bitch", "nigger", "faggot"] : ["mierda", "cabron", "maricón"]
for probe in probes {
for reading in read(probe, decoder, keys) {
XCTAssertFalse(words.blocks(reading, allowSwearing: swearing), "\(probe) read as \(reading), swear words \(swearing)")
}
}
}
}
let (swearing, keys) = try decoder(.english, swearing: true)
XCTAssertEqual(read("fuck", swearing, keys).first, "fuck", "with Suggest swear words on")
}
/// A straight stroke runs over the letters between its ends: "tired" and "tied" draw one
/// line from i to e, "wire" and "write" one from w to i and back to e. Only where the
/// finger slowed tells them apart (X tester: guesses didn't account for the letters under
/// the swipe), so a pause on a letter the stroke crosses reads it, and the same stroke
/// without one doesn't.
func testAPauseOnALetterTheStrokeCrossesReadsIt() throws {
let (en, keys) = try decoder(.english)
func read(_ word: String, pausingOn pauses: Set<Int>) -> String? {
let (points, times) = SwipePath.timed(word, keys: keys, pausingOn: pauses)
return en.decode(points, times: times, keys: keys).first?.text
}
// The pauses are on the letters' keys in order (t i r e d: r is 2).
XCTAssertEqual(read("tired", pausingOn: [2]), "tired")
XCTAssertEqual(read("tired", pausingOn: []), "tied", "the same path, no pause on r")
XCTAssertEqual(read("tire", pausingOn: [2]), "tire")
XCTAssertEqual(read("tie", pausingOn: []), "tie")
XCTAssertEqual(read("wire", pausingOn: [2]), "wire")
XCTAssertEqual(read("write", pausingOn: [1, 3]), "write")
XCTAssertEqual(read("greet", pausingOn: [1]), "greet")
XCTAssertEqual(read("get", pausingOn: []), "get")
}
/// A pause off the word's keys counts against it: p to t along the top row is "put" when
/// the finger slows on u and "pit" on i; slowing on o and u offers the much rarer "pout".
func testAPauseOffTheWordsKeysCountsAgainstIt() throws {
let (en, keys) = try decoder(.english)
func read(_ word: String, pausingOn pauses: Set<Int>) -> [String] {
let (points, times) = SwipePath.timed(word, keys: keys, pausingOn: pauses)
return en.decode(points, times: times, keys: keys).map(\.text)
}
XCTAssertEqual(read("put", pausingOn: [1]).first, "put")
XCTAssertEqual(read("pit", pausingOn: [1]).first, "pit")
XCTAssertTrue(read("pout", pausingOn: [1, 2]).prefix(3).contains("pout"))
XCTAssertFalse(read("put", pausingOn: [1]).prefix(3).contains("pout"), "no pause on o, no pout")
// Touches that all arrive at once (a test's, or no clock) read as before.
let (points, _) = SwipePath.timed("pout", keys: keys, pausingOn: [1, 2])
let still = [TimeInterval](repeating: 0, count: points.count)
XCTAssertEqual(en.decode(points, times: still, keys: keys).map(\.text), en.decode(points, keys: keys).map(\.text))
}
func testAPathTooShortReadsAsNothing() throws {
let (en, keys) = try decoder(.english)
let h = try XCTUnwrap(keys.centres["h"])
XCTAssertTrue(en.decode([h, CGPoint(x: h.x + 4, y: h.y)], keys: keys).isEmpty)
XCTAssertTrue(en.decode([h], keys: keys).isEmpty)
}
}
/// Swipes for tests: straight strokes through the centres of a word's keys.
enum SwipePath {
/// Through the centres of `word`'s keys on `keys` (an accented letter on its base
/// letter's key, an apostrophe on none), a point every `step` points, moved by `offset`.
static func through(_ word: String, keys: TypingDecoder.Keys, offset: CGVector = .zero, step: CGFloat = 6) -> [CGPoint] {
var centres: [CGPoint] = []
for letter in word.lowercased() where letter != "'" {
let base = String(letter).decomposedStringWithCanonicalMapping.unicodeScalars.first.map(Character.init)
guard let centre = keys.centres[letter] ?? base.flatMap({ keys.centres[$0] }) else { continue }
if centres.last != centre { centres.append(centre) }
}
guard let first = centres.first else { return [] }
var points = [first]
for (a, b) in zip(centres, centres.dropFirst()) {
let count = max(1, Int((hypot(b.x - a.x, b.y - a.y) / step).rounded(.up)))
for i in 1...count {
let t = CGFloat(i) / CGFloat(count)
points.append(CGPoint(x: a.x + (b.x - a.x) * t, y: a.y + (b.y - a.y) * t))
}
}
return points.map { CGPoint(x: $0.x + offset.dx, y: $0.y + offset.dy) }
}
/// Through `word`'s keys as `through` does, at an even 750 points a second, with the
/// touches' times: the finger stays `pause` seconds on the keys at `pausingOn` (indexes
/// into the word's keys, repeats run together), sending nothing meanwhile, as a still
/// finger does.
static func timed(_ word: String, keys: TypingDecoder.Keys, pausingOn pauses: Set<Int>, pause: TimeInterval = 0.15) -> (points: [CGPoint], times: [TimeInterval]) {
var centres: [CGPoint] = []
for letter in word.lowercased() where letter != "'" {
let base = String(letter).decomposedStringWithCanonicalMapping.unicodeScalars.first.map(Character.init)
guard let centre = keys.centres[letter] ?? base.flatMap({ keys.centres[$0] }) else { continue }
if centres.last != centre { centres.append(centre) }
}
guard let first = centres.first else { return ([], []) }
var points = [first]
var times: [TimeInterval] = [0]
let step: CGFloat = 6, perStep: TimeInterval = 0.008
for (index, (a, b)) in zip(centres, centres.dropFirst()).enumerated() {
let count = max(1, Int((hypot(b.x - a.x, b.y - a.y) / step).rounded(.up)))
for i in 1...count {
let t = CGFloat(i) / CGFloat(count)
points.append(CGPoint(x: a.x + (b.x - a.x) * t, y: a.y + (b.y - a.y) * t))
// The first point after a paused key comes only once the finger moves on.
times.append(times[times.count - 1] + perStep + (i == 1 && pauses.contains(index) ? pause : 0))
}
}
return (points, times)
}
}