hex/ios/benchmark/typing-replay/README.md
Dax ea1fb6be26
iOS keyboard: the typing replay reports all-words accuracy, contractions and taps read as swipes (#2287)
* iOS keyboard: the typing replay reports all-words accuracy, contractions and taps read as swipes

The swipe report's all-words-accuracy divided in-dictionary right words by in-dictionary words. Rename it in-dictionary-accuracy and add an all-words-accuracy over every word, for swipes and taps. right now means the typed word matches regardless of the dictionary; the in-dictionary counters keep their filter.

Tap mode keeps Swipe typing attached, so a drifting tap can be promoted. Count those as read-as-swipes from SwipeTyping's active touch.

Scoring keeps apostrophes (curly ones straightened) and the dictionary check accepts AutocorrectEngine.englishContractions. Taps compare letters, digits and apostrophes; swipes compare letters only, since a swipe path cannot draw an apostrophe.

* iOS keyboard: the typing replay keys taps on ASCII letters and apostrophes

Restore the tap key's ASCII-letter behaviour: digits, numbers and accented letters no longer enter the key, so accuracy for inputs without apostrophes is unchanged. Trim leading and trailing apostrophes so quoted words and plural possessives stay in the dictionary, and keep swipe keys ASCII letters only.

Correct the benchmark note: --set test keeps every phone test condition the default filters keep, not every condition in the dataset.

* iOS keyboard: a tap's apostrophe counts only when the recorded word has one

Autocorrect adds apostrophes to recorded words that had none (cant -> can't), and the ASCII tap key made those wrong. tapKey compares apostrophes only when the intended word has one, and align uses the same key per intended-typed pair. A recorded don't still needs its apostrophe.
2026-11-21 20:32:00 -05:00

227 lines
12 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

# Typing replay converters
`TypingReplayTests` can replay real recorded touches through the keyboard. The
recordings are not vendored here. Download a public dataset yourself and convert
it with `convert.py`; the script only reads local files and makes no network
requests.
Both output shapes are described in `ios/HEXMobileTests/TypingReplayTests.swift`.
Swipes have one gesture per word, taps have every committed touch of the
sentence in order. Coordinates are in the unit grid: x 0…1 spans the ten columns
of the top letter row (q's left edge to p's right edge, so their centres are 0.05
and 0.95), y 0…1 runs from the top of the first letter row to the bottom of the
third (row centres at 1/6, 1/2 and 5/6), and rows two and three start half and one
and a half columns in.
Times are milliseconds from the gesture's first point (swipes) or the sentence's first
touch (taps). Each swipe also records `start`: when its finger landed, in milliseconds from
the sentence's first landing, so the replay keeps the person's own gaps between words. A file
without `start` still replays: the test begins each gesture 0.5 s after the previous lift,
longer than the two windows the replay's clock drives (the double-space period and the strip's
touch guard). A start that would overlap the previous touch is moved to its lift, and the
report counts those as `clamped`. A space tap has y = 1.25 and x along 0.25…0.75 of the
dataset's own space bar.
Every subcommand writes the same JSON object and prints sentences, touches and
what it skipped to stderr:
```sh
python3 convert.py <dataset> <input…> -o recording.json [--limit N]
```
`--limit` caps emitted sentences. Run the replay with:
```sh
TEST_RUNNER_HEX_SWIPE_REPLAY=recording.json xcodebuild test \
-project HEXMobile.xcodeproj -scheme HEXMobile -destination "id=<SIM_UDID>" \
-only-testing:HEXMobileTests/TypingReplayTests
```
Use `TEST_RUNNER_HEX_TAP_REPLAY` for a tap recording.
The report line’s fields are described in
[`ios/README.md`](../../README.md#replaying-real-typing): swipes report `attempts` (the
decoder’s denominator, missed attempts included), `in-dictionary-accuracy` (every
in-dictionary word, taps included and skipped gestures wrong), `all-words-accuracy` (every
word, out-of-dictionary ones included), `taps`/`taps-right` (recordings that are taps),
`not-swipes`, `skipped` and `clamped`; taps report `accuracy` (in-dictionary words),
`all-words-accuracy` (every word), `read-as-swipes` (recorded taps the plane promoted to
swipes) and `clamped`.
## FUTO swipe
[Dataset](https://huggingface.co/datasets/futo-org/swipe.futo.org) · MIT ·
[paper](https://huggingface.co/papers/2606.25247).
`dev.jsonl`, `test.jsonl` and `train.jsonl` are in the repository root;
`swipe-2/swipe2.jsonl` through `swipe-5/swipe5.jsonl` are the later runs. For
example:
```sh
curl -L -o dev.jsonl \
https://huggingface.co/datasets/futo-org/swipe.futo.org/resolve/main/dev.jsonl
python3 convert.py futo dev.jsonl -o futo-dev.json
```
```sh
python3 convert.py futo swipe-5/swipe5.jsonl -o futo-swipe5.json
```
`--valid-only` skips records whose `potentially_invalid_sentence` is true; it is
off by default. `distance` is not used.
Each JSONL line is one word: `session`, `sentence`, `word_idx`, `word`, and
`data` with epoch-millisecond `t` and canvas-normalised `x`/`y`. Records are
grouped by `(session, sentence)` and ordered by `word_idx`; a sentence is only
emitted when every word from `sentence.split()` is present. Words keep their
punctuation, which the tests ignore when comparing letters. Each gesture's `start`
is its first sample's epoch `t` less the sentence's first word's, so the gaps the
epoch stamps record survive the conversion (median about 0.84 s and minimum 20 ms
over `dev`).
The root splits do not carry `layout`, `language` or `dual_finger`; those are
assumed qwerty/English/single-finger. `swipe-5` does carry them, and records
with another layout, another language, or `dual_finger = 1` are skipped (a
dual-finger `data` value is an `L`/`R` object, not a point list). The dataset
has deliberate gaps when a volunteer skipped a word, so incomplete sentences are
common; the counts show how many were dropped. A one-letter word is often recorded as
a single point; it is kept, and the replay reads it as a tap.
`dev.jsonl` is about 304 MB, so the converter streams it line by line.
## How We Swipe
[Dataset](https://osf.io/sj67f) · Leiva et al., MobileHCI 2021 · MIT.
Download `swipelogs.zip` (about 70 MB) from OSF. Pass the zip directly, a
directory of logs, or a single log:
```sh
python3 convert.py howweswipe swipelogs.zip -o howweswipe.json
python3 convert.py howweswipe swipelogs/03ft0n5ol992m393fb9lpc0uqt.log -o one-user.json
```
Rows are space-separated: `sentence timestamp keyb_width keyb_height event
x_pos y_pos x_radius y_radius angle word is_err`. Words in `sentence` are
joined with underscores. A swipe is a run of consecutive `touchmove` rows for
the same word, split on a word change or `touchend`, following the approach of
the study's `stats_create.py`; the last attempt for each word in a sentence is
used. Rows shorter than 12 columns and rows whose `is_err` is not 0/1 are
ignored, and extra columns are ignored.
Each gesture's `start` comes from its first row's `time`, so the replay keeps the
gaps between words. The geometry is inferred from the study's web keyboard, not
exported with the logs: ten columns across `keyb_width`, and three letter rows of
pitch `(keyb_height - 4) / 3.7` starting one pixel down; treat it as approximate.
The converter has been run over the full `swipelogs.zip`: 1,338 logs, 20,118
sentences, 86,831 words, 1,465 sentences skipped for a missing word, no empty
gestures. It never reads `touchstart` rows: a swipe is its `touchmove` rows, so a
touch whose start wasn't logged still begins at its first move. Some logs write a
row twice — the same sentence, millisecond, point, event and word, sometimes with a
13th column — and the converter keeps both, adding the same point and time to the
path; taking the last attempt for a word hides the retries that differ. A word
recorded as a single point is kept, since the replay reads it as a tap.
## Noisy Typing on QWERTY Keyboards
[Dataset](https://zenodo.org/records/24225302) · Vertanen and Kristensson, IUI
2023 · CC-BY-4.0.
Download `vertanen_noisy_typing_json.zip` (about 22.5 MB) from Zenodo:
```sh
python3 convert.py noisy vertanen_noisy_typing_json.zip -o noisy-phone-test.json --set test
```
The zip holds `noisy_typing/json/keyboards.json` and a 422 MB
`noisy_typing/json/noisy_typing.json` array. The array is read one record at a
time, so memory stays bounded; each record is one sentence attempt with
`chunks` of `taps` whose `input` holds the touch samples.
The default device is `phone` (`--device` selects another). `--set dev`,
`--set test` or `--set all` (the default) filters normal sets; `*_word` sets
are not emitted unless `--word-sets` is given, because they are force-aligned
variants of the same input, but their per-word tap counts are still read to
segment the plain records. Deleted chunks are skipped, and so are records
without a reference (the composition tasks in `vt_exp3`). A keyboard's
coordinates are mapped with a line through each letter row's first and last key
centres for x, and a line through q's and z's row-mean y centres for y; the grid
centres are 0.05…0.95 across and 1/6, 1/2 and 5/6 down. A tap whose nearest key
is `<sp>` becomes a space tap; taps nearest to other non-letter keys are skipped.
Samples keep the dataset's relative times, shifted to the sentence's first touch.
The eyes-free studies are left out unless `--eyes-free` is given:
`eyesfree_exp1`'s `blindfolded` condition and `eyesfree_exp2` tapped a keyboard
they only imagined, so their touches do not land where the keys are. The `small`
and `tiny` conditions of `vt_exp3` and `vt_potential` are left out unless
`--small-keyboards` is given: they used smaller-than-full-size keyboards, not the
full-size sighted typing the other datasets measure. `--limit` takes records in the
dataset's own order, not a sample: with both filters off, the first ~266 phone test
records are `eyesfree_exp1`'s sighted condition, so a small cut measures only that
study.
The logs do not always record spaces. `vt_exp1`, `vt_exp2`'s `nospace`
condition and `vt_exp3` typed a run of letters and let their decoder choose the
word breaks; the `swipe` condition and `eyesfree_exp2` drew a space as a right
swipe. A touch that travels more than two q-key widths along x is read as a space
when its record has no spacebar tap. A record with fewer space taps than the reference
has words is segmented by the tap counts of its force-aligned `*_word` twin, which holds
that attempt's letter taps split at the reference's words; a record whose twin is missing
stays one run of letters. The stderr line counts how many sentences were split either way
and how many ran together.
`--set test` keeps every phone test condition the default filters keep (they drop
`eyesfree_exp1`'s blindfolded condition, `eyesfree_exp2`, and the small/tiny keyboards),
including the no-space and swipe-for-space conditions and records whose words ran together, so
its number is not comparable to a sample of only normal-space sentences.
## Tap Typing with Touch Sensing Images
[Dataset](https://github.com/google-research-datasets/tap-typing-with-touch-sensing-images)
· CC-BY-4.0.
Download `touch_data.csv` (about 28 MB), `prompt_data.csv` and
`keyboard_data.json` from the repository, then:
```sh
python3 convert.py tsi touch_data.csv prompt_data.csv keyboard_data.json -o tsi.json
```
Rows are grouped by `(participant_id, task_id, trial_id)` and sorted by
`timestamp_ms`. `was_deleted` taps are dropped. The default `--prompt phrase`
keeps only `prompt_type == "phrase"`; `--prompt all` also keeps the random
letter strings, which are deliberate non-words. The CSV has very large heatmap
columns, so the converter raises Python's CSV field limit and only keeps the
coordinate columns.
Coordinates are mapped with the single Pixel 6 Pro keyboard in
`keyboard_data.json`: the x pitch comes from q and p's centres (with a half-key
margin so q's centre is 0.05), and the y pitch comes from q and z. A tap whose
nearest key is `SPACE` becomes a space tap, and one whose nearest key is `.` is
dropped; the reference's `ref_char` is never used to choose what to keep. Words
stay as `prompt.split()`, including their punctuation.
## Biases
- TSI drops `was_deleted` taps, so a mistake the participant fixed is invisible.
- Noisy records with fewer space taps than reference words and no `_word` twin stay one
run of letters (the stderr line's "run together"), so their later words cannot come out
right.
- FUTO ignores `distance`; `--valid-only` skips records marked
`potentially_invalid_sentence` and is off by default.
- How We Swipe's keyboard geometry is inferred from the study's web keyboard, not exported
with the logs.
- How We Swipe logs some rows twice, so a swipe's path can hold the same point and time two or
more times; the repeat adds no new sample. Retries that differ are hidden by taking the last
attempt for a word.
## Pitfalls
- The data is not vendored and the converters never download anything.
- FUTO root splits and How We Swipe logs are recordings of what people did, not
clean synthetic gestures; incomplete sentences and retries are normal.
- Noisy Typing's JSON is much larger than its zip; use `--limit` while
experimenting. `--set`, `--word-sets`, `--eyes-free` and `--small-keyboards`
change which attempts are emitted.
- TSI's `--prompt phrase` is the useful default; random prompts are not words.
- Output can be large. The tests report per-source accuracy, and `HEX_REPLAY_LIMIT`
caps sentences inside the test as well.