mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS keyboard: the typing replay reports all-words accuracy, contractions and taps read as swipes The swipe report's all-words-accuracy divided in-dictionary right words by in-dictionary words. Rename it in-dictionary-accuracy and add an all-words-accuracy over every word, for swipes and taps. right now means the typed word matches regardless of the dictionary; the in-dictionary counters keep their filter. Tap mode keeps Swipe typing attached, so a drifting tap can be promoted. Count those as read-as-swipes from SwipeTyping's active touch. Scoring keeps apostrophes (curly ones straightened) and the dictionary check accepts AutocorrectEngine.englishContractions. Taps compare letters, digits and apostrophes; swipes compare letters only, since a swipe path cannot draw an apostrophe. * iOS keyboard: the typing replay keys taps on ASCII letters and apostrophes Restore the tap key's ASCII-letter behaviour: digits, numbers and accented letters no longer enter the key, so accuracy for inputs without apostrophes is unchanged. Trim leading and trailing apostrophes so quoted words and plural possessives stay in the dictionary, and keep swipe keys ASCII letters only. Correct the benchmark note: --set test keeps every phone test condition the default filters keep, not every condition in the dataset. * iOS keyboard: a tap's apostrophe counts only when the recorded word has one Autocorrect adds apostrophes to recorded words that had none (cant -> can't), and the ASCII tap key made those wrong. tapKey compares apostrophes only when the intended word has one, and align uses the same key per intended-typed pair. A recorded don't still needs its apostrophe.
227 lines
12 KiB
Markdown
227 lines
12 KiB
Markdown
# Typing replay converters
|
||
|
||
`TypingReplayTests` can replay real recorded touches through the keyboard. The
|
||
recordings are not vendored here. Download a public dataset yourself and convert
|
||
it with `convert.py`; the script only reads local files and makes no network
|
||
requests.
|
||
|
||
Both output shapes are described in `ios/HEXMobileTests/TypingReplayTests.swift`.
|
||
Swipes have one gesture per word, taps have every committed touch of the
|
||
sentence in order. Coordinates are in the unit grid: x 0…1 spans the ten columns
|
||
of the top letter row (q's left edge to p's right edge, so their centres are 0.05
|
||
and 0.95), y 0…1 runs from the top of the first letter row to the bottom of the
|
||
third (row centres at 1/6, 1/2 and 5/6), and rows two and three start half and one
|
||
and a half columns in.
|
||
Times are milliseconds from the gesture's first point (swipes) or the sentence's first
|
||
touch (taps). Each swipe also records `start`: when its finger landed, in milliseconds from
|
||
the sentence's first landing, so the replay keeps the person's own gaps between words. A file
|
||
without `start` still replays: the test begins each gesture 0.5 s after the previous lift,
|
||
longer than the two windows the replay's clock drives (the double-space period and the strip's
|
||
touch guard). A start that would overlap the previous touch is moved to its lift, and the
|
||
report counts those as `clamped`. A space tap has y = 1.25 and x along 0.25…0.75 of the
|
||
dataset's own space bar.
|
||
|
||
Every subcommand writes the same JSON object and prints sentences, touches and
|
||
what it skipped to stderr:
|
||
|
||
```sh
|
||
python3 convert.py <dataset> <input…> -o recording.json [--limit N]
|
||
```
|
||
|
||
`--limit` caps emitted sentences. Run the replay with:
|
||
|
||
```sh
|
||
TEST_RUNNER_HEX_SWIPE_REPLAY=recording.json xcodebuild test \
|
||
-project HEXMobile.xcodeproj -scheme HEXMobile -destination "id=<SIM_UDID>" \
|
||
-only-testing:HEXMobileTests/TypingReplayTests
|
||
```
|
||
|
||
Use `TEST_RUNNER_HEX_TAP_REPLAY` for a tap recording.
|
||
|
||
The report line’s fields are described in
|
||
[`ios/README.md`](../../README.md#replaying-real-typing): swipes report `attempts` (the
|
||
decoder’s denominator, missed attempts included), `in-dictionary-accuracy` (every
|
||
in-dictionary word, taps included and skipped gestures wrong), `all-words-accuracy` (every
|
||
word, out-of-dictionary ones included), `taps`/`taps-right` (recordings that are taps),
|
||
`not-swipes`, `skipped` and `clamped`; taps report `accuracy` (in-dictionary words),
|
||
`all-words-accuracy` (every word), `read-as-swipes` (recorded taps the plane promoted to
|
||
swipes) and `clamped`.
|
||
|
||
## FUTO swipe
|
||
|
||
[Dataset](https://huggingface.co/datasets/futo-org/swipe.futo.org) · MIT ·
|
||
[paper](https://huggingface.co/papers/2606.25247).
|
||
|
||
`dev.jsonl`, `test.jsonl` and `train.jsonl` are in the repository root;
|
||
`swipe-2/swipe2.jsonl` through `swipe-5/swipe5.jsonl` are the later runs. For
|
||
example:
|
||
|
||
```sh
|
||
curl -L -o dev.jsonl \
|
||
https://huggingface.co/datasets/futo-org/swipe.futo.org/resolve/main/dev.jsonl
|
||
python3 convert.py futo dev.jsonl -o futo-dev.json
|
||
```
|
||
|
||
```sh
|
||
python3 convert.py futo swipe-5/swipe5.jsonl -o futo-swipe5.json
|
||
```
|
||
|
||
`--valid-only` skips records whose `potentially_invalid_sentence` is true; it is
|
||
off by default. `distance` is not used.
|
||
|
||
Each JSONL line is one word: `session`, `sentence`, `word_idx`, `word`, and
|
||
`data` with epoch-millisecond `t` and canvas-normalised `x`/`y`. Records are
|
||
grouped by `(session, sentence)` and ordered by `word_idx`; a sentence is only
|
||
emitted when every word from `sentence.split()` is present. Words keep their
|
||
punctuation, which the tests ignore when comparing letters. Each gesture's `start`
|
||
is its first sample's epoch `t` less the sentence's first word's, so the gaps the
|
||
epoch stamps record survive the conversion (median about 0.84 s and minimum 20 ms
|
||
over `dev`).
|
||
|
||
The root splits do not carry `layout`, `language` or `dual_finger`; those are
|
||
assumed qwerty/English/single-finger. `swipe-5` does carry them, and records
|
||
with another layout, another language, or `dual_finger = 1` are skipped (a
|
||
dual-finger `data` value is an `L`/`R` object, not a point list). The dataset
|
||
has deliberate gaps when a volunteer skipped a word, so incomplete sentences are
|
||
common; the counts show how many were dropped. A one-letter word is often recorded as
|
||
a single point; it is kept, and the replay reads it as a tap.
|
||
|
||
`dev.jsonl` is about 304 MB, so the converter streams it line by line.
|
||
|
||
## How We Swipe
|
||
|
||
[Dataset](https://osf.io/sj67f) · Leiva et al., MobileHCI 2021 · MIT.
|
||
|
||
Download `swipelogs.zip` (about 70 MB) from OSF. Pass the zip directly, a
|
||
directory of logs, or a single log:
|
||
|
||
```sh
|
||
python3 convert.py howweswipe swipelogs.zip -o howweswipe.json
|
||
python3 convert.py howweswipe swipelogs/03ft0n5ol992m393fb9lpc0uqt.log -o one-user.json
|
||
```
|
||
|
||
Rows are space-separated: `sentence timestamp keyb_width keyb_height event
|
||
x_pos y_pos x_radius y_radius angle word is_err`. Words in `sentence` are
|
||
joined with underscores. A swipe is a run of consecutive `touchmove` rows for
|
||
the same word, split on a word change or `touchend`, following the approach of
|
||
the study's `stats_create.py`; the last attempt for each word in a sentence is
|
||
used. Rows shorter than 12 columns and rows whose `is_err` is not 0/1 are
|
||
ignored, and extra columns are ignored.
|
||
|
||
Each gesture's `start` comes from its first row's `time`, so the replay keeps the
|
||
gaps between words. The geometry is inferred from the study's web keyboard, not
|
||
exported with the logs: ten columns across `keyb_width`, and three letter rows of
|
||
pitch `(keyb_height - 4) / 3.7` starting one pixel down; treat it as approximate.
|
||
The converter has been run over the full `swipelogs.zip`: 1,338 logs, 20,118
|
||
sentences, 86,831 words, 1,465 sentences skipped for a missing word, no empty
|
||
gestures. It never reads `touchstart` rows: a swipe is its `touchmove` rows, so a
|
||
touch whose start wasn't logged still begins at its first move. Some logs write a
|
||
row twice — the same sentence, millisecond, point, event and word, sometimes with a
|
||
13th column — and the converter keeps both, adding the same point and time to the
|
||
path; taking the last attempt for a word hides the retries that differ. A word
|
||
recorded as a single point is kept, since the replay reads it as a tap.
|
||
|
||
## Noisy Typing on QWERTY Keyboards
|
||
|
||
[Dataset](https://zenodo.org/records/24225302) · Vertanen and Kristensson, IUI
|
||
2023 · CC-BY-4.0.
|
||
|
||
Download `vertanen_noisy_typing_json.zip` (about 22.5 MB) from Zenodo:
|
||
|
||
```sh
|
||
python3 convert.py noisy vertanen_noisy_typing_json.zip -o noisy-phone-test.json --set test
|
||
```
|
||
|
||
The zip holds `noisy_typing/json/keyboards.json` and a 422 MB
|
||
`noisy_typing/json/noisy_typing.json` array. The array is read one record at a
|
||
time, so memory stays bounded; each record is one sentence attempt with
|
||
`chunks` of `taps` whose `input` holds the touch samples.
|
||
|
||
The default device is `phone` (`--device` selects another). `--set dev`,
|
||
`--set test` or `--set all` (the default) filters normal sets; `*_word` sets
|
||
are not emitted unless `--word-sets` is given, because they are force-aligned
|
||
variants of the same input, but their per-word tap counts are still read to
|
||
segment the plain records. Deleted chunks are skipped, and so are records
|
||
without a reference (the composition tasks in `vt_exp3`). A keyboard's
|
||
coordinates are mapped with a line through each letter row's first and last key
|
||
centres for x, and a line through q's and z's row-mean y centres for y; the grid
|
||
centres are 0.05…0.95 across and 1/6, 1/2 and 5/6 down. A tap whose nearest key
|
||
is `<sp>` becomes a space tap; taps nearest to other non-letter keys are skipped.
|
||
Samples keep the dataset's relative times, shifted to the sentence's first touch.
|
||
|
||
The eyes-free studies are left out unless `--eyes-free` is given:
|
||
`eyesfree_exp1`'s `blindfolded` condition and `eyesfree_exp2` tapped a keyboard
|
||
they only imagined, so their touches do not land where the keys are. The `small`
|
||
and `tiny` conditions of `vt_exp3` and `vt_potential` are left out unless
|
||
`--small-keyboards` is given: they used smaller-than-full-size keyboards, not the
|
||
full-size sighted typing the other datasets measure. `--limit` takes records in the
|
||
dataset's own order, not a sample: with both filters off, the first ~266 phone test
|
||
records are `eyesfree_exp1`'s sighted condition, so a small cut measures only that
|
||
study.
|
||
|
||
The logs do not always record spaces. `vt_exp1`, `vt_exp2`'s `nospace`
|
||
condition and `vt_exp3` typed a run of letters and let their decoder choose the
|
||
word breaks; the `swipe` condition and `eyesfree_exp2` drew a space as a right
|
||
swipe. A touch that travels more than two q-key widths along x is read as a space
|
||
when its record has no spacebar tap. A record with fewer space taps than the reference
|
||
has words is segmented by the tap counts of its force-aligned `*_word` twin, which holds
|
||
that attempt's letter taps split at the reference's words; a record whose twin is missing
|
||
stays one run of letters. The stderr line counts how many sentences were split either way
|
||
and how many ran together.
|
||
|
||
`--set test` keeps every phone test condition the default filters keep (they drop
|
||
`eyesfree_exp1`'s blindfolded condition, `eyesfree_exp2`, and the small/tiny keyboards),
|
||
including the no-space and swipe-for-space conditions and records whose words ran together, so
|
||
its number is not comparable to a sample of only normal-space sentences.
|
||
|
||
## Tap Typing with Touch Sensing Images
|
||
|
||
[Dataset](https://github.com/google-research-datasets/tap-typing-with-touch-sensing-images)
|
||
· CC-BY-4.0.
|
||
|
||
Download `touch_data.csv` (about 28 MB), `prompt_data.csv` and
|
||
`keyboard_data.json` from the repository, then:
|
||
|
||
```sh
|
||
python3 convert.py tsi touch_data.csv prompt_data.csv keyboard_data.json -o tsi.json
|
||
```
|
||
|
||
Rows are grouped by `(participant_id, task_id, trial_id)` and sorted by
|
||
`timestamp_ms`. `was_deleted` taps are dropped. The default `--prompt phrase`
|
||
keeps only `prompt_type == "phrase"`; `--prompt all` also keeps the random
|
||
letter strings, which are deliberate non-words. The CSV has very large heatmap
|
||
columns, so the converter raises Python's CSV field limit and only keeps the
|
||
coordinate columns.
|
||
|
||
Coordinates are mapped with the single Pixel 6 Pro keyboard in
|
||
`keyboard_data.json`: the x pitch comes from q and p's centres (with a half-key
|
||
margin so q's centre is 0.05), and the y pitch comes from q and z. A tap whose
|
||
nearest key is `SPACE` becomes a space tap, and one whose nearest key is `.` is
|
||
dropped; the reference's `ref_char` is never used to choose what to keep. Words
|
||
stay as `prompt.split()`, including their punctuation.
|
||
|
||
## Biases
|
||
|
||
- TSI drops `was_deleted` taps, so a mistake the participant fixed is invisible.
|
||
- Noisy records with fewer space taps than reference words and no `_word` twin stay one
|
||
run of letters (the stderr line's "run together"), so their later words cannot come out
|
||
right.
|
||
- FUTO ignores `distance`; `--valid-only` skips records marked
|
||
`potentially_invalid_sentence` and is off by default.
|
||
- How We Swipe's keyboard geometry is inferred from the study's web keyboard, not exported
|
||
with the logs.
|
||
- How We Swipe logs some rows twice, so a swipe's path can hold the same point and time two or
|
||
more times; the repeat adds no new sample. Retries that differ are hidden by taking the last
|
||
attempt for a word.
|
||
|
||
## Pitfalls
|
||
|
||
- The data is not vendored and the converters never download anything.
|
||
- FUTO root splits and How We Swipe logs are recordings of what people did, not
|
||
clean synthetic gestures; incomplete sentences and retries are normal.
|
||
- Noisy Typing's JSON is much larger than its zip; use `--limit` while
|
||
experimenting. `--set`, `--word-sets`, `--eyes-free` and `--small-keyboards`
|
||
change which attempts are emitted.
|
||
- TSI's `--prompt phrase` is the useful default; random prompts are not words.
|
||
- Output can be large. The tests report per-source accuracy, and `HEX_REPLAY_LIMIT`
|
||
caps sentences inside the test as well.
|