mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS keyboard: the live preview's motion, smooth and calm
What: one spring family (duration and bounce 0, UIKit's springDuration curve) for every
change while dictating, all of it drawn by one view so no memory comes back:
- Keys out, words in as recording starts: the keys fade (.2) as the words fade in rising
6 pt (.3), the same frame. Words out, keys back as the dictation goes in: the words fade
drifting 6 pt up toward the field (.15) as the keys fade in (.3); the words stop taking
touches at once, so the keys work the moment the dictation ends.
- Words arriving (LiveWordsView): each new word fades in rising 4 pt (.3), a burst's words
40 ms apart and all of them within 240 ms, so the last is in long before Apple's next
burst (about once a second). "Listening…" fades out a beat (60 ms) ahead of the first words.
- Tentative to settled: the muted tone eases to the keys' ink (.3).
- A new line: the block moves up by a critically damped spring (.3) that carries on from
where it is when another line comes, and the line leaving the top fades through a 14 pt band
instead of being cut.
- A changed word ("Okay" to "Hey", "730." to "7.30."): the old word fades out where it was
(.15) as the new one fades in (.2); words keep their place by position, so the words after
it never re-fade, and a word that only gains a mark stays as it is.
- The recording bar's morph has no bounce any more (was damping .82) and its fades are the
same spring (.2), so nothing bounces.
- Reduce Motion: fades only (.2), nothing rises, drifts or slides.
LiveWordsView draws the words with one TextKit 1 layout into one backing store the size of the
box, from a display link that runs only while something moves, and drops whole lines well out
of view so the lines that stay never re-break. It replaces the preview's label.
Why: Ryan asked for smoother motion with better easing after watching the phone recordings
(keys and words swapped in 180 ms with a double exposure; a burst's words popped in at once).
Verified: LiveDictationTests (10) on the Studio simulator; the Studio simulator with the 60 s
replay: footprint +2.7 MB at most while recording (+5.7 with #2410's label), 1,868 frames
drawn while moving, none late; frames of a recording checked for the start, a burst, a new
line and the end. code-shape.py passes.
* iOS keyboard: the preview's words move by Core Animation, not a display link
What: LiveWordsView no longer redraws every frame. It lays the words out and draws them once
per update, in one layer, and Core Animation does every frame of the motion: what changed
cross-fades in place (0.3 s, the cubic closest to a no-bounce spring: new words, a settled
tail turning to ink, a word the recogniser changed), the block moves up by whole lines with a
no-bounce spring (0.3 s) inside a box whose 14 pt band fades the line leaving the top, and once
at rest that line fades out of the band and the lines above the box are dropped from the layout
with nothing on screen changing. The per-word stagger and rise are gone.
Why: on the iPhone 17 Pro the display link drawing the words got only a quarter of its frames
on time while recording (intervals 12-20 ms 26%, 20-30 ms 23%, 30-45 ms 27%, longer 24%; the
drawing itself took 0.3 ms): the main thread is shared with the capture, the recogniser and the
Dot Scope. Core Animation's own animations keep the display's pace regardless.
Verified: LiveDictationTests (9, including whole-line scrolling and dropping the lines above the
box) and recordings on the Studio simulator: a burst, settling, a new line with the band's fade,
the drop at rest, the start and the end; 60 s replay +5.9 MB at most against +7.0 MB for the
same recording without the preview; 1.4 ms of main thread per update.
* iOS README: the live preview's motion, as it now runs
What: the live-words bullet in ios/README.md describes the motion as it ships: one family (a
no-bounce spring by its duration, and its closest cubic for the cross-fade), Core Animation
running every frame, what changes cross-fading in place while the rest stays exactly as it is,
whole-line scrolling through a 14 pt band, and Reduce Motion as cross-fades only. Replaces the
paragraph about per-word staggers and a display link, which the previous commit removed.
Why: the README is where the keyboard's behaviour is written down for the next person.
Verified on the iPhone 17 Pro with 9869899c installed, warm (Parakeet compiled before measuring):
- 60 s dictation, preview on: 59.1 MB at the end, 62.9 MB peak while recording, 76.3 MB at
transcription (peak 77.3); #2410 was 60.1 / 66.5 / 78.7. Preview off: 58.3 / 60.0 / 75.3.
- The preview's main-thread share: 1.69 ms per update on average, 3.40 ms at most (242 updates).
- In the 30 fps phone recordings, words that didn't change hold the same brightness to the
decimal through every cross-fade near them; a changing line eases over about 0.2 s with no
stall; the field's text appears on the frame the preview's words go.
Correction to 9869899c's message: its "+7.0 MB for the same recording without the preview" on
the simulator wasn't measured (the simulator's memory script only runs with the preview); the
phone numbers above are the comparison.
2303 lines
No EOL
183 KiB
Markdown
2303 lines
No EOL
183 KiB
Markdown
# Hex for iPhone
|
||
|
||
**Status:** Internal TestFlight app (`HEXMobile`), custom keyboard extension
|
||
(`HEXKeyboard`), and Live Activity + Control Center widget extension
|
||
(`HEXWidgets`). The containing app records a generation-scoped temporary WAV,
|
||
transcribes on-device with Apple Speech on iOS 26+ (`SpeechTranscriber`, or
|
||
`DictationTranscriber` for the languages it lacks, such as Norwegian, Danish,
|
||
Swedish and Hebrew) or FluidAudio Parakeet V2 (iOS 18+, English). Before iOS 26,
|
||
every other language, and English on 3 GB iPhones (iPhone XR, SE 2nd generation)
|
||
or after Parakeet couldn't load, goes to Apple's `SFSpeechRecognizer` on device
|
||
only (`OnDeviceSpeechTranscriber`, chosen by `SpeechEngineChoice`). Hex removes
|
||
each capture's file after inference, cancellation, or failure. During a warm session the keyboard
|
||
extension sends commands and inserts completed text over the shared App Group
|
||
(`group.anomaly.hex.mobile`). Otherwise it dictates by itself (in-keyboard
|
||
dictation, iOS 26+): iOS refuses to record from a keyboard extension
|
||
directly, so it captures through `getUserMedia` in a hidden `WKWebView`
|
||
under Hex's microphone permission, writes a temporary WAV, and transcribes
|
||
it on-device with Apple Speech, downloading the speech assets on first use
|
||
(`HEXKeyboard/KeyboardLocalDictation.swift`), or, with Experiments › Speech model in
|
||
keyboard on, with Parakeet on the Neural Engine in the keyboard's own process
|
||
(`HEXKeyboard/KeyboardParakeet.swift`, below). With Full Access, the web view
|
||
and its page load as the keyboard shows (unless memory is already short), without
|
||
the microphone, so no orange dot shows until you dictate; the full keyboard's
|
||
Hex key then starts capture as the finger lands, before the touch is a tap or
|
||
a hold, and a slide to the style menu or a cancelled touch drops it with
|
||
nothing transcribed (its orange dot shows for that moment). iOS ends a keyboard that passes
|
||
its memory limit (about 60–90 MB depending on the model), so while less is left
|
||
than starting needs (`os_proc_available_memory`: 32 MB for the first
|
||
dictation, 24 MB once the capture web view is loaded) Dictate and the Hex key
|
||
open Hex instead, as before iOS 26. Audio never leaves the iPhone. From iOS 26 Hex
|
||
never asks for Speech Recognition access: that permission covers `SFSpeechRecognizer`,
|
||
which Hex uses only before iOS 26, and `SpeechAnalyzer` runs without it in the app and
|
||
the keyboard alike (a TestFlight tester who turned it off dictated, iOS 27.0.1, build 51).
|
||
Before iOS 26 Hex asks for it as it gets `SFSpeechRecognizer` ready (setup, or Set up),
|
||
never at a launch by itself.
|
||
|
||
## Capabilities
|
||
|
||
- **One-tap keyboard dictation (`HEXKeyboard`):**
|
||
- **First run (`HEXMobile/Onboarding.swift`):** The desktop's pages and
|
||
order. **Setting up Hex** starts the speech model downloading (the engine
|
||
`transcriberFactory` picks; a launch that found Parakeet ended mid-load
|
||
stays failed until Try again) and moves on after a beat. **With Hex**, a
|
||
short demo, moves on by itself. **Let Hex hear you** has two rows, the
|
||
microphone (its pill and the page's Continue say Continue before iOS's
|
||
prompt, never Allow, per App Review 5.1.1(iv)) and the Hex keyboard with
|
||
Full Access (Open Settings, Hex's page); a done row shows only its check,
|
||
and the page moves on once both are done. Continue never needs the
|
||
microphone. **Almost ready** shows only while the model downloads, with
|
||
Continue without a model after a failure or 30 s without progress.
|
||
**Your turn** keeps the keyboard up and counts once a dictation arrives or
|
||
three words are typed with Full Access; an empty dictation says Hex heard
|
||
nothing (the keyboard notes it in the App Group). The keyboard also notes
|
||
when it comes on screen and leaves (`KeyboardPresence`): when Hex's
|
||
keyboard isn't up 2 s after a keyboard switch, the page says to tap the
|
||
globe until it is, and only words that arrive while it's up count as
|
||
typed. iOS's dictation and other keyboards take it off screen, so their
|
||
text never finishes the practice; the page says to tap the Hex key
|
||
instead. **Hex is ready** shows
|
||
only once every step is done, else Almost ready with what's left. The
|
||
model step counts the keyboard's own model too
|
||
(`KeyboardSpeechPreparation`): on iOS 26 the keyboard dictates with
|
||
Apple Speech even when Hex uses Parakeet, so Hex installs that language's
|
||
model as the keyboard would; where Apple has none on the iPhone, it marks
|
||
`keyboardNeedsHex` so the Hex key opens Hex from the first tap. Home's
|
||
Ready counts it the same way. The
|
||
model's progress pill sits in the top bar until it's ready; Skip shows on
|
||
the demo and the practice only. It shows once: `onboarding.completed` in
|
||
the app's UserDefaults records Start using Hex or Continue without a
|
||
model, `onboarding.page` resumes a quit part-way on the same page, and a
|
||
launch that finds the keyboard ready and the microphone allowed (an
|
||
existing user) records completion without showing it. **Run setup again**
|
||
replays it from page 1. Its `--snap-state` scenes are `onboarding-setup`,
|
||
`-demo`, `-permissions`, `-permissions-done`, `-waiting`, `-failed`,
|
||
`-try`, `-try-done`, `-try-empty`, `-try-off`, `-ready`, and
|
||
`onboarding-tour`, which walks every page by itself for a recording.
|
||
- **Without Full Access (App Review 4.4.1):** Hex is a keyboard first. The
|
||
full keyboard types without Full Access or network: letters, shift and caps
|
||
lock, numbers and symbols, accents, delete, return, the globe key, and
|
||
suggestions and autocorrect (`UITextChecker`, the supplementary lexicon,
|
||
and, where a typing model is bundled for the language, Spatial autocorrect's
|
||
model: English, Spanish and French, and German with its experiment on). Words
|
||
it learns may last only while the keyboard runs, as an iPhone may not let
|
||
it write the App Group; haptics, key clicks (Apple's keyboard guide), Paste and
|
||
dictation need Full Access. Without Full
|
||
Access the keyboard always mounts the full keyboard, whichever keyboard is
|
||
chosen (the dictation bar types only spaces and line breaks). Dictation needs
|
||
Full Access (Hex's App Group and the microphone): tapping the Hex key, or
|
||
sliding for its menu, shows for 8 s in the suggestion strip "Allow Full
|
||
Access to dictate" and "Settings › Apps › Hex › Keyboards", with an **Open
|
||
Hex** chip (`hex-dictation://keyboard/setup`) that opens the app's setup
|
||
sheet. Nothing else opens Hex without Full Access, and the strip shows no
|
||
notice until the Hex key is tapped.
|
||
- **Setup:** Home's status line counts the setup steps left ("Finish
|
||
setup · 1 step ›", else "Ready") and opens the same sheet: live status for
|
||
the keyboard, Full Access, the microphone ("For dictation" with
|
||
**Continue**, which iOS's prompt follows, until it's asked; never a button
|
||
that says Allow, per App Review 5.1.1(iv)), and the speech model, a field to
|
||
switch to Hex in while Full Access is unconfirmed, and **Open Settings**
|
||
(Hex's page in Settings) while the keyboard isn't ready. iOS reports Full
|
||
Access only inside the keyboard, so the keyboard reports it to the App Group
|
||
as it loads, each time it appears and every 5 s while shown
|
||
(`keyboard.fullAccessOn`/`Off`, with when its process started, and
|
||
`keyboard.fullAccessSeenAt` for an on), and posts a Darwin notification the
|
||
app refreshes on. Full Access shows as on once a keyboard has run with it,
|
||
and as off only when a keyboard said so since Hex last came to the
|
||
foreground (it changes only in Settings), from a keyboard process no older
|
||
than the one that last saw it on. Otherwise the sheet says "Not confirmed
|
||
yet" with "Switch to Hex below to confirm." under it, and focuses the field.
|
||
- **Opening Hex:** SwiftUI `Link`, `openURL`, and `extensionContext.open`
|
||
do nothing in a keyboard extension, so the keyboard opens Hex through
|
||
the `UIApplication` at the end of its responder chain
|
||
(`ContainingAppOpener`).
|
||
- **Cold start (before iOS 26):** Tapping **Dictate** opens the containing app
|
||
(`hex-dictation://keyboard/record?...`) with a 10-second
|
||
handoff window and starts recording. Hex never opens another app: the
|
||
handoff screen shows a quiet "Swipe bar to return" cue over the home
|
||
indicator, and you swipe back to the app you were typing in. The text goes in
|
||
once the keyboard is on screen there again: iOS often starts a new keyboard on
|
||
the way back, and one that inserted while it loaded, before its field was
|
||
connected, lost the text (iPad mini, iPadOS 18, TestFlight build 115).
|
||
- **GPU Dot Scope (`HEXShared/DotScope.swift`, `DotScopeShader.metal`):**
|
||
Both `HEXMobile` and `HEXKeyboard` render Hex's desktop menu-bar braided
|
||
oscilloscope (`src/dot_scope.rs`, `src/rune_hud.wgsl`) at native display
|
||
refresh via `CAMetalLayer` + `CADisplayLink`, pausing at 0% CPU/GPU once
|
||
settled in idle. The keyboard extension mounts it only while a capture or
|
||
transcription is active and shows a static `Canvas` dot row at rest, so a
|
||
fresh keyboard process builds no Metal device, pipeline, layer, display
|
||
link, or idle pre-roll before its first frame; the app's docked keyboard
|
||
preview keeps the live scope. The home-indicator swipe cue's pipeline is
|
||
compiled on first use, which only the app shows. Both layers turn off the Metal
|
||
Performance HUD (Settings › Developer › Graphics HUD), which would draw its frame
|
||
rate under the home well's Dictate pill, unless launched with `MTL_HUD_ENABLED=1`. Lock Screen / Dynamic
|
||
Island Live Activities render a
|
||
matching `Canvas` dot-matrix badge (`HexDotMatrixBadge`) because WidgetKit
|
||
archives static view snapshots rather than hosting live `CAMetalLayer`s.
|
||
Meter polling stays alive through a temporary non-recording state of the
|
||
same capture; it resumes app levels and keyboard snapshots when the recorder
|
||
is ready again. Capture completion, interruption, and Stop end polling.
|
||
- **Warm session (`Off`, `5m`, `15m`, `1h`):** Picking a duration on **Keep
|
||
mic warm** starts an `AVAudioEngine` input session immediately so
|
||
subsequent taps on **Dictate** record directly inside the keyboard with zero
|
||
app switches; picking `Off` or tapping the header timer ends it. Unless
|
||
chosen, it is `Off`, with the row hidden, where the keyboard dictates by
|
||
itself (`KeyboardSelfDictation`: iOS 26, and the keyboard hasn't opened Hex
|
||
for a dictation in the last 7 days, for any reason: the keyboard records its
|
||
own (memory, a missing speech model, the Multilingual model) and Hex records
|
||
every keyboard launch, since TestFlight build 129 showed keyboards that opened
|
||
Hex for reasons they left no record of; a dictation the keyboard finishes by
|
||
itself clears it), and `15m` elsewhere; worked out on each read, never
|
||
stored. The row shows whenever a duration is in effect, and stays on the
|
||
page once picked. Once on iOS 26, a stored `15m` (which tapping the old
|
||
default to warm the mic also stored) was taken for not chosen where the
|
||
keyboard dictates by itself. A Dictate that opens Hex there brings the `15m`
|
||
default back from that launch, so it warms as before 129.
|
||
- **Styles (`TranscriptFormatter`, run by `DictationPipeline` wherever Hex
|
||
dictates: the app, the keyboard's warm session and cold start, the Action
|
||
Button, Control Center and in-keyboard dictation):** `Polished` (full
|
||
sentences: fillers trimmed, each sentence capitalized, "I" too, and ended,
|
||
in Chinese and Japanese with "。"), `Casual` (texting: sentences may start
|
||
lowercase, though never an acronym or a word with inner capitals; "i" for
|
||
"I"; no comma after an opening "yeah" or "ok"; no final full stop on a
|
||
message of one line and up to three sentences), `Notes` (spoken `"bullet"`
|
||
/ `"new line"` markers, else several sentences become a bulleted list and
|
||
"Shopping list: eggs, milk and bread" bullets its items; no opening "so",
|
||
"yeah" or "and then", no final full stops), and `Verbatim` (as the speech
|
||
model wrote it), with cursor-aware insertion spacing and repeating
|
||
backspace. On an already punctuated sentence ("Sounds good, see you at
|
||
5.") Polished and Verbatim read the same; they differ on fillers, missing
|
||
capitals or endings, and with Clean up what I say on.
|
||
- **Lowercase (Settings › Dictation › Lowercase):** Off by default. After the
|
||
style and custom vocabulary's spellings, everything is lowercased, custom
|
||
vocabulary included, as desktop's Lowercase step (`str::to_lowercase` on the
|
||
final text) does, on every path (`DictationLowercaseSettings`, App Group key
|
||
`settings.dictationLowercase`).
|
||
- **French spacing (every style but Verbatim):** dictation in French, or translated
|
||
into it, gets the spaces French puts around punctuation, last in the pipeline
|
||
(`FrenchSpacing`): a narrow no-break space (U+202F) before `? ! ;` and a no-break
|
||
space (U+00A0) before `:` and inside `« »`; Canadian French none before `? ! ;`
|
||
(OQLF), Swiss French narrow ones inside `« »` too. Whatever was there becomes
|
||
exactly that one space, and only a mark that ends a word is spaced, so 10:30, URLs,
|
||
addresses, emoticons (":)", ";-)"), code in backticks and a mark at a line's start
|
||
stay as they are. Apple Speech writes a plain space before "?" in fr-FR and fr-CH and
|
||
none in fr-CA's `DictationTranscriber` (macOS 27.2.1, `say` voices), and every
|
||
style but Verbatim took the space out ("ça va?"). Apple's French keyboards type
|
||
no space before a mark (iOS 27.2 simulator, black box), so the full keyboard
|
||
doesn't either.
|
||
- **Format lists (Settings › Dictation › Format lists):** On by default
|
||
(`DictationListSettings`, App Group key `settings.dictationFormatLists`), in
|
||
English and Spanish, every style but Verbatim (`SpokenLists`, inside the
|
||
style after its fillers, so each style ends the lines its own way). "first …
|
||
second …" ("1st … 2nd …", "primero … segundo …", "en primer lugar …"),
|
||
"number one … number two …" (also "step", "item", "point", "número", "paso")
|
||
and "one, … two, …" become `1. …` lines; "bullet", "bullet point", "next
|
||
item", "dash" ("viñeta", "guion") said twice or more become `- …` lines; "new
|
||
line" and "new paragraph" ("nueva línea", "nuevo párrafo", "salto de línea")
|
||
break lines. Text before the list gets its own line (with a colon unless it
|
||
ends a sentence), and what follows the last item's sentence its own line.
|
||
Guards keep prose: markers count up from one, at least two non-empty items,
|
||
each marker at a clause start (the start, after a mark or line break, or
|
||
after "and"/"then"/"y"/"luego" that are), a bare number needs its comma, and
|
||
words like "the first time", "second guess", "a new line of", "bullet
|
||
points" stay as said. `TranscriptCleanupTests` holds the list cases and
|
||
62 prose sentences that must not change.
|
||
- **Spoken punctuation (Settings › Dictation › Spoken punctuation):** On by
|
||
default (`DictationPunctuationSettings`, App Group key
|
||
`settings.dictationSpokenPunctuation`), in English, Spanish, French, German,
|
||
Italian and Portuguese, every style but Verbatim (`SpokenPunctuation`, inside
|
||
the style after Format lists, so "dash … dash …" stays a list). Punctuation
|
||
said by name becomes the mark: "comma", "period"/"full stop", "question
|
||
mark", "exclamation point", "colon", "semicolon", "ellipsis", "dash",
|
||
"hyphen", "apostrophe", "hashtag", "dollar sign", "open quote … close
|
||
quote" (or "quote … end quote"/"unquote"), "open paren … close paren" and
|
||
"smiley" (":-)"), and each language's own words ("coma", "punto", "signo de
|
||
interrogación", "abrir comillas", "punto y aparte"; "virgule", "point
|
||
d'interrogation", "ouvrez les guillemets"; "Komma", "Fragezeichen",
|
||
"Anführungszeichen unten … oben", "Klammer auf … zu"; "virgola", "punto
|
||
interrogativo", "apri virgolette"; "vírgula", "ponto final", "abre aspas").
|
||
Quotes are “ ” (« » in French, „ “ in German), French spacing follows
|
||
(`FrenchSpacing`), and a word after "." "?" "!" gets a capital. Black box
|
||
(macOS 27.2.1, `say` voices): `DictationTranscriber` already writes every
|
||
mark, so its text has no names left; `SpeechTranscriber` writes "comma",
|
||
"question mark" and some French, German and Italian names, leaving the
|
||
rest as words, often between commas; Parakeet writes every name as words
|
||
("noon, comma, see you soon, period", "tonight question mark?"). A mark the
|
||
model already put beside a name is replaced, never doubled. Guards keep
|
||
prose: never after an article, possessive, preposition or a word that
|
||
makes the name a noun ("a period of time", "the comma is", "Oxford comma",
|
||
"trial period", "ein wichtiger Punkt"), before one that continues it
|
||
("question mark tattoo", "colon cancer", "punto de vista"), in a compound
|
||
("comma-separated"), or between numbers ("two comma five"); the names that
|
||
are everyday nouns ("period", "point"/"Punkt"/"punto"/"ponto", "question
|
||
mark", "dash", "smiley") also need the line's end, a mark after them or a
|
||
clause start after them; a colon needs words after it; a closing quote or
|
||
bracket closes only an open one. `TranscriptCleanupTests` holds the
|
||
models' transcripts and 141 prose sentences (84 English) that must not
|
||
change. Polished no longer ends a line that ends in "," or ";" ("Hi
|
||
Sarah," before a line break) with a period, and keeps the space before an
|
||
emoticon (":-)").
|
||
- **Spoken "@" (English, every style but Verbatim):** "at the rate" (India's
|
||
name for @) and "at sign" become "@", and an address said around it joins
|
||
up: "John at the rate Gmail dot com" becomes "john@gmail.com", "dot" turning
|
||
into "." only inside the address (`SpokenCleanupRules.spokenSymbols`, run by
|
||
`CleanupTagger.cleanIfEnabled` with Clean up what I say on or off, in the app
|
||
and in-keyboard dictation). "at the rate" needs a domain after it, so "at the
|
||
rate of 5%" and "at the rate we planned" stay; "at" alone is never touched.
|
||
Apple's `SpeechTranscriber` glues the domain to the word before ("at the
|
||
rategmail.com"), which is split again; `DictationTranscriber` already writes
|
||
"@" for "at sign" and for "john at gmail dot com". Polished leaves an address's
|
||
case alone, starts no sentence at a full stop inside a word ("gmail.com"),
|
||
and ends no lone address with a full stop.
|
||
- **Spoken counts (English, every style but Verbatim):** a count of three
|
||
numbers or more from one, in words or digits with only commas or spaces
|
||
between them, is written in digits: Apple's `SpeechTranscriber` writes
|
||
"Test one, 2, 3." where Parakeet writes "Test 1, 2, 3", and both become "Test
|
||
1, 2, 3." "123" right after "test" or "testing" and ending its clause, which
|
||
Parakeet and `DictationTranscriber` write for a quick "testing one two
|
||
three", becomes "1, 2, 3" too (`SpokenCleanupRules.spokenCounts`, run by
|
||
`CleanupTagger.cleanIfEnabled` before the tagger, with Clean up what I say on
|
||
or off). Other numbers stay ("Chapter one, two and three", "room 123", "test
|
||
123 passed"). The tagger alone took a count in words for a self-correction
|
||
("Test one two three" became "Test three."); `CleanupGuard` keeps it.
|
||
- **Numbers as digits (Settings › Dictation, on by default; English, every style but
|
||
Verbatim):** numbers said in words are written in digits, as Apple's
|
||
`SpeechTranscriber` writes them, so every engine reads the same
|
||
(`SpokenNumbers`, run by `CleanupTagger.cleanIfEnabled` after Spoken counts and before
|
||
the tagger, so the tagger never removes a number in digits outside a self-correction).
|
||
Parakeet V2 now and then writes a whole sentence in words ("The ratio is five point
|
||
five.", "november twenty second"), mostly in a real voice: 53 of 80 test sentences in
|
||
Ryan's cloned voice against 11 of 80 in macOS voices (macOS 27.2.1,
|
||
`experiments/2026-11-22-hex-numbers`). It writes decimals ("5.5", "3.1.4", "1.2
|
||
million"), percentages ("20%"), numbers from ten up and compounds ("25", "312",
|
||
"15,000"; commas from 10,000, as Apple writes "8000"), years ("1999", "2026"), money
|
||
with "dollars" ("$5", "$15.40"), two to nine before a plural noun or unit ("2 dogs", "6
|
||
foot"), ranges ending in one ("5 to 10 minutes"), times with am, pm or o'clock ("6 am",
|
||
"7:15 pm"), ordinals in dates ("November 22nd", "the 4th of July"; "May" and "March"
|
||
only with a capital), three or more spoken digits ("4729", "555-123-4567"), and minus or
|
||
negative before a number ("-5", "-$40"). Text already in digits keeps its form ("2,400",
|
||
"$4.99"). All of a number converts or none of it ("a hundred and fifty" 150, "one point
|
||
twenty one" 1.21, "one in ten people" 1 in 10, "five to ten percent" 5 to 10%, "five foot
|
||
six" 5 foot 6, "oh four seven two" 0472 mid-sentence); arithmetic stays as said ("ten
|
||
minus five is five"), as do "two to three hundred" and "one or two hundred". A number the
|
||
engine capitalised as a name stays: mid-sentence or with a capital inside, before another
|
||
capitalised word ("I love Twenty One Pilots", "Ten Thousand Villages", "Fifty Shades"), but
|
||
"Area fifty one", "Forever Twenty One" and "Twenty people came" convert. Custom vocabulary
|
||
and learned words are never converted (`protectedTerms`, matched word by word without
|
||
case): adding "Five Guys" or "Twenty One Pilots" keeps them, the fix for brands the engine
|
||
writes in lower case, which no rule can tell from a count. Not yet: "one" alone ("one of them", "no one", "one more"), times and prices
|
||
without a marker ("three thirty", "four ninety nine", "twelve fifty"), ranges and scores
|
||
without a unit ("eight to twelve", "three to one"), fractions, ordinals outside dates
|
||
("finished first", "twenty first floor", "a second"), small numbers before anything else
|
||
("at eight", "GPT five"), and any run of number words that reads two ways ("one thirty").
|
||
Other languages keep their words: each needs its own number words, compounds (German
|
||
"fünfundzwanzig", French "quatre-vingt-dix") and separators ("5,5" and "15.000" in
|
||
German, French, Spanish, Italian and Dutch). `SpokenCleanupRulesTests` holds the 80
|
||
sentences as both engines wrote them in both voices.
|
||
- **Mid-sentence dictation:** when the text before the cursor doesn't end a
|
||
sentence (`.` `!` `?` `…` or a line break), the keyboard lowercases the
|
||
dictation's first word, except "I" and its contractions, acronyms, words
|
||
with inner capitals, and words the dictation language's `UITextChecker`
|
||
dictionary doesn't know in lowercase (likely names). German keeps the case,
|
||
since its dictionary takes nouns in lowercase; so does a language with no
|
||
dictionary on the iPhone (`TranscriptFormatter.insertionText`).
|
||
- **Full keyboard Hex key (Settings › Keyboard › Full, the default):** A tap, or a
|
||
still hold of 0.35 s, starts dictation: in a warm session, else inside
|
||
the keyboard (in-keyboard dictation, iOS 26+), else by opening Hex.
|
||
In-keyboard dictation listens from touch-down, so a tap's or hold's first
|
||
words are kept; moving
|
||
more than 10 pt first opens the style menu, dropping that capture; releasing over a row picks it, and releasing elsewhere leaves
|
||
the menu open for taps (`HexKeyGesture`). VoiceOver activation dictates,
|
||
and the **Dictation styles** action opens the menu.
|
||
- **While the keyboard records: Show what I say and Type as I speak (Settings ›
|
||
Dictation, iOS 26+):** While the keyboard records by itself (in-keyboard dictation),
|
||
the capture's chunks also go to Apple's `SpeechAnalyzer` as they arrive
|
||
(`AppleLiveTranscriber`: `SpeechTranscriber` with volatile and fast results, or
|
||
`DictationTranscriber`'s progressive long dictation for its languages), only when the
|
||
language's model is already on the iPhone: it never downloads one. **Show what I say**
|
||
(on by default): the full keyboard's keys go and the words so far take their place
|
||
under the recording bar (`LiveDictationPreviewView`), the settled words in the keys'
|
||
ink and the tail the recogniser may still change muted, the newest line in view with
|
||
whole lines above it. It takes every touch where the keys were, so nothing types while
|
||
it shows; the recording bar's ✕ and ✓ work as before, and an iPhone with a Home button
|
||
keeps its globe key where it was. VoiceOver reads it as one element, never announced as
|
||
words change. Its motion is one family, a spring with no bounce by its duration
|
||
(`LiveWordsMotion`, `LiveDictationPreviewHost.Motion`, and the recording bar's morph and
|
||
fades), and Core Animation runs every frame of it, so it keeps the display's pace however
|
||
busy the main thread is with the capture, the recogniser and the Dot Scope (a display link
|
||
drawing the words got a quarter of its frames on time on an iPhone 17 Pro): the keys fade
|
||
(0.2 s) as the words fade in rising 6 pt (0.3 s); whatever changes in the words (new words,
|
||
a settled tail turning to ink, a word the recogniser changed) cross-fades in place (0.3 s,
|
||
the cubic closest to the spring) while what didn't change stays exactly as it is; a new line
|
||
moves the block up (0.3 s) and the line leaving the top fades through a 14 pt band, then out
|
||
of it once the words are at rest. With Reduce Motion, the cross-fade only (0.2 s) and lines
|
||
jump. `LiveWordsView` draws the words once per update into one layer (TextKit 1) and drops
|
||
the lines above the box, whole, at rest. When the dictation goes in, the preview's words
|
||
fade drifting 6 pt up (0.15 s) as the keys come back (0.3 s), and they stop taking touches
|
||
at once; the field gets the dictation's own text, with its style, clean-up,
|
||
punctuation, held results and Undo dictation as before, so it can differ from the
|
||
preview: with the speech model in keyboard (Parakeet, English's default) the preview's
|
||
words are Apple's and the text Parakeet's ("10" can go in as "ten"). **Type as I speak**
|
||
(off by default): the words are typed into the field as they come
|
||
(`LiveFieldTyping`), only the changed tail retyped. It keeps exactly what it typed and
|
||
edits only while the text before the cursor still ends with it; when the dictation's
|
||
text goes in, its words come out first, so the text goes in where and as it would without
|
||
them, in one step. ✕, a failure or nothing heard take them out. The keyboard hiding
|
||
mid-dictation takes them out where the app still takes edits; one gone to the background
|
||
doesn't, so they are noted (`LiveDictationSettings.leftover`) and the held text, inserted by
|
||
itself or with Paste dictation, replaces them where the field still ends with them.
|
||
Once the field changes under it (a key, a cursor move, a selection, or an app that
|
||
reports no text), it stops and leaves its words, and the text goes in at the cursor as
|
||
without it. Never in secure, password, one-time code or card number fields, over a
|
||
selection or a two-thumb Dictate's range, or with Translate what I say on. Neither runs
|
||
before iOS 26, for dictations recorded through Hex (Keep mic warm), when less than 32 MB
|
||
is left once the capture runs (`KeyboardLiveDictation`: the recogniser's share in the
|
||
keyboard with margin, and the speech model's transcription after it), on an error, or
|
||
when the recogniser isn't running 4–6 s into the audio; under 24 MB, or on a memory warning, the recogniser goes
|
||
and the dictation goes on. The dictation bar shows no preview (Type as I speak works
|
||
there, unverified on an iPhone). On an iPhone 17 Pro (iOS 27.2, keyboard limit 177 MB,
|
||
Parakeet loaded, a clip played from a Mac's speaker): the first words show about 1.2–1.3 s
|
||
after speech starts (2 s after the capture's first audio; 2.1 s and 3 s on the first
|
||
dictation after the keyboard starts), then about a second behind; over a 60 s dictation
|
||
the keyboard ended at 60.1 MB with the preview against 58.1 MB without (peak while
|
||
recording 66.5 against 60.1, at Parakeet's transcription 78.7 against 76.6), and two-sentence
|
||
dictations 0.4–0.8 MB higher. A scrolling text view holding the whole text had peaked
|
||
27 MB higher there, so the preview draws only about its box's worth. Measured on the Mac
|
||
Studio (macOS 27.2.1) with the keyboard's own code fed a clip in the page's
|
||
4,096-sample chunks in real time: first words 0.5–1.0 s after the first word ends, then
|
||
0.6–0.8 s behind (90% within 1.2 s), in bursts about once a second; 2% of words shown
|
||
were rewritten (7 of 336, "730." to "7.30."); words settle only at a pause. Without fast
|
||
results words came 2.7–3.5 s late, and `DictationTranscriber` rewrote 15% of them. The simulator has no speech models: in
|
||
simulator builds only, `LiveTranscriptReplay` plays a recorded run (`live-replay.json` in
|
||
the App Group, or the keyboard's own Library for an unsigned build) from the first audio.
|
||
`ParakeetLiveTranscriber` (the audio so far again about once a second, a 20 s window) is
|
||
written for when the speech model's own words should show, and not used yet.
|
||
- **Auto-start:** Settings › Keyboard › Auto-start (`FullKeyboardSettings.dictationBarAutoStart`)
|
||
lets the dictation bar start dictating by itself as it shows, as a tap on **Dictate**
|
||
would, 0.8 s after it appears (after the first frame and the launch's optional work),
|
||
only while idle, outside number pads and the emoji panel, and only where it dictates
|
||
without leaving the field (a warm session or in-keyboard dictation, never by opening
|
||
Hex). **Never**, **On switch** (the default) or **Always**. iOS doesn't tell a
|
||
keyboard whether you switched to it or a field opened with it, so On switch guesses
|
||
(`KeyboardSwitchGuess`): the keyboard comes back in the field it last hid in
|
||
(`documentIdentifier`), and the field's app didn't go to the background
|
||
(`NSExtensionHostDidEnterBackground`) from 2 s before that hide on. Switching to
|
||
another keyboard and back in one field starts; a field that opens with Hex already
|
||
chosen doesn't. Known gap: the first switch to Hex in a field another keyboard opened
|
||
is another document, so it doesn't start. Unproven on a device: whether
|
||
`documentIdentifier` stays the same across a keyboard switch and changes when a field
|
||
is focused again, which the guess relies on.
|
||
- **Push to talk:** Holding still for 0.35 s on the full keyboard's Hex key
|
||
or the dictation bar's **Dictate** starts dictation at once with a light
|
||
haptic (Settings › Keyboard › Haptics), and lifting the finger finishes it as
|
||
**Done** does; a quick tap still starts it and **Done** finishes it. On
|
||
the Hex key, sliding more than 10 pt before the hold fires opens the menu
|
||
instead, and once dictation has started, moving the finger changes
|
||
nothing. Push to talk needs dictation in the keyboard (a warm session or
|
||
in-keyboard dictation); a hold that opens Hex is only a tap. The full
|
||
keyboard's bar takes the same hold (0.4 s, Apple's long press) anywhere no
|
||
word is: an empty slot, beside the Hex key, or a bar with nothing in it,
|
||
through the Hex key's own path (`HexKeyGesture`, `hexKeyHeld`), but not
|
||
while the bar shows the Full Access hint or a notice. A tap there does nothing,
|
||
a slide cancels it, and a word's slot keeps its tap and its hold (Add to
|
||
vocabulary). The bar's Paste is plain text on the words' line, and Paste and
|
||
Next keyboard centre on the keyboard's middle, not the strip's. Paste shows
|
||
only while the field is empty, as Apple's keyboard offers a copy, and Code
|
||
from Messages only in an empty one-time-code field with a number pad or with
|
||
autocorrection set either way (as WebKit sets it on Safari's
|
||
`autocomplete="one-time-code"`): a chat box that marks itself one-time-code
|
||
keeps its suggestions. On a number pad, or with only Apple's keyboards
|
||
beside Hex, it switches to the next keyboard; elsewhere (Emoji or another
|
||
third-party keyboard on, where that switch lands on them about half the
|
||
time) a tap opens iOS's keyboard list, as the globe's hold does, to pick
|
||
Apple's. It needs no Full Access, which only Paste does. When iOS prompts
|
||
Allow Paste (read ≥ 0.3 s) or refuses a paste read, the strip shows a
|
||
tappable two-line notice (`Paste without asking` / `Paste from Other Apps ›
|
||
Allow` with a **Settings** chip that routes through
|
||
`hex-dictation://keyboard/paste-settings` straight into `Settings › Apps ›
|
||
Hex`), and Hex records `keyboard.clipboard.allowPastePrompted` in the App
|
||
Group so Home, Settings › Keyboard, and Setup surface the same one-tap Open
|
||
Settings banner once iOS has created that row. A later fast cross-app paste
|
||
(ignoring Hex's own copies via `markOwnedCopy`) or tapping dismiss clears
|
||
it.
|
||
- **Apple Intelligence cleanup (Settings › Experiments › Clean up with
|
||
Apple Intelligence, iOS 26+):** Off by default. For every style but `Verbatim`,
|
||
one on-device FoundationModels pass before formatting fixes clearly
|
||
misheard words and spoken self-corrections, within a 2 s budget. A
|
||
timeout, error, empty or implausible output keeps the raw transcript, and
|
||
the toggle is disabled with a reason when Apple Intelligence is unusable.
|
||
- **Multilingual model (Settings › Experiments › Multilingual model (Parakeet), iOS
|
||
18+):** Off by default, and disabled on iPhones under 6 GB
|
||
(`MultilingualParakeetSettings`, in the App Group). While on, Hex dictates
|
||
Parakeet Ultra's 25 languages (Moondream's post-training of NVIDIA Parakeet TDT
|
||
0.6B v3) with it, which hears them mixed within a sentence (Dutch with English
|
||
words, a TestFlight request); any other dictation language keeps the engine Hex
|
||
picks without it. The language Ultra takes is the one set, else the one Automatic
|
||
follows on the keyboard, else the iPhone's first. The experiment never takes
|
||
dictation away: until Ultra is ready, Hex and the keyboard dictate as they would
|
||
with it off, and Hex switches to Ultra once it has loaded (a dictation in flight
|
||
isn't interrupted). Turning it on downloads
|
||
FluidInference's Core ML build (632 MB, pinned to revision `95eaa59`, each file
|
||
checked against its SHA-256 and resumed from its `.partial`) into Application
|
||
Support, out of backups, beside that engine (`ParakeetUltraDownload`): home and the
|
||
switch say "Downloading the multilingual model: 123.4 MB of 632.2 MB" with a bar. A
|
||
dropped connection, or bytes that stop for 20 s, tries again by itself after 2, 4, 8,
|
||
16, then 30 s, starting over whenever it got further; in the background it waits for
|
||
Hex to come back to the front, which picks it up at once (and starts a fresh
|
||
connection after 25 s away). After six tries in a row that got nothing further, or a
|
||
refused request or a full iPhone, it says what happened in plain words with Try again
|
||
and Turn off. Once downloaded, Ultra's first load (Core ML's compile, a minute or
|
||
more) runs beside the usual engine too, except where Parakeet V2 dictates and leaves
|
||
no room for it (English before iOS 26), where Ultra takes V2's place for that load;
|
||
later launches load it directly when Core ML has compiled it for this iOS. Turning it off stops the download, then deletes it;
|
||
a launch deletes a leftover only when the switch reads off as stored, never when the
|
||
App Group can't be read (a launch before the first unlock). FluidAudio 0.15.5 has no
|
||
`.ultra`, so `ParakeetUltraModel` loads the four models itself and hands
|
||
FluidAudio v3 `AsrModels` (its own loader would replace a folder it can't load
|
||
with v3's download). It is chosen only with 1.5 GB left to load it
|
||
(`os_proc_available_memory`), and never again after a load that ended Hex
|
||
(`ParakeetLoadGuard`, its own keys) until Try again beside the status, or the switch
|
||
turned off and on. Unless Experiments › Speech model in keyboard has it loaded in the
|
||
keyboard (below), the keyboard doesn't use it, so once Hex has it ready
|
||
(`MultilingualParakeetSettings.isModelReady`, set after a load, cleared while it's
|
||
missing, set aside or off) and it takes the dictation language, Dictate and the Hex
|
||
key open Hex, as before iOS 26 (a warm session still dictates without leaving the
|
||
app); until then the keyboard dictates itself. Measured on a Mac (`~/dev/experiments/2026-11-21-hex-multilingual-parakeet`):
|
||
mixed Dutch-English test sentences 7.5% WER against Apple Speech nl-NL's 26.9%,
|
||
pure Dutch 0.9% (0.4%), pure English 0% (0%). Load memory, compile time and speed
|
||
on an iPhone are unmeasured.
|
||
- **Speech model in keyboard (Settings › Experiments › Speech model in keyboard, iOS
|
||
26+):** On unless the person turns it off, and disabled (shown off) on iPhones under
|
||
6 GB (`KeyboardParakeetSettings`, in the App Group; a value stored by hand wins, so a
|
||
switch turned off stays off). Off, nothing of it loads and the keyboard dictates exactly
|
||
as before. On, Hex downloads Parakeet V2 (464 MB, about 700 MB with the keyboard's
|
||
compiled copy) into the App Group (`KeyboardParakeetDownload`; turning it off stops and
|
||
deletes it unless Hex itself dictates with V2). By itself it downloads only where the
|
||
keyboard would use it (`KeyboardParakeetDownload.hold`): English dictation without
|
||
Two-language dictation or the Multilingual model, the keyboard's own memory limit
|
||
measured on this iPhone (the keyboard writes footprint plus `os_proc_available_memory`
|
||
to the App Group at each launch and tells Hex with a Darwin notification) and at least
|
||
`KeyboardParakeetBudget.requiredLimit`, a connection that isn't expensive or constrained
|
||
(`NWPathMonitor`: not cellular, a personal hotspot or Low Data Mode), and 2 GB free.
|
||
It checks at launch, whenever Hex comes to the front, when such a connection returns and
|
||
when the keyboard reports its limit. Turned on by hand (a stored true, including from
|
||
the builds where it was off by default) it downloads at once on any connection. Until
|
||
then the row says what it waits for: "Downloads after you first use the Hex keyboard",
|
||
"Downloads on Wi-Fi", "Downloads once the iPhone has 2 GB free", "Only for English
|
||
dictation". The keyboard's own dictation transcribes with Parakeet in
|
||
the keyboard's process (`HEXKeyboard/KeyboardParakeet.swift`), with the app closed:
|
||
V2 for English, and Parakeet Ultra for a language Experiments › Multilingual model
|
||
dictates once its files are there, in which case Dictate and the Hex key stop opening
|
||
Hex for it. Parakeet's files moved from the app's Application Support into the App
|
||
Group (`SpeechModelStore`), renamed in place at launch (`moveLegacyDownloads`) rather
|
||
than downloaded again; each folder is marked complete (`.hex-complete`) once every file
|
||
is there, and the keyboard loads only a marked one. The keyboard loads it after its
|
||
first frame and launch work, off the main thread (the preprocessor on the CPU, the rest
|
||
on the CPU and Neural Engine, one chunk at a time, never the GPU, which iOS gives a
|
||
keyboard only in the background role), and keeps it while its process lives, until a
|
||
memory warning. Its first load compiles it for the Neural Engine in the keyboard's own
|
||
Caches (17 s for V2, 12 s for Ultra on an iPhone 17 Pro); a dictation never waits for
|
||
that, and uses Apple's speech recognition exactly as with the experiment off whenever
|
||
the model isn't loaded, isn't on the Neural Engine (the keyboard's Neural Engine memory,
|
||
else Core ML's compute plan before iOS 27), another transcription runs, the keyboard's
|
||
memory limit (footprint plus `os_proc_available_memory`) is under 131 MB (V2 and Ultra
|
||
alike: with the model loaded the keyboard records at up to 91 MB, and Ultra's
|
||
transcription rose 9 MB in Hex's keyboard, V2's 16 MB), too little is
|
||
left, the recording is longer than 90 s or there isn't room for it (FluidAudio holds the
|
||
whole recording in memory, about 0.42 MB a second measured, counted at 0.5 MB on top of
|
||
the 24 MB transcription headroom; Apple's speech recognition streams the file, so a long
|
||
dictation goes there and arrives a little later), it hears nothing, or it fails
|
||
(`KeyboardParakeetBudget` has the measured numbers). Core ML keys what it compiled to the files themselves: a model downloaded
|
||
again, or moved, compiles again, so the log calls a load "cold" by its time (5 s or
|
||
more), and moving a legacy folder forgets the app's compile record for it
|
||
(`ParakeetCompileRecord.forget`), so V2 says it's optimizing for this iPhone and Ultra
|
||
gets ready beside the engine that dictates. The Experiments row says what it's doing: downloading,
|
||
downloaded, getting ready in the keyboard, ready, or that the keyboard can't use it now
|
||
(`KeyboardParakeetSettings.keyboardState`, written by the keyboard). Each dictation logs
|
||
its engine, the keyboard's limit, footprint before and at peak, load and transcription
|
||
time to `keyboard-capture-probe.log`. Measured in Hex's keyboard on an iPhone 17 Pro
|
||
(iOS 27.2, keyboard limit 177 MB) with the app force-quit: V2's first load 15.4 s, the
|
||
keyboard recording at 56 MB and peaking at 72 MB while V2 transcribed 9.4 s of audio in
|
||
0.28 s (Apple's speech recognition, same clip: 0.34 s), with 468 MB on the Neural Engine
|
||
outside the footprint. Twenty dictations in one keyboard process stayed on V2 with a
|
||
peak of 96 MB, ten hide-and-show cycles and a five-minute hide kept the process and the
|
||
model, a new process reloaded it in 0.3 s, and Ultra peaked at 66 MB (607 MB on the
|
||
Neural Engine). A probe keyboard (`~/dev/experiments/2026-11-22-hex-keyboard-asr/native`)
|
||
put 60 s of audio at 0.25–0.5 s; on the CPU, four chunks at once passed the limit
|
||
there, so the keyboard never runs that. Linking FluidAudio into the keyboard grows its binary from
|
||
5.9 to 12.6 MB and, with the experiment off, its footprint at launch by about 0.2 MB
|
||
(writable data dyld fixes up), with no change to its first frame.
|
||
- **Clean up what I say (Settings › Dictation › Clean up what I say):** Off by
|
||
default. For every style but `Verbatim`, Hex's dictation cleanup tagger
|
||
labels each spoken word KEEP or DELETE on the iPhone and drops ums,
|
||
repeats, and the words a spoken correction replaces ("3pm, actually 4pm"
|
||
becomes "4pm"); a word goes only above P(DELETE) 0.98.
|
||
`CleanupReconstruction`, a port of the tagger's `reconstruct.js`, then
|
||
repairs the commas and capitals next to a deletion. It never adds,
|
||
reorders, or changes words. The tagger only knows English (run on
|
||
Spanish it dropped negations: "Te digo que no puedo ir" became "Te
|
||
puedo ir"), so Spanish, German and Norwegian get word rules instead
|
||
(`SpokenCleanupRules`): the language's hesitation sounds go (the
|
||
styles' list below), a filler between commas goes ("este", "o sea"),
|
||
and a correction marker set off by commas ("digo", "perdón", "no",
|
||
"mejor dicho"; "nein", "ich meine"; "nei", "nei vent", "jeg mener")
|
||
drops the words it replaces, counted back to the nearest earlier word
|
||
that matches the repair's first word or is of its kind (a number,
|
||
weekday, month, or today and tomorrow): "a las 3, digo, a las 4" becomes
|
||
"a las 4". With no such word the sentence stays as said, and other
|
||
languages stay as heard. It runs after transcription and learned fixes
|
||
and custom vocabulary, before Apple Intelligence
|
||
cleanup, translation, and formatting, in in-keyboard dictation and in the
|
||
app. The tagger is a BERT L-4/H-512 token classifier fine-tuned from
|
||
Google's Apache-2.0 `bert_uncased_L-4_H-512_A-8` on generated examples,
|
||
shipped as an int8 Core ML model (28.5 MB) with its WordPiece vocabulary
|
||
once, in `HEXKeyboard.appex` (`HEXKeyboard/CleanupTagger/`). The app loads
|
||
it from there. `CleanupTokenizer` matches Hugging Face's `BertTokenizer`,
|
||
including its Unicode 8.0 character classes. The model loads on the Neural
|
||
Engine or CPU when a dictation starts recording with the experiment on,
|
||
stays loaded, and is freed on a memory warning. Cleanup gets 150 ms; a
|
||
timeout, an error, or a result with no words left keeps the transcript as
|
||
heard.
|
||
- **Custom vocabulary (Settings › Dictation › Custom vocabulary):** Off by default.
|
||
A local list of up to 100 names and phrases, applied to every engine, in the
|
||
app and in-keyboard dictation. The full keyboard uses the same list whether or not the
|
||
switch is on (Settings › Keyboard › Vocabulary edits it too; TestFlight build 56:
|
||
"custom dictionary for uncommon programmer slang"): its words, and each word of a
|
||
phrase, are never autocorrected away, a different case is put right on space ("github"
|
||
→ "GitHub"), the strip completes them before the dictionary's words ("kub" →
|
||
"kubectl") with no spelling guesses for them, Swipe typing reads them as your own words,
|
||
and a bilingual partner shares them. Holding the word you typed in the strip's left slot
|
||
offers **Add to vocabulary** there (`FullKeyboardView+Vocabulary`; a capital only a
|
||
sentence's start gave the word is left out); without Full Access
|
||
the keyboard can't write the App Group, so the word is kept for that session only.
|
||
Before this the keys knew only iOS's text replacements and Contacts names
|
||
(`requestSupplementaryLexicon`), never Hex's list. After
|
||
transcription, matching words are rewritten to the listed spelling across
|
||
case, spaces, tabs, a single hyphen, and possessive apostrophes, before
|
||
cleanup and again after formatting, so a style cannot undo a spelling. In
|
||
English, a name heard with another spelling then takes the listed one
|
||
(`SoundAlikeSpelling`; every engine writes a name's commonest spelling, so
|
||
TestFlight build 130's "Stacey" kept coming back "Stacy"): a word the engine
|
||
capitalized that reads the same as a listed term of three or more letters
|
||
starting with a capital, once interchangeable spellings are evened out (ph
|
||
and f; a hard c, k and ck; chr and kr; a soft c and s; ae and e; a leading
|
||
ai and a; a final y, ey, ie, ee and i; any other y and i; a silent h;
|
||
doubled letters), takes the term ("Stacy" → "Stacey", "John" → "Jon",
|
||
"Catherine's" → "Katherine's", "Amy" → "Aimee"). Inside a sentence, and not
|
||
right after another capitalized word, it may instead sound the same by a
|
||
Metaphone-style key (its consonants with voicing kept, first vowel sound,
|
||
vowel ending and vowel groups) and be at most two edits away ("Jeff" →
|
||
"Geoff", "Versal" → "Vercel"). Lowercase words, words in capitals, listed
|
||
words and two equally close terms stay as heard. The
|
||
listed spelling is kept exactly, even at a sentence start (an all-lowercase
|
||
term stays lowercase), and a term that spells its own boundaries ("iPhone 7",
|
||
"X-ray") matches those boundaries exactly. A word that carries its own
|
||
apostrophe only matches a term that has one, several transcript words are
|
||
joined only when the term reads as a compound, and compound terms like
|
||
"WeChat" can also match the plain phrase ("we chat"). On iOS 26 Apple Speech
|
||
the list is also sent as a best-effort recognition hint, in the app and in
|
||
in-keyboard dictation (through build 130 the keyboard sent only Learn how I
|
||
type's words: #2103 added the list's rewrite there but not its hint);
|
||
Parakeet's batch path gets no hint.
|
||
- **Learn how I type (Settings › Privacy › Learn how I type):** On by default (an
|
||
unset switch reads on; turning it off is saved and stays off), and
|
||
shown whether or not the full keyboard is on. It learns only from typing and
|
||
dictating with Hex, on this iPhone, never from what you write elsewhere. A
|
||
private journal in the App Group (`typing-journal.json`) that never stores
|
||
what you write: per-key tap offsets, and word-level pairs with counts and at
|
||
most one word of context on each side. Only plain text fields that allow
|
||
autocorrect count (`TypingJournal.allows`). Nothing leaves the iPhone.
|
||
- **Typing:** words you erase and retype, autocorrect fixes you undo from the
|
||
strip, suggestions you pick, and autocorrections you turn down by
|
||
tapping what you typed. A typo you fixed the same way twice (never a
|
||
completion, never fixed two ways) becomes a fix on space, only a suggestion
|
||
when what you typed is a real word; an autocorrection you turned down twice
|
||
never applies again (`TypingLearning`, read by `AutocorrectEngine`).
|
||
- **Dictation:** from the full keyboard and the dictation bar alike, the
|
||
keyboard remembers each dictation by the text around it
|
||
(`DictationEditTracker`), finds it again after edits through Hex's keys and
|
||
on `textDidChange`, and diffs it word by word against what Hex inserted and
|
||
the raw transcript the app or in-keyboard dictation sends with it
|
||
(`DictationEditInference.swift`). It settles after 60 s, when the keyboard
|
||
goes away or the field changes, on a Send return key, or when the host
|
||
clears the field (a send). Each dictation gets an outcome (kept, edited,
|
||
rewritten when more than half changed, undone, deleted, or redictated when
|
||
deleted and dictated again within 20 s) and its pairs: substitutions,
|
||
insertions, deletions, words cleanup dropped that you added back, and
|
||
fillers Hex left that you deleted, each as heard, inserted, and final words.
|
||
New writing before or after a dictation, changes that aren't plausibly the
|
||
same words, edges the host's cut-short context can't show, and translated
|
||
dictations teach nothing.
|
||
- **Applied:** a substitution seen twice and never contradicted (you kept the
|
||
heard words in a dictation you corrected, changed the fix back, or fixed
|
||
the same words another way) is applied right after transcription, in
|
||
in-keyboard dictation and in the app before custom vocabulary and cleanup,
|
||
whole words only (never inside a longer, hyphenated, dotted, or slashed
|
||
word) and following the original's case unless the fix has capitals. Its
|
||
meant words join the Apple Speech hints after custom vocabulary's.
|
||
Typing insights lists these fixes with a switch each and the share of
|
||
dictations kept and edited; Delete what Hex learned erases everything,
|
||
including what a running keyboard still holds.
|
||
- **Per language (Settings › Experiments › Learn each language separately):**
|
||
Off by default (`experiments.learnPerLanguage`, App Group). Every correction and
|
||
dictation fix is tagged with the language it was typed or dictated in either way,
|
||
so turning it on needs nothing learned again; a fix made in two languages is
|
||
tagged with neither and counts for every one. On, the keys learn a language's typos
|
||
and rejections from that language's corrections and the untagged ones, and a
|
||
dictation applies only the fixes made in its language and the untagged ones (a fix
|
||
learned in en-US applies in en-GB). Learned words are one list per keyboard
|
||
language (`keyboard.learnedWords.<code>`), started from the shared
|
||
`keyboard.learnedWords` list the first time that language is used; bilingual
|
||
autocorrect's partner keeps its own list and its own fixes. Where fingers land and
|
||
Adaptive touch stay shared. Delete what Hex learned erases the shared list and every
|
||
language's. Two limits: a dictation fix you kept as heard in one language still
|
||
counts against that fix in every language, and retyped words and picked suggestions
|
||
are tagged with the keyboard's main language, never the bilingual partner's.
|
||
Off, everything is the one shared list and learning it was before.
|
||
- **Auto-capitalization (Settings › Keyboard › Auto-capitalization):** On by default,
|
||
as Apple's (`FullKeyboardSettings.autoCapitalization`, App Group key
|
||
`settings.keyboardAutoCapitalization`). Off, shift never turns itself on, whatever the
|
||
field's `autocapitalizationType` asks (a field's or sentence's start, after "." "!" "?"
|
||
or the double-space period, which still types ". "), and autocorrect and swipe typing add no
|
||
capital you didn't type: "i" stays "i", "im" becomes "i'm", "mary" stays "mary" and
|
||
"mondya" becomes "monday", as Apple's keyboard does with Auto-Capitalization off (seen on
|
||
the iOS 27.2 simulator). Words you typed with a capital, and your own spellings (text
|
||
replacements, Contacts names, words you kept, fixes Hex learned, your vocabulary), keep
|
||
theirs; shift and caps lock work as before. The strip offers no capital you didn't type
|
||
either (TestFlight build 56: "even with auto-capitalization off, many suggested words are
|
||
capitalized"): a spelling guess or completion iOS knows capitalised ("Mandy", "Monday")
|
||
and the contraction "I'd" for "id" come lowercase there; sentence and clause starters
|
||
keep "I" and "I'm" (#2135), and next words and swipe readings already had no capital.
|
||
Dictation has its own switch (Settings › Dictation › Lowercase).
|
||
- **Spatial autocorrect (Settings › Keyboard › Spatial autocorrect):**
|
||
On by default, shown under Autocorrect while it's on (it needs the full keyboard
|
||
with Autocorrect on). Stored as `experiments.spatialAutocorrect` from when it was an
|
||
experiment off by default: unset or true is on, false off, so testers who turned it
|
||
off keep it off. A noisy-channel decoder
|
||
(`TypingDecoder`) replaces `UITextChecker`'s one-edit guesses on space and in the
|
||
strip for English, Spanish and French (France): each tap is scored against every letter with a 2D
|
||
Gaussian around the key (shifted by Learn how I type's tap offsets when it has 20
|
||
or more taps on a letter), words are found with a beam search over a memory-mapped
|
||
word list and over your own words (Contacts names and other `UILexicon` terms,
|
||
one-word text replacements, words you kept, fixes Hex learned), a word-pair language
|
||
model weighs them by the word before, a missed space is tried by splitting the taps,
|
||
accented letters read from their base key (é from e), and what you typed is replaced
|
||
only when the best reading beats it by a margin (more when what you typed is a word,
|
||
including a name, day or place typed in lowercase that the dictionary knows only
|
||
capitalised, which is capitalised as the dictionary path does: "mary" -> "Mary").
|
||
A run that is no word and no dictionary's at all is also read as a word plus one slip
|
||
in the next ("specialchsracter" -> "special character") or as three to six words typed
|
||
together ("seeyoutomorrow" -> "see you tomorrow"), when the words without their missed
|
||
spaces are no less likely than the run; those can replace a run the whole-word margin
|
||
would keep. Deutsch keeps the ordinary readings: its lowercase model cannot give a
|
||
split's nouns their capitals.
|
||
Shortcuts, learned fixes, contractions, and words you kept still come first. Other
|
||
layouts (Norsk, Deutsch, Italiano, Français (Canada), Français (Suisse), Português, Polski, Română, Русский, Українська, Ελληνικά) keep the dictionary
|
||
engine in their own language, and Srpski (no iOS dictionary) none, never the English
|
||
model, except as bilingual autocorrect's second language (below). The
|
||
models (`HEXKeyboard/TypingModel/en.hextm` and `es.hextm`, about 3.6 MB each) are
|
||
built from Google Books Ngram v3 by `scripts/build-typing-model.py` (see
|
||
Acknowledgements below); English's contractions count as often as the words they stand
|
||
for ("do not"), and it adds Hex's own chat and tech words (`scripts/en-extra-words.tsv`),
|
||
byte-identical to Android's `en.hextm`; `de.hextm` is the German model Android builds and ships
|
||
(`android/benchmark/typing/build-layout-model.py`), the same v3 format and, on iOS,
|
||
off until the experiment below is turned on. `fr.hextm` is Android's French model, built
|
||
the same way from Google Books v3's French shards (corpus spellings, with the books'
|
||
elided forms such as "c'est") and byte-identical to
|
||
`android/app/src/main/assets/typing/fr.hextm` (its `SHA256SUMS` hash). On the held-out
|
||
French replay (2,000 Tatoeba sentences typed on AZERTY, scored as
|
||
`android/benchmark/typing/quality-gate.py` does) it makes 76.4% fewer word errors than
|
||
the raw taps (35.3% with `UITextChecker` alone), changes 0.10% of correctly typed words
|
||
(0.56%) and restores 81.1% of the accents left out where the plain spelling is in neither
|
||
the model nor `fr.valid` (78.7%), short of the gate's 85%. Canada's and Switzerland's French don't read it yet (`fr-CA`, `fr-CH`).
|
||
Beside a model, a language with its own word list (`HEXKeyboard/TypingModel/<code>.valid`,
|
||
Android's `android/benchmark/typing/build-valid-words.py` output: today French's) takes what
|
||
is a word from the model and that list alone, not from `UITextChecker` (`HexWordList`,
|
||
memory-mapped and searched in place; `SystemTextChecker.ownWords`), and a missing system
|
||
dictionary no longer turns its model off (`AutocorrectEngine.hasWords`). `UITextChecker`
|
||
still gives guesses and completions while it is in that language. Beside the list, the
|
||
language's `<code>.capitals` (Android's `build-capitals.py` output, read in place the same way)
|
||
says which words are names: one the books write with a capital at least 95% of the time is no
|
||
word typed all lowercase, so it takes its capital as `UITextChecker` gives one ("paris" ->
|
||
"Paris", "jean" -> "Jean"; "marie", 94%, stays), and a listed name outside the model is a word.
|
||
German's `de.capitals` is read in place through the same reader (`GermanCapitals`). With `fr.valid` French
|
||
restores 86.1% of the accents the gate counts (81.1% with `UITextChecker`). English, Spanish
|
||
and German keep `UITextChecker`: their own lists moved a gate number the wrong way.
|
||
Polish reads `pl.hextm` (150,000 words, 184,327 pairs, 3.60 MB) and `pl.valid` (525,084 words,
|
||
8.29 MB), both built by `scripts/build-typing-model.py pl` from NKJP's n-grams, ParlaMint and
|
||
PoliMorf (Settings › Acknowledgements). Its letters are held on their base keys, so the model
|
||
puts them back ("zolw" -> "żółw", "wiem ze" -> "wiem że"). iOS's own Polish dictionary is no
|
||
use without Apple's word data (it takes any letters for a word), and the list replaces it. On
|
||
the held-out Polish replay (`scripts/typing-gate.py run … pl`): 85.98% fewer word errors than
|
||
the raw taps, 0.22% of correctly typed words changed, 95.46% of the marks
|
||
restored where the plain spelling is no word, 5/5 critical cases. `pl.capitals`
|
||
(`android/benchmark/typing/build-capitals.py pl`, case counted in ParlaMint) gives names
|
||
their capital: of the names typed all lowercase, 10.65% come out capitalised (0% with the
|
||
phone's dictionary on the simulator, which has no Polish word data, and 0% without the list).
|
||
With English as the second language, Polish makes 85.0% fewer errors (86.4% alone: English
|
||
takes "cos", "troche", "beda" as its words) and English is unchanged.
|
||
- **German spatial autocorrect (Settings › Experiments › German spatial autocorrect):**
|
||
Off by default (`experiments.germanSpatialAutocorrect`, App Group), and only while
|
||
Spatial autocorrect is on too. With both on, the Deutsch layout reads taps with
|
||
`HEXKeyboard/TypingModel/de.hextm`, byte-identical to
|
||
`android/app/src/main/assets/typing/de.hextm` and the hash its `SHA256SUMS` lists: the
|
||
word list feeds the decoder, the letter odds and the strip, and a bilingual partner may
|
||
read German taps too. German's model is lowercase throughout, so a noun takes its capital
|
||
from the list beside it, whether the decoder corrected it or it was typed right ("hsus"
|
||
reads "haus" and comes out "Haus"; "das haus" becomes "das Haus"): 95% and up anywhere,
|
||
and 50-94% after an article or determiner, while Auto-capitalization is on. A 50-94%
|
||
word with no determiner before it, and any word the list lacks, keeps the dictionary's
|
||
rule: a verb stays lowercase ("zu leben", "ich frage", "wollt ihr essen") and a word the
|
||
dictionary knows only capitalised still takes its capital ("heute abend" becomes "heute
|
||
Abend"). The builder caps at 94, so a capital arrives only after a determiner: a word that
|
||
is also a common first-person or imperative verb form ("frage", "reise", "drucke") or the
|
||
stem people type alone ("kauf", "schlaf", "sitz": "ich kauf noch brot", "schlaf gut"); the
|
||
separable prefixes "teil" and "preis", the adjective "klasse", and the verb forms the
|
||
counts miss ("filme", "rolle", "presse", "pumpe", "campe", "pflanze", "beichte", "fische",
|
||
"taufe"); and every two-letter "word", a chat abbreviation ("gl", "rn") rather than a
|
||
noun. The determiner list includes "in" ("in Ruhe", "in Frage") but not "ohne" ("ohne
|
||
essen" is a verb). The list is
|
||
`HEXKeyboard/TypingModel/de.capitals`, built by `scripts/build-german-capitals.py`
|
||
from the same Google Books shards: for each model word it compares the 1980–2019 counts
|
||
of the all-lowercase and first-letter-capitalised spellings and lists
|
||
`word<TAB>percent` for the words at least 50% capitalised and seen at least 40 times,
|
||
leaving out pronouns and possessives capitalised for other reasons (`FUNCTION_WORDS`:
|
||
"sie", "ihr", "mein", and the rest). Rebuild it from `ios/` with
|
||
`python3 scripts/build-german-capitals.py HEXKeyboard/TypingModel/de.hextm HEXKeyboard/TypingModel/de.capitals`;
|
||
it streams the eight shards through curl and gzip four at a time, caches each shard's
|
||
counts under `scripts/.build-german-capitals`, keeps their merged per-token counts there
|
||
so a rule change renders without streaming again, and applies the caps after counting.
|
||
Off, Deutsch keeps the dictionary engine exactly as before: no decoder, no word list for
|
||
the letter odds or model completions, and no model for a bilingual partner. Swipe typing
|
||
stays English and Spanish only.
|
||
- **Adaptive touch (Settings › Experiments › Adaptive touch):** Off by default
|
||
(`experiments.adaptiveTouch`). The keyboard learns where you land on each physical
|
||
letter key, on this device, and Spatial autocorrect's decoder and the strip read your
|
||
taps with those personal centres (`AdaptiveTouch.swift`). Keycaps, hit rectangles,
|
||
the next-letter claims and Swipe typing are unchanged, and it never makes an automatic
|
||
correction the keyboard wouldn't make with it off: space puts in a personal reading
|
||
only when the centres read without it also clear it by the margin
|
||
(`AutocorrectEngine.decoderBaselineKeys`, `TypingDecoder.replaces`), else their own
|
||
decision, and joining two words at the space bar's edge is theirs alone to decide;
|
||
what only the personal centres read better is offered in the strip. While
|
||
on, Learn how I type's per-letter offsets stop moving the decoder's centres (they stay
|
||
that baseline), and its word-level learning carries on.
|
||
- **What teaches it (`AdaptiveTouchLabels.swift`):** only taps with evidence of the
|
||
letter meant, and only once the next word shows the word stood: a word left as typed
|
||
(letters the key under the finger typed, not ones the next-letter odds claimed), a
|
||
picked suggestion of the same length (at most two letters changed), a word kept over
|
||
an autocorrection by reverting it from the strip or tapping it, and a letter erased and retyped as a
|
||
neighbour (the erased tap is labelled with the letter that replaced it). Never an
|
||
autocorrection left in place (or a word before it revised, or two joined), a swipe, a
|
||
slide, a strip touch, an accent pick, or a password, code, email, URL, secure or
|
||
no-autocorrect field. Taps and words wait in
|
||
memory for at most two words and 30 s, and are dropped when the cursor moves, the
|
||
host changes the text, the field or the keyboard's size changes.
|
||
- **What it keeps:** sums per physical key in four rotating bins of 200 labelled taps,
|
||
for each context (letters, posture, orientation, split, Hex key side, width) and one
|
||
of two grip profiles: offsets in keycaps (so a new key plane of the same size keeps
|
||
them), shrunk toward the row and the keyboard, at most 0.3 keycap and 0.05 per
|
||
update, fixed spread. No letters, words, times or sequences: 23,588 bytes, in a file
|
||
in the keyboard's own container with Full Access (protected, excluded from backup,
|
||
never in the App Group or its exports, written in batches off the main thread), in
|
||
memory only without it. Settings shows that it resets without Full Access, Learning (n of 400 taps) or
|
||
Active from counts the keyboard reports; Reset touch learning, and Delete what Hex
|
||
learned, bump a reset generation that drops it from memory and disk (the file is
|
||
the keyboard's, so it goes the next time the keyboard runs) and that a write
|
||
queued before can't undo.
|
||
- **When the decoder uses it:** a profile is used only after 400 labelled taps over
|
||
three sessions, while it fits your recent taps better than the centres read without
|
||
it (log-likelihood per tap, each axis, over 32 labels). Once it fits worse (a new
|
||
grip, another hand), the keyboard goes back to them at once and labels teach the
|
||
second profile; a grip that comes back gets its profile back. Before the first 400
|
||
taps nothing is in use, and a new grip just mixes into what is learned.
|
||
- **Offensive words (Settings › Keyboard › Suggest swear words):** like Apple's
|
||
keyboard, the strip and autocorrect never offer slurs, and offer swear words only
|
||
with Suggest swear words on (off by default; App Store rating 4+). The list
|
||
(`HEXShared/FullKeyboard/never-suggest.tsv`: layout, `slur` or `swear`, spelling or
|
||
two-word expression; `*` lines apply to every layout) is LDNOOBW as curated for Hex
|
||
for Android, entry for entry the same as `android/app/src/main/assets/typing/never-suggest.tsv`
|
||
(see Acknowledgements below); change both together. `OffensiveWords` is checked when a
|
||
suggestion is looked up: `UITextChecker`'s guesses and completions, learned fixes, and
|
||
Spatial autocorrect's readings (`TypingDecoder.NeverSuggest`, so the models still score
|
||
them as words). An expression is blocked only as the pair ("blow job"), never its words
|
||
alone. A word it may not offer, typed in full, is never corrected (as on Android); your own `UILexicon` words
|
||
(Contacts names, text replacements) are always offered.
|
||
- **Swipe typing (Settings › Keyboard › Swipe typing):** On by default (stored as
|
||
`experiments.swipeTyping` from when it was an experiment off by default: unset or true
|
||
is on, false off, so testers who turned it off keep it off), for the
|
||
English and Spanish layouts only (every other layout keeps a drag as a slide to
|
||
another key) and off under VoiceOver. A touch that goes down on a letter and travels
|
||
more than a key's pitch (width plus gap) is a swipe (`SwipeTyping`): its key no longer
|
||
types, a light trail (a shape layer, the last 0.45 s of the path) follows the finger,
|
||
and the strip shows what the path reads as so far, decoded off the main thread at most
|
||
every 70 ms. Lifting types the best reading after a space, unless the text before
|
||
ends in a space, a line break, an attachment (U+FFFC: a shared link's card in Messages,
|
||
a picture in the text) or opening punctuation (`( [ { « “ ‘ ¿ ¡ @ # /`, or a
|
||
straight quote after a space), as Apple's QuickPath does; shift capitalises its first
|
||
letter (caps lock all of it), the space it adds can start a sentence ("Hi." then a
|
||
swipe types " There"), and a name the dictionary only knows capitalised is
|
||
capitalised. The strip then offers the next three readings, the likeliest in the
|
||
centre; tapping one puts it in the word's place. Until anything else changes the
|
||
text, delete takes the whole word back (with the space it added), unless Settings ›
|
||
Experiments › Delete swiped words letter by letter is on
|
||
(`experiments.swipeDeleteLetterByLetter`, off by default), which makes delete act as
|
||
anywhere else and take one letter at a time; space never autocorrects it. As on Apple's
|
||
keyboard (watched on iOS 27.2), a space stays pending
|
||
after the word: a tapped letter, digit, opening mark or symbol goes in after a space
|
||
("hello" then w-o-r-l-d types "hello world", "(" types "hello ("), while
|
||
`, . ! ? ; : ) ] } ' - /` go against the word ("hello,") and take that space; shift you
|
||
set stays, and delete then takes the letters and the space one at a time. A second
|
||
finger going down ends the swipe first, so its word goes a space before that key's
|
||
letter. On the iPhone Duo's split
|
||
keyboard a swipe reads across the middle between the halves: its points are read with
|
||
the right half pushed against the left and the middle shrunk to one key's gap
|
||
(`KeyboardLayout.joinedForSwipe`), so either half's G and V is the one letter, as for
|
||
Spatial autocorrect's taps. Taps, holds for accents, slides
|
||
shorter than a key, delete's word slide, the space bar's trackpad, 123 and the Hex key
|
||
work as before: no swipe starts on them, and none once an accent menu is open. A swipe
|
||
that lifts on the key it went down on, never more than 1.6 key pitches from it, is a
|
||
tap that slid: its letter goes in as a tap's would ("a" dragged a little too far and
|
||
back), unless it read as a word that starts and ends on that key ("lol").
|
||
`SwipeDecoder` is a shape-matching decoder after SHARK2: the path is resampled to 32
|
||
points in key pitches; words come from the language's typing model by the keys the
|
||
path starts and ends near (each start letter is one range of the sorted word list,
|
||
read in place from the memory-mapped model, so nothing is built or kept between
|
||
swipes), kept when every key lies near the path and their ideal path (the polyline
|
||
through their keys' centres) is about as long; each is scored on location (point by
|
||
point, then dynamic time warping for the best 80), shape, and how near its first and
|
||
last keys the finger went down and lifted, the best 80 also on how their keys account
|
||
for where the finger turned and, from the touches' times, where it slowed (an edit
|
||
distance: a turn or pause no key of the word explains costs, and so does a key the word
|
||
goes straight through without a pause on it), and weighed with the model's word-pair
|
||
frequency after the word before. A straight stroke runs over the letters between its
|
||
ends, so the path alone can't tell "tired" from "tied" or "wire" from "write" (X
|
||
tester: guesses didn't account for the letters under the swipe); a pause on r can.
|
||
English contractions, which the Google Books model barely counts, and your own words
|
||
(Contacts names and other `UILexicon` terms, one-word
|
||
text replacements, words you kept) are read beside it. It never reads a slur, and reads
|
||
swear words only with Suggest swear words on (`TypingDecoder.NeverSuggest` and
|
||
`mayOffer`, as the strip). Its weights are tuned on synthetic swipes of the
|
||
autocorrect benchmark's held-out dev sentences and on mined sets of common words that
|
||
share a path; on the held-out test sentences (typical fingers, 390 pt) top-1 is 97.1%
|
||
in English and 94.2% in Spanish (94.9 and 90.9 without the turns and pauses), and on
|
||
held-out words that share a path 96.0% and 94.5% (85.1 and 81.2); a decode takes
|
||
0.91 ms at the median and 2.85 ms at p99 on an M5 Max (experiment folder
|
||
2026-11-20-hex-swipe-typing, results/2026-11-21). Record typing logs keeps a swipe's
|
||
touch moves and a `swipe_end` line (points, decode time, readings) for fitting it to
|
||
real fingers.
|
||
- **Record typing logs (Settings › Privacy › Advanced › Record typing logs):** Off by default,
|
||
and apart from Learn how I type. A one-time migration (`migrations.typingLogsOff.v1` in the
|
||
App Group, `TypingLogSettings.turnOffOnce`, run by the app and the keyboard at launch)
|
||
turned it off for everyone who had it on, keeping the logs already recorded.
|
||
Its description says what it records, and
|
||
turning it on asks first in a dialog (App Review 2.5.14): each tap and the
|
||
text you type or dictate with Hex, including the text around the cursor,
|
||
for 14 days, on this iPhone, to export or delete, never password, code or
|
||
card fields. **Turn On** turns it on; **Cancel** leaves it off. Turning it
|
||
off doesn't ask. For testing and training prediction and
|
||
per-user touch models: the keyboard writes every touch on Hex's keys (raw
|
||
position, radius, `UITouch` timestamp, the key it resolved to), every layout
|
||
with each key's frame and touch target, what each key did to the text
|
||
(inserts, deletes, autocorrections and undos, suggestion picks, double-space
|
||
periods, returns), the suggestions shown, the host's text around the cursor
|
||
after each word and when the host changes it, and each dictation's steps
|
||
(heard, learned fixes, cleanup, inserted) with its outcome. One NDJSON file a
|
||
day in the App Group (`TypingLogs/`), kept 14 days, at most 64 MB a day;
|
||
Privacy › Advanced › Typing logs shares a day or all of them (AirDrop to a Mac) and
|
||
deletes them. Password, one-time code, and card fields write one `skipped`
|
||
line and nothing else. Events are buffered on the main thread and encoded and
|
||
appended on a utility queue every 2 s. Nothing is sent. The keyboards look
|
||
the same with it on as with it off: it is only the switch. The schema is in
|
||
[`docs/typing-logs.md`](docs/typing-logs.md).
|
||
- **Settings (`ContentView`, `SettingsPages.swift`):** Home fits one screen: the
|
||
Hex header, one status line ("Ready" with a check, or "Finish setup · 1 step ›"),
|
||
Dictate, and four rows that each open a short page, then **Keyboard settings**
|
||
(Hex's page in iOS Settings) and **Run setup again**, and last **Privacy
|
||
policy**, **Support** and **Acknowledgements**. Privacy policy (App Review
|
||
5.1.1(i)) and Support open in Safari inside Hex (`HexLinks`, one base URL:
|
||
`https://dev.anoma.ly/hex` until the production site serves
|
||
`anoma.ly/hex/privacy`). Each row has a one-line description at most. The pages:
|
||
- **Dictation:** Style, Lowercase, Format lists, Spoken punctuation, End with a
|
||
period, Dictation language, Show what I say and Type as I speak (iOS 26 and later),
|
||
Translate what I say, Clean up what I say,
|
||
Custom vocabulary (its list below the switch), Keep mic warm,
|
||
and Microphone.
|
||
- **Keyboard:** Keyboard, **Full** (the default) or **Dictation bar** (stored
|
||
as `FullKeyboardSettings.isEnabled`, App Group key `experiments.fullKeyboard`
|
||
from when the full keyboard was an experiment: unset or true is Full, false
|
||
the dictation bar, so testers who turned that experiment off keep the bar;
|
||
an iPad follows the same choice, where until build 117 it always showed the bar), Auto-start (Never, On
|
||
switch by default, or Always) while the dictation bar is chosen, then Languages, Number row (off by default), Auto-capitalization, Autocorrect (with Spatial autocorrect below it while it's
|
||
on), Suggest swear words, Vocabulary (Hex's custom vocabulary list, how many words it
|
||
holds) and Key pop-ups only while Full is chosen, Haptics (the dictation bar uses it too),
|
||
then Swipe typing and Emoji (Clear emoji history while it's on), which say they need the
|
||
full keyboard while the dictation bar is chosen, and a field to try the keyboard in.
|
||
- **Privacy:** "Everything stays on this iPhone", Learn how I type (what it
|
||
learned below the switch), and under **Advanced**, Record typing logs (off
|
||
by default) with its Typing logs viewer while it's on.
|
||
- **Experiments:** Hinglish (Hindi in Latin letters), Greeklish (Greek in Latin
|
||
letters), Two-language dictation,
|
||
Clean up with Apple Intelligence, Adaptive touch (its status and Reset touch learning
|
||
below the switch), German spatial autocorrect, Korean keyboard (두벌식), Dvorak punctuation
|
||
keys, Comma and period keys, Quick accents, ABC under the emoji key, Keyboard themes (its
|
||
Theme menu below the switch while it's on), Two-thumb selection, Delete swiped words
|
||
letter by letter, Hide word suggestions, Join the keyboard when unfolded, and Multilingual
|
||
model (Parakeet).
|
||
|
||
Simulator scenes: `--snap-state settings-home` (ready), `setup` (keyboard
|
||
added, Full Access not confirmed yet), `setup-sheet`, `setup-sheet-microphone`
|
||
(before iOS has asked for the microphone), and
|
||
`settings-<dictation|keyboard|privacy|experiments>[-on][-end]`. Each writes
|
||
its page's switches to the App Group: off by default, on with `-on`; switches that are
|
||
on by default (Spatial autocorrect, Swipe typing, Emoji) stay on.
|
||
`-end` starts the page scrolled to the bottom. `settings-privacy-confirm`
|
||
opens Record typing logs' consent dialog. `--experiment-keyboard-themes` turns
|
||
Keyboard themes on, and `--keyboard-theme automatic|black|white|navy|forest|ember`
|
||
picks the theme the full keyboard draws;
|
||
the dictation bar keeps the system look, and iOS draws its own strip above or below
|
||
a third-party keyboard, which a theme can't color.
|
||
- **Dictation language (Settings › Dictation › Dictation language, iOS 26+):** Automatic by
|
||
default: the first of the iPhone's preferred languages (then its region's
|
||
locale, then US English) that Apple Speech transcribes, so a Norwegian iPhone
|
||
dictates Norwegian with no setup; Nynorsk tries Bokmål, the only Norwegian
|
||
Apple has. Apple Speech's Spanish includes es-ES, es-MX and es-US (and es-CL on
|
||
recent systems), and Spanish follows the iPhone rather than always Spain: a region
|
||
Apple has keeps it (es-CL on a Chilean phone), es-419, es-CO, es-AR and any other
|
||
Latin American, United States or unlisted region is the Americas' Spanish (es-US,
|
||
then es-MX), plain `es` is the iPhone's Region's (es-MX in Mexico, es-US in the
|
||
United States, Puerto Rico and Latin America, es-ES elsewhere), and only Spain and
|
||
Equatorial Guinea stay on es-ES (`DictationLanguageSwitch.spanishCandidates`,
|
||
`spanish(forRegion:)`). Before iOS 26 `SFSpeechRecognizer` lists es-CO and es-419
|
||
itself, so the on-device pick keeps them, then falls through es-US, es-MX and
|
||
Spain's. While
|
||
the keyboard switches between two or more languages, Automatic follows the one
|
||
it shows (Keyboard languages, below). The menu also lists every locale
|
||
`SpeechTranscriber` or
|
||
`DictationTranscriber` supports, by name. `SpeechTranscriber` is used wherever
|
||
it has the language (as before); `DictationTranscriber` (preset
|
||
`longDictation`) covers the rest (`SpeechLanguagePicker`,
|
||
`AppleSpeechLanguages`, `AppleSpeechModule`). The setting lives in the App Group
|
||
(`settings.dictationLanguage`, nil for Automatic) so in-keyboard dictation uses
|
||
the same rule and the same downloaded assets; both record the language they
|
||
resolved (`settings.dictationLanguage.resolved`). Picking a language prepares
|
||
(and if needed downloads) its model at once, or after the dictation in flight.
|
||
An iPhone without `SpeechTranscriber` (A13: iPhone 11, SE 2nd generation, all
|
||
3–4 GB) dictates English with `DictationTranscriber` on iOS 26+: loading
|
||
Parakeet there sat at 50% (Core ML's compile) until iOS ended Hex, at every
|
||
launch. Parakeet stays where it fits (6 GB) and before iOS 26, for English only, on
|
||
an iPhone without `SpeechTranscriber`: an iPhone that has it keeps English on Apple
|
||
Speech, on `DictationTranscriber` too while `SpeechTranscriber`'s English is set
|
||
aside, and goes to Parakeet only once neither Apple module's English installed twice
|
||
in a row (TestFlight build 126: one failed install sent an iPhone 15 Pro Max to a
|
||
Parakeet download that stalled, until a reinstall). When Apple Speech doesn't answer,
|
||
English stays on it there too, and any other language stays on Apple Speech, which
|
||
says it can't dictate it, rather than going to Parakeet as English
|
||
(`DictationController.appleSpeechLanguage`). After Hex ended twice in a row while
|
||
loading Parakeet, the next launches wait for Set up instead of loading it again, for
|
||
a day and on that build only (`ParakeetLoadGuard`: iOS ending Hex in the background
|
||
or a swipe away mid-load once is no evidence), and a Parakeet download that sits at
|
||
one fraction for 90 s ends with a message. A preparation that says nothing for 30 s
|
||
("Checking the speech model": Speech or its asset store not answering), or nothing
|
||
new for 120 s after, ends with "The speech model isn't ready. Try again."
|
||
(`watchPreparation`), as does the keyboard model's check behind setup's last step,
|
||
whose row then offers **Try again**. Apple's reservation and asset status are watched
|
||
at 60 s the same way (`SpeechModelDeadline`), a reservation or release that never
|
||
answers is forgotten so the next attempt asks again, and a keyboard check that hangs
|
||
counts no strike toward Hex's handoff (`SpeechModelNeedsHex.keyboardError`).
|
||
**When a model won't install**
|
||
(`SpeechModelInstall`, shared by the app and in-keyboard dictation): Apple lists
|
||
some languages whose `SpeechTranscriber` asset never arrives ("transcription.ru
|
||
asset not found after attempted download"), so Hex then installs the same
|
||
language on `DictationTranscriber`, and after the second such failure in a row goes
|
||
straight to it for three days. A download whose progress doesn't move for 60 s is
|
||
given up on. When neither module installs while the iPhone is online, twice in a row,
|
||
Automatic sets the language aside for three days and dictates the next one ("Russian
|
||
isn't available on this iPhone yet. Using English."); a language picked by hand says
|
||
so and offers that one ("Try English", with **Use English** in the app). A model that
|
||
installs clears the failures so far. Only Hex's own failures set anything aside:
|
||
in-keyboard dictation's are often its own (each process holds its own reservation, one
|
||
released, removed or over Apple's limit reads as not installed there, "Too many asset
|
||
locales allocated"). With nothing else to move
|
||
to (an English-only iPhone), Automatic tries the set-aside language again
|
||
rather than none, so "Apple's speech recognition can't dictate this language" means neither
|
||
module lists any of the iPhone's languages. Offline or stalled, it
|
||
says the download failed and tries again next time. In the keyboard, a model it
|
||
can't get itself says "Hex couldn't transcribe that. Try again." the first time, and the
|
||
second time in a row (within 6 h, no model in place since) "Open Hex to download
|
||
English.": Dictate and the Hex key then open Hex, which downloads it, until a model
|
||
installs (in either process; Hex checks again each time it comes to the front), a
|
||
language is chosen, or a day passes (`SpeechModelNeedsHex`,
|
||
`settings.dictationLanguage.keyboardNeedsHex`); a move to another language keeps its
|
||
own sentence. Coming back to Hex after it was in the background starts over a check
|
||
that failed or never answered, and a check that gives up bumps its generation so a
|
||
late answer can't reopen the row (`KeyboardSpeechPreparation`). Reservations make room
|
||
under `AssetInventory.maximumReservedLocales` by releasing others first. Only
|
||
Hex's own error text reaches the app or keyboard (`HexDescribedError`); Apple's
|
||
and other packages' descriptions go to logs, however readable.
|
||
The setup sheet's Speech model row names the language ("Norwegian Bokmål ·
|
||
Apple"). Steps that only know English follow the language: Clean up what I
|
||
say (an English tagger) uses word rules in Spanish, German and Norwegian and
|
||
skips other languages; Clean up with Apple
|
||
Intelligence runs only when the on-device model supports the language; the
|
||
styles trim English fillers (um, uh, er, ah) only in English, Nordic ones (eh,
|
||
ehm, øh, hm) in Norwegian, Danish and Swedish, Spanish ones (eh, em, mmm) in
|
||
Spanish, German ones (äh, ähm, öhm) in German, and none elsewhere (Norwegian
|
||
"er" is "is", Portuguese "um" is "a"); translation's source is the dictation
|
||
language rather than a guess. Custom vocabulary works in any language.
|
||
- **Two-language dictation (Settings › Experiments › Two-language dictation, iOS 26+):**
|
||
Off by default (`experiments.twoLanguageDictation`, App Group). While on, and while
|
||
dictation uses Apple Speech's `SpeechTranscriber`, one recording is read twice: in the
|
||
language dictation resolved (unchanged) and in the first other language in Settings ›
|
||
Keyboard › Languages whose base language differs from it, mapped by the same rule the
|
||
keyboard-follow language uses (`LanguageSettings.dictationLanguage(for:)`). Both must
|
||
resolve to `SpeechTranscriber`, and the second language's model must already be installed
|
||
(`AssetInventory.status`): dictation never downloads for it, and a language whose
|
||
`SpeechTranscriber` model was set aside as failing is skipped. Two `SpeechAnalyzer`s
|
||
read the same WAV, each with its own `AVAudioFile`, prepared with
|
||
`prepareToAnalyze(in: nil)` before starting, and a `SpeechTranscriber(locale:`,
|
||
`transcriptionOptions: [], reportingOptions: [],`
|
||
`attributeOptions: [.audioTimeRange, .transcriptionConfidence])`; both get custom
|
||
vocabulary's contextual strings. Each final result's timed runs give the merge its words
|
||
(the run's own characters, leading whitespace included, start, end, confidence), and a
|
||
run without letters or without a time range keeps its characters on the word before it.
|
||
`TwoLanguageMerger` cuts both readings at the earlier of two word ends within 150 ms,
|
||
scores each reading in a segment by its confidence-weighted seconds clipped to the
|
||
segment, and picks a reading per segment with a two-state Viterbi that charges 0.35 s of
|
||
confident audio for each language change, ties going to the primary. A word the new
|
||
reading starts with is dropped only when it repeats the last word kept, case and
|
||
punctuation aside, and the two spans overlap by at least half of the shorter one. The
|
||
winning runs are joined with their own characters, inserting one space at a language
|
||
change only when neither side has whitespace there. Whenever the second language wins no
|
||
piece, the primary's original text is used unchanged, so Chinese, Thai, punctuation and
|
||
untimed runs are exactly as Apple gave them. The text then runs through the same pipeline
|
||
with the primary language. No second installed language, a second analyzer that throws,
|
||
or no timed words in either reading dictates exactly as before; a primary reading that
|
||
finished without hearing anything reports empty so its caller throws its own no-speech
|
||
error without transcribing again; with the switch off no two-language work starts.
|
||
`TwoLanguageMergeTests` cover the merge and the run extraction. The app, keyboard,
|
||
widgets, and tests built on a macOS 26 GitHub runner with 33 tests passing (run
|
||
45005571072, the latest observed verification); nothing here has been measured on an
|
||
iPhone, and the simulator can't transcribe.
|
||
- **Switching the dictation language in the keyboard (iOS 26+, Full Access):**
|
||
the dictation bar shows the language beside the style ("Norsk ⌄") and opens a
|
||
tray of chips; the full keyboard's Hex key menu has a Language row with its
|
||
own page, and the space bar names the language faintly ("English", "Norsk"),
|
||
unless the keyboard switches between two or more languages: then it names the keyboard's.
|
||
Both offer Automatic plus up to five languages: the one set, the ones picked
|
||
lately (`settings.dictationLanguage.recent`), then the iPhone's preferred
|
||
languages, each matched to one Apple Speech has (`DictationLanguageSwitch`;
|
||
the keyboard lists Apple's languages once per process and caches them in
|
||
`settings.dictationLanguage.supported`). One variant per language, the first
|
||
of the current, the recents and the preferred winning, so Spain's Spanish and
|
||
Mexico's never both take a chip; picking one drops its siblings from the
|
||
recents (Norwegian's nb/nn/no aliases stay one language). Chinese stays
|
||
apart, so Mandarin's zh-CN, Taiwan's zh-TW and Cantonese's zh-HK each keep a
|
||
chip and a pick keeps the others in the recents. Names are each language's
|
||
own ("Español", "Deutsch", "עברית"), with the region where Apple Speech has
|
||
more than one of that language ("Español (US)" or "Español (ES)", "English
|
||
(GB)"; Norwegian stays "Norsk"), so the menu and the dictation bar's chips
|
||
say which variant dictation actually uses. Under Automatic, Settings ›
|
||
Dictation language says the same itself while the keyboard follows: "Follows
|
||
keyboard: English (US) · Spanish (US)" names each keyboard language's
|
||
dictation language, and is refreshed when Hex comes back to the foreground,
|
||
not only when the row
|
||
appears. A long press on a language (a chip, or a row on the Hex key menu's Language
|
||
page) shows **Remove**, which takes it off (`settings.dictationLanguage.removed`, so one
|
||
of the iPhone's own languages stays off too) and puts the one set back on Automatic;
|
||
Settings › Dictation language's **Remove from keyboard** does the same (TestFlight build
|
||
56: a language couldn't be removed, and one could only be added in Hex). **More
|
||
languages…** lists every other language Apple Speech has, the iPhone's own first, and
|
||
picking one there adds it (in the dictation bar the list takes the keys' place). Removing
|
||
or adding one never changes Automatic, which still follows the keyboard's language.
|
||
A pick writes the same App Group setting as Hex's menu, so
|
||
in-keyboard dictation uses it at once; a warm Hex session follows it within
|
||
100 ms and prepares the language between dictations, and a handoff or the
|
||
Action Button picks it up before recording.
|
||
- **Translate what I say (Settings › Dictation, iOS 26+):** Off by default. Translates
|
||
each dictation on the iPhone with Apple's Translation framework into the
|
||
language picked under Insert in, once its pack is downloaded. When it
|
||
can't (no pack, a pair Apple can't translate, or a translation that fails
|
||
twice, each in a new `TranslationSession`, never one kept), the
|
||
transcript goes in as spoken and the keyboard says so for 8 s:
|
||
"Couldn't translate, pasted as spoken" over the fix ("Download Chinese in
|
||
Settings"), whose Open Hex (the dictation bar: a tap) opens Settings ›
|
||
Dictation (`hex-dictation://keyboard/translation`). Settings keeps the
|
||
last failure (`experiments.translateDictation.lastFailure`, with Apple's
|
||
error code, never the text) beside the download until a dictation
|
||
translates. With Hinglish on, Hindi going into another language is
|
||
translated from its Devanagari, and romanized only if it goes in as
|
||
spoken; Greeklish does the same for Greek, translated from Greek letters.
|
||
Where you dictate, a "Translate to
|
||
English" switch turns it on or off without opening Hex (testers asked for
|
||
a button, not a setting): a row in the full keyboard's Hex key menu, lit
|
||
with a checkmark while on, and a chip in the dictation bar's language tray.
|
||
It writes Settings' own switch (`experiments.translateDictation`), the
|
||
only switch: nothing else turns translation off (build 56's "Paste in
|
||
Hindi" left `experiments.translateDictation.pastesTranslation` false
|
||
with Settings still on; that reads as off, and the next switch removes
|
||
it), and Settings re-reads it when it shows. Below it, Translation language lists
|
||
up to six targets (the current one, English, the dictation and preferred
|
||
languages, then common ones, only those Apple translates into); picking one
|
||
sets `experiments.translateDictation.target` and turns translation on.
|
||
Settings keeps the full language list and the pack download under Insert in. While dictations paste translated, the Hex
|
||
key wears the target's code ("EN") in its corner, the dictation bar shows
|
||
"→ EN" beside the Hex mark, and the space bar reads "Español → English".
|
||
The App Group carries the switch, so in-keyboard dictation, a warm
|
||
session, and the Action Button all follow it. The keyboard shares the
|
||
app's String Catalog, so the Hex key menu is in Chinese on a Chinese iPhone.
|
||
- **Keyboard languages (Settings › Keyboard › Languages, with Full chosen):** Automatic (the
|
||
default, as before: the dictation language's layout, below), or one or more layouts ticked
|
||
and dragged into order (`KeyboardLanguagesPage`, App Group `settings.keyboardLanguages`,
|
||
which reads the one-layout `settings.keyboardLayout` until first written and writes its
|
||
first language there). The layouts of the iPhone's own keyboards and preferred languages
|
||
come first ("On this iPhone"); ticking one from Automatic keeps Automatic's language beside
|
||
it, and unticking the last goes back to Automatic. With two or more, the space bar names
|
||
the current one ("Français", with the region only beside another French layout) and a
|
||
quick swipe along it switches (left the next, right the one before): letters, accents
|
||
and autocorrect. English and Español chosen together are Apple's bilingual "English &
|
||
Spanish" keyboard (`KeyboardLanguageList.layout`): both type on the first one's keys and
|
||
long presses, so English first is QWERTY with ñ on a held n in both, Español first keeps
|
||
its ñ key in both, and the swipe switches the language they autocorrect and dictate in
|
||
(black box, iOS 27.2 simulator: Spanish added to Apple's English keyboard has no ñ key and
|
||
English's long presses; English added to its Spanish keyboard keeps the ñ key and
|
||
Spanish's). English and Deutsch together are its "English & German" keyboard the same way
|
||
(iOS 27.2 and 26.5 simulators): German added to the English keyboard is QWERTY with
|
||
English's long presses, English added to the German keyboard keeps its layout, QWERTZ –
|
||
Deutsch's ü ö ä keys or, on QWERTZ, ten keys a row with ä ö ü and ß held. Apple and
|
||
Gboard switch with a long press on the globe, but a Face ID
|
||
iPhone shows no globe in a third-party keyboard and Hex's space-bar slide is its cursor
|
||
trackpad, so this follows SwiftKey: with two or more languages the cursor slides once the
|
||
space bar was held still for 0.35 s (the hold that starts Apple's trackpad, below), a slide
|
||
before that is a swipe, and one shorter than 24 pt types a space
|
||
(`FullKeyboardEngine.trackpadHoldDelay`, `languageSwipeDistance`).
|
||
The globe key is unchanged. The keyboard comes back in the language it was left in, in
|
||
every app and field, as Apple's keyboards do (black box: a switch in Messages held in
|
||
Safari and back; `settings.keyboardLanguages.current`; without Full Access an iPhone may
|
||
keep it only while the keyboard runs). The input mode says what it types: Hex sets
|
||
`UIInputViewController.primaryLanguage` to the language's BCP 47 identifier
|
||
(`KeyboardLanguage.inputModeLanguage`) as the language changes (unit-tested), meant to
|
||
make the host spell and autocorrect in the language typing; Apple documents it as
|
||
superseding the extension's Info.plist en-US, but whether the host follows it (Notes's red
|
||
underlines, for one) is not verified on a device. Switching from the full keyboard to the
|
||
dictation bar leaves the last reported language in place. A field gives a third-party
|
||
keyboard nothing else to go on: before this change `documentInputMode` was observed as nil
|
||
in Messages and the plist's en-US in a field that asks for English; what it reports with
|
||
the runtime language is unverified. **Dictation follows the keyboard:** with
|
||
two or more languages and Dictation language on Automatic, dictation is in the language
|
||
the space bar shows, in the iPhone's region where Apple Speech has it
|
||
(`LanguageSettings.keyboardDictationLanguage`,
|
||
`KeyboardLanguage.dictationLanguage`; none for Srpski), in the keyboard and in Hex,
|
||
which prepares the new language between dictations (TestFlight build 56: with the
|
||
keyboard on Español, dictation stayed English and Spanish came out "incomprehensible").
|
||
Spanish is the iPhone's Spanish, not always Spain's: a region Apple has keeps it
|
||
(es-CL on a Chilean phone), es-419, es-CO and any other Latin American or United
|
||
States region is es-US, then es-MX, plain `es` and no Spanish preferred language
|
||
follow the iPhone's Region (es-MX in Mexico, es-US in the United States, Puerto Rico
|
||
and Latin America, es-ES elsewhere), and Spain and Equatorial Guinea stay on es-ES.
|
||
A Spanish variant picked by hand for that language is used again
|
||
when the space bar comes back to it: the recent pick goes ahead of the preferred
|
||
languages, so es-US or es-MX survives the switch away and back. It doesn't where the
|
||
iPhone's own language has no layout (Hebrew), as dictating is how that language gets
|
||
written. **Switch dictation too** (off by default, shown with two or
|
||
more on iOS 26+) follows there too, and puts a dictation language picked by hand back on
|
||
Automatic at each switch (it used to write the keyboard's language as a pick, which
|
||
stayed after Spanish was unticked or the switch turned off). **Bilingual autocorrect:**
|
||
the first other chosen language with Latin letters and
|
||
an iOS dictionary (`KeyboardLanguageList.partner`: English beside French, French beside
|
||
English; none for Русский, Українська, Ελληνικά, Srpski, or two French layouts) is checked on the same keys,
|
||
with its Spatial model where it has one, as Apple's "Français et English" keyboard does: a
|
||
word either language knows as typed is never changed, a typo only one can fix gets that
|
||
fix ("teh" → "the" while French types), and when both can, the language of the last words
|
||
only one dictionary knows wins, else the clearly cheaper slip, else the keyboard's own;
|
||
the strip shows fixes and completions in the language the words before are in, never the
|
||
other's, even where that language has none (TestFlight build 56, Čeština and English:
|
||
"autocomplete suggests English words in Czech too"; the iOS 27.2 simulator's Czech
|
||
dictionary completes nothing, and English filled the strip), and with nothing before to
|
||
tell, both languages', in turn. In the simulator the English
|
||
keyboard alone changed 34 of 77 common French words and the French keyboard alone 27 of
|
||
80 English words; with both chosen, none. Any two Latin-script languages with an iOS
|
||
dictionary pair this way (English with Français, Deutsch, Italiano, Português, Polski,
|
||
Čeština, Norsk, Română, Español, or two of those), whatever their order; Polski's simulator
|
||
dictionary takes every word, so the any-word guard (#2143) leaves it out there. The words before
|
||
count for a language when only its dictionary knows them, or, from three letters, when
|
||
both do but only its typing model has them among its commonest (iOS's French and German
|
||
dictionaries take "the", "you", "your" and "please"). Once they are the keyboard's own
|
||
language, a word the other dictionary knows still gets its sure fixes (a missing
|
||
apostrophe or common slip from `unambiguousFixes`, a two-letter slip); a word without
|
||
a vowel never counts as the other's ("dk", "tkm" are abbreviations there); the other's
|
||
name guesses ("Zuri" for "Ziri") are never fixes; Spatial autocorrect may still read the
|
||
taps as another word, counting the other's word as one it changes only on its valid-word
|
||
margin; and at a sentence's start, "Im" and "Dont" are revised to "I'm" and "Don't" once
|
||
the next word leans English (`bilingualRevision`; German "Im Haus" stays). A word kept
|
||
because only one language knows it as typed takes the other's accents once the next word
|
||
leans that one, as Apple's keyboard puts "nao sei" right at "sei" (TestFlight build 56,
|
||
English and Português): "voce esta" becomes "você esta" ("voce" is an English word), only
|
||
for a lowercase guess of that dictionary's that adds accents alone, and not when the words
|
||
before lean the other way (`accentRevision`). "nao" alone was already "não" beside
|
||
Português. Held-out
|
||
English benchmark (2,000 Tatoeba sentences' synthetic taps, Spatial autocorrect on,
|
||
iPhone 17 simulator, iOS 27.2, keys as of #2180): English alone 5.75% WER; with Français
|
||
beside it 7.56% before, 5.79% now (99 correct words broken against 125 before and 113
|
||
alone); with Deutsch 7.53% before, 5.79% now; with the German Spatial experiment on,
|
||
7.18% before, 5.70% now. Typed at a key's centre after "we said that", 1 of 80 common
|
||
French words changes ("dont" to "don't") and 3 of 72 German ones ("auch", "ist", "habe"),
|
||
none before; none at a message's start or after French or German words, and a slip in a
|
||
French or German word at a message's start is fixed as often as before (26 of 50, 18 of
|
||
40). Per-letter engine p95 with a partner 4.1-5.4 ms (3.9-5.1 ms before), 2.2 ms alone.
|
||
Black box (iOS 27.2 simulator): Settings › Keyboards offers "Add to English (US)
|
||
Keyboard" for French (France, Canada, Switzerland), "Type in English and French on the
|
||
same keyboard", which makes one "English & French" bilingual keyboard on English's QWERTY
|
||
with no language switch; Portuguese, Polish and Turkish offer "Add to Existing
|
||
Keyboard". There it fixed "I dont know" and "Dont do it" to "don't", kept "est" and
|
||
"merci" after English words, and wrote "Im" as "I m"; the simulator's French word data is
|
||
thin ("je pense" became "he pence"), so how it weighs French there is unverified.
|
||
**한국어 (두벌식):** Settings › Experiments › Korean keyboard (off by
|
||
default, `experiments.koreanKeyboard`) adds 한국어 to the list: the standard Dubeolsik rows
|
||
with shift's ㅃㅉㄸㄲㅆ and ㅒㅖ, and no qwerty/azerty choice. `HangulComposer` composes each
|
||
syllable in the field (한, then delete to 하, ㅎ, nothing; ㅘ splits to ㅗ; 닭 + ㅗ → 달고,
|
||
한 + ㅏ → 하나), the keyboard replacing only the run's changed characters through
|
||
`deleteBackward`/`insertText` (no marked text). Space, return, punctuation, a page or
|
||
language switch, emoji, dictation, paste, undo, a selection action, the delete-word swipe, a
|
||
cursor move or hiding the keyboard commits the run. The cursor check records the last 32
|
||
characters before the run, the text after the cursor and the field, plus the host's selected
|
||
text, so a move to another place whose text before it ends alike commits too while a host
|
||
that trims the front of its window still matches if the window holds at least those 32
|
||
characters plus the run; a host that trims more than the tail or reports no selection is
|
||
best-effort, and a host that reports no text before the cursor joins syllables only while
|
||
jamo come within about half a second (a cursor move followed by a jamo inside that window
|
||
can't be detected). A field whose text after the cursor changes while you type (inline
|
||
autocomplete) commits each jamo, so syllables don't join there. Korean has no accent
|
||
holds, spelling dictionary, autocorrect, word suggestions or typing model, and with
|
||
the experiment off the language isn't listed, Automatic never picks it, and a stored "ko" is
|
||
ignored. Unit tests only (`HangulComposerTests`, `HangulKeyboardTests`,
|
||
`KoreanKeyboardGatingTests`): no CI compiles the iOS targets and no device run checks it.
|
||
- **Layout per language (Settings › Keyboard › Languages › Layout):** each language the keys
|
||
show (every chosen one, Automatic's while none is, and only the first of English and
|
||
Español together) gets a menu of the layouts Apple's Keyboard Layout list offers for it
|
||
(iOS 27.0 simulator, black box), its own (today's) first: English QWERTY, AZERTY, QWERTZ,
|
||
Dvorak; Deutsch "QWERTZ – Deutsch" (with ü ö ä), QWERTZ, QWERTY, AZERTY; Español
|
||
"QWERTY – Español" (with ñ), QWERTY, AZERTY, QWERTZ; Français (France) AZERTY, QWERTY,
|
||
QWERTZ, (Canada) QWERTY, AZERTY, QWERTZ, (Suisse) QWERTZ, AZERTY, QWERTY (Français (France)'s
|
||
AZERTY is Apple's "AZERTY – French", the first layout Apple lists for France and Belgium, with
|
||
the apostrophe key; Canada's QWERTY and Switzerland's QWERTZ are Apple's "QWERTY – French" and
|
||
"QWERTZ – French", the first each lists, with the key ending row 2; every other French layout
|
||
is the plain one); Italiano and Português QWERTY,
|
||
AZERTY, QWERTZ; Polski and Română QWERTY, QWERTZ, AZERTY; Čeština "QWERTZ – Čeština" and
|
||
"QWERTY – Čeština", Apple's "QWERTZ – Czech/Slovak" and "QWERTY – Czech/Slovak" (iOS 27.2
|
||
simulator), both its own rows with the ´ and ˇ keys, Z and Y swapped on the second; Magyar
|
||
"QWERTZ – Magyar" and "QWERTY – Magyar", the standard Hungarian QWERTZ with every accented
|
||
vowel on a held key, Z and Y swapped on the second (Apple's own Hungarian layout list wasn't
|
||
read on a simulator for this change); Srpski
|
||
QWERTZ (Hex's, though Apple now adds it as QWERTY), QWERTY, AZERTY. Norsk, Русский, Українська and Ελληνικά keep their
|
||
one layout (Apple also offers a phonetic "Russian – QWERTY", not built). The plain
|
||
layouts are the same 26 letters in every language, with the language's bottom row and
|
||
123 pages; only the letters move (`KeyboardLetterLayout`, `KeyboardLanguage.letterRows(_:)`).
|
||
Holds and autocorrect stay the language's: English on QWERTZ (the TestFlight request,
|
||
build 56) holds what English QWERTY does, and a plain layout without the language's own
|
||
letter keys holds them right after their base letter, as Apple's does (German QWERTZ:
|
||
A Ä À…, O Ö…, U Ü…; Spanish QWERTY: N Ñ Ń). Apple's Dvorak is P Y F G C R L and delete,
|
||
A O E U I D H T N S from the edge, shift and Q J K X B M W V Z, on keys narrowed so that
|
||
row fits (38.3 pt pitch on a 402 pt iPhone); split on the Duo it keeps delete on row 3
|
||
(Apple's split Dvorak unobserved). Settings › Experiments › Dvorak punctuation keys (off by
|
||
default, `experiments.dvorakPunctuation`) puts `,` and `.` ahead of P on row 1, whose nine keys
|
||
narrow to start at the left margin and end at the same delete key, puts `;` just left of the
|
||
space bar where it can give up a letter's width, and types `<`, `>` and `:` with a shift you
|
||
press or caps lock (auto-capitalization leaves the comma); on the split Duo the two
|
||
punctuation keys join only the left half, on a pitch narrowed so it keeps its right edge (its
|
||
top-row keys are narrower), and `;` takes from the left space bar. A URL or email field keeps its
|
||
own `.` key instead of row 1's, and Experiments › Comma and period keys adds no second pair
|
||
beside the space bar where this experiment put them on row 1. Off, every layout is key for key
|
||
the one it was. Kept per
|
||
language in the App Group
|
||
(`settings.keyboardLanguages.layouts`, `LanguageSettings.letterLayout(for:)`); unset,
|
||
not offered, or unreadable (an iPhone that keeps the App Group from a keyboard without
|
||
Full Access), a language types on its own layout. Spatial autocorrect, the touch decoder,
|
||
swipe typing, next-key growth and the neighbouring-key cost read the keys as drawn, and
|
||
Adaptive touch learns each arrangement as its own key plane; typing logs' `layout` lines
|
||
name it (`letters`).
|
||
- **Keyboard layouts (each language's letters):** English,
|
||
Norsk (`å` ending row 1, `ø æ` row 2), Español (`ñ` ending row 2; beside English, the
|
||
first of the two's keys, above),
|
||
Deutsch (QWERTZ with `ü`, `ö ä`), Italiano and Português on the English letters
|
||
(Apple's keyboards for them are QWERTY), Français (France) on Apple's "AZERTY – French" (rows of
|
||
10, 10 and 7 keys: W to N and the apostrophe key, a letter wide and centred between shift
|
||
and delete at their English size, as on Apple's; Hex's keycaps fall on Apple's to the pixel on
|
||
an iPhone 17 Pro and SE, upright and turned, iOS 27.2 simulator). The apostrophe key
|
||
(`FrenchTyping`) shows ’ right after a letter and types the apostrophe, curled as the field
|
||
curls quotes (l’homme, qu’il), and holds ' ’ ‘ `; anywhere else it shows and types ´ (never a
|
||
dead key) and holds ´ ˋ ˆ ¨, as Apple's does. It is no letter, so the decoder, Adaptive touch,
|
||
next-letter odds and swipe typing never read it, and French autocorrect fixes a slip on either
|
||
side of an elision's apostrophe as a word of its own ("l'hpmme" → "l'homme", "k'homme" →
|
||
"l'homme", `frenchElisionFix`). The iPad keeps its own ' and : key instead; Français (Canada) on QWERTY and Français (Suisse) on QWERTZ, as on Apple's
|
||
"QWERTY – French" and "QWERTZ – French": the same apostrophe key ends row 2, ten keys a
|
||
letter wide edge to edge under row 1's, its rules and holds France's (keycaps on Apple's to the
|
||
pixel on an iPhone 17 Pro and SE, upright and turned)
|
||
(automatic picks them for `fr-CA` and `fr-CH`, AZERTY for any other French, Belgium
|
||
included), Русский (Apple's ЙЦУКЕН: 11, 11 and 9 letters, ё on a
|
||
held е, ъ on a held ь), Українська (the standard Ukrainian ЙЦУКЕН, ДСТУ 4862:2007, in the
|
||
rows the iPhone's own Ukrainian keyboard shows on the iOS it runs on, because Apple changed
|
||
them in iOS 27: from iOS 27 (27.0 and 27.2 simulators) 12, 12 and 10 keys,
|
||
`йцукенгшщзхї`, `фівапролджєʼ`, `ячсмитьбюґ`, with ʼ the U+02BC apostrophe Ukrainian
|
||
spells with, and only ʼ offers anything when held, the other apostrophes (' ’ ‘ and the
|
||
backtick); before iOS 27 (26.2 to 26.5 simulators) Russian's 11, 11 and 9, `йцукенгшщзх`,
|
||
`фівапролджє`, `ячсмитьбю`, every letter the two share on Russian's key, ї on a held і and
|
||
ґ on a held г. On both the 123 and #+= pages' apostrophe key is ʼ, held like the letter, and
|
||
₴, which the system keyboard offers on its held currency key, is held on $; no ы, э, ъ or ё), Polski (QWERTY, as the
|
||
standard Polish layout is, on the English letters; it matches the system keyboard's
|
||
observed long-press options: Polish's own letter right after the base on A, C, E, L, N, O,
|
||
S and Z, so ą ć ę ł ń ó ś ż, with ź after ż, and nothing on U, I or Y; its currency key
|
||
is €, held for € $ £ ¥ ₩ ₽ and no zł, which Hex holds on the #+= page's €), Română (QWERTY
|
||
on the English letters, as Apple's iPhone Romanian keyboard is: ă and â held on A, î on I,
|
||
ș on S and ț on T right after the base, the comma-below ș and ț (U+0219, U+021B), never the
|
||
cedilla ş ţ, and nothing on the other letters; Apple's Romanian layout with keys of its own
|
||
for them is the iPad's; its currency key is €, held for € $ £ ¥ ₩ ₽ as Polish's is, and its
|
||
held " offers " „ ” » « “), Ελληνικά (it matches the standard Greek layout and the system
|
||
keyboard's observed long-press options, in the rows the iOS 27.2 simulator's own Greek
|
||
keyboard shows: `ερτυθιοπ´`, `ασδφγηξκλ`, `ζχψωβνμ`, nine, nine and seven keys, with no ;
|
||
or ς key; the tonos key ´ (U+00B4) accents the vowel right before it, α´ → ά, Α´ → Ά,
|
||
ϊ´ → ΐ, and after anything else, an accented vowel included, types ´ itself, as the system
|
||
keyboard was observed to with real touches in Messages (no dead key, no ¨ key); held, each
|
||
vowel offers its tonos form, ι and υ also ϊ ΐ and ϋ ΰ, σ offers ς and κ the kai sign ϗ, and
|
||
in capitals Σ offers nothing and Ι, Υ only Ί Ϊ, Ύ Ϋ, each callout laid out as the system's,
|
||
the key's own on the left (`AccentCatalog.arrangement`, as for English and Spanish); σ types ς while it ends the word and
|
||
turns back into σ when another letter follows, as the system keyboard does; its 123 page
|
||
has the ano teleia · in ;'s place and ; (the Greek question mark, which also starts the next
|
||
sentence with a capital) in ?'s, its #+= page ? in |'s, its currency key is $ held as
|
||
English's, and its held " offers French's order), Čeština (it matches the standard Czech
|
||
layout and the system keyboard's observed long-press options, in the rows the iOS 27.2
|
||
simulator's own Czech keyboard, "QWERTZ – Czech/Slovak", shows: `qwertzuiop`, `asdfghjkl´`,
|
||
`yxcvbnmˇ`, ten keys edge to edge on rows 1 and 2 and shift and delete a letter wide on row
|
||
3; the ´ (U+00B4) and ˇ (U+02C7) keys are no dead keys: with real touches in Messages, each
|
||
accents the letter right before it, ´ giving á ć é í ĺ ń ó ŕ ś ú ý ź and ˇ č ď ě ľ ň ř š ť ž
|
||
and ů after u, capitals too (A´ → Á, Uˇ → Ů), and after anything else, an accented letter
|
||
included, types itself ("aˇ", "á´"); delete then takes the accented letter whole
|
||
(`CzechTyping`); held, E offers ě é ê è ę ë ē ė, R ř ŕ, T ť, Z ž ź ż, U ú ů û ü ù ū, I í î ï
|
||
ì į ī, O ó ô ö ò õ œ ø ō, A á à ą â ä æ ã å ā, S š ß ś, D ď, L ł ľ ĺ, Y ý ÿ, C č ç ć, N ň ñ ń
|
||
and the ˇ key the ring ˚ (which makes ů and å), each callout laid out as the system's
|
||
(`AccentCatalog.arrangement`), and every other key, ´ included, nothing; its 123 and #+=
|
||
pages and their held keys are English's), Magyar (the standard Hungarian QWERTZ, so
|
||
`qwertzuiop`, `asdfghjkl`, `yxcvbnm`, on the English keys' QWERTZ; its nine accented vowels
|
||
are all held, á on A, é on E, í on I, ó ö ő on O and ú ü ű on U, the base first and then the
|
||
neighbouring Latin accents, ő and ű the double-acute letters U+0151 and U+0171, and every
|
||
consonant holds nothing; QWERTY is offered too) and Srpski (Latin: Apple's QWERTZ option for it; Apple's iPhone
|
||
Serbian Latin keyboard has no keys of its own for č ć đ š ž, they are held on c, d, s
|
||
and z), as Apple's iOS keyboards
|
||
lay them out (`KeyboardLanguage`: letter rows, long-press accents, spelling dictionary).
|
||
Automatic follows the dictation language (Hebrew, Cyrillic Serbian and other
|
||
languages without a layout get English; Serbian only in Latin script, `sr-Latn`);
|
||
before any dictation language is known, the first iPhone
|
||
keyboard, then preferred language, that has a layout. It doesn't read back the input mode
|
||
Hex reports (`primaryLanguage`), which is derived from this choice. Rows of 11 keys narrow
|
||
only the letters: shift, delete and the bottom row keep their English sizes,
|
||
and row 3's letters sit centred between shift and delete. Russian's nine-letter row 3
|
||
wouldn't fit beside them, so there shift and delete narrow instead, as on Apple's;
|
||
Ukrainian's rows (twelve keys from iOS 27, Russian's before it) leave them one letter wide, as on Apple's. Long-press options
|
||
match each language's system keyboard and its observed long-press options (iOS 27.2
|
||
simulator: every letter, 123 and #+= key held for about 3 s with a real touch, the
|
||
callout read from the accessibility tree; `KeyboardLanguage.letterHolds` and
|
||
`symbolHolds`, `AccentCatalog`): a key the system keyboard holds nothing on holds
|
||
nothing (Spanish Z, Serbian E), letters with their own key drop out (Norwegian A has no
|
||
å or æ, and Æ and Ø offer Ä and Ö; Spanish N offers only ń), German keeps ß on a held
|
||
S, and Hex's callout shows the options in the system's left-to-right order with the base
|
||
letter at the end under the thumb, in one row where the system wraps a long list
|
||
(English A) into two. Autocorrect and suggestions use the layout's
|
||
`UITextChecker` dictionary (`nb`, `es`, `de`, `it`, `fr`, `pt_BR` then
|
||
`pt_PT`, `pl`, `cs`, `hu`, `ro`, `ru`, `uk`, `el`; Serbian would use `sr_Latn`, which iOS 27.2 doesn't have, nor a
|
||
Croatian one); where the iPhone has none
|
||
they stay off except your own text replacements, and the English contractions
|
||
and typo list (`i` → `I`) apply only to English. Neighbouring-key costs follow
|
||
the layout (on QWERTZ, Z sits beside T), and an accent or tilde you left out
|
||
is the cheapest fix of all ("aqui" → "aquí", "manana" → "mañana"; a word the
|
||
dictionary knows, such as "esta", is never changed). Italian also restores common
|
||
accented spellings independently of the checker's guesses and a bilingual partner's
|
||
dictionary, where the plain spelling is no word of its own ("perche" → "perché", "citta" →
|
||
"città", the days, the -ità words; `KeyboardLanguage.accentRestorations`); "e"/"è" and
|
||
"ce"/"c'è" stay as typed, because "e" is "and" and "ce" is "to us": the strip offers "è"
|
||
and "c'è", and space never applies them (`KeyboardLanguage.accentSuggestions`). In Polish
|
||
every missing diacritic in a word counts as one such fix
|
||
("zolw" → "żółw", "maly" → "mały"). The
|
||
iOS 27.2 simulator lists `pl_PL` but its checker flags no word, so Polish fixes are
|
||
checked against a stub dictionary there. Greek likewise: every tonos or diaeresis left out
|
||
counts as one such fix, and so does σ where the word ends in ς ("καλημερα" → "καλημέρα",
|
||
"λογοσ" → "λόγος"); Greek fixes are checked against a stub dictionary too. Czech likewise:
|
||
every háček or čárka left out counts as one such fix ("cesky" → "český", "prilis" →
|
||
"příliš"), checked against a stub dictionary, and Hungarian too ("koszonom" → "köszönöm",
|
||
"kesobb" → "később"), while a word of its own keeps its spelling ("meg" and "kor" stay
|
||
though "még" and "kör" exist). A two-letter non-word one
|
||
neighbouring key (or one swap) from a very common two-letter word is fixed too
|
||
("tk" → "to", Spanish "nl" → "no", Hungarian "es" → "és"; `KeyboardLanguage.commonTwoLetterWords`,
|
||
English, Spanish, Italian and Hungarian). Auto-capitalization is unchanged.
|
||
The 123 and #+= pages are English's in every language but Greek (above); their held keys follow each
|
||
language's system keyboard where it differs (# holds № in Russian and Ukrainian, the
|
||
digits º and ª in Spanish and Italian, . holds · in French, $ holds ₴ in Ukrainian and
|
||
kr in Norwegian).
|
||
- **Size (full keyboard):** On an iPhone the suggestion strip is Apple's prediction bar,
|
||
as thick as Apple's (iOS 26.5 and 27.2 simulators, iPhone 17 Pro, Air, 17 Pro Max and SE,
|
||
upright and turned, light and dark, beside Hex in the same field): 44 pt from the
|
||
keyboard's top edge in every state (suggestions, Undo dictation, Paste, the Full Access
|
||
hint, the recording bar, the emoji panel's top row), with Apple's 8 pt
|
||
under it upright (6 pt turned), so row 1's keycaps start 52 pt down (50 turned), the keys
|
||
stay on Apple's rows, and where iOS adds nothing above the keyboard Hex is exactly as tall
|
||
as Apple's keyboard: 253 pt on an iPhone 17 or 17 Pro, 260 pt on an SE, 272 pt on an Air or
|
||
Pro Max (`KeyboardMetrics.withAppleBar`). On an 844 to 874 pt iPhone from iOS 27 Apple's
|
||
43 pt row 4 ends 4 pt below where iOS ends a third-party keyboard's view, so those 4 pt come
|
||
out of all four rows alike: 42 pt keycaps with Apple's 11 pt gaps, row 1 on Apple's line and
|
||
rows 2 to 4 1, 2 and 3 pt above Apple's, no row shorter than another; every bottom-row key
|
||
takes taps down to the view's bottom edge. The gap between rows 3 and 4 is the space bar's
|
||
across its width (row 3's keys whose middle is over it end at their keycaps), as on Apple's
|
||
keyboard, whose space bar takes taps up to 2.5 pt into row 3's keycaps (iOS 27.2); over 123,
|
||
emoji, return and the rest the gap splits, so a low m or delete above return never sends
|
||
return (Hex's rows sit 1 to 3 pt above Apple's, so a finger trained on Apple's lands lower on
|
||
Hex's row 3). Apple's space bar takes taps 13 pt below its
|
||
keycap, down to its own key plane's end; the 17 pt under Hex's view belong to iOS's globe band
|
||
and never reach a third-party keyboard. The 17 pt band iOS adds above a third-party
|
||
keyboard in some fields (Notes, Spotlight, a re-tapped Safari field; on iOS 26 even a plain
|
||
first-focused text view) is an exception Hex doesn't design for: there Hex is 17 pt taller
|
||
than Apple's. Inside the bar Hex draws what Apple's does: the words, SF 17 pt, centred 25 pt
|
||
down; a pending autocorrection in Apple's bubble, its whole cell across and a 38 pt capsule
|
||
from 6 pt down to the bar's bottom; 1 × 24 pt dividers between filled cells, hidden beside the
|
||
bubble; and the Hex key in a 59 pt cell of its own at its end of the bar, as Apple's writing
|
||
tools icon has at the other (a TestFlight screenshot, iPhone 17 Pro Max, iOS 27.2.1), with a
|
||
divider between it and the word beside it; the three cells share the rest. iOS 18 is
|
||
unmeasured and keeps the 38 pt strip it had (247, 254 and 266 pt). Apple's keyboard draws other rows before iOS 27 (iOS 26.2 and
|
||
26.5 simulators, iPhone 12, 16, 17 Pro, Air and 17 Pro Max, beside Hex in the same field):
|
||
the same keycaps, but upright its panel is 7 pt taller under the keys on an 844 to 874 pt
|
||
iPhone and 2 pt shorter on a 900 pt or taller one, so on iOS 26 Hex's rows sit 7 pt higher
|
||
(row 4 whole, 260 pt) or 2 pt lower (270 pt) to stay on Apple's (`KeyboardMetrics.appleRowsAreIOS26`);
|
||
the SE is the same on both. iOS 18 is unmeasured (the Studio has no iOS 18 simulator) and
|
||
keeps the iOS 27 rows Hex drew there before.
|
||
Turned, the bar is 44 pt with 6 pt to row 1 and the keyboard 204 pt (32 pt keycaps on a 40 pt pitch); on iOS
|
||
26 Apple's turned keycaps are 27.33 pt on a 35.33 pt pitch ending 4 pt above the bottom, and
|
||
Hex's are too, 187.33 pt in all (iOS 18, unmeasured, keeps 204 pt) (the SE keeps iOS 27's turned keys for now: Apple's turned SE
|
||
on iOS 26 is its upright keyboard across the whole screen).
|
||
Where iOS gives the keyboard a shorter view than it asks for (Spotlight on an iPhone
|
||
17 Pro Max, iOS 27.3, TestFlight: about 20 pt short), the rows fit the height it has:
|
||
Apple's bar gives up its 6 pt over the old 38 pt strip first, so a view up to 6 pt short
|
||
keeps every key whole and in place, and beyond that the keycaps and the gaps between rows shrink alike
|
||
(`KeyboardMetrics.fitted`), instead of row 4 falling under iOS's globe band.
|
||
Dictating no longer grows the keyboard. On the iPhone Duo (`KeyboardPosture`,
|
||
model `iPhone19,4`) iOS sets third-party keyboards in an inset platter with a band
|
||
of its own above them, so the strip has no gap there, and the keys follow Apple's
|
||
for each screen: 56 pt rows folded, 59.7 pt unfolded and turned, 51.7 pt unfolded
|
||
upright, starting as far in from the screen's side as Apple's. Folded, the cover
|
||
screen was measured beside Apple's keyboard on a Duo: its keycaps start 18.5 pt in
|
||
(54.5 px of 1375 at 2.95 px per point), and row 4 stands on row 3's columns: 123
|
||
from the edge to a gap before z, the globe on z (without it, space starts there),
|
||
space up to return, and return from a gap after m to the edge. A Hex key right of
|
||
space takes m's column, where Apple's dictation key is; left of space it follows the
|
||
globe and emoji. Its labels are 0.875 of the iPhone's there, as Apple's are: its 123
|
||
is 70 px wide folded against Hex's 80 at 2.95 px per point. The unfolded postures are
|
||
measured in the simulator.
|
||
- **Number row (full keyboard, iPhone):** Settings › Keyboard › Number row (off by default,
|
||
`settings.keyboardNumberRow`, App Group) puts 1 to 0 in a row of their own above row 1, in
|
||
every language and posture, upright and turned. Off, nothing changes. On, the keyboard
|
||
grows upward by the row and every row below stays on Apple's lines (iOS keeps a keyboard
|
||
on the screen's bottom): Apple's iPhone keyboard has no number row, so it follows
|
||
SwiftKey's (0.85 of a letter row's pitch, the same row gap; Gboard on iPhone has no row,
|
||
only digits held on row 1): 34.9 pt keycaps and 45.9 pt more keyboard on a 17 Pro (293 pt,
|
||
granted in a text view, a Safari form and Spotlight on the iOS 27.2 simulator), 47.6 pt on
|
||
a Pro Max, 34 pt turned. The digits sit on the 123 page's columns, take the gap under the
|
||
strip as row 1 did, and hold what the 123 page's digits hold (0 holds °). On the 123 page
|
||
the row hides, since its row 1 is the same digits, but its band stays so nothing jumps, and
|
||
that row 1 reaches up through it; the #+= page keeps the row. The digits never reach
|
||
Spatial autocorrect, next-letter touch help or Swipe typing (a drag from a digit is no
|
||
swipe), and a word holding digits is never corrected. Where iOS gives less height, the
|
||
number row shrinks with the others (`KeyboardMetrics.fitted`). The iPad keeps Apple's rows
|
||
for now: its docked keyboards would need Apple's own number row on the 13-inch and room
|
||
above the strip on the others.
|
||
- **Comma and period keys (full keyboard, iPhone):** Settings › Experiments › Comma and
|
||
period keys (off by default, `experiments.commaPeriodKeys`, App Group) puts a comma just
|
||
left of the space bar and a period just right of it, both a letter wide, on the letters page
|
||
in a plain text field. On the joined keyboard the comma starts where the bar used to start
|
||
and the period ends where it used to end, the bar moving inward at both ends; on the Duo's
|
||
split keyboard the comma takes from the left half's bar and the period from the right
|
||
half's, each bar keeping its inner edge. Neither is added unless the bar still holds a
|
||
letter-wide space after them (on an upright iPhone showing both the globe and emoji keys
|
||
with the Hex key beside space there isn't room, so they're skipped). They type `,` and `.`
|
||
with shift on too (they have no shifted symbols). The 123 and #+= pages, a URL or email
|
||
field (which keeps its own `.`), an iPad, and English's Dvorak letters while Dvorak
|
||
punctuation puts both on row 1 are unchanged. Off, every layout is key for key the one it
|
||
was. Unit tests only (`KeyboardLayoutTests`, `FullKeyboardEngineTests`): no CI compiles the
|
||
iOS targets, and no physical iPhone run checks it.
|
||
- **Word suggestions (full keyboard):** The strip above the keys shows the words the engine
|
||
offers as you type, and after a space the finished word's emoji. Settings › Experiments ›
|
||
Hide word suggestions (off by default, `experiments.hideWordSuggestions`, App Group) stops
|
||
the suggestions: the strip keeps its height, the Hex key, Undo, status, notices and the Paste
|
||
chip, but shows no typing or next-word suggestions, swipe readings, finished-word emoji, or
|
||
the chip that keeps what you typed while a correction is still ahead (keep literal).
|
||
Autocorrect still corrects on space, and the chip that puts back what it replaced (`.revert`)
|
||
stays after Delete. With no words offered, tapping the left slot to keep what you typed
|
||
mid-word and holding a word to add it to Custom vocabulary are gone. Off, the strip suggests
|
||
exactly as it did.
|
||
- **Background (full keyboard and dictation bar):** Hex draws nothing behind its keys, so
|
||
the keyboard's background is iOS's own glass panel: the same colour as the band iOS adds
|
||
above a third-party keyboard and the globe row under it, over any app or wallpaper, with
|
||
or without that band (no step at Hex's edges on an iPhone 17 Pro, iOS 27.2, in Notes,
|
||
Safari on a light page and Spotlight over a wallpaper, and in the iOS 27.0 and 27.2
|
||
simulators, light and dark). One case gets a cover, because iOS can leave Apple's keyboard
|
||
under a third-party one. In a field that takes focus as its screen appears and has an
|
||
input accessory view, opened with Hex, switching to Apple's keyboard never tells Hex it
|
||
disappeared and keeps its view; switching back puts a new Hex over both, and Apple's keys
|
||
show through Hex's translucent ones (every switch in the iOS 27.0 and 27.2 simulators,
|
||
under a bare clear keyboard extension too; one TestFlight report, build 56, iPhone 17,
|
||
iOS 27.0.1; never in Notes, Safari or Spotlight on an iPhone 17 Pro). Hex can't see the
|
||
app's views, but it can see its own earlier keyboard still on screen in the same field
|
||
(`KeyboardsOnScreen`), and only then draws a thin blurred material under itself
|
||
(`KeyboardBackdropView`), through which Apple's keys are an even, slightly lighter tone
|
||
with no shapes. Opened with Apple's keyboard, or focused by a tap, the same field frees
|
||
Hex as usual and nothing is left behind. Until build 117 the material was always on: 5 to
|
||
9 off the glass in dark mode and neutral grey where the glass takes a wallpaper's tint, so
|
||
Hex showed as a panel of another colour inside iOS's panel, with a seam under iOS's band.
|
||
- **Split keyboard (iPhone Duo):** Unfolded and held upright (the inner screen wider than
|
||
tall), the full keyboard splits as Apple's does there (`SplitGeometry`): each half at
|
||
the keyboard's side with Apple's 46.3 pt keys on a 57 pt pitch, row 2 half a key in
|
||
and row 3's letters a key and a half, an odd row's middle key on both halves (G and V
|
||
in English), a space bar on each half, and nothing in the middle, where a tap types
|
||
nothing. The strip sits over the right half, where Apple's predictions are; the
|
||
recording bar still spans the top. Every layout splits by the same rule (measured
|
||
against Apple's English split only; AZERTY, QWERTZ, Russian and the rest follow the
|
||
rule unchecked), and every page (123, #+=). Spatial autocorrect reads taps as on the
|
||
unsplit keyboard (`KeyboardLayout.unsplitX`), so either G is one key. Apple's split
|
||
follows its own Split Keyboard preference, which a keyboard extension can't read, so
|
||
Hex splits whenever the Duo is in that posture (the iOS 27.2 simulator has Apple's off
|
||
until `KeyboardShouldSplitInDefaultContext` is set). Settings › Experiments › Join the
|
||
keyboard when unfolded (off by default, `experiments.joinKeyboardWhenUnfolded`, App
|
||
Group) draws one joined keyboard there instead: unfoldedWide's own 39 pt rows on a
|
||
51.7 pt row pitch, 19 pt in from the keyboard's own side as unfoldedTall's keys sit
|
||
from its screen's, with the iPhone's portrait bottom row (123 and the globe to z's
|
||
right edge, the emoji key on x's column when the globe shows, space after them,
|
||
return from m). Off, nothing changes.
|
||
- **iPad:** The app is iPhone-only (`TARGETED_DEVICE_FAMILY` 1), but iPadOS runs its keyboard
|
||
extension as an iPad keyboard across the screen, so the full keyboard shows there too
|
||
(Settings › Keyboard picks it or the dictation bar, as on an iPhone). Docked, it has
|
||
Apple's iPad mini rows on every iPad (`KeyboardPadRows.swift`; the 11- and 13-inch's own,
|
||
with tab, caps lock and the 13-inch's number row, are a later step): delete ends row 1 and
|
||
return row 2, a shift key opens and closes row 3, after the letters Apple's two-symbol keys
|
||
(, and ! then . and ?; AZERTY , and ?, . and !, ' and :), the shifted symbol drawn over the
|
||
one a tap types and typed with shift (`KeyboardKey.shifted`), and the bottom row the globe,
|
||
123, the Hex key where Apple's dictation key is (whatever Settings › Hex key says: the strip
|
||
is all suggestions), the space bar, emoji (or 123 when Emoji is off) where Apple's second
|
||
123 is, and the hide-keyboard key. The letter keys are as wide as the longest row allows
|
||
with its delete or return at least 1.236 and 1.24 letters wide (English 54.5 pt, French
|
||
53.25 pt on the mini upright), row 2 is set in 0.4 of a key pitch (0.2 where it holds as
|
||
many letters as row 1, as AZERTY's does), the left shift is a letter wide (1.061 on AZERTY),
|
||
and delete, return and the right shift take what's left; the bottom row's keys are the same
|
||
in every language. The 123 and #+= pages follow Apple's: digits and delete, @ # $ & * ( ) '
|
||
" (€ £ ¥ _ ^ [ ] { } on #+=) and return, a page key either side of % - + = / ; : and the two
|
||
two-symbol keys (§ | ~ … \ < > ! ? on #+=), and row 4 as the letters' with ABC; Hex has no
|
||
undo or redo key there, so ABC takes the room of Apple's undo and second ABC.
|
||
German is QWERTZ without ü, ö and ä as Apple's iPad German is, holding them on u, o and a.
|
||
Measured against Apple's iPad mini in English, German and French (every key within a point);
|
||
the other languages' rows follow the same rule unmeasured. Docked, a held key's accents open
|
||
in Apple's iPad cells (below). The sizes are Apple's iPad key sizes (`KeyboardPadSize`,
|
||
`KeyboardPadPlacement`, `KeyboardMetrics.pad`), measured beside Apple's keyboard in the iPadOS 27.2 simulator:
|
||
docked, Apple's key heights, row pitch, gaps and side inset for the iPad mini (under 800 pt
|
||
across), 11-inch and 13-inch (1,000 pt and up), upright or turned, so its four rows sit
|
||
exactly on Apple's bottom four (letters and the bottom row; the 13-inch's number row and
|
||
the 11- and 13-inch's tab, caps lock and extra keys are a later step), counted from the
|
||
screen's bottom, where iPadOS leaves 20 pt empty under a third-party keyboard (3 pt under
|
||
Apple's). Which way up comes from the keyboard's width against the screen, not the size
|
||
classes, which an iPad keeps regular either way. Above the rows the strip is Apple's 44 pt
|
||
bar with the words centred, and above it 44 pt stay clear, as the dictation bar keeps them:
|
||
iPadOS always adds its own 55 pt bar above a docked keyboard (undo, redo, paste; empty in
|
||
Windowed Apps), and in some apps it draws that bar over the keyboard's top instead (Notes on
|
||
an iPad mini, iPadOS 26.6, TestFlight). So Hex is taller than Apple's (353 against 282 pt
|
||
on the mini and 11-inch upright). Docked, the labels keep their upright sizes turned, and
|
||
keys show no balloons, as Apple's don't.
|
||
Floating (iPadOS makes every keyboard 320 pt wide, with no bar above and its own globe and
|
||
dictation band below), Hex is Apple's floating keyboard: 26 × 38.25 pt keys on a 54.3 pt
|
||
pitch, 35.5 pt shift and delete, 123, space and return on row 3's columns, and the strip
|
||
where Apple's bar is, 271 pt in all as Apple's; it shows no globe of its own there.
|
||
iPadOS grants a custom keyboard any height it asks for (up to 900 pt tried); one that asks
|
||
for none gets 258 pt (343 turned) on the 11-inch. It always lays a third-party keyboard out
|
||
at that default first and switches to the height asked for about 15 to 20 ms after
|
||
`viewDidAppear`, even when the height was set before the view had a window (iPhone: 216 to
|
||
247 pt), and a tap while the keyboard still moves lands where the key was a frame earlier.
|
||
Measured with a scratch build logging every touch and layout (iPadOS 27.2 simulators, three
|
||
at once): taps 100, 200, 400 and 800 ms after the keyboard appeared, 100 launches each, all
|
||
went in on an 11-inch docked (400 of 400) and floating (399; one second tap never reached the
|
||
keyboard), as on an iPhone 17 Pro (399; one first tap at 100 ms typed the key above). Launched
|
||
into a turn, at the same moment after launch Hex takes taps sooner than Apple's keyboard
|
||
(1.4 s: 75 of 80 against 2 of 80; from 1.6 s both every one); earlier the keyboard is still
|
||
sliding in. Docked, a held key's accents show in Apple's iPad cells, half a key wide and
|
||
53 pt tall, two rows once there are more than five (`AccentMenu.padArrangement`,
|
||
`KeyboardMetrics.accentCellWidth`); floating, as on an iPhone. In the floating keyboard's
|
||
emoji panel the category icons shrink to fit its 18.6 pt slots (17 pt in 22 pt and wider).
|
||
- **Touch targets (full keyboard):** Every point of the key area types a key: gaps
|
||
split down the middle, the outer keys reach the screen's edges, and row 1 also
|
||
takes the 8 pt gap under the strip (on the Duo, the strip's bottom 4 pt below its
|
||
pills, `KeyboardLayout.stripTouchBand`), so a tap just above a keycap types it
|
||
rather than picking a suggestion (Apple's keys reach about 8 pt above theirs,
|
||
measured in the simulator). While you type (a
|
||
key is down, or one went down or came up in the last 0.25 s) the strip's whole
|
||
lower half goes to the row-1 key below the finger
|
||
(`FullKeyboardEngine.takesStripTouch`): Record typing logs showed every strip
|
||
pick that soon after a key was an e aimed high landing on the left slot, which
|
||
kept "th" and typed a space where "the" was meant; nobody reads a new suggestion
|
||
and taps it that fast. Taps are not
|
||
shifted: moving every logged tap up by 2 to 8 pt landed no more of them on the
|
||
meant key, and Apple's row borders sit within about 2 pt of the gaps' middles
|
||
when nothing is typed. While autocorrect is on in a text field that allows it,
|
||
the English and Spanish word lists (Spatial autocorrect's models, whether or not
|
||
it is on) let the likelier next letter claim up to 5 pt of a
|
||
neighbouring letter's target, 1.25 pt for each e-fold it is likelier
|
||
(`FullKeyboardEngine.letterOdds`): after "th" a tap just inside d types e, as
|
||
Apple's does (there e reaches about 10 pt into d). A finger that rolls within
|
||
that reach keeps the letter. Other layouts keep fixed targets. The keyboard's
|
||
window no longer holds back touches near the screen's side edges for system
|
||
gestures: Record typing logs showed them reaching the keys about 80 ms later
|
||
than others on an iPhone (whether this removes it is unverified on hardware;
|
||
the simulator never delayed them).
|
||
- **Spaces (full keyboard):** A suggestion (a word, what you typed, or an emoji)
|
||
types its own space, so a space pressed right after it is swallowed, as on Apple's
|
||
keyboard; it still counts towards double-space for a period. Holding the space bar
|
||
still for 0.35 s starts the trackpad, as Apple's does, and letting go then types no
|
||
space (`FullKeyboardEngine.trackpadHoldDelay`). A tap that slides before then is still a
|
||
space, however far it goes, as on Apple's (black box, iOS 27.2: taps of 60 to 150 ms
|
||
sliding 8 to 24 pt, and slides of 40 pt in 150 ms, all typed one); a slide of 10 pt used
|
||
to start the trackpad at once and type nothing (TestFlight build 56: "missing the space
|
||
key"). With two or more languages such a slide is a language swipe (Keyboard languages,
|
||
above). In a rollover space keeps its place: a key going down while the space bar is
|
||
still held types the space first, as a held letter's does. The
|
||
full keyboard's trackpad moves as Apple's does (black box, iOS 27.2): the finger drags a
|
||
point through the text at the text's own scale, in two dimensions, and the cursor goes to
|
||
the character nearest it. Slowly a point of finger is a point of text, and faster it goes
|
||
further (1.4 times at 400 pt/s, twice at 1,000, 2.6 times at 2,000, `TrackpadCursor.gain`);
|
||
a line takes the font's line height (20 pt), the first half of one; the x carries across
|
||
short lines; it stops at a line's start and end and at the text's first and last line. A
|
||
keyboard can only move the cursor by UTF-16 offsets, so `TrackpadCursor` lays out the text
|
||
the field shares (the body font's own widths, rows wrapped at the keyboard's width less
|
||
40 pt) and moves by the offset to the nearest position, a 👍🏽 in one step. The field shares
|
||
only the cursor's sentence or line and a little before it, so at that text's edge the cursor
|
||
waits for the host to show more, and past its first or last line it steps just past it in
|
||
one move, which the host makes only if there's text there. The keys blank to keycaps 60%
|
||
of the way to the backdrop, as Apple's, with no haptic while it moves. Apple's second-finger
|
||
tap to select text has no host counterpart: `UITextDocumentProxy` (iOS 27.2 SDK) can read the
|
||
selection but not make one. Off (the default), a second finger does nothing. Settings ›
|
||
Experiments › Two-thumb selection (off by default, `experiments.twoThumbSelection`, App Group)
|
||
gives it one: while the space bar is the trackpad, a second finger starts a selection from
|
||
the cursor, and the finger further left moves the start of the range, the other the end, each
|
||
with its own `TrackpadCursor.State` (the trackpad's gain, acceleration and rows). Settling the
|
||
trackpad at the start can move the host cursor once; after that, no host cursor moves. The
|
||
strip shows the text around the range with the selected part highlighted,
|
||
within the text the host shares: the range is clamped to what the field shares around the
|
||
cursor when the selection starts (the field shares only the cursor's sentence or line). If
|
||
the host's context hasn't caught up at the start the whole span is kept; either way an
|
||
action runs only while the host still shares the whole range, and otherwise dismisses
|
||
without acting. Both thumbs up shows Delete · Cut · Copy · Dictate · Cancel (Cut and
|
||
Copy only with Full Access, as the strip's clipboard; without it they aren't offered): Delete
|
||
puts the cursor at the range's end with `adjustTextPosition` and deletes its characters one by
|
||
one (a 👍🏽 is one); Cut copies then deletes; Copy copies and puts the cursor at the end;
|
||
Delete and Cut leave it at the range's start; Cancel changes nothing; and any other key
|
||
dismisses the actions without acting. Dictate leaves the range in place and starts the
|
||
keyboard's usual dictation there (the warm Hex session or in-keyboard dictation). When its
|
||
words are about to insert and the field still shows the range unchanged (the same field, and
|
||
the text around it), they replace it; otherwise they insert at the cursor as any dictation
|
||
would, with the range left alone. Cancelling or hearing nothing leaves the range untouched.
|
||
Where dictation would open Hex instead (no Full Access, or the keyboard can't dictate here),
|
||
tapping Dictate dismisses the actions and opens Hex (or shows the Full Access hint) with the
|
||
text left in place. An empty range ends like a normal
|
||
trackpad lift. Delete and Cut ask the host to delete once per Character: in a script whose
|
||
host removes one code point per call (some Devanagari and Thai clusters), the last code
|
||
points of a cluster can stay behind. The dictation bar's space bar is the same trackpad along the line (the same still hold or a 10 pt slide, a
|
||
character per 7.5 pt, the keys' glyphs blanking, a selection tick per step; TestFlight
|
||
build 56 asked for it), and a tap still types a space.
|
||
- **Smart punctuation (full keyboard):** As on Apple's keyboard (black box, iOS 27.2), the
|
||
' and " keys type opening quotes at the start, after a space, a bracket or an opening
|
||
quote, and closing ones elsewhere (‘Hi’ it’s “yes” 5’9”); a second hyphen makes an em dash
|
||
(a--b is a—b); autocorrect's and the strip's apostrophes are ’ (don’t, I’m). Each language
|
||
types its own quotes (`SmartQuotes`): „ “ ‚ ‘ in German, « » with no-break spaces in French,
|
||
„ ” ‚ ’ in Polish, « » in Norwegian, Russian and Ukrainian, “ ” ‘ ’ elsewhere. A field's
|
||
`smartQuotesType` or `smartDashesType` of `.no` turns each off (Messages' composer has
|
||
both off: Apple's keyboard types straight quotes and -- there). The keys reason about the
|
||
text with straight quotes (`SmartPunctuation.plain`). A mark that ends a clause (, . ! ? ;
|
||
: )) right after a picked suggestion takes the space the suggestion typed ("Hello,"), as
|
||
Apple's does with `smartInsertDeleteType` on or off.
|
||
- **Delete after an autocorrection (full keyboard):** As on Apple's keyboard (black box,
|
||
iOS 27.2), delete takes only the space or mark after the corrected words, and the strip's
|
||
left slot offers what you typed, highlighted with an undo arrow (Apple's text view shows
|
||
it in a bubble under the word, which a keyboard can't draw); tapping it puts that back,
|
||
without the space, and turns the fix down for the session. Space keeps the fix. The offer
|
||
comes back whenever the cursor is at the end of the latest autocorrection again, revisions
|
||
("well see" to "we'll see") and joins included. Text replacements offer none, as on Apple's.
|
||
- **Key clicks:** Every key of the full keyboard (delete, shift, 123 and return included, and
|
||
each delete repeat), its strip, and the dictation bar's keys ask for the system click on
|
||
touch-down (`UIDevice.playInputClick`), which plays only because the keyboard's root view
|
||
(`KeyboardInputView`) adopts `UIInputViewAudioFeedback`: the full keyboard view adopted it
|
||
before, as a subview, and never clicked. It follows Settings › Sounds & Haptics › Keyboard
|
||
Feedback › Sound, with no setting of Hex's. Apple's keyboard has other sounds for delete and
|
||
the modifier keys, but the click is the only keyboard sound a keyboard may play, so every key
|
||
gets it. In the iOS 27.2 simulator each press reaches the system click (system sound 1104,
|
||
in the keyboard's log) with Full Access or without, and none with the Sound switch off; on
|
||
an iPhone, Apple's guide says it needs Allow Full Access.
|
||
- **Slides from 123, #+= and shift (full keyboard):** As on Apple's keyboard, holding 123
|
||
(or ABC, or #+=) shows its page, and sliding shows the pop-up of the key under the
|
||
finger (or darkens it with Key pop-ups off); letting go types that key and goes back to
|
||
the page it came from. Letting go back on the page key types nothing and goes back too;
|
||
off the keys it types nothing and the page stays. Holding shift and sliding to a letter
|
||
shows and types its capital. None of them is a swipe, and a slid-to key opens no
|
||
accent menu (Apple's does after a hold).
|
||
- **Quick accents (full keyboard, Settings › Experiments):** Off by default
|
||
(`experiments.quickAccents`, App Group). Holding a letter opens its callout with the letter's
|
||
main accent selected — the language's preferred one (`KeyboardLanguage.quickAccentPrimaries`:
|
||
Spanish o → ó, English e → é, Italian a → à, and Á with shift), else the first variant after
|
||
the key's own in its holds (`AccentCatalog.primaryVariant`) — so letting go where the finger
|
||
landed types it. The cell under the finger when the callout opened is remembered
|
||
(`accentOpenCell`), so the preselection stands until the finger moves into another cell,
|
||
where sliding picks that one as before; letting go far below still cancels, and another key
|
||
going down while the callout is held commits the preselection. Only letters: the 123 and
|
||
#+= pages' held symbols, and the mark keys ˇ and ʼ, keep the key's own selected. Off, the
|
||
callout opens on the key's own letter. Unit tests only (`FullKeyboardEngineTests`): no CI
|
||
compiles the iOS targets, and no physical iPhone run checks it.
|
||
- **Emoji (Settings › Keyboard › Emoji, for the full keyboard):** On by default (stored as
|
||
`experiments.emoji` from when it was an experiment off by default: unset or true is on,
|
||
false off, so testers who turned it off keep it off). An emoji key
|
||
right of 123 (after the globe) in plain text fields, not web address, email or
|
||
ASCII-only ones. It opens a panel in place of the strip and the keys: a row of
|
||
emoji for the text before the cursor, then a grid that scrolls sideways through
|
||
Frequently used (eight columns: your emoji, then common ones) and the eight
|
||
categories, at the system emoji keyboard's 46 pt pitch, with ABC, the categories and
|
||
delete below, drawn without keycaps as the system's are; a tap anywhere under the
|
||
grid goes to the key under it, each category the full width of its slot. With
|
||
Settings › Experiments › ABC under the emoji key on (off by default,
|
||
`experiments.emojiLettersUnderEmojiKey`, App Group), ABC sits exactly where the emoji
|
||
key that opened the panel was, its x and width, so switching back to the letters needs
|
||
no thumb move; a tap left of it still goes to ABC, and the categories take the room
|
||
between it and delete. Off, or in a panel with no emoji key behind it (the dictation
|
||
bar's), ABC keeps its leading-end place. As on the
|
||
system emoji keyboard (iOS 27.2 simulator), a tap on a category jumps the grid there as
|
||
the finger lifts, and a finger sliding along the bar scrubs the grid live, each icon's
|
||
slot running through its own category from start to end, with the chip on the category
|
||
under the finger and a light tick for each new one (`EmojiCategoryBar`); scrolling the
|
||
grid moves the chip. The row reads the last eight words (the newest counts most), their plurals
|
||
and endings, letters held down ("lmaooo"), phrases ("new year") and the word still
|
||
being typed, matched against Unicode's CLDR names and keywords for the layout's
|
||
language and English and against Hex's own chat words ("congrats", "bday"), one
|
||
emoji per family ("woman running", "man running"); your emoji break ties and fill
|
||
the rest (`EmojiSuggester`). Search (the magnifier) types with the letter keys into
|
||
a field in the strip, results above it, by name first. Holding an emoji with skin
|
||
tones offers five; the one you pick is typed and kept for that emoji. After a space,
|
||
an empty strip offers the finished word's own emoji ("pizza " → 🍕), never in place
|
||
of word suggestions. The App Group keeps a use count per emoji that halves every 10
|
||
days and each emoji's skin tone, no words (`EmojiUsageStore`); Clear emoji history
|
||
empties it. The index is `HEXShared/FullKeyboard/Emoji/emoji-index-<lang>.txt`,
|
||
built by `scripts/build-emoji-data.py` from Unicode's emoji-test.txt 18.0 and CLDR
|
||
49 (Unicode License v3, `LICENSE-Unicode.txt` beside it, which permits commercial
|
||
use with the notice) and `scripts/emoji-aliases-en.txt`; the keyboard maps it and
|
||
binary-searches it in place (about 0.7 MB on the heap), loading it on the first
|
||
emoji key press or finished word. Emoji newer than 15.1 are listed only where the
|
||
emoji font draws them. Nothing leaves the iPhone.
|
||
- **Screen stays on while dictating in Hex:** recording and transcribing set
|
||
`isIdleTimerDisabled`, and every way out (done, cancel, failure) hands the
|
||
screen back to Auto-Lock. iOS honors it only while Hex is in front: the
|
||
handoff screen and in-app recording, not a warm session or in-keyboard
|
||
dictation, where another app is in front and a keyboard extension has no
|
||
public way to keep the screen on.
|
||
- **Resilient capture & expiry contract (`DictationController`):**
|
||
- Monotonic `jobGeneration`, retained `transcriptionTask`, and unique
|
||
per-capture WAV filenames ensure an expiring warm session never drops an
|
||
already-accepted transcription (deliverable for 30s), while a stale
|
||
winding-down inference can never overwrite a newer capture or delete its
|
||
audio file.
|
||
- A keyboard's dictation never leaves the keyboard stuck. Hex reads the
|
||
keyboard's Done and Cancel and heartbeats the job's snapshot until its
|
||
text is in, in a warm session and in One shot alike (One shot used to
|
||
ignore them, so a handed-off recording ran to its 60 s limit), and asks
|
||
iOS for background time to finish the text. The keyboard takes a
|
||
recording or transcription whose heartbeat is more than 6 s old as over
|
||
(`KeyboardSnapshot.isOrphanedJob`): a Hex that ended, crashed or was
|
||
suspended mid-job used to hold every keyboard in recording for good. A
|
||
result that missed its 30 s to go in by itself stays offered as **Paste
|
||
dictation** for 10 minutes, in the full keyboard's strip and in the dictation
|
||
bar's top row (on an iPad it once offered nothing). A failure or note
|
||
from Hex ("Another app is using the microphone. Try again.") shows for 8 s
|
||
after Hex wrote it (`KeyboardSnapshot.messageAt`), not for as long as it
|
||
stays in the App Group; meanwhile both keyboards dictate in the keyboard, and
|
||
after it a warm session's next start is Hex's Try again.
|
||
- An in-keyboard dictation the keyboard is dismissed from (or another app
|
||
comes forward) stops its capture at once, keeps what was said, and
|
||
finishes without the keyboard; its text is held in the App Group
|
||
(`keyboard.heldDictation`), goes into the same field if the keyboard
|
||
comes back there within 30 s, and is offered as Paste dictation
|
||
anywhere else or later. A capture that goes quiet for 6 s finishes with
|
||
what it heard, and a transcription stuck for 2 minutes is given up.
|
||
- Phone/FaceTime/Siri audio interruptions finalize captures `>= 300 ms` so
|
||
spoken words are not lost, cancel shorter taps cleanly out of `Listening`,
|
||
and resume warm keepalive when iOS signals `.shouldResume`. Hex's own
|
||
recording (the app, a warm session's keepalive and captures) is
|
||
`playAndRecord` with `mixWithOthers` and A2DP output, never `duckOthers`,
|
||
so it doesn't interrupt another app's video or music, and ending a session
|
||
deactivates with `notifyOthersOnDeactivation`. It sets no preferred sample
|
||
rate: `playAndRecord` runs the microphone and the output at one rate, and
|
||
the 16 kHz it used to ask for moved everything the iPhone played and
|
||
recorded (a screen recording's audio too) to 16 kHz while Hex held the
|
||
session (48 kHz to 16 kHz in the simulator). The recorder writes 16 kHz
|
||
itself, and the keepalive is input only (an `AVAudioEngine` input tap, no
|
||
output node: its output element is off). A keepalive that keeps stopping by
|
||
itself is restarted at once, then after 1 s, 2 s, … up to 30 s, not every
|
||
100 ms. **Microphone** (`MicrophoneChoice`, `MicrophoneRoute`) picks each
|
||
dictation's input: **Automatic** (the default) a wired or USB headset's
|
||
microphone, the car's in CarPlay (the driver's microphone Siri uses, where
|
||
the iPhone's hears the speakers and the road), else the iPhone's, so AirPods
|
||
and other Bluetooth headphones keep playing at full quality and nothing
|
||
waits for call audio; **iPhone** always the built-in one; **Bluetooth** also
|
||
Bluetooth hands-free microphones (AirPods, Bluetooth car kits, which iOS
|
||
can't tell from headphones), in call audio (8–16 kHz) and still mixing. Only
|
||
Bluetooth adds `allowBluetoothHFP`, and the input is always set
|
||
explicitly. The warm keepalive stays on the iPhone's microphone whatever the
|
||
choice, so a warm session never holds a car or headset in call audio between
|
||
dictations; a dictation moves to its input as it starts (a Bluetooth
|
||
headset's call link can take 0.5–2 s, unmeasured here). The setting replaced
|
||
Always use iPhone microphone: left on it reads as Automatic, turned off as
|
||
Bluetooth. In-keyboard dictation asks WebKit for the same input by name
|
||
(`deviceId`), read from the keyboard's own session; left to itself WebKit
|
||
takes the input of the route already playing, the iPhone's while a car or
|
||
headphones play over A2DP. Its capture session (`playAndRecord`,
|
||
`videoChat`, no `mixWithOthers`) no app can change: other apps' video and
|
||
music pause while it records and WebKit ends it without telling them, so
|
||
once it has ended the keyboard activates and deactivates its own session with
|
||
`notifyOthersOnDeactivation` (`InterruptedAudio`), for players that resume
|
||
on `.shouldResume`. Unverified on an iPhone: the simulator has no audio
|
||
interruptions.
|
||
- **Action Button, Control Center, Back Tap & Live Activity:**
|
||
- `DictateWithHexIntent` (`AudioRecordingIntent` + `LiveActivityIntent`) and
|
||
`HexDictationControl` (`ControlWidget`) start recording on first press with
|
||
a Dynamic Island / Lock Screen Live Activity (`Stop` and `Cancel` buttons)
|
||
and transcribe + copy to the clipboard on second press.
|
||
- Only these dictations get a Live Activity; the keyboard and the app show
|
||
their own recording UI. It shows Copied for 2 s, and a cancel, failure, or
|
||
empty capture ends it at once, under a background task so a suspending app
|
||
can't leave it behind (`LiveActivityCoordinator`). Every end also ends any
|
||
other Hex activity, and launch, returning to Hex, and ending the warm
|
||
session (Keep mic warm › Off) end any no dictation owns.
|
||
|
||
## Acknowledgements
|
||
|
||
The app shows these under Settings › Acknowledgements (`HEXMobile/AcknowledgementsView.swift`).
|
||
|
||
- **Google Books Ngram data:** Spatial autocorrect's word and word-pair frequencies,
|
||
German's capitalisation shares and French's list of words outside its model come from the Google Books Ngram Viewer data, version 3
|
||
(2020), by Google, licensed under
|
||
[Creative Commons Attribution 3.0](https://creativecommons.org/licenses/by/3.0/). Counts
|
||
are summed over 1980–2019 and kept only for words in the lists below.
|
||
- **SCOWL:** English words are checked against SCOWL (Spell Checker Oriented Word Lists)
|
||
and VarCon by Kevin Atkinson, sizes up to 70 (<http://wordlist.aspell.net/>). Copyright
|
||
2000-2018 by Kevin Atkinson. Permission to use, copy, modify, distribute and sell these
|
||
word lists, the associated scripts, the output created from the scripts, and its
|
||
documentation for any purpose is hereby granted without fee, provided that the above
|
||
copyright notice appears in all copies and that both that copyright notice and this
|
||
permission notice appear in supporting documentation. Kevin Atkinson makes no
|
||
representations about the suitability of this array for any purpose. It is provided
|
||
"as is" without express or implied warranty. SCOWL's inflections come from WordNet 1.6,
|
||
Copyright 1997 by Princeton University, all rights reserved, provided "as is" without
|
||
warranties (the full notice is in the app); its other sources are public domain.
|
||
- **LDNOOBW:** The keyboard's suggestion filter uses the
|
||
[List of Dirty, Naughty, Obscene, and Otherwise Bad Words](https://github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words/tree/5faf2ba42d7b1c0977169ec3611df25a3c08eb13)
|
||
(LDNOOBW), © 2012–2020 Shutterstock, Inc., and contributors: its English, Spanish,
|
||
German and Norwegian lists, licensed under
|
||
[Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/).
|
||
Hex lowercased and classified the entries, omitted ordinary, medical and search-only
|
||
terms, and added common slurs, inflections and spelling variants.
|
||
- **Parakeet V2:** Hex dictates English with NVIDIA's
|
||
[Parakeet TDT 0.6B v2](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v2), licensed under
|
||
[Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/),
|
||
downloaded as FluidInference's
|
||
[Core ML conversion](https://huggingface.co/FluidInference/parakeet-tdt-0.6b-v2-coreml)
|
||
(CC BY 4.0, encoder weights at reduced precision), used unchanged.
|
||
- **Parakeet Ultra:** Experiments › Multilingual model downloads
|
||
[Parakeet Ultra](https://huggingface.co/moondream/parakeet-ultra) by Moondream, a
|
||
post-training of NVIDIA's
|
||
[Parakeet TDT 0.6B v3](https://huggingface.co/nvidia/parakeet-tdt-0.6b-v3), both licensed
|
||
under [Creative Commons Attribution 4.0 International](https://creativecommons.org/licenses/by/4.0/),
|
||
as FluidInference's [Core ML conversion](https://huggingface.co/FluidInference/parakeet-ultra-coreml)
|
||
(revision `95eaa59`: 8-bit integer encoder weights, decoder and joint re-exported), used
|
||
unchanged.
|
||
- **RLA-ES:** Spanish words are checked against RLA-ES, the Spanish spelling dictionaries
|
||
for LibreOffice by Santiago Bosio and contributors (Spain and Mexico), used under the
|
||
[Mozilla Public License 1.1](https://www.mozilla.org/en-US/MPL/1.1/)
|
||
(<https://github.com/sbosio/rla-es>).
|
||
|
||
Polish's three entries show while the keyboard bundles `pl.hextm`
|
||
(`AcknowledgementsView.polishEntries`):
|
||
|
||
- **National Corpus of Polish n-grams:** Polish word and word-pair frequencies come from
|
||
the n-grams of NKJP's balanced subcorpus (Institute of Computer Science, Polish Academy
|
||
of Sciences; <http://zil.ipipan.waw.pl/NKJPNGrams>), made available under a
|
||
[Creative Commons Attribution](https://creativecommons.org/licenses/by/) licence (the page
|
||
names no version).
|
||
- **ParlaMint:** which Polish spellings are missing their marks, and which words are names, is
|
||
checked against the Polish part of ParlaMint 5.0 (Erjavec et al., CLARIN.SI, <http://hdl.handle.net/11356/2004>),
|
||
licensed under [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/).
|
||
- **PoliMorf:** Polish words are checked against PoliMorf 2018-07-23
|
||
(<http://hdl.handle.net/11321/577>), under the 2-clause BSD licence whose notice the app
|
||
reproduces in full: portions copyright © 2011 Marek Futrega, Piotr Gackiewicz, Mirosław
|
||
Prywata, Włodzimierz Macewicz, Marcin Miłkowski; © 2007–2018 Zygmunt Saloni, Włodzimierz
|
||
Gruszczyński, Marcin Woliński, Robert Wołosz, Danuta Skowrońska; © 2011–2018 Institute of
|
||
Computer Science, Polish Academy of Sciences.
|
||
|
||
## Build & Test
|
||
|
||
Prerequisites: Xcode 26+ and [XcodeGen](https://github.com/yonaskolb/XcodeGen) 2.46.0+.
|
||
|
||
```sh
|
||
cd ios
|
||
scripts/fetch-cleanup-model.sh # the cleanup tagger's Core ML model, kept out of git
|
||
xcodegen generate
|
||
xcodebuild test -project HEXMobile.xcodeproj -scheme HEXMobile -destination "id=<SIM_UDID>"
|
||
```
|
||
|
||
The emoji index files are generated and committed: after changing
|
||
`scripts/emoji-aliases-en.txt` or the sources, run `scripts/build-emoji-data.py`
|
||
(it downloads the pinned Unicode and CLDR files, or reads them from `--sources`).
|
||
|
||
The cleanup tagger's model (`HEXKeyboard/CleanupTagger/HexCleanupTagger.mlpackage`,
|
||
25 MB) is a GitHub release asset (`ios-model-cleanup-tagger-1`), not a git file.
|
||
`fetch-cleanup-model.sh` downloads it and checks its SHA-256; Xcode Cloud runs it
|
||
from `ci_scripts/ci_post_clone.sh`. A new model gets a new release tag and hash.
|
||
|
||
### Replaying real typing
|
||
|
||
`TypingReplayTests` runs recordings of real fingers through the keyboard as it runs: the
|
||
key plane, Swipe typing or taps with Spatial autocorrect, then the engine's spacing. It is
|
||
opt-in, one test per mode, and the datasets below are not vendored, so convert a recording
|
||
to one of the two JSON shapes first. `xcodebuild` passes environment variables to the test
|
||
runner only when they are prefixed with `TEST_RUNNER_`, and strips the prefix, so
|
||
`TEST_RUNNER_HEX_SWIPE_REPLAY` becomes `HEX_SWIPE_REPLAY`:
|
||
|
||
```sh
|
||
TEST_RUNNER_HEX_SWIPE_REPLAY=/path/swipes.json xcodebuild test \
|
||
-project HEXMobile.xcodeproj -scheme HEXMobile -destination "id=<SIM_UDID>" \
|
||
-only-testing:HEXMobileTests/TypingReplayTests
|
||
```
|
||
|
||
Swipe recordings are `{"sentences":[{"source":String,"session":String,"words":[String],
|
||
"gestures":[{"x":[Double],"y":[Double],"t":[Double],"start":Double?}]}]}`: one gesture per
|
||
word, x 0…1 across the ten columns of the top letter row, from q's left edge to p's right
|
||
edge (their centres at 0.05 and 0.95), y 0…1 from the top of the first letter row to the
|
||
bottom of the third (row centres at 1/6, 1/2 and 5/6), t milliseconds since the finger
|
||
landed, and the optional `start` milliseconds from the sentence's first landing to that
|
||
one, so gestures keep the person's gaps; without it each begins 0.5 s after the previous
|
||
lift, longer than the two windows the replay's clock drives that a lift leaves open (the
|
||
double-space period and the strip's touch guard). A start that would put a finger down before
|
||
the previous one came up is moved to that lift, in both modes, and `clamped` counts those. A
|
||
gesture recorded as one point is a tap. Hex's second and third letter rows are offset half a
|
||
column and one and a half columns, as the format assumes. Tap recordings replace `gestures`
|
||
with `taps`, one `{x, y, t}` per touch, each array holding that touch's samples (a tap is
|
||
usually one) on the sentence's clock; y > 1 is the space bar, x 0.25…0.75 across it.
|
||
`TEST_RUNNER_HEX_TAP_REPLAY=/path/taps.json` does the same for taps, with Spatial
|
||
autocorrect reading each word on its space.
|
||
|
||
Both report a line per source inside `TYPING-REPLAY … TYPING-REPLAY-END`: total words,
|
||
words outside the dictionary and their share (a word is in the dictionary when the decoder's
|
||
model has it or English's contractions in `AutocorrectEngine.englishContractions` do; the
|
||
model lookup gets only the word's ASCII a–z letters and apostrophes, so accented, hyphenated
|
||
and numeric tokens normally count as outside), and accuracy among the in-dictionary words.
|
||
Scoring lowercases, straightens curly `’`, trims leading and trailing apostrophes, and keeps
|
||
ASCII letters: a tap compares letters, and an apostrophe only when the recorded word has one;
|
||
a swipe compares letters only. Taps report that as `accuracy` over every in-dictionary word and
|
||
also `all-words-accuracy` over every word, so the gap between the two is what the
|
||
out-of-dictionary words contribute. Swipes report `accuracy` and `typed-or-offered` over the
|
||
swipe attempts (`attempts`): gestures that became swipes, ones whose path left the key by a
|
||
swipe's pitch and weren't promoted, and ones that read nothing. They also report
|
||
`in-dictionary-accuracy` over every in-dictionary word — taps included, a skipped gesture
|
||
counted wrong — and `all-words-accuracy` over every word, which is the end-to-end number:
|
||
what the person reading the field saw, with an out-of-dictionary word now able to count as
|
||
right when the field has it. `taps` and `taps-right` are the recordings whose path never
|
||
leaves its key by a swipe's pitch: the keyboard types that key's letter, and the field keeps
|
||
it, as for a person typing a one-letter word. Tap mode keeps Swipe typing attached as the
|
||
keyboard runs it, so a recorded tap whose samples drift can be promoted to a swipe;
|
||
`read-as-swipes` counts the recorded taps the plane took that way. `not-swipes` counts swipe
|
||
attempts the plane never took as swipes or that read nothing; `skipped` counts gestures that
|
||
began off the letter keys, and `clamped` the touches moved forward to the previous lift;
|
||
unlike `taps` and `taps-right`, which count in-dictionary words, those three count every
|
||
recorded touch, and `read-as-swipes` also counts every delivered tap, whatever its word. A
|
||
tap's words compare an apostrophe only when the recorded word has one; a swipe's by letters
|
||
only, because a swipe's path runs over letter keys and cannot draw one. The decoder's own
|
||
median and p90 milliseconds close the line, though a swipe's preview decode can overlap its
|
||
timed final decode, so a `decode-ms` sample may include that overlap; the words are
|
||
unaffected. `HEX_REPLAY_LIMIT` caps sentences per source; `HEX_REPLAY_OUT` writes each
|
||
sentence's intended and typed words (prefix both with `TEST_RUNNER_` too).
|
||
|
||
The test with no recording checks Hex's key geometry against the grid and runs clean
|
||
gestures through the same mapping to type the right words; it cannot catch a converter that
|
||
emits wrong coordinates.
|
||
|
||
Numbers can move between simulator iOS versions because autocorrect also consults the system
|
||
text checker for word validity and names, and its dictionary changes with iOS.
|
||
|
||
Public datasets of real typing it is meant for:
|
||
|
||
- [FUTO swipe dataset](https://huggingface.co/datasets/futo-org/swipe.futo.org) (MIT).
|
||
- [How We Swipe](https://osf.io/sj67f) (Leiva et al., MobileHCI 2021; MIT).
|
||
- [Noisy Typing on QWERTY Keyboards](https://zenodo.org/records/24225302) (Vertanen and
|
||
Kristensson, IUI 2023; CC-BY-4.0).
|
||
- [Tap Typing with Touch Sensing Images](https://github.com/google-research-datasets/tap-typing-with-touch-sensing-images)
|
||
(CC-BY-4.0).
|
||
|
||
See [`benchmark/typing-replay/README.md`](benchmark/typing-replay/README.md) for each
|
||
dataset's download and its `convert.py` command.
|
||
|
||
## Simulator vs. Physical iPhone Proof Limits
|
||
|
||
- **Verified in Simulator (`HEXMobileTests` + `--snap-state` scenes):** Compilation
|
||
across `HEXMobile`, `HEXKeyboard`, and `HEXWidgets`; App Group command/snapshot
|
||
framing; the `DictationController` generation/expiry race and 30s post-close
|
||
transcript delivery; 10s handoff window; interruption handling; meter recovery after injected startup and mid-capture
|
||
recorder unavailability (including keyboard levels and heartbeat); deterministic
|
||
style formatting; UI layouts; and first-run onboarding's launch decision (a
|
||
fresh install shows it, a saved page resumes, a ready keyboard and
|
||
microphone skip it).
|
||
- **Requires a physical iPhone:** Real `AVAudioSession` background keepalive
|
||
survival across lock/app switches, live orange indicator behavior, on-device
|
||
`SpeechTranscriber` / Neural Engine Parakeet latency, Dynamic Island
|
||
Live Activity interaction, and hardware Action Button invocation. Transient
|
||
`AVAudioRecorder.isRecording` changes during warm handoff or Bluetooth route
|
||
changes have not been reproduced on hardware; the meter recovery tests inject
|
||
that condition through `ActiveAudioCapture`.
|
||
- **Screen recordings with the microphone:** The simulator can't screen-record,
|
||
so whether one picks up noise while Hex records shows only on an iPhone:
|
||
record with the microphone on with Hex closed, Hex open and idle, a warm
|
||
session on, and during in-keyboard dictation (WebKit's `videoChat` session,
|
||
which Hex can't change), and compare.
|
||
- **Show what I say and Type as I speak:** `LiveDictationTests` cover how results become the
|
||
words shown, the switches' defaults, the end of a long dictation, and Type as I speak's
|
||
typing, revising, taking back, stopping when the field changes or an app reports no text, and
|
||
private fields. The simulator has no speech models, so there the preview and typing ran on a
|
||
replay of Apple's results recorded on a Mac (`LiveTranscriptReplay`), in a host app, light and
|
||
dark, upright and turned, on an iPhone SE's globe key and with no model at all (no preview).
|
||
On an iPhone 17 Pro (iOS 27.2): the preview, Type as I speak, a 60 s dictation, memory with
|
||
and without, and the keyboard hidden mid-dictation with Type as I speak (Home, then back: the
|
||
held text pasted over its words). Unchecked: VoiceOver on a device (the test runner there has
|
||
no VoiceOver control), hosts that lag or show part of the text (web views, Notes) and real
|
||
speech in a room rather than a clip from a speaker.
|
||
- **In-keyboard dictation's start time:** The simulator's WebKit captures from a
|
||
mock microphone, and on a Mac without a microphone the warm session's
|
||
recorder can't start, so how long a real microphone takes to open
|
||
(getUserMedia, the audio session, the recorder) shows only on an iPhone, in
|
||
the probe log's `first audio` and `warm start` lines.
|
||
- **Apple Speech hint:** Custom vocabulary's contextual strings are best effort
|
||
and unverified on a physical iPhone. Apple documents contextual strings only
|
||
for `DictationTranscriber`; Hex sets them on both modules, and Parakeet's
|
||
batch path gets no hint. On macOS 27.2.1 `SpeechTranscriber`'s text didn't
|
||
change with them (0 of 126 clips of names and terms), and
|
||
`DictationTranscriber` wrote the listed spelling in 28 of 132 places against
|
||
4 without them, so for English on a recent iPhone `SoundAlikeSpelling` is
|
||
what writes a name the listed way.
|
||
- **Dictation languages:** `SpeechLanguagePickerTests` cover the picking rule
|
||
and `DictationControllerTests` a language change during a dictation, Automatic
|
||
following the keyboard's language, and Spanish never reaching Parakeet. The
|
||
simulator can't transcribe: `SpeechTranscriber.isAvailable` is false there
|
||
and `DictationTranscriber` fails with "Model not available", so
|
||
`AppleSpeechClipTests` (skipped unless `HEX_SPEECH_CLIPS` names a folder of
|
||
`<language>.wav` clips) reports that. The same code (`AppleSpeechTranscriber`,
|
||
`AppleSpeechModule`, the picker) built for macOS 26 transcribed `say` clips
|
||
word for word: German and Spanish through `SpeechTranscriber`, Norwegian
|
||
(`Nora`) and Hebrew (`Carmit`) through `DictationTranscriber`, which
|
||
downloaded each model in 11–16 s. `DictationTranscriber` returned Norwegian
|
||
and Hebrew without punctuation under every preset (the short-form presets
|
||
returned nothing), so Polished only capitalizes and ends the sentence.
|
||
Ukrainian is `DictationTranscriber`'s only (macOS 27.2 lists `uk-UA` there and
|
||
not for `SpeechTranscriber`): a `say -v Lesya` clip came back word for word in
|
||
Cyrillic, without punctuation, after an 18 s model download.
|
||
Polish is too (macOS 27.2.1 lists `pl-PL` for `DictationTranscriber` only):
|
||
`say -v Zosia` clips came back with every diacritic (ą ć ę ł ń ó ś ź ż), numbers as
|
||
digits ("10 min", "19:30"), and no punctuation.
|
||
Czech is too (macOS 27.2.1 lists `cs-CZ` for `DictationTranscriber` only, a 12 s model
|
||
download): `say -v Zuzana` clips came back with every háček and čárka ("Příliš žluťoučký kůň
|
||
úpěl"), numbers and times as digits ("10 min.", "19:30", "8:15") and no punctuation. It
|
||
never goes to Parakeet, which is English-only.
|
||
Greek is too (macOS 27.2 lists `el-GR` for `DictationTranscriber` only): `say -v Melina`
|
||
clips came back with every tonos, final ς, numbers as digits ("10 λεπτά", "7:30"), and no
|
||
punctuation. Lowercase lowers with Unicode's context-aware mapping, so a word-final Σ
|
||
becomes ς ("ΟΔΟΣ" → "οδος"), as desktop's `to_lowercase` does.
|
||
**Spanish variants** have unit tests only: a region Apple has keeps it (es-CL on a
|
||
Chilean phone), es-419 and es-CO resolve to es-US then es-MX, before iOS 26
|
||
`SFSpeechRecognizer` keeps es-CO and es-419 as themselves and falls to es-US, then
|
||
es-MX, then Spain's for an unlisted region, plain `es` by the iPhone's Region,
|
||
a picked variant survives a swipe away and back, the switch offers one variant per
|
||
language with Chinese's variants kept apart, and the Hex key menu names the region.
|
||
No iPhone or simulator ran the Spanish variants for this change, and the Settings
|
||
line and menu labels weren't captured.
|
||
Unverified on a physical iPhone.
|
||
- **Speech model fallback:** `SpeechModelInstallTests` and `StallWatchTests` run
|
||
the fallback, set-aside, offline, stall and keyboard-handoff rules against fakes
|
||
of Apple Speech's calls (the simulator has no speech models),
|
||
`SpeechLanguagePickerTests` the set-aside and no-`SpeechTranscriber` picks, and `--snap-state
|
||
full-keyboard-language-unavailable`, `keyboard-language-unavailable`,
|
||
`language-unavailable` and `language-switched` render the messages. Which
|
||
module fails for which language on iOS 26.x, whether a stalled Apple download
|
||
ever moves again, the reservation limit on a full iPhone, and Parakeet's
|
||
memory on 4 GB iPhones are unverified on hardware.
|
||
- **Keyboard languages:** `KeyboardLayoutTests` cover the list (resolution, switching,
|
||
space-bar label, pairing, migration, the dictation pick, English & Spanish's keys,
|
||
dictation following the keyboard, Spanish's Americas variants and the iPhone's Region),
|
||
`HangulComposerTests` the two-set composer and the keyboard typing through it,
|
||
`FullKeyboardEngineTests` the swipe against the
|
||
hold-then-slide trackpad and switching in a `FullKeyboardView` (English & Spanish too,
|
||
and a picked Spanish variant surviving the swipe away and back), and
|
||
`AutocorrectEngineTests` bilingual autocorrect on stub dictionaries and iOS's own
|
||
(`testABilingualKeyboardKeepsBothLanguagesWords` prints the break counts). Hungarian's
|
||
QWERTZ rows and held vowels, typing through them in a `FullKeyboardView`, and its accent
|
||
fixes are unit tests too (`testHungariansRowsAndHoldMatchItsLayout`,
|
||
`testHungarianTypesAndSwitches`, `testHungarianRestoresMissingDiacritics`); no
|
||
simulator or iPhone typed on its keys for this change. Checked live in
|
||
Messages in the simulator (English, Français, Русский). Memory (Release, iPhone 17 Pro
|
||
simulator, iOS 27.2, Full Access, three runs, `footprint` after typing a sentence): Français
|
||
alone 28.7–28.8 MB, with English beside it 29.7–29.9 MB; English alone 28.9–29.9 MB, with
|
||
Français 29.5–29.9 MB; resting 18.7–19.1 MB and peaks 37–41 MB either way, so the second
|
||
language costs about 1 MB (the simulator counts no GPU memory). A real iPhone, the strip's
|
||
latency with two Spatial models (English and Spanish), and whether a keyboard without Full
|
||
Access keeps the current language are unverified.
|
||
- **Keyboard layouts:** `KeyboardLayoutTests` check every language's rows,
|
||
widths, hit frames (tiling the key area and row 1's band above it with no gaps
|
||
or overlaps), balloons
|
||
and accent menus at 320–440 pt, in landscape and in the iPhone Duo's postures, and the
|
||
iPad's (`testPad…`: each size docked either way up against Apple's rows, floating, the
|
||
hide-keyboard key, targets); `AutocorrectEngineTests` and
|
||
`FullKeyboardEngineTests` check typing, dictionaries and English-only fixes.
|
||
Whether the simulator or an iPhone has a `UITextChecker` dictionary for
|
||
`nb` decides Norwegian autocorrect; how good those corrections are is
|
||
unmeasured.
|
||
- **Learn how I type:** Dictation edit inference (span location, word diff,
|
||
outcomes, the learning threshold, and applying fixes) is covered by unit
|
||
tests only. How much `documentContextBeforeInput` and
|
||
`documentContextAfterInput` each host app shows, and whether it calls
|
||
`textDidChange` after the keyboard's own edits, varies by app and is
|
||
unverified on a physical iPhone; when the context is cut short or missing,
|
||
the dictation is settled on where it was last seen or not learned from.
|
||
- **Adaptive touch:** `FullKeyboardEngineTests` and `AutocorrectEngineTests` cover what
|
||
teaches it (words that stood, picks, undone fixes, retypes; never autocorrections,
|
||
swipes, claimed letters, strip touches or sensitive fields), contexts, the evidence
|
||
floor, the grip fallback, the file (size, backup exclusion, damage, reset) and the guard
|
||
(no new rewrite, no new join, names and your words kept). The autocorrect benchmark's
|
||
held-out sentences (Tatoeba text, synthetic taps; iPhone 17 simulator, iOS 27.2) with the
|
||
Experiment off, or on without evidence, match the keyboard without it item for item
|
||
(2,000 of 2,000 outputs and strip hits, English and Spanish). With a profile learned from
|
||
the 300 dev sentences, English WER goes from 5.24% to 5.16% (16 words fixed, 5 broken,
|
||
none typed right) and Spanish from 8.34% to 8.26% (12 fixed, 3 broken); the strip's hit
|
||
after two letters rises 0.8 and 0.3 points. With the personal centres deciding alone it
|
||
would be 5.04% and 8.00%: the guard holds back most of the gain. Learn how I type's
|
||
journal alone does better (5.00% and 8.02% in the same replay, offline): with both on,
|
||
5.04% and 8.04%. In an offline replay of a grip change (learned on taps 6 pt right,
|
||
typed 6 pt left), the profile without the fallback broke 38 English words stock got
|
||
right; the fallback stops at the stock centres and breaks none, and then learns the new
|
||
grip. Per-letter engine p95 changed by -0.09 to +0.04 ms against the Experiment off
|
||
(simulator noise), peak footprint by at most +0.25 MB. Real iPhones, real grips and
|
||
hands, host apps' text reports, Full Access off on a device (whether the switch is
|
||
readable, the extension's own container), and how often people retype or undo are
|
||
unverified.
|
||
- **Record typing logs:** `TypingLogTests` cover the line format, day files,
|
||
the cap, retention, the skip rule, and a real `FullKeyboardView` typing
|
||
through the logger, whose edits replay to the field's text. Touches there are
|
||
synthetic samples fed to the same path `UITouch`es take; real touch radii,
|
||
sampling rates, how often hosts call `textDidChange`, and the extension's
|
||
write cost on an iPhone are unmeasured.
|
||
- **Clean up what I say:** `CleanupTaggerTests` check, in the simulator
|
||
(CPU only), the tokenizer against Hugging Face on 1,891 words,
|
||
reconstruction against `reconstruct.js` on 670 cases, and the model on the
|
||
137 hand-written sentences at 0.98. Its labels match the fp32 checkpoint's
|
||
on all 1,128 words, and 136 of 137 outputs match the ONNX int8 pipeline.
|
||
The other sentence differs on two words within 0.01 of the threshold,
|
||
which ONNX's int8 activations keep and the checkpoint deletes. In the
|
||
simulator the model loads in about 150 ms, cleans a sentence in about
|
||
3 ms (p95 under 8 ms), and adds about 3 MB to the process footprint,
|
||
because the CPU leaves the int8 weights memory-mapped from the bundle. The
|
||
simulator's GPU path took 0.8 s to load and added 30 MB, so Hex doesn't use
|
||
the GPU. Neural Engine load time, latency, and the keyboard extension's
|
||
footprint next to Apple Speech are unmeasured on an iPhone;
|
||
`keyboard-capture-probe.log` records the footprint around each in-keyboard
|
||
dictation.
|
||
- **In-keyboard dictation memory:** Measured only in the simulator (Release,
|
||
iPhone 13, iOS 27.2, which enforces no limit and leaves out GPU memory; three
|
||
runs each, the same before and after the full keyboard became the default):
|
||
the full keyboard rests at 18.4–18.6 MB with Full Access and 18.0–18.3 MB
|
||
without, and its first dictation reaches 30.8–31.8 MB within 3 s of the tap
|
||
(peak 32.1–33.0 MB, so about 13 MB); the dictation bar rests at 20.7–20.8 MB
|
||
and its first dictation reaches 27.3–27.9 MB (about 7 MB). Later dictations
|
||
add 1–5 MB. The 32 and 24 MB thresholds rest on that plus the Metal scope's
|
||
GPU memory, not on an iPhone's; `keyboard-capture-probe.log` records each
|
||
step's footprint, peak and headroom, and keeps only its newest 256 KB once it
|
||
passes 512 KB.
|
||
- **Without Full Access:** Verified in the simulator with the keyboard in a
|
||
real hosted field (`--snap-state hosted-keyboard`) and Full Access off: a
|
||
fresh install mounts the full keyboard and offers no in-keyboard dictation;
|
||
keys typed through the engine autocorrect ("thw" became "the", "tk" "to",
|
||
and with Spatial autocorrect, on by default, "ofthw" "of the"); the Hex key shows the
|
||
hint. Scenes: `full-keyboard-no-access[-hint][-dark]`; `--keyboard
|
||
full|dictation-bar` picks the keyboard. The simulator's keyboard reads the App
|
||
Group without Full Access; whether an iPhone's does is unverified, which is
|
||
why the keyboard mounts the full keyboard without Full Access whatever was
|
||
chosen. Full Access in a simulator: a `kTCCServiceKeyboardNetwork` row for
|
||
the containing app (`anomaly.hex.mobile`, auth 2) in the shut-down
|
||
simulator's `data/Library/TCC/TCC.db`. Unverified on a physical iPhone.
|
||
- **First run:** Reading Full Access from the keyboard on page 3, the
|
||
keyboard-switch signal behind "Full Access looks off"
|
||
(`UITextInputMode.currentInputModeDidChangeNotification`), and Settings
|
||
opening on Hex's page are unverified on a physical iPhone. Before iOS 26
|
||
without a warm session, dictating on page 3 takes the cold-start path, whose
|
||
handoff screen replaces the page.
|
||
- **Emoji:** `EmojiSuggesterTests` cover the index, ranking, search, the strip and
|
||
the history, and score 100 sentences
|
||
(`HEXMobileTests/Fixtures/emoji-context-sentences.json`) for a sensible emoji in
|
||
the top five: 30 hand-written to tune (30/30, top-1 27), 30 hand-written and held
|
||
out (30/30, top-1 27; written by the same hand as the chat words, so optimistic),
|
||
and 40 written by a separate agent that saw no code or word lists (37/40, top-1 28
|
||
before any change; 38/40, top-1 29, now). CLDR's words alone scored 26, 22 and 30.
|
||
Apple's on-device word embedding (`EmojiSemanticIndex`, app and tests only) changed
|
||
none of these, cost 5.3 MB that stayed after release and 100–120 ms to build, so the
|
||
keyboard doesn't use it; Apple's sentence embedding scored 18/40. In the keyboard
|
||
extension (Release, iPhone 17 Pro simulator, iOS 27.2, Full Access, a hosted field, no
|
||
GPU memory, no headroom reading, `footprint` resident and peak): drawn as text, each
|
||
emoji's image (about 48 KB) stayed in a cache of the shared emoji font that nothing
|
||
public empties, so browsing every category reached 62.7 MB (peak 67.5) and scrolling
|
||
past every emoji 137.7 MB (peak 142.3), still held once closed. The panel now draws
|
||
its emoji itself (`EmojiGlyphRenderer`): a private copy of the emoji font, from the same
|
||
memory-mapped file, replaced every 32 emoji and once drawing pauses for 0.3 s, and a
|
||
2 MB least-recently-used cache of the bitmaps, freed with the panel and on a memory
|
||
warning. Two runs: typing 30.6–34.6 MB, the panel open 36.4–40.0 MB, every category
|
||
39.4–43.0 MB, every emoji 40.0–42.8 MB (peak 44.3–47.9), 31.9–35.3 MB once closed, and
|
||
an in-keyboard dictation after it no higher. Measured again with the bar's scrubbing
|
||
(one run, same setup): tapping every category peaked at 44.5 MB and scrolling past
|
||
every emoji at 44.4; one slide across five categories peaked at 46.0, three slides
|
||
back and forth across the bar at 47.5 and ten at 48.3, each back to 42–43 MB resident
|
||
after. `EmojiSuggesterTests` check that a cell
|
||
lands where a label drew its emoji and that drawing every emoji stays in the cache's
|
||
limit; `--snap-state full-keyboard-emoji-probe` prints the app's footprint, the
|
||
renderer's counts and each frame's work scrolling every emoji at 40 pt a frame
|
||
(p50 1.6–1.8 ms, p95 2.5–6 ms against 0.6–1.0 ms drawn as text). The panel opens in
|
||
24–29 ms the first time and 15–17 ms after (8–9 ms drawn as text, which reused the
|
||
shared font's images).
|
||
- **No iOS CI:** Custom vocabulary is covered by unit tests only. No CI job
|
||
compiles or tests the iOS targets, so these checks run only in a local
|
||
`xcodebuild test`.
|
||
|
||
## Internal TestFlight releases
|
||
|
||
[Hex Mobile in App Store Connect](https://appstoreconnect.apple.com/apps/6836837368/testflight/ios)
|
||
uses Anomaly's signing team. The GitHub **Release Hex Mobile** workflow checks
|
||
`dev` at noon and 6 p.m. Chicago time, or on manual dispatch, and cuts an
|
||
`ios-beta-*` prerelease when iOS inputs have changed. These tags are separate
|
||
from desktop releases. The marketing version stays fixed across beta builds.
|
||
`MARKETING_VERSION` stays 9.0.0 for TestFlight, so every `dev` build reaches testers
|
||
without a new beta review; it's never submitted to the App Store. App Store releases are
|
||
1.0.x and use their own version. Apple rejects any upload below the highest approved App
|
||
Store version, so TestFlight's version must stay above every App Store release. To build one, run `scripts/marketing-release.sh <version> [commit]`, which pushes
|
||
the tag `ios-beta-appstore-<version>` (e.g. `ios-beta-appstore-1.0.0`) on that `dev` commit:
|
||
it matches the Xcode Cloud start condition, and `ci_scripts/ci_post_clone.sh` stamps that
|
||
version into `project.yml` before generating the project. Once Apple approves a
|
||
version, later uploads must be higher than it. Hex Mobile is iPhone only (`TARGETED_DEVICE_FAMILY` 1).
|
||
An iPad still installs it (as an iPhone app) and runs its keyboard as an iPad keyboard: the full
|
||
keyboard or the dictation bar, as Settings › Keyboard chooses (see **iPad** under the full
|
||
keyboard; until build 117 an iPad always showed the bar). Docked, the bar keeps its top 44 pt clear:
|
||
iPadOS adds its shortcut bar (undo, redo, paste) to a custom keyboard, usually above it but
|
||
in some apps over its top (Notes, iPadOS 26.6), where it covered the bar's top row. A
|
||
floating keyboard (iPhone-wide) has no shortcut bar and no strip. The bar's top row is
|
||
40 pt tall around its 26 pt pills, and every control in it takes taps across the row's
|
||
full height, at least 44 pt wide, with the Hex mark and each tray's close reaching the
|
||
screen's edges.
|
||
|
||
The Xcode Cloud workflow needs a tag start condition with prefix `ios-beta-`,
|
||
container `ios/HEXMobile.xcodeproj`, shared scheme `HEXMobile`, an iOS Archive
|
||
action, and a TestFlight post-action for **Hex Internal**. Cloud owns the build
|
||
number. Verify the run's source tag and the processed build in that group. |