hex/ios/HEXShared/KeyboardSurface+ViewModel.swift
Ryan Vogel 130835701d
iOS keyboard: Show what I say and Type as I speak (words while you dictate) (#2410)
* iOS keyboard: live words over the keys while dictating (stage 1, two looks)

What: while the keyboard records (in-keyboard dictation, iOS 26+), the capture's
16-bit chunks also feed Apple's SpeechAnalyzer (SpeechTranscriber with volatile and
fast results, DictationTranscriber's progressive preset for its languages), and the
words so far show over the keys: settled words in the keys' ink, the tentative tail
muted, newest line in view, whole lines at the top. Two looks behind a temporary App
Group key (debug.liveDictationLook): dim (keys at 10%) and hide. The preview takes every
touch on the keys, so nothing types while it shows; the recording bar, its cancel and
done, and the globe (drawn over the words where the keyboard has its own) work as before.
The final text still comes from the recording exactly as today; the live recogniser is
stopped when the capture stops (model lingering for the final pass).

Why: Ryan asked for the dictation to show in real time over a hidden or dimmed keyboard.
LiveTranscriptSource is one small protocol so the source can be swapped:
ParakeetLiveTranscriber (the audio so far again about once a second, a 20 s window) is
written but not wired. Never worse than today: no preview when neither switch wants it,
when less than 24 MB is left, when the language's model isn't on the iPhone (no
download), on an error, or before iOS 26. Hooks in KeyboardLocalDictation stay in the
capture's events and phase, away from the transcription call.

Verified: Studio simulator (iPhone 17 Pro iOS 27.2, SE iOS 26.5), real keyboard in a
host app with results recorded from Apple's recogniser on macOS 27.2.1 replayed (the
simulator has no speech models; LiveTranscriptReplay is simulator-only): both looks,
light and dark, upright and turned, a 60 s dictation scrolling, taps on Q and space
while recording type nothing with the preview (and "Q " without it), the globe switches
keyboards on the SE, no model -> no preview and the keys as today. Measured on macOS
with the same source: first words 0.5-1.0 s after the first word ends, word lag median
0.56-0.76 s, 2.1% of shown words rewritten, +0.4-2.6 MB in the feeding process.
code-shape.py passes. No settings UI yet (stage 2).

* iOS keyboard: Show what I say (hide look) and Type as I speak

What: look (b) only. While the keyboard records by itself the keys go and the words
so far take their place under the recording bar; the dim look and its temporary
debug.liveDictationLook key are gone. Two switches in Settings › Dictation on their
own card under the style card, iOS 26 and later only, Simplified Chinese:
- Show what I say (on by default): the preview.
- Type as I speak (off by default, "Words can change until you stop"):
  LiveFieldTyping types the live words through the text document proxy, retyping only
  the changed tail, and keeps exactly what it typed. It edits only while the text
  before the cursor still ends with that (smart quotes allowed for; a host showing only
  the end must show the part that changes); otherwise it stops and leaves the field,
  and the dictation goes in at the cursor as today. When the dictation's text goes in,
  KeyboardLiveTyping.handOver takes the words out first and returns the field as it was,
  so spacing, Undo dictation (the whole dictation), Learn how I type and typing logs see
  exactly what they did before. Cancel, a failure, nothing heard and the keyboard hiding
  take the words out (a held dictation goes in at the next keyboard as before). Never
  in secure, password, one-time code or card number fields, over a selection or a
  two-thumb Dictate's range, or with Translate what I say on (Ryan's call).
- Never worse than today: live words only with 32 MB left once the capture runs (the
  speech model's 24 MB transcription headroom plus the recogniser's share), stopped under
  24 MB or on a memory warning; the preview never shows before a recogniser is running,
  so a dictation without one never flashes it.
- The hand-over: the preview fades with the words it had while the keys and the
  dictation's own text come in; with Parakeet now English's default the field's words
  can differ slightly from the preview's (Apple's), and never flash in the preview.

Why: Ryan picked look (b) and the two switch names at the checkpoint.

Verified: Studio simulator (iPhone 17 Pro, iOS 27.2) with the replayed recogniser
results: the field typed and revised as the words came ("Quick note: Hey, I am running
about 10 minutes late. Can you grab us a table by...") and back to "Quick note:" after
Cancel; Settings › Dictation in English, dark with both on, and Simplified Chinese.
New tests in DictationLearningTests (LiveDictationTests: accumulator, joining, switch
defaults, typing and revising, stopping when the field changes or reports nothing,
private fields). README describes what the keyboard shows while recording.
code-shape.py passes.

* iOS keyboard: the preview draws only the end of a long dictation

What: LiveDictationPreviewView draws its words with one label in a clipping box, moved up by
whole lines once they overflow it, and only about the box's worth of text and three lines more
(from a word's start). It replaces the scrolling UITextView, so the words can no longer be
dragged back.

Why: on the iPhone 17 Pro a 60 s dictation with the preview held up to 27 MB more in the
keyboard (peak 103 MB, against 60 MB with the recogniser on and the preview off); the Studio
simulator showed the same, +24.5 MB.

Verified: Studio simulator, the 60 s replay: +24.5 MB (text view) -> +17.3 (capped text view)
-> +5.7 MB with the label, against +7.0 MB for the same recording without the preview. Test
for the tail in LiveDictationTests.

* iOS keyboard: the preview's tail keeps a word that starts right at the cut

Verified: LiveDictationTests on the Studio simulator.

* iOS keyboard: Type as I speak's words are replaced by the held text after the keyboard hides

What: when the keyboard hides mid-dictation, KeyboardLiveTyping notes the words it typed
(LiveDictationSettings.leftover, in the App Group) as well as taking them out. When the held
dictation goes in (inserted by itself or by Paste dictation), it replaces those words if the
text before the cursor still ends with them (LiveFieldTyping(resuming:in:)); otherwise it goes in
at the cursor as before. A new dictation clears the note.

Why: on the iPhone 17 Pro, pressing Home mid-dictation left the typed words in the field (an
app gone to the background takes no edits) and, back in the app (a new document identifier),
the strip offered Paste dictation, which would have put the dictation after them.

Verified: LiveDictationTests on the Studio simulator (9 tests).

* iOS README: what the keyboard shows while recording, measured on the iPhone 17 Pro

Adds the phone's numbers (first words about 1.2-1.3 s after speech starts; 60 s dictation 60.1 MB
with the preview against 58.1 MB without, peaks 66.5 against 60.1), the hidden-keyboard case for
Type as I speak, and what is verified where and what isn't (VoiceOver on a device, lagging hosts).
2026-11-22 13:37:37 -05:00

346 lines
14 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import Combine
import SwiftUI
import UIKit
/// The dictation bar's state: its phase, the style and language trays, the message and
/// transcript, and the clock the surface reads. Kept beside the surface view.
@MainActor
final class KeyboardViewModel: ObservableObject {
enum PickerTray: Equatable {
case none
case style
case language
/// More languages…: every other language Apple Speech has, in place of the keys.
case moreLanguages
}
enum StagePhase: Equatable {
case idle
case recording
case transcribing
case failed
}
struct PendingCommand: Equatable {
let kind: KeyboardCommandKind
let jobID: String
let sentAt: TimeInterval
}
static let commandAckTimeout: TimeInterval = 1.6
let clock = KeyboardClockModel()
/// The words so far while the keyboard records with live words on (`LiveDictationPreviewHost`).
let liveText = LiveDictationText()
@Published var hasFullAccess = true
@Published var needsGlobeKey = true
/// The bar's return key for the field's type, as the full keyboard's, and whether it's on.
@Published var returnKey = ReturnKeyStyle(.default)
@Published var returnKeyEnabled = true
@Published var isModelInstalled = false
@Published private(set) var isWarmAvailable = false
@Published private(set) var stagePhase: StagePhase = .idle
@Published private(set) var statusMessage: String?
/// A tap on Dictate or the Hex key records through Hex's warm session. Not while the last
/// dictation shows as failed: the keyboard's own failure keeps its snapshot live for a few
/// seconds, which isn't a session, and the bar already dictates in the keyboard then.
var dictatesThroughHex: Bool {
hasFullAccess && isWarmAvailable && stagePhase != .failed
}
@Published var style: DictationStyle = .polished
@Published var activeTray: PickerTray = .none
/// Settings › Dictation language (nil for Automatic), what Automatic resolves to, and the
/// languages the keyboard's quick switch offers. Empty choices hide the switch.
@Published var dictationLanguage: String?
@Published var automaticLanguage: String?
@Published var languageChoices: [String] = []
/// Every other language Apple Speech has, for More languages… (`DictationLanguageSwitch.more`).
@Published var moreLanguages: [String] = []
/// The language a long press on its chip offered to remove: Remove shows beside it.
@Published var removingLanguage: String?
/// Translate what I say's target (BCP 47) while it is on, else nil.
@Published var translationTarget: String?
/// The last dictation went in as spoken while Translate what I say was on: both keyboards
/// say so for a few seconds, with the fix, which opens Hex's Settings.
@Published var translationNote: TranslationFailure?
/// Translate what I say's target whether it is on or off, for the keyboards' Translate to
/// <language> switch; nil hides the switch (no Apple translation before iOS 26).
@Published var translationLanguage: String?
/// The targets the full keyboard's Hex key menu offers (`DictationTranslationSettings.targetChoices`).
@Published var translationTargetChoices: [String] = []
/// The dictation bar's emoji key: Settings › Keyboard › Emoji, in a field that takes emoji.
@Published var showsEmojiKey = true
/// A finished dictation the keyboard didn't insert by itself (it missed its moment, or
/// the keyboard came back in another field): the full keyboard's strip and the dictation
/// bar's top row offer Paste dictation, which calls `onPasteDictation`.
@Published var offeredDictation: String?
// Non-published live telemetry read directly by `DotScopeHostView` / button tap handlers
// so 25–30 Hz polling never invalidates SwiftUI views mid-spring.
var liveLoudnessDB: Float = -62.0
var currentTimestamp: TimeInterval = Date().timeIntervalSince1970
private(set) var currentSnapshot: KeyboardSnapshot = .offline
var coldStartURL: URL = KeyboardLaunchRequest.startSession(requestedAt: nil).url
private(set) var pendingCommand: PendingCommand?
var onDictateTap: (() -> Void)?
var onLocalDictateTap: (() -> Void)?
/// The finger landed on the full keyboard's Hex key (wall clock), before it is a tap,
/// a hold or a slide.
var onHexKeyPress: ((TimeInterval) -> Void)?
/// That press became a slide (the style menu) or was cancelled before it was a tap or a hold.
var onHexKeyAbandon: (() -> Void)?
var onStopTap: (() -> Void)?
var onCancelTap: (() -> Void)?
var onStyleSelected: ((DictationStyle) -> Void)?
/// A language picked in the keyboard's switch; nil is Automatic.
var onLanguageSelected: ((String?) -> Void)?
/// A language taken off the keyboard's switch (a long press on it, then Remove).
var onLanguageRemoved: ((String) -> Void)?
/// Translate to <language> switched on (true) or off where you dictate.
var onTranslationToggled: ((Bool) -> Void)?
/// A target picked in the Hex key menu; it also turns translation on.
var onTranslationTargetSelected: ((String) -> Void)?
var onSpace: (() -> Void)?
/// The dictation bar's space bar slid as a trackpad: move the cursor by this many characters.
var onMoveCursor: ((Int) -> Void)?
/// A trackpad step or a long press that opened Remove, for the selection haptic.
var onSelectionStep: (() -> Void)?
/// A punctuation key on the dictation bar.
var onInsertText: ((String) -> Void)?
/// The dictation bar's emoji key: the full keyboard's emoji panel opens in its place.
var onEmojiTap: (() -> Void)?
var onDelete: (() -> Void)?
var onReturn: (() -> Void)?
var onOpenURL: ((URL) -> Void)?
var onPasteDictation: (() -> Void)?
/// The dictation bar's light haptic for each press; the keyboard allows it with Full
/// Access and Experiments › Haptics, as the full keyboard's keys.
var onKeyFeedback: (() -> Void)?
weak var inputViewController: UIInputViewController?
/// The language dictation uses now, when known.
var effectiveLanguage: String? {
dictationLanguage ?? automaticLanguage
}
/// Its own short name ("Norsk", "English"), for the switch and the space bar.
var languageLabel: String? {
effectiveLanguage.map(DictationLanguageSwitch.nativeName(for:))
}
var showsLanguageSwitch: Bool {
hasFullAccess && !(languageChoices.isEmpty && moreLanguages.isEmpty)
}
/// "Paste in English" or "Paste in Spanish", when Translate what I say is on and you
/// dictate in another language than its target.
var pasteChoice: TranslationPasteChoice? {
guard hasFullAccess, let translationTarget else { return nil }
return TranslationPasteChoice(target: translationTarget, source: effectiveLanguage)
}
/// The keyboards' Translate to <language> switch can show: Full Access and a target.
var showsTranslationSwitch: Bool {
hasFullAccess && translationLanguage != nil
}
/// Dictations paste translated: Translate what I say is on. The Hex key wears the target's
/// badge ("EN") the whole time.
var isTranslating: Bool {
hasFullAccess && translationTarget != nil
}
/// The target while `isTranslating`, for the badge beside the Hex key ("EN").
var translatingTo: String? {
isTranslating ? translationTarget : nil
}
/// The language label with where dictations go while they paste translated:
/// "Español → English".
var languageSummary: String? {
guard let languageLabel else { return nil }
guard let target = pasteChoice?.target else { return languageLabel }
return "\(languageLabel) → \(DictationLanguageSwitch.nativeName(for: target))"
}
var dotScopePhase: DotScopePhase {
switch stagePhase {
case .idle:
return .idle
case .recording:
return .recording
case .transcribing:
return .processing
case .failed:
return .failed
}
}
func setActiveTray(_ tray: PickerTray, reduceMotion: Bool) {
guard activeTray != tray else { return }
withAnimation(HexMotion.trayAnimation(reduceMotion: reduceMotion)) {
activeTray = tray
removingLanguage = nil
}
}
func setPendingCommand(_ command: PendingCommand?, now: TimeInterval, animated: Bool) {
currentTimestamp = now
pendingCommand = command
reconcileVisualState(
hasFullAccess: hasFullAccess,
snapshot: currentSnapshot,
isModelInstalled: isModelInstalled,
style: style,
now: now,
animated: animated
)
}
func applyBridgeState(
hasFullAccess newFullAccess: Bool,
snapshot newSnapshot: KeyboardSnapshot,
isModelInstalled newModelInstalled: Bool,
style newStyle: DictationStyle,
now: TimeInterval,
clearPending: Bool = false,
animated: Bool
) {
// A job Hex stopped heartbeating for is over (Hex ended, crashed or was suspended
// mid-dictation): without this its last snapshot held every keyboard in recording or
// transcribing for good, with Done and Cancel sent to no one.
// A failure Hex wrote a while ago is over too (`KeyboardSnapshot.hasStaleMessage`).
let newSnapshot = newSnapshot.droppingOrphanedJob(at: now).droppingStaleMessage(at: now)
currentTimestamp = now
currentSnapshot = newSnapshot
liveLoudnessDB = newSnapshot.state == .recording ? newSnapshot.effectiveLoudnessDB : -62.0
if clearPending {
pendingCommand = nil
} else if let pending = pendingCommand {
let expired = now - pending.sentAt >= Self.commandAckTimeout
let acknowledged = (pending.kind == .start && newSnapshot.state == .recording)
|| (pending.kind == .stop && newSnapshot.state != .recording)
|| (pending.kind == .cancel && newSnapshot.state != .recording)
if expired || acknowledged {
pendingCommand = nil
}
}
let recordingElapsed: TimeInterval? = if newSnapshot.state == .recording {
newSnapshot.recordingElapsed(at: now)
} else if let pending = pendingCommand, pending.kind == .start {
max(0, now - pending.sentAt)
} else {
nil
}
let warmRemaining: TimeInterval? = newSnapshot.isAvailable(at: now)
? max(0, newSnapshot.expiresAt - now)
: nil
clock.update(recordingElapsed: recordingElapsed, warmRemaining: warmRemaining)
reconcileVisualState(
hasFullAccess: newFullAccess,
snapshot: newSnapshot,
isModelInstalled: newModelInstalled,
style: newStyle,
now: now,
animated: animated
)
}
/// Convenience for deterministic preview/snapshot fixtures (`KeyboardPreviewHost`).
func configureFixture(
snapshot: KeyboardSnapshot,
now: TimeInterval,
hasFullAccess: Bool = true,
isModelInstalled: Bool = true,
style: DictationStyle = .polished,
activeTray: PickerTray = .none
) {
self.activeTray = activeTray
applyBridgeState(
hasFullAccess: hasFullAccess,
snapshot: snapshot,
isModelInstalled: isModelInstalled,
style: style,
now: now,
clearPending: true,
animated: false
)
}
private func reconcileVisualState(
hasFullAccess newFullAccess: Bool,
snapshot snap: KeyboardSnapshot,
isModelInstalled newModelInstalled: Bool,
style newStyle: DictationStyle,
now: TimeInterval,
animated: Bool
) {
let nextWarm = newFullAccess && snap.isAvailable(at: now)
let nextStage = computeStagePhase(snapshot: snap, now: now, hasFullAccess: newFullAccess)
let trimmedMsg = snap.message?.trimmingCharacters(in: .whitespacesAndNewlines)
let nextMessage = (trimmedMsg?.isEmpty == false) ? trimmedMsg : nil
let changed = hasFullAccess != newFullAccess
|| isWarmAvailable != nextWarm
|| isModelInstalled != newModelInstalled
|| stagePhase != nextStage
|| statusMessage != nextMessage
|| style != newStyle
guard changed else { return }
let commit = {
self.hasFullAccess = newFullAccess
self.isWarmAvailable = nextWarm
self.isModelInstalled = newModelInstalled
self.stagePhase = nextStage
self.statusMessage = nextMessage
self.style = newStyle
}
if animated {
withAnimation(HexMotion.stageAnimation(reduceMotion: UIAccessibility.isReduceMotionEnabled)) {
commit()
}
} else {
commit()
}
}
private func computeStagePhase(
snapshot snap: KeyboardSnapshot,
now: TimeInterval,
hasFullAccess: Bool
) -> StagePhase {
guard hasFullAccess else { return .idle }
if let pending = pendingCommand, now - pending.sentAt < Self.commandAckTimeout {
switch pending.kind {
case .cancel:
return .idle
case .stop where snap.state == .recording:
return .transcribing
case .start where snap.state != .recording:
return .recording
default:
break
}
}
switch snap.state {
case .recording:
return .recording
case .transcribing:
return .transcribing
case .failed:
return .failed
case .offline, .ready:
return .idle
}
}
}