mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS keyboard: live words over the keys while dictating (stage 1, two looks)
What: while the keyboard records (in-keyboard dictation, iOS 26+), the capture's
16-bit chunks also feed Apple's SpeechAnalyzer (SpeechTranscriber with volatile and
fast results, DictationTranscriber's progressive preset for its languages), and the
words so far show over the keys: settled words in the keys' ink, the tentative tail
muted, newest line in view, whole lines at the top. Two looks behind a temporary App
Group key (debug.liveDictationLook): dim (keys at 10%) and hide. The preview takes every
touch on the keys, so nothing types while it shows; the recording bar, its cancel and
done, and the globe (drawn over the words where the keyboard has its own) work as before.
The final text still comes from the recording exactly as today; the live recogniser is
stopped when the capture stops (model lingering for the final pass).
Why: Ryan asked for the dictation to show in real time over a hidden or dimmed keyboard.
LiveTranscriptSource is one small protocol so the source can be swapped:
ParakeetLiveTranscriber (the audio so far again about once a second, a 20 s window) is
written but not wired. Never worse than today: no preview when neither switch wants it,
when less than 24 MB is left, when the language's model isn't on the iPhone (no
download), on an error, or before iOS 26. Hooks in KeyboardLocalDictation stay in the
capture's events and phase, away from the transcription call.
Verified: Studio simulator (iPhone 17 Pro iOS 27.2, SE iOS 26.5), real keyboard in a
host app with results recorded from Apple's recogniser on macOS 27.2.1 replayed (the
simulator has no speech models; LiveTranscriptReplay is simulator-only): both looks,
light and dark, upright and turned, a 60 s dictation scrolling, taps on Q and space
while recording type nothing with the preview (and "Q " without it), the globe switches
keyboards on the SE, no model -> no preview and the keys as today. Measured on macOS
with the same source: first words 0.5-1.0 s after the first word ends, word lag median
0.56-0.76 s, 2.1% of shown words rewritten, +0.4-2.6 MB in the feeding process.
code-shape.py passes. No settings UI yet (stage 2).
* iOS keyboard: Show what I say (hide look) and Type as I speak
What: look (b) only. While the keyboard records by itself the keys go and the words
so far take their place under the recording bar; the dim look and its temporary
debug.liveDictationLook key are gone. Two switches in Settings › Dictation on their
own card under the style card, iOS 26 and later only, Simplified Chinese:
- Show what I say (on by default): the preview.
- Type as I speak (off by default, "Words can change until you stop"):
LiveFieldTyping types the live words through the text document proxy, retyping only
the changed tail, and keeps exactly what it typed. It edits only while the text
before the cursor still ends with that (smart quotes allowed for; a host showing only
the end must show the part that changes); otherwise it stops and leaves the field,
and the dictation goes in at the cursor as today. When the dictation's text goes in,
KeyboardLiveTyping.handOver takes the words out first and returns the field as it was,
so spacing, Undo dictation (the whole dictation), Learn how I type and typing logs see
exactly what they did before. Cancel, a failure, nothing heard and the keyboard hiding
take the words out (a held dictation goes in at the next keyboard as before). Never
in secure, password, one-time code or card number fields, over a selection or a
two-thumb Dictate's range, or with Translate what I say on (Ryan's call).
- Never worse than today: live words only with 32 MB left once the capture runs (the
speech model's 24 MB transcription headroom plus the recogniser's share), stopped under
24 MB or on a memory warning; the preview never shows before a recogniser is running,
so a dictation without one never flashes it.
- The hand-over: the preview fades with the words it had while the keys and the
dictation's own text come in; with Parakeet now English's default the field's words
can differ slightly from the preview's (Apple's), and never flash in the preview.
Why: Ryan picked look (b) and the two switch names at the checkpoint.
Verified: Studio simulator (iPhone 17 Pro, iOS 27.2) with the replayed recogniser
results: the field typed and revised as the words came ("Quick note: Hey, I am running
about 10 minutes late. Can you grab us a table by...") and back to "Quick note:" after
Cancel; Settings › Dictation in English, dark with both on, and Simplified Chinese.
New tests in DictationLearningTests (LiveDictationTests: accumulator, joining, switch
defaults, typing and revising, stopping when the field changes or reports nothing,
private fields). README describes what the keyboard shows while recording.
code-shape.py passes.
* iOS keyboard: the preview draws only the end of a long dictation
What: LiveDictationPreviewView draws its words with one label in a clipping box, moved up by
whole lines once they overflow it, and only about the box's worth of text and three lines more
(from a word's start). It replaces the scrolling UITextView, so the words can no longer be
dragged back.
Why: on the iPhone 17 Pro a 60 s dictation with the preview held up to 27 MB more in the
keyboard (peak 103 MB, against 60 MB with the recogniser on and the preview off); the Studio
simulator showed the same, +24.5 MB.
Verified: Studio simulator, the 60 s replay: +24.5 MB (text view) -> +17.3 (capped text view)
-> +5.7 MB with the label, against +7.0 MB for the same recording without the preview. Test
for the tail in LiveDictationTests.
* iOS keyboard: the preview's tail keeps a word that starts right at the cut
Verified: LiveDictationTests on the Studio simulator.
* iOS keyboard: Type as I speak's words are replaced by the held text after the keyboard hides
What: when the keyboard hides mid-dictation, KeyboardLiveTyping notes the words it typed
(LiveDictationSettings.leftover, in the App Group) as well as taking them out. When the held
dictation goes in (inserted by itself or by Paste dictation), it replaces those words if the
text before the cursor still ends with them (LiveFieldTyping(resuming:in:)); otherwise it goes in
at the cursor as before. A new dictation clears the note.
Why: on the iPhone 17 Pro, pressing Home mid-dictation left the typed words in the field (an
app gone to the background takes no edits) and, back in the app (a new document identifier),
the strip offered Paste dictation, which would have put the dictation after them.
Verified: LiveDictationTests on the Studio simulator (9 tests).
* iOS README: what the keyboard shows while recording, measured on the iPhone 17 Pro
Adds the phone's numbers (first words about 1.2-1.3 s after speech starts; 60 s dictation 60.1 MB
with the preview against 58.1 MB without, peaks 66.5 against 60.1), the hidden-keyboard case for
Type as I speak, and what is verified where and what isn't (VoiceOver on a device, lagging hosts).
261 lines
11 KiB
Swift
261 lines
11 KiB
Swift
import Foundation
|
||
import os
|
||
import Speech
|
||
|
||
/// One in-keyboard dictation's live words (Settings › Dictation › Show what I say and Type as
|
||
/// I speak):
|
||
/// started with the capture's first audio when a switch wants them, fed every chunk the page
|
||
/// sends, and stopped when the capture stops, keeping its words on screen until the
|
||
/// dictation goes in. It never changes the dictation itself: whenever it can't run (a switch
|
||
/// off, short of memory, the language's model not on the iPhone, an error, or no first words
|
||
/// within `firstWordsPatience`) the keyboard records and inserts exactly as without it.
|
||
@MainActor
|
||
final class KeyboardLiveDictation {
|
||
private let text: LiveDictationText
|
||
private let sampleRate: Double
|
||
private var source: LiveTranscriptSource?
|
||
/// Audio from before the source exists (the language's lookup), replayed into it.
|
||
private var held: [Data] = []
|
||
private var heldBytes = 0
|
||
private var stopped = false
|
||
private var resolving: Task<Void, Never>?
|
||
private var latest: LiveTranscript?
|
||
private var publishScheduled = false
|
||
private let startedAt = Date()
|
||
private var firstAudioAt: Date?
|
||
private var firstWordsAt: Date?
|
||
private var updates = 0
|
||
private var rewrites = 0
|
||
private var shownWords: [Substring] = []
|
||
private let footprintAtStart = KeyboardParakeet.memory().footprint
|
||
private var peakFootprint = 0
|
||
|
||
/// Memory the keyboard must still have once its capture runs: the live recogniser's own
|
||
/// share in the keyboard (about 2 MB measured, its model runs in Apple's speech service)
|
||
/// with margin, and the speech model's transcription after it
|
||
/// (`KeyboardParakeetBudget.transcribeHeadroom`, 24 MB).
|
||
static let headroomNeeded = 32 * 1_048_576
|
||
/// Below this while recording, the recogniser goes and the dictation goes on without it.
|
||
static let floor = 24 * 1_048_576
|
||
/// Audio held while the language is looked up; past it the live words give up.
|
||
static let maximumHeldSeconds: Double = 4
|
||
|
||
private let settings: LiveDictationSettings
|
||
|
||
private init(text: LiveDictationText, sampleRate: Double, settings: LiveDictationSettings) {
|
||
self.text = text
|
||
self.sampleRate = sampleRate
|
||
self.settings = settings
|
||
}
|
||
|
||
/// Starts live words for a capture at `sampleRate`, or returns nil (today's behaviour)
|
||
/// when neither switch wants them or memory is short.
|
||
static func start(sampleRate: Double, text: LiveDictationText?) -> KeyboardLiveDictation? {
|
||
guard let text else { return nil }
|
||
let settings = LiveDictationSettings()
|
||
// Words an earlier keyboard left are another dictation's: never replaced by this one's.
|
||
if settings.leftover != nil { settings.leftover = nil }
|
||
guard settings.wantsLiveWords else { return nil }
|
||
let available = os_proc_available_memory()
|
||
guard available == 0 || available >= headroomNeeded else {
|
||
KeyboardProbeLog.shared.write("live: skipped, short of memory \(KeyboardProbeLog.memory())")
|
||
return nil
|
||
}
|
||
let live = KeyboardLiveDictation(text: text, sampleRate: sampleRate, settings: settings)
|
||
live.resolve()
|
||
return live
|
||
}
|
||
|
||
func append(_ pcm: Data) {
|
||
guard !stopped else { return }
|
||
if firstAudioAt == nil { firstAudioAt = Date() }
|
||
if let source {
|
||
source.append(pcm)
|
||
return
|
||
}
|
||
held.append(pcm)
|
||
heldBytes += pcm.count
|
||
if Double(heldBytes / 2) / sampleRate > Self.maximumHeldSeconds {
|
||
giveUp("no recogniser after \(Int(Self.maximumHeldSeconds)) s of audio")
|
||
}
|
||
}
|
||
|
||
/// The capture stopped: the recogniser goes now, its words stay until the dictation is in.
|
||
func stop() {
|
||
guard !stopped else { return }
|
||
stopped = true
|
||
resolving?.cancel()
|
||
source?.stop()
|
||
source = nil
|
||
held = []
|
||
publish()
|
||
let first = firstWordsAt.map { "\(Int($0.timeIntervalSince(firstAudioAt ?? startedAt) * 1000)) ms after first audio" } ?? "none"
|
||
KeyboardProbeLog.shared.write(
|
||
"live: stopped after \(String(format: "%.1f", Date().timeIntervalSince(startedAt))) s, first words \(first), \(updates) updates, \(rewrites) words rewritten, footprint +\(String(format: "%.1f", Double(max(0, peakFootprint - footprintAtStart)) / 1_048_576)) MB at most \(KeyboardProbeLog.memory())"
|
||
)
|
||
}
|
||
|
||
/// iOS warned the keyboard about memory: the recogniser goes, the dictation goes on.
|
||
func memoryWarning() {
|
||
guard !stopped else { return }
|
||
KeyboardProbeLog.shared.write("live: stopped for a memory warning")
|
||
stop()
|
||
}
|
||
|
||
/// Finds the language the dictation will be transcribed in and starts Apple's recogniser
|
||
/// for it, only if its model is already on the iPhone.
|
||
private func resolve() {
|
||
guard #available(iOS 26.0, *) else {
|
||
giveUp("needs iOS 26")
|
||
return
|
||
}
|
||
let rate = sampleRate
|
||
resolving = Task { [weak self] in
|
||
let settings = LanguageSettings()
|
||
let picked = await AppleSpeechLanguages.resolve(settings: settings)
|
||
var installed = false
|
||
if let picked {
|
||
installed = await AssetInventory.status(forModules: [AppleSpeechModule(picked).module]) == .installed
|
||
}
|
||
guard let self, !self.stopped else { return }
|
||
guard let picked, installed else {
|
||
if let replay = LiveTranscriptReplay.load(log: { KeyboardProbeLog.shared.write($0) }) {
|
||
self.attach(replay)
|
||
return
|
||
}
|
||
self.giveUp(picked.map { "\($0.identifier) model not on this iPhone" } ?? "no language")
|
||
return
|
||
}
|
||
guard let source = AppleLiveTranscriber(
|
||
locale: picked.locale,
|
||
module: picked.module == .dictation ? .dictation : .speech,
|
||
sampleRate: rate,
|
||
log: { KeyboardProbeLog.shared.write($0) }
|
||
) else {
|
||
self.giveUp("no audio format")
|
||
return
|
||
}
|
||
self.attach(source)
|
||
}
|
||
}
|
||
|
||
/// The preview shows from here (empty until the first words), never while the language is
|
||
/// still looked up, so a dictation without live words never flashes it. Type as I speak
|
||
/// never types a dictation Translate what I say turns into another language.
|
||
private func attach(_ source: LiveTranscriptSource) {
|
||
text.begin(
|
||
showsPreview: settings.showsPreview,
|
||
typesIntoField: settings.typesIntoField && !DictationTranslationSettings().isEnabled
|
||
)
|
||
source.onUpdate = { [weak self] transcript in self?.received(transcript) }
|
||
source.onFailure = { [weak self] reason in self?.giveUp(reason) }
|
||
self.source = source
|
||
let audio = held
|
||
held = []
|
||
heldBytes = 0
|
||
audio.forEach { source.append($0) }
|
||
}
|
||
|
||
private func received(_ transcript: LiveTranscript) {
|
||
guard !stopped else { return }
|
||
let available = os_proc_available_memory()
|
||
if available > 0, available < Self.floor {
|
||
KeyboardProbeLog.shared.write("live: stopped, short of memory \(KeyboardProbeLog.memory())")
|
||
stop()
|
||
return
|
||
}
|
||
if firstWordsAt == nil, !transcript.isEmpty { firstWordsAt = Date() }
|
||
latest = transcript
|
||
peakFootprint = max(peakFootprint, KeyboardParakeet.memory().footprint)
|
||
// A recogniser's results come in bursts, a word at a time: the screen takes the last.
|
||
guard !publishScheduled else { return }
|
||
publishScheduled = true
|
||
DispatchQueue.main.async { [weak self] in
|
||
MainActor.assumeIsolated { self?.publish() }
|
||
}
|
||
}
|
||
|
||
private func publish() {
|
||
publishScheduled = false
|
||
guard let latest else { return }
|
||
self.latest = nil
|
||
updates += 1
|
||
let words = latest.text.split(whereSeparator: \.isWhitespace)
|
||
rewrites += Self.changedWords(from: shownWords, to: words)
|
||
shownWords = words
|
||
text.update(latest)
|
||
}
|
||
|
||
/// How many words already on screen `next` changes or takes away.
|
||
nonisolated static func changedWords(from shown: [Substring], to next: [Substring]) -> Int {
|
||
var same = 0
|
||
while same < min(shown.count, next.count), shown[same] == next[same] { same += 1 }
|
||
return shown.count - same
|
||
}
|
||
|
||
private func giveUp(_ reason: String) {
|
||
guard !stopped else { return }
|
||
KeyboardProbeLog.shared.write("live: none for this dictation (\(reason))")
|
||
stop()
|
||
// Without words, the keys come back as they are without the preview.
|
||
if text.transcript?.isEmpty ?? true { text.end() }
|
||
}
|
||
}
|
||
|
||
/// The simulator has no speech models: a recorded run of Apple's recogniser (`live-replay.json`
|
||
/// in the App Group, from `AppleLiveTranscriber` on a Mac) replayed at its own pace from the
|
||
/// capture's first audio, so the preview can be seen in the real keyboard there. Never on an
|
||
/// iPhone.
|
||
@MainActor
|
||
final class LiveTranscriptReplay: LiveTranscriptSource {
|
||
private struct Step: Decodable {
|
||
let t: Double
|
||
let settled: String
|
||
let tentative: String
|
||
}
|
||
|
||
var onUpdate: ((LiveTranscript) -> Void)?
|
||
var onFailure: ((String) -> Void)?
|
||
private let steps: [Step]
|
||
private var task: Task<Void, Never>?
|
||
|
||
private init(steps: [Step]) {
|
||
self.steps = steps
|
||
}
|
||
|
||
static func load(log: (String) -> Void) -> LiveTranscriptReplay? {
|
||
#if targetEnvironment(simulator)
|
||
// The App Group, or the keyboard's own Library where an unsigned build has no group.
|
||
let folders = [
|
||
FileManager.default.containerURL(forSecurityApplicationGroupIdentifier: KeyboardBridge.appGroup),
|
||
FileManager.default.urls(for: .libraryDirectory, in: .userDomainMask).first,
|
||
].compactMap { $0 }
|
||
guard let data = folders.lazy.compactMap({ try? Data(contentsOf: $0.appendingPathComponent("live-replay.json")) }).first,
|
||
let steps = try? JSONDecoder().decode([Step].self, from: data), !steps.isEmpty else { return nil }
|
||
log("live: replaying \(steps.count) recorded results (simulator)")
|
||
return LiveTranscriptReplay(steps: steps)
|
||
#else
|
||
return nil
|
||
#endif
|
||
}
|
||
|
||
func append(_ pcm: Data) {
|
||
guard task == nil else { return }
|
||
let steps = steps
|
||
let start = Date()
|
||
task = Task { [weak self] in
|
||
for step in steps {
|
||
let wait = step.t - Date().timeIntervalSince(start)
|
||
if wait > 0 { try? await Task.sleep(for: .seconds(wait)) }
|
||
guard !Task.isCancelled else { return }
|
||
self?.onUpdate?(LiveTranscript(settled: step.settled, tentative: step.tentative))
|
||
}
|
||
}
|
||
}
|
||
|
||
func stop() {
|
||
task?.cancel()
|
||
task = nil
|
||
onUpdate = nil
|
||
}
|
||
}
|