hex/ios/HEXShared/KeyboardParakeetSettings.swift
Ryan Vogel 5467eca7d0
iOS keyboard: a long dictation goes to Apple's engine instead of Parakeet, which holds it all in memory (#2412)
What changed
- KeyboardParakeet.transcribe falls back to Apple's speech recognition (which streams the
  file) when the recording is longer than KeyboardParakeetBudget.longestAudio (90 s), or when
  the room left (os_proc_available_memory) doesn't cover the transcription headroom (24 MB)
  plus 0.5 MB per second of audio (transcribeHeadroom(audioSeconds:)). Each reason is logged
  like the other fallbacks ("parakeet: dictation engine=apple (longer than 90 s)", "... (short
  of memory for N s of audio)"). The person sees nothing different: Apple's text arrives a
  moment later.
- allowsTranscription takes the recording's length.

Why
- FluidAudio reads the whole recording into memory before it transcribes (the 48 kHz file
  as floats, a copy of it and the 16 kHz result, AudioConverter.resampleAudioFile): measured
  in Hex's keyboard at about 0.42 MB a second (7.9 MB for 18.9 s). With the keyboard
  recording at 74-91 MB once the model is loaded, three to four minutes of speech reached
  the iPhone 17 Pro's 177 MB limit, and less on a phone with a lower limit: iOS would end the
  keyboard mid-transcription and the dictation would be lost. The fixed 24 MB check before
  didn't grow with the recording.
- 90 s: 45 MB of audio at 0.5 MB a second, which with the 91 MB recording and the 16 MB
  transcription leaves 25 MB under 177 MB on the phone we measured, a margin the room check
  alone can't promise if memory moves while it runs. Almost every dictation is far shorter.

Verified
- DictationControllerTests: the room for 60 s is 54 MB; on the iPhone 17 Pro's keyboard at
  91 MB, 60 s and 90 s go to Parakeet and 91 s to Apple; with 50 MB left, 40 s goes to
  Parakeet and 60 s to Apple; the cap holds where iOS sets no limit. The existing budget
  cases pass with their lengths. Studio suite and the pull request's head in the PR.
2026-11-22 13:47:38 -05:00

165 lines
8.8 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import Foundation
/// Experiments › Speech model in keyboard, in the App Group so the keyboard reads it. On unless
/// the person turned it off. While it's on, the keyboard's own dictation transcribes with Hex's
/// speech model (Parakeet V2 for English, Parakeet Ultra where Experiments › Multilingual model
/// dictates) on the Neural Engine, without opening Hex, once the files are in the App Group and
/// the keyboard has compiled the model. Whenever that isn't so, or isn't safe, it uses Apple's
/// speech recognition exactly as with the switch off. Hex downloads the model by itself only
/// where the keyboard can use it (`KeyboardParakeetDownload.hold`).
nonisolated struct KeyboardParakeetSettings {
static let enabledKey = "experiments.keyboardParakeet"
/// What the keyboard last said about it (`KeyboardState`), for the app's Experiments row.
static let keyboardStateKey = "experiments.keyboardParakeet.keyboardState"
static let keyboardStateAtKey = "experiments.keyboardParakeet.keyboardStateAt"
/// Whether that state is the multilingual model's (Parakeet Ultra) rather than V2's.
static let keyboardStateMultilingualKey = "experiments.keyboardParakeet.keyboardStateMultilingual"
/// The keyboard's memory limit on this iPhone, as the keyboard measured it.
static let keyboardLimitKey = "experiments.keyboardParakeet.keyboardLimit"
static let keyboardLimitNotification = "anomaly.hex.keyboardParakeet.keyboardLimit"
static let launchArgument = "--experiment-keyboard-parakeet"
/// What the keyboard did with the model, written by the keyboard.
enum KeyboardState: String {
/// Loading it, the first time a minute or less while Core ML compiles it for the
/// Neural Engine. The keyboard dictates with Apple's speech recognition meanwhile.
case preparing
case ready
/// This keyboard can't use it (its memory limit, no Neural Engine, a load that failed).
case unavailable
}
private let defaults: UserDefaults?
init(defaults: UserDefaults? = UserDefaults(suiteName: KeyboardBridge.appGroup)) {
self.defaults = defaults
}
var isEnabled: Bool {
get { defaults?.object(forKey: Self.enabledKey) as? Bool ?? true }
nonmutating set { defaults?.set(newValue, forKey: Self.enabledKey) }
}
/// What the switch shows: on where the iPhone has the memory and the person didn't turn it off.
var isOn: Bool { isEnabled && Self.fits() }
/// Turned on by the person (in this build or when it was off by default), as stored: its
/// download starts at once on any connection.
var isTurnedOnByHand: Bool {
defaults?.object(forKey: Self.enabledKey) as? Bool == true
}
/// Turned off by the person, as stored: an App Group iOS doesn't let Hex read yet (before
/// the first unlock) reads nil, which must not delete a download.
var isTurnedOff: Bool {
defaults?.object(forKey: Self.enabledKey) as? Bool == false
}
var keyboardState: (state: KeyboardState, at: Date)? {
guard let raw = defaults?.string(forKey: Self.keyboardStateKey), let state = KeyboardState(rawValue: raw) else {
return nil
}
let at = defaults?.double(forKey: Self.keyboardStateAtKey) ?? 0
return (state, Date(timeIntervalSince1970: at))
}
func setKeyboardState(_ state: KeyboardState?, multilingual: Bool = false, now: Date = Date()) {
defaults?.set(state?.rawValue, forKey: Self.keyboardStateKey)
defaults?.set(multilingual, forKey: Self.keyboardStateMultilingualKey)
defaults?.set(now.timeIntervalSince1970, forKey: Self.keyboardStateAtKey)
}
/// The keyboard has the multilingual model loaded, so it dictates that model's language
/// itself instead of opening Hex (`KeyboardSelfDictation`).
var keyboardHasMultilingualModel: Bool {
isEnabled && keyboardState?.state == .ready && defaults?.bool(forKey: Self.keyboardStateMultilingualKey) == true
}
/// The keyboard's memory limit (bytes), nil until the keyboard first opened on this iPhone.
var keyboardLimit: Int? {
defaults?.object(forKey: Self.keyboardLimitKey) as? Int
}
/// The keyboard, at each launch: its footprint plus `os_proc_available_memory`. Written, and
/// Hex told, only when it moved by a megabyte or more.
func recordKeyboardLimit(_ limit: Int) {
guard limit > 0 else { return }
if let known = keyboardLimit, abs(known - limit) < KeyboardParakeetBudget.megabyte { return }
defaults?.set(limit, forKey: Self.keyboardLimitKey)
CFNotificationCenterPostNotification(
CFNotificationCenterGetDarwinNotifyCenter(),
CFNotificationName(Self.keyboardLimitNotification as CFString),
nil,
nil,
true
)
}
/// As the app's Parakeet: 6 GB iPhones report about 5.5 GB.
static func fits(physicalMemory: UInt64 = ProcessInfo.processInfo.physicalMemory) -> Bool {
physicalMemory >= MultilingualParakeetSettings.minimumPhysicalMemory
}
/// Simulator scenes pass `--experiment-keyboard-parakeet` to turn the experiment on.
func applyLaunchArguments(_ arguments: [String] = ProcessInfo.processInfo.arguments) {
if arguments.contains(Self.launchArgument) {
isEnabled = true
}
}
}
/// How much memory Parakeet needs in the keyboard, measured in Hex's keyboard on an iPhone 17 Pro
/// on iOS 27.2, whose keyboard limit is 177 MB. The Neural Engine's copy of the model (468 MB for
/// V2, 607 MB for Ultra) is outside the footprint iOS limits; what counts:
/// - Hex's keyboard records at up to 91 MB with the model loaded: over twenty dictations in one
/// keyboard process, recordings started at 74.7–78.0 MB through the ninth and at 87.9–91.4 MB
/// after (56 MB for a first dictation, and with Apple's speech recognition throughout).
/// - V2's first transcription rose 16 MB above the recording; Ultra's rose 9 MB, so both use
/// V2's numbers. The first load, compiling, rose 42 MB in an idle probe keyboard
/// (`~/dev/experiments/2026-11-22-hex-keyboard-asr/native`).
/// - FluidAudio reads the whole recording into memory before it transcribes (the 48 kHz file as
/// floats, a copy of it and the 16 kHz result, `AudioConverter.resampleAudioFile`): 7.9 MB
/// for 18.9 s of audio in Hex's keyboard, about 0.42 MB a second, counted at 0.5.
/// So a keyboard needs a limit of the recording footprint, the transcription's rise and `margin`
/// (131 MB), as much left (`os_proc_available_memory`) as the load's rise plus 16 MB before it
/// loads, and the transcription's rise plus 8 MB plus the recording's own cost before each
/// transcription, which is at most `longestAudio` long. Longer or tighter goes to Apple's speech
/// recognition, which streams the file.
nonisolated enum KeyboardParakeetBudget {
static let megabyte = 1_048_576
static let recordingFootprint = 91 * megabyte
static let margin = 24 * megabyte
static let transcriptionRise = 16 * megabyte
static let loadRise = 42 * megabyte
static var requiredLimit: Int { recordingFootprint + transcriptionRise + margin }
static var loadHeadroom: Int { loadRise + 16 * megabyte }
static var transcribeHeadroom: Int { transcriptionRise + 8 * megabyte }
/// What a second of audio costs while FluidAudio holds the recording.
static let audioCostPerSecond = megabyte / 2
/// The longest recording Parakeet transcribes in the keyboard: 45 MB of audio at
/// `audioCostPerSecond`, which with the 91 MB recording and the 16 MB transcription leaves
/// 25 MB under an iPhone 17 Pro's 177 MB, a margin the room check alone can't promise if
/// memory moves while it runs. Almost every dictation is far shorter.
static let longestAudio: TimeInterval = 90
/// Room left for a transcription of `audioSeconds` of audio.
static func transcribeHeadroom(audioSeconds: Double) -> Int {
transcribeHeadroom + Int(max(0, audioSeconds) * Double(audioCostPerSecond))
}
/// Whether a keyboard with `footprint` and `available` (bytes) may load the model.
/// `available` 0 is a process iOS sets no limit for (the simulator): it may.
static func allowsLoad(footprint: Int, available: Int) -> Bool {
guard available > 0 else { return true }
return footprint + available >= requiredLimit && available >= loadHeadroom
}
/// Whether Parakeet may transcribe `audioSeconds` of audio now: no longer than `longestAudio`,
/// and (where iOS sets a limit) the room it needs left.
static func allowsTranscription(footprint: Int, available: Int, audioSeconds: Double) -> Bool {
guard audioSeconds <= longestAudio else { return false }
guard available > 0 else { return true }
return footprint + available >= requiredLimit && available >= transcribeHeadroom(audioSeconds: audioSeconds)
}
}