hex/ios/HEXMobile/KeyboardParakeetDownload.swift
Ryan Vogel 7e4ee0b483
iOS: Speech model in keyboard on by default for English, downloaded only where the keyboard can use it (#2390)
* iOS: Speech model in keyboard on by default for English, downloaded only where the keyboard can use it

What changed
- KeyboardParakeetSettings.isEnabled reads on unless the person turned it off (a stored value
  wins, so a switch turned off stays off). The switch shows off and disabled under 6 GB (isOn).
  When Parakeet is used is unchanged: English (V2) or the Multilingual model's language
  (Ultra), 6 GB or more, not with Two-language dictation, Apple's engine whenever the model
  isn't ready or doesn't fit, never the GPU, one chunk at a time.
- Automatic download (KeyboardParakeetDownload.hold): only for English dictation without
  Two-language dictation or the Multilingual model, once the keyboard has measured its own
  memory limit on this iPhone and it is at least KeyboardParakeetBudget.requiredLimit (the
  keyboard writes footprint + os_proc_available_memory to the App Group at each launch and
  posts a Darwin notification when it changed), on a connection that is neither expensive nor
  constrained (NWPathMonitor: not cellular, a personal hotspot or Low Data Mode), with 2 GB
  free. Checked at launch, when Hex comes to the front, when such a connection returns and
  when the keyboard reports its limit; an automatic download stops when the connection
  becomes expensive (its files resume). Turned on by hand (a stored true, also from the
  builds where it was off by default) it still downloads at once on any connection.
- The row: "For English. Uses about 700 MB, downloaded on Wi-Fi. Until then, the keyboard
  uses Apple's speech recognition." (464 MB download plus the keyboard's 234 MB compiled
  copy), and while it waits: "Downloads after you first use the Hex keyboard", "Downloads on
  Wi-Fi", "Downloads once the iPhone has 2 GB free", "Only for English dictation"; the
  keyboard's statuses are unchanged. Simplified Chinese for all; the old detail string is
  replaced. Still under Experiments.
- Moving a legacy Parakeet folder into the App Group forgets that model's compile record
  (ParakeetCompileRecord.forget): Core ML compiles moved files again (18 s for Ultra on an
  iPhone 17 Pro, measured), which the app showed as "Loading the speech model" with a quick
  estimate. Now V2 says it's optimizing for this iPhone with the first-load estimate, and
  Ultra gets ready beside the engine that dictates, as after its download.
- KeyboardParakeetBudget: one set of numbers for V2 and Ultra (Ultra's transcription rose
  9 MB in Hex's keyboard, under V2's 16 MB); required limit 96 MB for both (was 115 MB for
  Ultra, from the probe keyboard).
- The keyboard's log calls a load cold by its time (5 s or more) instead of a per-iOS-version
  record, which called real recompiles after a re-download "warm"; the record is removed.
- README updated.

Why
- Ryan: Parakeet V2 becomes the default for English dictation in the keyboard in the next
  TestFlight build, without anyone downloading 464 MB for a keyboard that can't use it.

Verified
- Studio simulator (iPhone 17 Pro, iOS 27.2, fresh), on origin/dev 962164fc: xcodebuild test,
  857 tests, 0 failures, 3 skipped. scripts/code-shape.py passes.
- iPhone 17 Pro, iOS 27.2, Release build installed over the previous one: the keyboard wrote
  its limit (177.0 MB) at its first open; the row shows the new detail; the first open after
  the update recompiled V2 (18.2 s, logged cold) and that dictation fell back to Apple's
  engine (0.42 s); switch off deleted the files and dictated with Apple (peak 56 MB, no
  Parakeet log lines); switch on by hand downloaded at once (30 s), the keyboard compiled it
  (15.1 s, cold) and a dictation with Hex force-quit ran on V2 (56 MB before, 71.7 MB peak,
  0.32 s for 9.9 s of audio); the row then said "Ready in the keyboard".
- Not yet on a phone: the automatic download from a never-set switch (needs a fresh install).

* iOS: the warm mic's default waits for the keyboard to load the multilingual model itself

What changed
- KeyboardSelfDictation.dictatesItself ended with KeyboardParakeetSettings.isEnabled ||
  !keyboardOpensHex. Written while isEnabled meant "turned on by hand"; with Speech model in
  keyboard on by default it said the keyboard dictates by itself for everyone whose language
  the Multilingual model takes, before the keyboard had that model loaded, so Keep mic warm
  defaulted to Off and hid while Dictate still opened Hex.
- Now the keyboard dictates that language itself only once it has the multilingual model
  loaded: the keyboard records which model its "ready" state is
  (KeyboardParakeetSettings.setKeyboardState(_:multilingual:), keyboardHasMultilingualModel),
  and dictatesItself asks that instead of the switch.

Why
- People whose language the Multilingual model takes keep their warm mic until the keyboard
  really dictates by itself (#2393's intent), with the switch on by default.

Verified
- KeyboardBridgeTests.testKeepMicWarmDefaultFollowsWhetherTheKeyboardDictatesItself gains the
  cases: on by default with only Hex having the model (15 minutes), the keyboard having V2
  ready (15 minutes), loading the multilingual model (15 minutes), having it ready (Off), and
  the switch turned off (15 minutes); its old assertion "only Hex loads the multilingual
  model" contradicts the old line with the switch unset.
- Studio simulator (iPhone 17 Pro, iOS 27.2, fresh) on this head, rebased on origin/dev
  bd8da457: xcodebuild test, 861 tests, 0 failures, 3 skipped. scripts/code-shape.py passes.

* iOS: the keyboard's Parakeet budget counts the recording at 91 MB, so it needs a 131 MB limit

What changed
- KeyboardParakeetBudget.recordingFootprint 56 MB -> 91 MB, so requiredLimit 96 MB -> 131 MB
  (V2 and Ultra alike). Both the keyboard's load and transcription guards and Hex's automatic
  download (KeyboardParakeetDownload.hold, keyboardTooSmall) use it.

Why
- 56 MB is a first dictation's recording. With the model loaded, over twenty dictations in one
  keyboard process on an iPhone 17 Pro, recordings started at 74.7-78.0 MB through the ninth
  and at 87.9-91.4 MB after (with Apple's speech recognition they stayed at 55.8-56.3 MB), so a
  96 MB requirement let in a keyboard whose limit is barely above what it records at. The
  iPhone 17 Pro's 177 MB still passes.

Verified
- DictationControllerTests: requiredLimit is 131 MB; a 100 MB keyboard neither loads the model
  nor gets it downloaded by itself; the 177 MB cases still pass.
- Studio simulator suite on the rebased head (see the PR).
2026-11-22 12:18:29 -05:00

277 lines
11 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import Foundation
import Network
import Observation
import SwiftUI
/// Experiments › Speech model in keyboard: gets Parakeet V2 into the App Group for the keyboard,
/// with FluidAudio's download (each file resumes from its `.partial`), at launch, whenever Hex
/// comes back to the front, on Wi-Fi coming back and once the keyboard measured its memory,
/// until every file is there. By itself only where the keyboard would use it (`hold`);
/// turned on by hand, at once on any connection.
@MainActor
@Observable
final class KeyboardParakeetDownload {
static let shared = KeyboardParakeetDownload()
enum State: Equatable {
case idle
/// The share downloaded.
case downloading(Double)
case failed(String)
}
/// Why it doesn't download by itself yet.
enum Hold: Equatable {
/// Two-language dictation or the Multilingual model takes the keyboard's dictation.
case otherEngine
case notEnglish
/// The keyboard hasn't opened on this iPhone yet, so its memory limit isn't known.
case keyboardNotMeasured
case keyboardTooSmall
/// Cellular, a personal hotspot, Low Data Mode or no connection.
case notOnWiFi
case needsSpace
}
private(set) var state = State.idle
@ObservationIgnored private var task: Task<Void, Never>?
@ObservationIgnored private var network: NWPathMonitor?
/// Whether the connection is one to download 464 MB on, once `network` said.
@ObservationIgnored private var isUnmetered: Bool?
/// What the download weighs: Parakeet V2's four models and vocabulary, about 464 MB.
static let bytes: Int64 = 464_400_000
/// The keyboard's compiled copy in its own Caches, measured on an iPhone 17 Pro.
static let compiledBytes: Int64 = 234_000_000
/// Free space it waits for: the model and the keyboard's copy, with room to spare.
nonisolated static let freeSpaceNeeded: Int64 = 2_000_000_000
/// What it takes on the iPhone, the download and the keyboard's compiled copy, to 10 MB.
static var size: String {
let total = (bytes + compiledBytes + 5_000_000) / 10_000_000 * 10_000_000
return format(total)
}
static func format(_ bytes: Int64) -> String {
ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file)
.replacingOccurrences(of: " ", with: "\u{00A0}")
}
var isDownloaded: Bool { SpeechModelStore.isComplete(ParakeetTranscriber.cacheDirectory) }
/// What holds the download back now, nil once it may start by itself.
var hold: Hold? {
let wanted = DictationController.wantedLanguage()
return Self.hold(
otherEngine: TwoLanguageDictationSettings().isEnabled || MultilingualParakeetSettings().language() != nil,
english: SpeechLanguagePicker.isEnglish(wanted),
keyboardLimit: KeyboardParakeetSettings().keyboardLimit,
unmetered: isUnmetered,
freeBytes: Self.freeBytes()
)
}
/// Only for a keyboard that would use it: English, the keyboard's own limit measured and
/// enough (`KeyboardParakeetBudget`), a connection that isn't cellular, a hotspot or Low
/// Data Mode, and `freeSpaceNeeded` free (unknown space doesn't hold it).
nonisolated static func hold(
otherEngine: Bool,
english: Bool,
keyboardLimit: Int?,
unmetered: Bool?,
freeBytes: Int64?
) -> Hold? {
if otherEngine { return .otherEngine }
guard english else { return .notEnglish }
guard let keyboardLimit else { return .keyboardNotMeasured }
guard keyboardLimit >= KeyboardParakeetBudget.requiredLimit else { return .keyboardTooSmall }
guard unmetered == true else { return .notOnWiFi }
if let freeBytes, freeBytes < freeSpaceNeeded { return .needsSpace }
return nil
}
nonisolated static func freeBytes() -> Int64? {
let home = URL(fileURLWithPath: NSHomeDirectory(), isDirectory: true)
return try? home.resourceValues(forKeys: [.volumeAvailableCapacityForImportantUsageKey])
.volumeAvailableCapacityForImportantUsage
}
func startIfWanted() {
let settings = KeyboardParakeetSettings()
guard settings.isEnabled, KeyboardParakeetSettings.fits(), !isDownloaded, task == nil else {
return
}
if !settings.isTurnedOnByHand {
watch()
guard hold == nil else { return }
}
state = .downloading(0)
task = Task { [weak self] in
do {
try await ParakeetTranscriber.downloadModel { fraction in
Task { @MainActor in KeyboardParakeetDownload.shared.show(fraction) }
}
guard let self, !Task.isCancelled else { return }
self.task = nil
self.state = .idle
} catch {
guard let self, !Task.isCancelled else { return }
print("speech model in keyboard: download stopped: \(error)")
self.task = nil
self.state = .failed(String(localized: "The speech model didn't download. Check your connection and try again."))
}
}
}
private func show(_ fraction: Double) {
guard task != nil, case .downloading(let shown) = state, fraction > shown else { return }
state = .downloading(fraction)
}
/// Stops it and waits until nothing writes to the folder any more.
func stop() async {
let running = task
task = nil
running?.cancel()
await running?.value
state = .idle
}
func retry() {
if case .failed = state { state = .idle }
startIfWanted()
}
/// The connection, and the keyboard's word that it measured its memory, both of which can
/// let a held download start while Hex is open.
private func watch() {
guard network == nil else { return }
let monitor = NWPathMonitor()
monitor.pathUpdateHandler = { path in
let unmetered = path.status == .satisfied && !path.isExpensive && !path.isConstrained
Task { @MainActor in KeyboardParakeetDownload.shared.networkChanged(unmetered) }
}
monitor.start(queue: DispatchQueue.global(qos: .utility))
network = monitor
CFNotificationCenterAddObserver(
CFNotificationCenterGetDarwinNotifyCenter(),
nil,
{ _, _, _, _, _ in
Task { @MainActor in DictationController.shared.startKeyboardParakeetDownloadIfWanted() }
},
KeyboardParakeetSettings.keyboardLimitNotification as CFString,
nil,
.deliverImmediately
)
}
/// Off Wi-Fi, a download Hex started by itself stops (its files resume on Wi-Fi).
private func networkChanged(_ unmetered: Bool) {
guard unmetered != isUnmetered else { return }
isUnmetered = unmetered
if unmetered {
DictationController.shared.startKeyboardParakeetDownloadIfWanted()
} else if task != nil, !KeyboardParakeetSettings().isTurnedOnByHand {
Task { await stop() }
}
}
}
extension DictationController {
/// The switch in Experiments: on downloads Parakeet V2 for the keyboard unless it's there;
/// off stops the download and deletes it, unless Hex itself dictates with Parakeet V2.
func keyboardParakeetChanged() {
let settings = KeyboardParakeetSettings()
settings.setKeyboardState(nil)
if settings.isEnabled {
startKeyboardParakeetDownloadIfWanted()
return
}
let keeps = transcriber is ParakeetTranscriber && (isModelReady || phase == .preparingModel)
Task {
await KeyboardParakeetDownload.shared.stop()
if !keeps {
try? FileManager.default.removeItem(at: ParakeetTranscriber.cacheDirectory)
}
}
}
/// Not while Hex's own Parakeet V2 gets ready: it downloads into the same folder.
func startKeyboardParakeetDownloadIfWanted() {
guard !(transcriber is ParakeetTranscriber && phase == .preparingModel) else { return }
KeyboardParakeetDownload.shared.startIfWanted()
}
}
/// Under the switch: what the download waits for, downloading, ready to get ready in the
/// keyboard, getting ready, ready, or why the keyboard isn't using it, in a few words. The
/// keyboard's word stays only while fresh.
struct KeyboardParakeetStatus: View {
let download: KeyboardParakeetDownload
var body: some View {
TimelineView(.periodic(from: .now, by: 2)) { context in
if let line = line(now: context.date) {
VStack(alignment: .leading, spacing: 8) {
Text(line)
.font(.system(size: 13))
.foregroundStyle(HexTheme.textBase)
.fixedSize(horizontal: false, vertical: true)
.accessibilityIdentifier("keyboard-parakeet-status")
if case .downloading(let fraction) = download.state {
ProgressView(value: fraction)
.tint(HexTheme.textBase)
}
if case .failed = download.state {
Button("Try again") { download.retry() }
.font(.system(size: 13, weight: .medium))
.foregroundStyle(HexTheme.textBase)
.accessibilityIdentifier("keyboard-parakeet-try-again")
}
}
.frame(maxWidth: .infinity, alignment: .leading)
.padding(14)
.background(Color.white.opacity(0.055), in: RoundedRectangle(cornerRadius: 14, style: .continuous))
}
}
}
private func line(now: Date) -> String? {
switch download.state {
case .downloading:
return String(localized: "Downloading the speech model")
case .failed(let message):
return message
case .idle:
break
}
guard download.isDownloaded else { return Self.line(for: download.hold) }
switch KeyboardParakeetSettings().keyboardState {
case (.preparing, let at)? where now.timeIntervalSince(at) < 180:
return String(localized: "Getting ready in the keyboard")
case (.ready, _)?:
return String(localized: "Ready in the keyboard")
case (.unavailable, _)?:
return String(localized: "The keyboard can't use it right now, so it uses Apple's speech recognition.")
default:
return String(localized: "Downloaded. The keyboard gets it ready the next time it opens.")
}
}
private static func line(for hold: KeyboardParakeetDownload.Hold?) -> String? {
switch hold {
case .notEnglish:
String(localized: "Only for English dictation.")
case .keyboardNotMeasured:
String(localized: "Downloads after you first use the Hex keyboard.")
case .keyboardTooSmall:
String(localized: "The keyboard can't use it right now, so it uses Apple's speech recognition.")
case .notOnWiFi:
String(localized: "Downloads on Wi-Fi.")
case .needsSpace:
String(localized: "Downloads once the iPhone has \(KeyboardParakeetDownload.format(KeyboardParakeetDownload.freeSpaceNeeded)) free.")
case .otherEngine, nil:
nil
}
}
}