mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS: Speech model in keyboard on by default for English, downloaded only where the keyboard can use it What changed - KeyboardParakeetSettings.isEnabled reads on unless the person turned it off (a stored value wins, so a switch turned off stays off). The switch shows off and disabled under 6 GB (isOn). When Parakeet is used is unchanged: English (V2) or the Multilingual model's language (Ultra), 6 GB or more, not with Two-language dictation, Apple's engine whenever the model isn't ready or doesn't fit, never the GPU, one chunk at a time. - Automatic download (KeyboardParakeetDownload.hold): only for English dictation without Two-language dictation or the Multilingual model, once the keyboard has measured its own memory limit on this iPhone and it is at least KeyboardParakeetBudget.requiredLimit (the keyboard writes footprint + os_proc_available_memory to the App Group at each launch and posts a Darwin notification when it changed), on a connection that is neither expensive nor constrained (NWPathMonitor: not cellular, a personal hotspot or Low Data Mode), with 2 GB free. Checked at launch, when Hex comes to the front, when such a connection returns and when the keyboard reports its limit; an automatic download stops when the connection becomes expensive (its files resume). Turned on by hand (a stored true, also from the builds where it was off by default) it still downloads at once on any connection. - The row: "For English. Uses about 700 MB, downloaded on Wi-Fi. Until then, the keyboard uses Apple's speech recognition." (464 MB download plus the keyboard's 234 MB compiled copy), and while it waits: "Downloads after you first use the Hex keyboard", "Downloads on Wi-Fi", "Downloads once the iPhone has 2 GB free", "Only for English dictation"; the keyboard's statuses are unchanged. Simplified Chinese for all; the old detail string is replaced. Still under Experiments. - Moving a legacy Parakeet folder into the App Group forgets that model's compile record (ParakeetCompileRecord.forget): Core ML compiles moved files again (18 s for Ultra on an iPhone 17 Pro, measured), which the app showed as "Loading the speech model" with a quick estimate. Now V2 says it's optimizing for this iPhone with the first-load estimate, and Ultra gets ready beside the engine that dictates, as after its download. - KeyboardParakeetBudget: one set of numbers for V2 and Ultra (Ultra's transcription rose 9 MB in Hex's keyboard, under V2's 16 MB); required limit 96 MB for both (was 115 MB for Ultra, from the probe keyboard). - The keyboard's log calls a load cold by its time (5 s or more) instead of a per-iOS-version record, which called real recompiles after a re-download "warm"; the record is removed. - README updated. Why - Ryan: Parakeet V2 becomes the default for English dictation in the keyboard in the next TestFlight build, without anyone downloading 464 MB for a keyboard that can't use it. Verified - Studio simulator (iPhone 17 Pro, iOS 27.2, fresh), on origin/dev962164fc: xcodebuild test, 857 tests, 0 failures, 3 skipped. scripts/code-shape.py passes. - iPhone 17 Pro, iOS 27.2, Release build installed over the previous one: the keyboard wrote its limit (177.0 MB) at its first open; the row shows the new detail; the first open after the update recompiled V2 (18.2 s, logged cold) and that dictation fell back to Apple's engine (0.42 s); switch off deleted the files and dictated with Apple (peak 56 MB, no Parakeet log lines); switch on by hand downloaded at once (30 s), the keyboard compiled it (15.1 s, cold) and a dictation with Hex force-quit ran on V2 (56 MB before, 71.7 MB peak, 0.32 s for 9.9 s of audio); the row then said "Ready in the keyboard". - Not yet on a phone: the automatic download from a never-set switch (needs a fresh install). * iOS: the warm mic's default waits for the keyboard to load the multilingual model itself What changed - KeyboardSelfDictation.dictatesItself ended with KeyboardParakeetSettings.isEnabled || !keyboardOpensHex. Written while isEnabled meant "turned on by hand"; with Speech model in keyboard on by default it said the keyboard dictates by itself for everyone whose language the Multilingual model takes, before the keyboard had that model loaded, so Keep mic warm defaulted to Off and hid while Dictate still opened Hex. - Now the keyboard dictates that language itself only once it has the multilingual model loaded: the keyboard records which model its "ready" state is (KeyboardParakeetSettings.setKeyboardState(_:multilingual:), keyboardHasMultilingualModel), and dictatesItself asks that instead of the switch. Why - People whose language the Multilingual model takes keep their warm mic until the keyboard really dictates by itself (#2393's intent), with the switch on by default. Verified - KeyboardBridgeTests.testKeepMicWarmDefaultFollowsWhetherTheKeyboardDictatesItself gains the cases: on by default with only Hex having the model (15 minutes), the keyboard having V2 ready (15 minutes), loading the multilingual model (15 minutes), having it ready (Off), and the switch turned off (15 minutes); its old assertion "only Hex loads the multilingual model" contradicts the old line with the switch unset. - Studio simulator (iPhone 17 Pro, iOS 27.2, fresh) on this head, rebased on origin/devbd8da457: xcodebuild test, 861 tests, 0 failures, 3 skipped. scripts/code-shape.py passes. * iOS: the keyboard's Parakeet budget counts the recording at 91 MB, so it needs a 131 MB limit What changed - KeyboardParakeetBudget.recordingFootprint 56 MB -> 91 MB, so requiredLimit 96 MB -> 131 MB (V2 and Ultra alike). Both the keyboard's load and transcription guards and Hex's automatic download (KeyboardParakeetDownload.hold, keyboardTooSmall) use it. Why - 56 MB is a first dictation's recording. With the model loaded, over twenty dictations in one keyboard process on an iPhone 17 Pro, recordings started at 74.7-78.0 MB through the ninth and at 87.9-91.4 MB after (with Apple's speech recognition they stayed at 55.8-56.3 MB), so a 96 MB requirement let in a keyboard whose limit is barely above what it records at. The iPhone 17 Pro's 177 MB still passes. Verified - DictationControllerTests: requiredLimit is 131 MB; a 100 MB keyboard neither loads the model nor gets it downloaded by itself; the 177 MB cases still pass. - Studio simulator suite on the rebased head (see the PR).
277 lines
11 KiB
Swift
277 lines
11 KiB
Swift
import Foundation
|
||
import Network
|
||
import Observation
|
||
import SwiftUI
|
||
|
||
/// Experiments › Speech model in keyboard: gets Parakeet V2 into the App Group for the keyboard,
|
||
/// with FluidAudio's download (each file resumes from its `.partial`), at launch, whenever Hex
|
||
/// comes back to the front, on Wi-Fi coming back and once the keyboard measured its memory,
|
||
/// until every file is there. By itself only where the keyboard would use it (`hold`);
|
||
/// turned on by hand, at once on any connection.
|
||
@MainActor
|
||
@Observable
|
||
final class KeyboardParakeetDownload {
|
||
static let shared = KeyboardParakeetDownload()
|
||
|
||
enum State: Equatable {
|
||
case idle
|
||
/// The share downloaded.
|
||
case downloading(Double)
|
||
case failed(String)
|
||
}
|
||
|
||
/// Why it doesn't download by itself yet.
|
||
enum Hold: Equatable {
|
||
/// Two-language dictation or the Multilingual model takes the keyboard's dictation.
|
||
case otherEngine
|
||
case notEnglish
|
||
/// The keyboard hasn't opened on this iPhone yet, so its memory limit isn't known.
|
||
case keyboardNotMeasured
|
||
case keyboardTooSmall
|
||
/// Cellular, a personal hotspot, Low Data Mode or no connection.
|
||
case notOnWiFi
|
||
case needsSpace
|
||
}
|
||
|
||
private(set) var state = State.idle
|
||
@ObservationIgnored private var task: Task<Void, Never>?
|
||
@ObservationIgnored private var network: NWPathMonitor?
|
||
/// Whether the connection is one to download 464 MB on, once `network` said.
|
||
@ObservationIgnored private var isUnmetered: Bool?
|
||
|
||
/// What the download weighs: Parakeet V2's four models and vocabulary, about 464 MB.
|
||
static let bytes: Int64 = 464_400_000
|
||
/// The keyboard's compiled copy in its own Caches, measured on an iPhone 17 Pro.
|
||
static let compiledBytes: Int64 = 234_000_000
|
||
/// Free space it waits for: the model and the keyboard's copy, with room to spare.
|
||
nonisolated static let freeSpaceNeeded: Int64 = 2_000_000_000
|
||
|
||
/// What it takes on the iPhone, the download and the keyboard's compiled copy, to 10 MB.
|
||
static var size: String {
|
||
let total = (bytes + compiledBytes + 5_000_000) / 10_000_000 * 10_000_000
|
||
return format(total)
|
||
}
|
||
|
||
static func format(_ bytes: Int64) -> String {
|
||
ByteCountFormatter.string(fromByteCount: bytes, countStyle: .file)
|
||
.replacingOccurrences(of: " ", with: "\u{00A0}")
|
||
}
|
||
|
||
var isDownloaded: Bool { SpeechModelStore.isComplete(ParakeetTranscriber.cacheDirectory) }
|
||
|
||
/// What holds the download back now, nil once it may start by itself.
|
||
var hold: Hold? {
|
||
let wanted = DictationController.wantedLanguage()
|
||
return Self.hold(
|
||
otherEngine: TwoLanguageDictationSettings().isEnabled || MultilingualParakeetSettings().language() != nil,
|
||
english: SpeechLanguagePicker.isEnglish(wanted),
|
||
keyboardLimit: KeyboardParakeetSettings().keyboardLimit,
|
||
unmetered: isUnmetered,
|
||
freeBytes: Self.freeBytes()
|
||
)
|
||
}
|
||
|
||
/// Only for a keyboard that would use it: English, the keyboard's own limit measured and
|
||
/// enough (`KeyboardParakeetBudget`), a connection that isn't cellular, a hotspot or Low
|
||
/// Data Mode, and `freeSpaceNeeded` free (unknown space doesn't hold it).
|
||
nonisolated static func hold(
|
||
otherEngine: Bool,
|
||
english: Bool,
|
||
keyboardLimit: Int?,
|
||
unmetered: Bool?,
|
||
freeBytes: Int64?
|
||
) -> Hold? {
|
||
if otherEngine { return .otherEngine }
|
||
guard english else { return .notEnglish }
|
||
guard let keyboardLimit else { return .keyboardNotMeasured }
|
||
guard keyboardLimit >= KeyboardParakeetBudget.requiredLimit else { return .keyboardTooSmall }
|
||
guard unmetered == true else { return .notOnWiFi }
|
||
if let freeBytes, freeBytes < freeSpaceNeeded { return .needsSpace }
|
||
return nil
|
||
}
|
||
|
||
nonisolated static func freeBytes() -> Int64? {
|
||
let home = URL(fileURLWithPath: NSHomeDirectory(), isDirectory: true)
|
||
return try? home.resourceValues(forKeys: [.volumeAvailableCapacityForImportantUsageKey])
|
||
.volumeAvailableCapacityForImportantUsage
|
||
}
|
||
|
||
func startIfWanted() {
|
||
let settings = KeyboardParakeetSettings()
|
||
guard settings.isEnabled, KeyboardParakeetSettings.fits(), !isDownloaded, task == nil else {
|
||
return
|
||
}
|
||
if !settings.isTurnedOnByHand {
|
||
watch()
|
||
guard hold == nil else { return }
|
||
}
|
||
state = .downloading(0)
|
||
task = Task { [weak self] in
|
||
do {
|
||
try await ParakeetTranscriber.downloadModel { fraction in
|
||
Task { @MainActor in KeyboardParakeetDownload.shared.show(fraction) }
|
||
}
|
||
guard let self, !Task.isCancelled else { return }
|
||
self.task = nil
|
||
self.state = .idle
|
||
} catch {
|
||
guard let self, !Task.isCancelled else { return }
|
||
print("speech model in keyboard: download stopped: \(error)")
|
||
self.task = nil
|
||
self.state = .failed(String(localized: "The speech model didn't download. Check your connection and try again."))
|
||
}
|
||
}
|
||
}
|
||
|
||
private func show(_ fraction: Double) {
|
||
guard task != nil, case .downloading(let shown) = state, fraction > shown else { return }
|
||
state = .downloading(fraction)
|
||
}
|
||
|
||
/// Stops it and waits until nothing writes to the folder any more.
|
||
func stop() async {
|
||
let running = task
|
||
task = nil
|
||
running?.cancel()
|
||
await running?.value
|
||
state = .idle
|
||
}
|
||
|
||
func retry() {
|
||
if case .failed = state { state = .idle }
|
||
startIfWanted()
|
||
}
|
||
|
||
/// The connection, and the keyboard's word that it measured its memory, both of which can
|
||
/// let a held download start while Hex is open.
|
||
private func watch() {
|
||
guard network == nil else { return }
|
||
let monitor = NWPathMonitor()
|
||
monitor.pathUpdateHandler = { path in
|
||
let unmetered = path.status == .satisfied && !path.isExpensive && !path.isConstrained
|
||
Task { @MainActor in KeyboardParakeetDownload.shared.networkChanged(unmetered) }
|
||
}
|
||
monitor.start(queue: DispatchQueue.global(qos: .utility))
|
||
network = monitor
|
||
CFNotificationCenterAddObserver(
|
||
CFNotificationCenterGetDarwinNotifyCenter(),
|
||
nil,
|
||
{ _, _, _, _, _ in
|
||
Task { @MainActor in DictationController.shared.startKeyboardParakeetDownloadIfWanted() }
|
||
},
|
||
KeyboardParakeetSettings.keyboardLimitNotification as CFString,
|
||
nil,
|
||
.deliverImmediately
|
||
)
|
||
}
|
||
|
||
/// Off Wi-Fi, a download Hex started by itself stops (its files resume on Wi-Fi).
|
||
private func networkChanged(_ unmetered: Bool) {
|
||
guard unmetered != isUnmetered else { return }
|
||
isUnmetered = unmetered
|
||
if unmetered {
|
||
DictationController.shared.startKeyboardParakeetDownloadIfWanted()
|
||
} else if task != nil, !KeyboardParakeetSettings().isTurnedOnByHand {
|
||
Task { await stop() }
|
||
}
|
||
}
|
||
}
|
||
|
||
extension DictationController {
|
||
/// The switch in Experiments: on downloads Parakeet V2 for the keyboard unless it's there;
|
||
/// off stops the download and deletes it, unless Hex itself dictates with Parakeet V2.
|
||
func keyboardParakeetChanged() {
|
||
let settings = KeyboardParakeetSettings()
|
||
settings.setKeyboardState(nil)
|
||
if settings.isEnabled {
|
||
startKeyboardParakeetDownloadIfWanted()
|
||
return
|
||
}
|
||
let keeps = transcriber is ParakeetTranscriber && (isModelReady || phase == .preparingModel)
|
||
Task {
|
||
await KeyboardParakeetDownload.shared.stop()
|
||
if !keeps {
|
||
try? FileManager.default.removeItem(at: ParakeetTranscriber.cacheDirectory)
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Not while Hex's own Parakeet V2 gets ready: it downloads into the same folder.
|
||
func startKeyboardParakeetDownloadIfWanted() {
|
||
guard !(transcriber is ParakeetTranscriber && phase == .preparingModel) else { return }
|
||
KeyboardParakeetDownload.shared.startIfWanted()
|
||
}
|
||
}
|
||
|
||
/// Under the switch: what the download waits for, downloading, ready to get ready in the
|
||
/// keyboard, getting ready, ready, or why the keyboard isn't using it, in a few words. The
|
||
/// keyboard's word stays only while fresh.
|
||
struct KeyboardParakeetStatus: View {
|
||
let download: KeyboardParakeetDownload
|
||
|
||
var body: some View {
|
||
TimelineView(.periodic(from: .now, by: 2)) { context in
|
||
if let line = line(now: context.date) {
|
||
VStack(alignment: .leading, spacing: 8) {
|
||
Text(line)
|
||
.font(.system(size: 13))
|
||
.foregroundStyle(HexTheme.textBase)
|
||
.fixedSize(horizontal: false, vertical: true)
|
||
.accessibilityIdentifier("keyboard-parakeet-status")
|
||
if case .downloading(let fraction) = download.state {
|
||
ProgressView(value: fraction)
|
||
.tint(HexTheme.textBase)
|
||
}
|
||
if case .failed = download.state {
|
||
Button("Try again") { download.retry() }
|
||
.font(.system(size: 13, weight: .medium))
|
||
.foregroundStyle(HexTheme.textBase)
|
||
.accessibilityIdentifier("keyboard-parakeet-try-again")
|
||
}
|
||
}
|
||
.frame(maxWidth: .infinity, alignment: .leading)
|
||
.padding(14)
|
||
.background(Color.white.opacity(0.055), in: RoundedRectangle(cornerRadius: 14, style: .continuous))
|
||
}
|
||
}
|
||
}
|
||
|
||
private func line(now: Date) -> String? {
|
||
switch download.state {
|
||
case .downloading:
|
||
return String(localized: "Downloading the speech model")
|
||
case .failed(let message):
|
||
return message
|
||
case .idle:
|
||
break
|
||
}
|
||
guard download.isDownloaded else { return Self.line(for: download.hold) }
|
||
switch KeyboardParakeetSettings().keyboardState {
|
||
case (.preparing, let at)? where now.timeIntervalSince(at) < 180:
|
||
return String(localized: "Getting ready in the keyboard")
|
||
case (.ready, _)?:
|
||
return String(localized: "Ready in the keyboard")
|
||
case (.unavailable, _)?:
|
||
return String(localized: "The keyboard can't use it right now, so it uses Apple's speech recognition.")
|
||
default:
|
||
return String(localized: "Downloaded. The keyboard gets it ready the next time it opens.")
|
||
}
|
||
}
|
||
|
||
private static func line(for hold: KeyboardParakeetDownload.Hold?) -> String? {
|
||
switch hold {
|
||
case .notEnglish:
|
||
String(localized: "Only for English dictation.")
|
||
case .keyboardNotMeasured:
|
||
String(localized: "Downloads after you first use the Hex keyboard.")
|
||
case .keyboardTooSmall:
|
||
String(localized: "The keyboard can't use it right now, so it uses Apple's speech recognition.")
|
||
case .notOnWiFi:
|
||
String(localized: "Downloads on Wi-Fi.")
|
||
case .needsSpace:
|
||
String(localized: "Downloads once the iPhone has \(KeyboardParakeetDownload.format(KeyboardParakeetDownload.freeSpaceNeeded)) free.")
|
||
case .otherEngine, nil:
|
||
nil
|
||
}
|
||
}
|
||
}
|