mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
TestFlight build 126, iPhone 15 Pro Max (8 GB), iOS 26.6 (the same tester as the
build 120 report): Apple's speech dictated instantly, then "after some use" the
keyboard said "Open Hex to download English.", Hex asked for a download, home sat
on "Checking the speech model" and setup on "Finish setup · 1 step", and only a
reinstall brought dictation back, twice.
Cause, from the code (the simulator has no Apple speech assets):
- One failed install of Apple's English model in either process set
SpeechTranscriber's English aside for three days (markSpeechModuleFailed, also
markUnavailable), in the App Group. In the keyboard such failures are often the
keyboard's own: each process holds its own asset reservation, and one released,
removed or over Apple's limit ("Too many asset locales allocated", a case
already in the tests) reads as not installed there.
- With SpeechTranscriber set aside, English went to DictationTranscriber, and
AppleSpeechTranscriber.shouldUse gave English on DictationTranscriber to
Parakeet V2 wherever it fits (6 GB+): Hex's engine became a 463 MB Parakeet
download (Set up), the keyboard handed its dictations to Hex for a day
(re-marked at each launch), and nothing cleared it but three days or a
reinstall. The 632.2 MB in his older screenshot is the multilingual model's
download from build 120.
- Several awaits before a preparation reports anything have no time limit
(Speech's language list, the asset reservation, AssetInventory.status,
prepareToAnalyze), so "Checking the speech model" can stay up for good, and
setup's last step with it.
What changed:
- English stays on Apple Speech on an iPhone that has SpeechTranscriber, on
DictationTranscriber too, and also when Speech doesn't answer at launch;
Parakeet is the way left only once neither Apple module's English installed
twice in a row (shouldUse/appleSpeechLanguage: hasSpeechTranscriber,
appleSetAside). iPhones without SpeechTranscriber are unchanged.
- SpeechModelInstall sets a module or language aside only after the second
failure in a row (LanguageSettings.noteSpeechModuleFailure/noteUnavailable
strikes); a model that installs clears them. In-keyboard dictation's failures
set nothing aside (setsAside: false).
- The keyboard hands its dictations to Hex only after its second failure in a row
within 6 h (noteKeyboardModelFailure); the first says "Hex couldn't transcribe
that. Try again." and the keyboard keeps dictating itself. Hex checks again each
time it comes to the front and ends the handoff when the model is in place
(KeyboardSpeechPreparation.recheckKeyboardHandoff), rather than for a day.
- ParakeetLoadGuard sets a load aside after two in a row that ended Hex, not one
(iOS ending a backgrounded Hex, a swipe away or an update mid-load is no
evidence), and the decision lasts a day and only for that build. Replaced: the
lasting loadInterrupted flag, which only a successful load (never chosen again)
or Try again cleared; an older build's flag is forgotten once.
- A model preparation that says nothing for 30 s, or nothing new for 120 s, ends
with "The speech model isn't ready. Try again." (watchPreparation); the
keyboard model's check behind setup's last step ends the same way after 30 s,
and the setup sheet's Speech model row offers Try again when it failed.
- New string with zh-Hans; README's speech model rules updated.
Verified:
- xcodebuild test, iPhone 15 Pro Max iOS 26.5 simulator on the Mac Studio, on
the rebased head: all tests pass, 3 skipped. New cases in
DictationControllerTests: a set-aside SpeechTranscriber never sends English to a
download; a silent preparation ends in plain words and Try again works; one
failed install sets nothing aside, two in a row do, a success in between
resets; the keyboard's failures set nothing aside; the keyboard hands off on
the second failure in a row only, and a stale first failure no longer counts;
one ended Parakeet load is tried again, two set it aside, a day or a new
build tries again, an older build's flag is forgotten.
- iOS 27.2 simulator (Speech lists en-US but its assets never install), fresh
install, three launches: before, launch 1 already set English aside
(unavailable[en-US]) and handed the keyboard to Hex; after, launch 1 records a
strike only and launch 2 sets it aside. The simulator has no SpeechTranscriber,
so it takes Parakeet either way: the English-stays-on-Apple change can't run
there.
- Same simulator, a model listing that never answers (stand-in server): before,
"Checking the speech model" for 90 s until Parakeet's own stall limit; after,
"The speech model isn't ready. Try again." within 30 s. Download server down
(503): "The speech model didn't download. Check your connection and try
again." with Set up.
- Not reproduced: Apple's speech on a real iPhone (installs, reservations,
"Too many asset locales allocated"), and ending Hex mid-load (the simulator's
cached Parakeet load is too short: 0 of 10 kill -9s at 0.3-1.5 s landed in
it).
305 lines
14 KiB
Swift
305 lines
14 KiB
Swift
import Foundation
|
||
import Speech
|
||
|
||
extension DictationController {
|
||
// MARK: - Which engine dictates (`SpeechEngineChoice`)
|
||
|
||
/// At launch. A launch after Hex ended while loading Parakeet doesn't load it again by
|
||
/// itself, or it would end the same way at every launch: from iOS 26 Apple Speech takes
|
||
/// over where it has the language, before it the on-device recognizer always does.
|
||
func prepareAtLaunch(hasCachedModel: Bool) {
|
||
let parakeetLoads = ParakeetLoadGuard()
|
||
parakeetLoads.noteLaunch()
|
||
let parakeetInterrupted = parakeetLoads.lastLoadWasInterrupted
|
||
if prepareMultilingualParakeetAtLaunch() { return }
|
||
guard #available(iOS 26.0, *) else {
|
||
prepareBeforeAppleSpeechAtLaunch(hasCachedModel: hasCachedModel)
|
||
return
|
||
}
|
||
if hasCachedModel && !parakeetInterrupted {
|
||
engineName = "Parakeet V2"
|
||
prepareModel()
|
||
return
|
||
}
|
||
let fits = SpeechEngineChoice.current.parakeetFitsBesideAppleSpeech
|
||
Task { [weak self] in
|
||
guard let self else { return }
|
||
if await Self.appleSpeechLanguage(picked: AppleSpeechTranscriber.preferredLanguage(), parakeetFits: fits) != nil {
|
||
self.engineName = "Apple Speech"
|
||
self.prepareModel()
|
||
} else {
|
||
self.engineName = "Parakeet V2"
|
||
if hasCachedModel {
|
||
self.phase = .failed(ParakeetLoadGuard.message)
|
||
}
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Experiments › Multilingual model (Parakeet) at launch: true when Parakeet Ultra takes
|
||
/// the dictation language and loads now, downloaded and compiled for this iOS (seconds).
|
||
/// Otherwise Hex gets ready with the engine it uses without it, and the download or a
|
||
/// first load runs beside it (`syncMultilingualParakeet`). With the experiment turned
|
||
/// off, it only deletes a download turning it off left mid-way.
|
||
private func prepareMultilingualParakeetAtLaunch() -> Bool {
|
||
let loads = ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel)
|
||
loads.noteLaunch()
|
||
let settings = MultilingualParakeetSettings()
|
||
guard settings.isEnabled else {
|
||
if settings.isTurnedOff {
|
||
ParakeetUltraModel.remove()
|
||
}
|
||
return false
|
||
}
|
||
if !ParakeetUltraModel.isDownloaded() {
|
||
settings.isModelReady = false
|
||
}
|
||
if loads.lastLoadWasInterrupted, settings.language() != nil {
|
||
settings.isModelReady = false
|
||
multilingualDownload.fail(ParakeetUltraTranscriber.setAsideMessage)
|
||
}
|
||
guard Self.multilingualStep(
|
||
language: ParakeetUltraTranscriber.chosenLanguage(),
|
||
downloaded: ParakeetUltraModel.isDownloaded(),
|
||
compiled: ParakeetUltraTranscriber.isCompiled
|
||
) == .dictate else {
|
||
// A first load waits for the engine Hex gets ready meanwhile (its end syncs).
|
||
startMultilingualDownloadIfWanted()
|
||
return false
|
||
}
|
||
engineName = ParakeetUltraTranscriber.name
|
||
prepareModel()
|
||
return true
|
||
}
|
||
|
||
/// What Hex does about Parakeet Ultra: nothing (off, a language it doesn't take, set
|
||
/// aside), download it, load it beside the engine that dictates meanwhile (a first load
|
||
/// compiles for a minute or more), or dictate with it.
|
||
enum MultilingualStep: Equatable {
|
||
case none
|
||
case download
|
||
case loadBeside
|
||
case dictate
|
||
}
|
||
|
||
nonisolated static func multilingualStep(language: String?, downloaded: Bool, compiled: Bool) -> MultilingualStep {
|
||
guard language != nil else { return .none }
|
||
guard downloaded else { return .download }
|
||
return compiled ? .dictate : .loadBeside
|
||
}
|
||
|
||
/// Starts Parakeet Ultra's download when the experiment wants it and it isn't on the
|
||
/// iPhone, whatever engine is getting ready meanwhile.
|
||
func startMultilingualDownloadIfWanted() {
|
||
// Memory left only matters for a load, not a download.
|
||
guard ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0) != nil,
|
||
!ParakeetUltraModel.isDownloaded() else { return }
|
||
MultilingualParakeetSettings().isModelReady = false
|
||
multilingualDownload.start()
|
||
}
|
||
|
||
/// Once an engine is ready, and once the download finishes: downloads Parakeet Ultra, or
|
||
/// loads it beside the engine that dictates, when the experiment wants it and it isn't the
|
||
/// one dictating already.
|
||
func syncMultilingualParakeet() {
|
||
guard MultilingualParakeetSettings().isEnabled else {
|
||
multilingualDownload.settle()
|
||
return
|
||
}
|
||
let language = ParakeetUltraTranscriber.chosenLanguage()
|
||
let wanted = ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0)
|
||
switch Self.multilingualStep(
|
||
language: wanted,
|
||
downloaded: ParakeetUltraModel.isDownloaded(),
|
||
compiled: true
|
||
) {
|
||
case .none:
|
||
if !multilingualDownload.isFailed { multilingualDownload.settle() }
|
||
case .download:
|
||
startMultilingualDownloadIfWanted()
|
||
case .loadBeside, .dictate:
|
||
if engineName == ParakeetUltraTranscriber.name, isModelReady {
|
||
multilingualDownload.settle()
|
||
} else if let language {
|
||
loadMultilingualParakeetBeside(language: language)
|
||
} else if let wanted, replaceParakeetWithMultilingual(language: wanted) {
|
||
multilingualDownload.loading()
|
||
} else {
|
||
// Too little memory left to load it now: said, with Try again.
|
||
multilingualDownload.fail(
|
||
String(localized: "Not enough memory for the multilingual model. Hex is using its usual model.")
|
||
)
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Parakeet Ultra loaded: the keyboard opens Hex for its languages from now on.
|
||
func multilingualParakeetLoaded() {
|
||
MultilingualParakeetSettings().isModelReady = true
|
||
multilingualDownload.settle()
|
||
}
|
||
|
||
/// Parakeet Ultra didn't load. Missing files download again; anything else sets it aside
|
||
/// (`ParakeetLoadGuard`) until Try again, so Hex doesn't try at every launch.
|
||
func multilingualParakeetFailedToLoad(_ error: Error) {
|
||
let settings = MultilingualParakeetSettings()
|
||
settings.isModelReady = false
|
||
guard settings.isEnabled, !(error is CancellationError) else {
|
||
multilingualDownload.settle()
|
||
return
|
||
}
|
||
if (error as? ParakeetUltraTranscriber.TranscriberError) == .notDownloaded {
|
||
multilingualDownload.settle()
|
||
startMultilingualDownloadIfWanted()
|
||
return
|
||
}
|
||
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).setAside()
|
||
multilingualDownload.fail(
|
||
Self.switchNotice(for: error) ?? UserFacingError.message(for: error, context: .model)
|
||
)
|
||
}
|
||
|
||
/// Try again beside the multilingual model's status: lets a set-aside load try again, and
|
||
/// picks up the download from what's on disk.
|
||
func retryMultilingualParakeet() {
|
||
ParakeetLoadGuard(model: ParakeetUltraTranscriber.loadGuardModel).forget()
|
||
multilingualDownload.settle()
|
||
if ParakeetUltraModel.isDownloaded() {
|
||
syncMultilingualParakeet()
|
||
} else if ParakeetUltraTranscriber.chosenLanguage(availableMemory: 0) != nil {
|
||
MultilingualParakeetSettings().isModelReady = false
|
||
multilingualDownload.restart()
|
||
}
|
||
}
|
||
|
||
/// Turn off beside the multilingual model's status, as the switch in Experiments does.
|
||
func turnOffMultilingualParakeet() {
|
||
MultilingualParakeetSettings().isEnabled = false
|
||
multilingualParakeetChanged()
|
||
}
|
||
|
||
/// Before iOS 26: Parakeet loads at launch once downloaded, as before. The recognizer gets
|
||
/// ready at launch only once Speech Recognition is allowed; otherwise setup asks for it
|
||
/// (onboarding's first page, or Set up on home), never a launch by itself.
|
||
private func prepareBeforeAppleSpeechAtLaunch(hasCachedModel: Bool) {
|
||
let choice = SpeechEngineChoice.current
|
||
let wanted = Self.wantedLanguage()
|
||
switch choice.engine(for: wanted, appleSpeechPicks: false) {
|
||
case .parakeet:
|
||
engineName = "Parakeet V2"
|
||
if hasCachedModel { prepareModel() }
|
||
case .onDeviceRecognizer, .appleSpeech:
|
||
engineName = "Apple Speech"
|
||
languageNotice = choice.notice(for: wanted)
|
||
if SFSpeechRecognizer.authorizationStatus() == .authorized {
|
||
// The keyboard's Hex key starts recording in Hex rather than only opening it.
|
||
keyboardBridge.setModelInstalled(true)
|
||
prepareModel()
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Whether getting the engine ready downloads a model in Hex: not Apple's on-device
|
||
/// recognizer, the only "Apple Speech" before iOS 26, whose model iOS keeps.
|
||
var engineDownloads: Bool {
|
||
if #available(iOS 26.0, *) { return true }
|
||
return engineName != "Apple Speech"
|
||
}
|
||
|
||
/// The language dictation is meant to be in: the one set, else the keyboard's Automatic
|
||
/// follows, else the iPhone's.
|
||
nonisolated static func wantedLanguage(settings: LanguageSettings = LanguageSettings()) -> String? {
|
||
settings.dictationLanguage ?? settings.keyboardDictationLanguage() ?? Locale.preferredLanguages.first
|
||
}
|
||
|
||
static func defaultTranscriber() async -> any LocalTranscribing {
|
||
// Experiments › Multilingual model (Parakeet), for its languages once it's downloaded and
|
||
// compiled; until then (and off) the engine Hex uses without it.
|
||
if let language = ParakeetUltraTranscriber.chosenLanguage(),
|
||
multilingualStep(
|
||
language: language,
|
||
downloaded: ParakeetUltraModel.isDownloaded(),
|
||
compiled: ParakeetUltraTranscriber.isCompiled
|
||
) == .dictate {
|
||
return ParakeetUltraTranscriber(language: language)
|
||
}
|
||
let choice = SpeechEngineChoice.current
|
||
guard #available(iOS 26.0, *) else {
|
||
return transcriberBeforeAppleSpeech(choice: choice)
|
||
}
|
||
if let language = await appleSpeechLanguage(
|
||
picked: AppleSpeechTranscriber.preferredLanguage(),
|
||
parakeetFits: choice.parakeetFitsBesideAppleSpeech
|
||
) {
|
||
return AppleSpeechTranscriber(language: language)
|
||
}
|
||
return ParakeetTranscriber()
|
||
}
|
||
|
||
/// Before iOS 26: Parakeet or the on-device recognizer (`SpeechEngineChoice`), in the
|
||
/// language the recognizer has on this iPhone, else the one wanted, which then says it
|
||
/// isn't available here.
|
||
nonisolated static func transcriberBeforeAppleSpeech(
|
||
choice: SpeechEngineChoice,
|
||
suite: String = KeyboardBridge.appGroup,
|
||
access: SpeechRecognitionAccess = .system,
|
||
resolve: (LanguageSettings) -> String? = { OnDeviceSpeechLanguages.resolve(settings: $0) }
|
||
) -> any LocalTranscribing {
|
||
let settings = LanguageSettings(defaults: UserDefaults(suiteName: suite))
|
||
let wanted = wantedLanguage(settings: settings)
|
||
guard choice.engine(for: wanted, appleSpeechPicks: false) == .onDeviceRecognizer else {
|
||
return ParakeetTranscriber()
|
||
}
|
||
let language = resolve(settings) ?? wanted ?? "en-US"
|
||
return OnDeviceSpeechTranscriber(language: language, access: access, suite: suite)
|
||
}
|
||
|
||
/// The language Apple Speech dictates in, or nil for Parakeet V2, which knows only
|
||
/// English: Apple Speech's pick, unless it is English that Parakeet takes
|
||
/// (`AppleSpeechTranscriber.shouldUse`). With no pick (Speech not answering), English goes
|
||
/// to Parakeet only on an iPhone without `SpeechTranscriber`, and any other language
|
||
/// (`wanted`: the one set, else the keyboard's Automatic follows, else the iPhone's) stays
|
||
/// on Apple Speech, which says it can't dictate it rather than Parakeet hearing Spanish as
|
||
/// English words.
|
||
@available(iOS 26.0, *)
|
||
nonisolated static func appleSpeechLanguage(
|
||
picked: SpeechLanguage?,
|
||
wanted: String? = wantedLanguage(),
|
||
parakeetFits: Bool = ParakeetTranscriber.fitsInMemory,
|
||
hasSpeechTranscriber: Bool = SpeechTranscriber.isAvailable,
|
||
appleSetAside: Bool? = nil
|
||
) -> SpeechLanguage? {
|
||
if let picked {
|
||
return AppleSpeechTranscriber.shouldUse(
|
||
picked,
|
||
parakeetFits: parakeetFits,
|
||
hasSpeechTranscriber: hasSpeechTranscriber,
|
||
appleSetAside: appleSetAside
|
||
) ? picked : nil
|
||
}
|
||
guard let wanted else { return nil }
|
||
if SpeechLanguagePicker.isEnglish(wanted) {
|
||
let english = SpeechLanguage(locale: Locale(identifier: wanted), module: .speech)
|
||
let setAside = appleSetAside ?? LanguageSettings().isUnavailable(english.identifier)
|
||
guard hasSpeechTranscriber, !setAside else { return nil }
|
||
return english
|
||
}
|
||
return SpeechLanguage(locale: Locale(identifier: wanted), module: .dictation)
|
||
}
|
||
|
||
/// What home says when a preparation failed in a way the next pick gets past: Automatic
|
||
/// set a language aside, or Parakeet found too little memory to load (it set itself aside,
|
||
/// so the next pick is Apple's).
|
||
nonisolated static func switchNotice(for error: Error) -> String? {
|
||
if let unavailable = error as? SpeechModelUnavailable, unavailable.switchedAutomatically {
|
||
return UserFacingError.message(for: unavailable, context: .model)
|
||
}
|
||
if (error as? ParakeetTranscriber.TranscriberError) == .notEnoughMemory {
|
||
return String(localized: "Not enough memory for the speech model. Hex is using Apple's on-device speech recognition.")
|
||
}
|
||
if (error as? ParakeetUltraTranscriber.TranscriberError) == .notEnoughMemory {
|
||
return String(localized: "Not enough memory for the multilingual model. Hex is using its usual model.")
|
||
}
|
||
return nil
|
||
}
|
||
}
|