mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
TestFlight build 126, iPhone 15 Pro Max (8 GB), iOS 26.6 (the same tester as the
build 120 report): Apple's speech dictated instantly, then "after some use" the
keyboard said "Open Hex to download English.", Hex asked for a download, home sat
on "Checking the speech model" and setup on "Finish setup · 1 step", and only a
reinstall brought dictation back, twice.
Cause, from the code (the simulator has no Apple speech assets):
- One failed install of Apple's English model in either process set
SpeechTranscriber's English aside for three days (markSpeechModuleFailed, also
markUnavailable), in the App Group. In the keyboard such failures are often the
keyboard's own: each process holds its own asset reservation, and one released,
removed or over Apple's limit ("Too many asset locales allocated", a case
already in the tests) reads as not installed there.
- With SpeechTranscriber set aside, English went to DictationTranscriber, and
AppleSpeechTranscriber.shouldUse gave English on DictationTranscriber to
Parakeet V2 wherever it fits (6 GB+): Hex's engine became a 463 MB Parakeet
download (Set up), the keyboard handed its dictations to Hex for a day
(re-marked at each launch), and nothing cleared it but three days or a
reinstall. The 632.2 MB in his older screenshot is the multilingual model's
download from build 120.
- Several awaits before a preparation reports anything have no time limit
(Speech's language list, the asset reservation, AssetInventory.status,
prepareToAnalyze), so "Checking the speech model" can stay up for good, and
setup's last step with it.
What changed:
- English stays on Apple Speech on an iPhone that has SpeechTranscriber, on
DictationTranscriber too, and also when Speech doesn't answer at launch;
Parakeet is the way left only once neither Apple module's English installed
twice in a row (shouldUse/appleSpeechLanguage: hasSpeechTranscriber,
appleSetAside). iPhones without SpeechTranscriber are unchanged.
- SpeechModelInstall sets a module or language aside only after the second
failure in a row (LanguageSettings.noteSpeechModuleFailure/noteUnavailable
strikes); a model that installs clears them. In-keyboard dictation's failures
set nothing aside (setsAside: false).
- The keyboard hands its dictations to Hex only after its second failure in a row
within 6 h (noteKeyboardModelFailure); the first says "Hex couldn't transcribe
that. Try again." and the keyboard keeps dictating itself. Hex checks again each
time it comes to the front and ends the handoff when the model is in place
(KeyboardSpeechPreparation.recheckKeyboardHandoff), rather than for a day.
- ParakeetLoadGuard sets a load aside after two in a row that ended Hex, not one
(iOS ending a backgrounded Hex, a swipe away or an update mid-load is no
evidence), and the decision lasts a day and only for that build. Replaced: the
lasting loadInterrupted flag, which only a successful load (never chosen again)
or Try again cleared; an older build's flag is forgotten once.
- A model preparation that says nothing for 30 s, or nothing new for 120 s, ends
with "The speech model isn't ready. Try again." (watchPreparation); the
keyboard model's check behind setup's last step ends the same way after 30 s,
and the setup sheet's Speech model row offers Try again when it failed.
- New string with zh-Hans; README's speech model rules updated.
Verified:
- xcodebuild test, iPhone 15 Pro Max iOS 26.5 simulator on the Mac Studio, on
the rebased head: all tests pass, 3 skipped. New cases in
DictationControllerTests: a set-aside SpeechTranscriber never sends English to a
download; a silent preparation ends in plain words and Try again works; one
failed install sets nothing aside, two in a row do, a success in between
resets; the keyboard's failures set nothing aside; the keyboard hands off on
the second failure in a row only, and a stale first failure no longer counts;
one ended Parakeet load is tried again, two set it aside, a day or a new
build tries again, an older build's flag is forgotten.
- iOS 27.2 simulator (Speech lists en-US but its assets never install), fresh
install, three launches: before, launch 1 already set English aside
(unavailable[en-US]) and handed the keyboard to Hex; after, launch 1 records a
strike only and launch 2 sets it aside. The simulator has no SpeechTranscriber,
so it takes Parakeet either way: the English-stays-on-Apple change can't run
there.
- Same simulator, a model listing that never answers (stand-in server): before,
"Checking the speech model" for 90 s until Parakeet's own stall limit; after,
"The speech model isn't ready. Try again." within 30 s. Download server down
(503): "The speech model didn't download. Check your connection and try
again." with Set up.
- Not reproduced: Apple's speech on a real iPhone (installs, reservations,
"Too many asset locales allocated"), and ending Hex mid-load (the simulator's
cached Parakeet load is too short: 0 of 10 kill -9s at 0.3-1.5 s landed in
it).
271 lines
12 KiB
Swift
271 lines
12 KiB
Swift
import AVFAudio
|
||
import Foundation
|
||
import Speech
|
||
|
||
/// The languages Apple Speech transcribes on this iPhone, shared by the app and in-keyboard
|
||
/// dictation so both pick the same language and the same downloaded model.
|
||
@available(iOS 26.0, *)
|
||
nonisolated enum AppleSpeechLanguages {
|
||
/// The language dictation uses: Settings › Dictation language, else the keyboard's
|
||
/// language Automatic follows (`LanguageSettings.keyboardDictationLanguage`), else
|
||
/// Automatic. Records the result in the App Group for the keyboard layout and translation.
|
||
static func resolve(settings: LanguageSettings = LanguageSettings()) async -> SpeechLanguage? {
|
||
let explicit = settings.dictationLanguage ?? settings.keyboardDictationLanguage()
|
||
let picked = await pick(explicit: explicit, settings: settings)
|
||
if let picked, settings.resolvedDictationLanguage != picked.identifier {
|
||
settings.resolvedDictationLanguage = picked.identifier
|
||
}
|
||
if explicit == nil, let picked, settings.automaticDictationLanguage != picked.identifier {
|
||
settings.automaticDictationLanguage = picked.identifier
|
||
}
|
||
return picked
|
||
}
|
||
|
||
/// What Automatic picks on this iPhone, whatever the setting says. Records it for the
|
||
/// keyboard's language switch.
|
||
static func automatic(settings: LanguageSettings = LanguageSettings()) async -> SpeechLanguage? {
|
||
let picked = await pick(explicit: nil, settings: settings)
|
||
if let picked, settings.automaticDictationLanguage != picked.identifier {
|
||
settings.automaticDictationLanguage = picked.identifier
|
||
}
|
||
return picked
|
||
}
|
||
|
||
private static func pick(explicit: String?, settings: LanguageSettings) async -> SpeechLanguage? {
|
||
await SpeechLanguagePicker.pick(
|
||
explicit: explicit,
|
||
candidates: SpeechLanguagePicker.automaticCandidates(
|
||
preferred: Locale.preferredLanguages,
|
||
current: Locale.current.identifier
|
||
),
|
||
speech: { locale in
|
||
guard SpeechTranscriber.isAvailable,
|
||
let found = await SpeechTranscriber.supportedLocale(equivalentTo: locale) else { return nil }
|
||
// Its model wouldn't install lately: `DictationTranscriber`'s, if it has one.
|
||
guard !settings.speechModuleFailed(found.identifier(.bcp47)) else { return nil }
|
||
return found
|
||
},
|
||
dictation: { locale in
|
||
await DictationTranscriber.supportedLocale(equivalentTo: locale)
|
||
},
|
||
skip: { settings.isUnavailable($0.identifier(.bcp47)) }
|
||
)
|
||
}
|
||
|
||
/// Whether neither module lists a single language here: Speech not answering (in a
|
||
/// keyboard, say), which isn't Speech lacking the iPhone's language.
|
||
static func listsNone() async -> Bool {
|
||
if SpeechTranscriber.isAvailable, !(await SpeechTranscriber.supportedLocales).isEmpty {
|
||
return false
|
||
}
|
||
return (await DictationTranscriber.supportedLocales).isEmpty
|
||
}
|
||
|
||
/// Every language either module transcribes, sorted by name (Hindi as Hinglish and Greek as
|
||
/// Greeklish while those experiments are on). Records the list for the keyboard's language
|
||
/// switch.
|
||
static func options(
|
||
display: Locale = .current,
|
||
settings: LanguageSettings = LanguageSettings(),
|
||
hinglish: HinglishSettings = HinglishSettings(),
|
||
greeklish: GreeklishSettings = GreeklishSettings()
|
||
) async -> [SpeechLanguageOption] {
|
||
var identifiers: [String] = []
|
||
var seen = Set<String>()
|
||
let speech = SpeechTranscriber.isAvailable ? await SpeechTranscriber.supportedLocales : []
|
||
for locale in speech + (await DictationTranscriber.supportedLocales) {
|
||
let identifier = locale.identifier(.bcp47)
|
||
if seen.insert(identifier).inserted {
|
||
identifiers.append(identifier)
|
||
}
|
||
}
|
||
if !identifiers.isEmpty, settings.supportedDictationLanguages != identifiers {
|
||
settings.supportedDictationLanguages = identifiers
|
||
}
|
||
return identifiers
|
||
.map {
|
||
SpeechLanguageOption(
|
||
id: $0,
|
||
name: hinglish.name(for: $0) ?? greeklish.name(for: $0)
|
||
?? SpeechLanguagePicker.displayName(for: $0, in: display)
|
||
)
|
||
}
|
||
.sorted { $0.name.localizedStandardCompare($1.name) == .orderedAscending }
|
||
}
|
||
}
|
||
|
||
/// Apple Speech's model downloads, for the app and in-keyboard dictation alike.
|
||
@available(iOS 26.0, *)
|
||
nonisolated enum AppleSpeechAssets {
|
||
/// Installs `language`'s model (`SpeechModelInstall`: `SpeechTranscriber`, then
|
||
/// `DictationTranscriber`) and returns the language it installed. When neither installs,
|
||
/// the error names what Automatic picks instead, to offer or to say it switched.
|
||
static func install(
|
||
_ language: SpeechLanguage,
|
||
settings: LanguageSettings = LanguageSettings(),
|
||
setsAside: Bool = true,
|
||
log: @escaping @Sendable (String) -> Void = { _ in },
|
||
progress: @escaping @Sendable (Double) -> Void = { _ in }
|
||
) async throws -> SpeechLanguage {
|
||
do {
|
||
let installed = try await SpeechModelInstall.run(
|
||
language,
|
||
steps: steps,
|
||
settings: settings,
|
||
setsAside: setsAside,
|
||
stallLimit: .seconds(60),
|
||
log: log,
|
||
progress: progress
|
||
)
|
||
settings.clearUnavailable(language.identifier)
|
||
return installed
|
||
} catch var unavailable as SpeechModelUnavailable {
|
||
unavailable.automatic = settings.dictationLanguage == nil && settings.keyboardDictationLanguage() == nil
|
||
if unavailable.reason == .notOnThisIPhone,
|
||
let next = await AppleSpeechLanguages.automatic(settings: settings),
|
||
next.languageCode != language.languageCode {
|
||
unavailable.fallback = next.identifier
|
||
}
|
||
throw unavailable
|
||
}
|
||
}
|
||
|
||
static let steps = SpeechModelInstall.Steps(
|
||
dictationEquivalent: { locale in
|
||
await DictationTranscriber.supportedLocale(equivalentTo: locale)
|
||
},
|
||
status: { language in
|
||
switch await AssetInventory.status(forModules: [AppleSpeechModule(language).module]) {
|
||
case .installed: .installed
|
||
case .unsupported: .unsupported
|
||
default: .downloadable
|
||
}
|
||
},
|
||
download: { language, progress in
|
||
let module = AppleSpeechModule(language)
|
||
await reserve(language.locale)
|
||
guard let request = try await AssetInventory.assetInstallationRequest(supporting: [module.module]) else {
|
||
return
|
||
}
|
||
let watched = request.progress
|
||
let monitor = Task {
|
||
while !Task.isCancelled {
|
||
progress(watched.fractionCompleted)
|
||
try? await Task.sleep(for: .milliseconds(200))
|
||
}
|
||
}
|
||
defer { monitor.cancel() }
|
||
try await withTaskCancellationHandler {
|
||
try await request.downloadAndInstall()
|
||
} onCancel: {
|
||
watched.cancel()
|
||
}
|
||
},
|
||
isOnline: { await SpeechModelInstall.isOnline() }
|
||
)
|
||
|
||
/// Reserves `locale`'s assets, first releasing others when Hex already holds
|
||
/// `AssetInventory.maximumReservedLocales` (each language picked in the keyboard reserves
|
||
/// one, and the limit shrinks with free storage). Best effort: without a reservation the
|
||
/// download still runs, or fails with its own reason.
|
||
static func reserve(_ locale: Locale) async {
|
||
let key = locale.identifier(.bcp47)
|
||
var reserved = await AssetInventory.reservedLocales
|
||
guard !reserved.contains(where: { $0.identifier(.bcp47) == key }) else { return }
|
||
let limit = AssetInventory.maximumReservedLocales
|
||
while reserved.count >= max(1, limit), let oldest = reserved.first {
|
||
reserved.removeFirst()
|
||
_ = await AssetInventory.release(reservedLocale: oldest)
|
||
}
|
||
_ = try? await AssetInventory.reserve(locale: locale)
|
||
}
|
||
}
|
||
|
||
/// One dictation's Apple Speech module: `SpeechTranscriber` or `DictationTranscriber`.
|
||
@available(iOS 26.0, *)
|
||
nonisolated enum AppleSpeechModule {
|
||
case speech(SpeechTranscriber)
|
||
case dictation(DictationTranscriber)
|
||
|
||
init(_ language: SpeechLanguage) {
|
||
switch language.module {
|
||
case .speech:
|
||
self = .speech(SpeechTranscriber(locale: language.locale, preset: .transcription))
|
||
case .dictation:
|
||
// Punctuated, without the short-form hint: dictations run up to a minute.
|
||
self = .dictation(DictationTranscriber(locale: language.locale, preset: .longDictation))
|
||
}
|
||
}
|
||
|
||
var module: any SpeechModule {
|
||
switch self {
|
||
case .speech(let transcriber): transcriber
|
||
case .dictation(let transcriber): transcriber
|
||
}
|
||
}
|
||
|
||
var name: String {
|
||
switch self {
|
||
case .speech: "SpeechTranscriber"
|
||
case .dictation: "DictationTranscriber"
|
||
}
|
||
}
|
||
|
||
/// The final results joined (`SpeechSegments.joined`), once the analyzer has finished the file.
|
||
func finalText() async throws -> String {
|
||
var segments: [String] = []
|
||
switch self {
|
||
case .speech(let transcriber):
|
||
for try await result in transcriber.results where result.isFinal {
|
||
segments.append(String(result.text.characters))
|
||
}
|
||
case .dictation(let transcriber):
|
||
for try await result in transcriber.results where result.isFinal {
|
||
segments.append(String(result.text.characters))
|
||
}
|
||
}
|
||
return SpeechSegments.joined(segments)
|
||
}
|
||
}
|
||
|
||
/// How one dictation's final results become its transcript.
|
||
nonisolated enum SpeechSegments {
|
||
/// `segments` back to back, as Apple writes them, except between two in Chinese or
|
||
/// Japanese where the first ends without a mark: `SpeechTranscriber` cuts a Chinese
|
||
/// dictation at a pause of about 1.5 s or more and leaves the cut unpunctuated
|
||
/// ("其实我一直在想" + "要不要换一份工作"), which back to back reads as one run-on clause.
|
||
/// A comma goes there (",", or "、" beside kana), the mark Apple's
|
||
/// `DictationTranscriber` writes for the same pause. Never a full stop: a pause is no
|
||
/// sentence end.
|
||
static func joined(_ segments: [String]) -> String {
|
||
var transcript = ""
|
||
for segment in segments {
|
||
if let last = transcript.last(where: { !$0.isWhitespace }),
|
||
let first = segment.first(where: { !$0.isWhitespace }),
|
||
isWritten(last), isWritten(first) {
|
||
transcript = String(transcript.reversed().drop(while: \.isWhitespace).reversed())
|
||
let kana = (transcript + segment).contains(where: isKana)
|
||
transcript += kana ? "、" : ","
|
||
transcript += String(segment.drop(while: \.isWhitespace))
|
||
} else {
|
||
transcript += segment
|
||
}
|
||
}
|
||
return transcript.trimmingCharacters(in: .whitespacesAndNewlines)
|
||
}
|
||
|
||
/// A Chinese or Japanese character, not punctuation.
|
||
private static func isWritten(_ character: Character) -> Bool {
|
||
guard let scalar = character.unicodeScalars.first?.value else { return false }
|
||
return (0x3041...0x30FF).contains(scalar) && scalar != 0x30FB // kana, not "・"
|
||
|| (0x3400...0x4DBF).contains(scalar)
|
||
|| (0x4E00...0x9FFF).contains(scalar)
|
||
|| (0xF900...0xFAFF).contains(scalar)
|
||
|| (0x20000...0x3134F).contains(scalar)
|
||
}
|
||
|
||
private static func isKana(_ character: Character) -> Bool {
|
||
guard let scalar = character.unicodeScalars.first?.value else { return false }
|
||
return (0x3041...0x30FA).contains(scalar)
|
||
}
|
||
}
|