mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS keyboard: French (France) autocorrects with Hex's own model
Add Android's French typing model (fr.hextm, byte-identical to
android/app/src/main/assets/typing/fr.hextm and its SHA256SUMS hash) to the
keyboard's bundle. The existing model path (FullKeyboardSettings.typingModel ->
TypingDecoderModel.bundled) already maps any bundled <code>.hextm, so French
now reads taps with Spatial autocorrect's decoder, the likelier-next-letter
odds and the model's word pairs instead of UITextChecker's one-edit guesses.
No decoder logic changes.
One guard changed: with a bilingual partner, a word the partner's dictionary
knows as typed is no longer respelled with only accents
(AutocorrectEngine.cautiousReplacement, partner branch). French's model
otherwise turned English "these" into "thèse" on a French + English keyboard
(testABilingualKeyboardKeepsBothLanguagesWords). Slips the decoder reads as
another word still apply ("tkm" -> "Tom").
Held-out gate (2,000 Tatoeba sentences on AZERTY, scored as
android/benchmark/typing/quality-gate.py): fewer word errors than raw taps
37.8% -> 77.9%, correctly typed words changed 0.56% -> 0.10%, accents
restored where the plain spelling is no word 70.2% -> 86.9%, critical cases
1/2 -> 2/2. Real touches on the AZERTY keyboard in an iOS 27.2 simulator,
apostrophe key included, typed five sentences right. The live keyboard's
footprint with French active is unchanged (about 19 MB shown, 29.6 MB after
typing, against English's 30.4 MB); the model maps in under 0.4 ms.
Known gap: Canada's and Switzerland's French (fr-CA, fr-CH) have no model of
their own and keep UITextChecker. Settings shows "English, Spanish and French
only" under Spatial autocorrect.
Verified: xcodebuild test on the Studio (857 tests, 0 failures), new cases in
AutocorrectEngineTests and FullKeyboardEngineTests, code-shape.py with no
hard problems.
* iOS keyboard: French model's gate numbers from the replay on today's layout
The replay's taps now land on the nearest letter or space bar when the
synthetic typist's taller rows put them outside Hex's letter keys (on 123,
shift or delete with today's row metrics): 78.7% fewer word errors than raw
taps, against 37.4% with UITextChecker alone (was 77.9% / 37.8% before #2379's
row change). README only.
* iOS keyboard: French model's numbers as the gate on dev scores them
quality-gate.py on dev counts an accent word as unambiguous only when its
plain spelling is in neither the model nor the .valid list; by that rule
French restores 81.1% (78.7% without the model), short of the 85% bar. The
replay now types French's apostrophe on the AZERTY key (FrenchTyping.key),
not from the 123 page: 76.4% fewer word errors than raw taps (35.3% before).
README only.
* iOS keyboard: French reads Hex's own word list, not the phone's dictionary
Add HexWordList (ios/HEXShared/HexWordList.swift): a language's list of real
words outside its typing model (HEXKeyboard/TypingModel/<code>.valid, the
file Android ships, built by android/benchmark/typing/build-valid-words.py),
memory-mapped read-only and searched in place by bisection over its sorted
UTF-8 lines. No copy, no Set; lookups touch only clean pages.
Where the decoder's language has a bundled list, SystemTextChecker answers
isWordValid from the model and that list alone (HexOwnWords, set by
AutocorrectEngine.useOwnWords whenever the decoder changes), and its guesses
and completions count only while UITextChecker is in that language. This
covers both cases where the phone's dictionary is no use: none for the
language, and one that takes any letters for a word.
One guard changed with it (AutocorrectEngine.hasWords): a missing system
dictionary no longer switches off space corrections, the strip, the revision
of the word before and the merge when Hex's model is there. AutocorrectEngine
.swift keeps 1,498 lines (five lines changed in place).
Only French ships a list (fr.valid, 1.45 MB, Google Books v3, CC BY 3.0).
Held-out gate (ios/scripts/typing-gate.py, as quality-gate.py scores it):
French 76.3% -> 76.6% fewer errors than raw taps, correct words changed
0.10% -> 0.23%, unambiguous accents 81.1% -> 86.1% (passes 85%), critical
2/2. With their own lists English (0.27% -> 0.71% changed, fails), Spanish
(0.17% -> 0.36% changed) and German (0.12% -> 0.15%) got worse, so they keep
UITextChecker and ship no list.
Behaviour change: with a list, a name typed in lowercase that UITextChecker
knew only capitalised no longer takes its capital ("mary"), until a capitals
list reader exists.
Verified: xcodebuild test on the Studio (866 tests, 0 failures), new case in
AutocorrectEngineTests; live keyboard in an iOS 27.2 simulator types French
sentences right, footprint 29.8 MB after typing (fr.valid 272 KB resident,
0 dirty); code-shape.py no hard problems.
* iOS keyboard: names take their capital from French's own capitals list
Reading Hex's word list instead of UITextChecker (previous commit) cost
French its names: a name typed in lowercase ("paris", "tom") no longer got
its capital. The gate ignores case, so it could not see that.
HexWordList now also reads <code>.capitals (Android's build-capitals.py
output, the same sorted format, mapped and searched in place) and gives a
line's number (value(of:)). With a list, HexOwnWords takes a word the books
capitalise at least 95% of the time as no word when typed all lowercase
("paris" -> "Paris", as UITextChecker's "only capitalised" rule does), and a
listed name outside the model as a word. A word the books capitalise 50-94%
of the time ("tom", "mary": also ordinary words, capped by the builder) is
a name where the phone's own dictionary for the language knows it only
capitalised, and only when that dictionary tells words from other letters
(one "qzxjvkq" probe per language): a UITextChecker call for those words
alone.
GermanCapitals reads de.capitals through the same reader instead of a
55,000-entry dictionary in memory; its rules are unchanged (German gate
83.6% / 0.12% / 2/2).
ios/scripts/typing-gate.py reports capitals, which its three numbers can't
see: every reference typed all lowercase, then, after a sentence's first
word, the reference words with a capital that keep it and the words
without that gain one. French: names kept 45.4% on UITextChecker, 0.0%
with the list alone, 47.5% with the list and fr.capitals; capitals added
0.01% / 0.00% / 0.00%. The gate's numbers hold: 76.6% fewer errors than
raw taps, 0.23% correct words changed, 86.1% unambiguous accents, 2/2.
Ships fr.capitals (272 KB, Google Books v3 counts, CC BY 3.0).
Verified: xcodebuild test on the Studio (884 tests, 0 failures),
code-shape.py no hard problems; AutocorrectEngine.swift unchanged.
154 lines
7.4 KiB
Swift
154 lines
7.4 KiB
Swift
import Darwin
|
||
import Foundation
|
||
|
||
/// One of Hex's own sorted word lists beside a language's typing model, the same files Android
|
||
/// ships: `<code>.valid`, the real words outside the model (`build-valid-words.py`), and
|
||
/// `<code>.capitals`, how often the books write a word with a capital (`build-capitals.py`).
|
||
/// One lowercase NFC spelling per line, alone or followed by a tab and a number (a cost, a
|
||
/// percent), sorted by its UTF-8 bytes, newline-terminated. The file is memory-mapped read-only, like the `.hextm` models,
|
||
/// and searched in place by bisection: no copy and no set, so a 5 MB list costs only the clean
|
||
/// pages its lookups touch, which iOS can drop under pressure.
|
||
nonisolated final class HexWordList: @unchecked Sendable {
|
||
let byteCount: Int
|
||
private let base: UnsafePointer<UInt8>
|
||
|
||
init?(url: URL) {
|
||
let fd = open(url.path, O_RDONLY)
|
||
guard fd >= 0 else { return nil }
|
||
defer { close(fd) }
|
||
var info = stat()
|
||
guard fstat(fd, &info) == 0, info.st_size > 0 else { return nil }
|
||
let size = Int(info.st_size)
|
||
guard let mapped = mmap(nil, size, PROT_READ, MAP_PRIVATE, fd, 0), mapped != MAP_FAILED else { return nil }
|
||
let bytes = UnsafePointer(mapped.assumingMemoryBound(to: UInt8.self))
|
||
guard bytes[size - 1] == 10 else { munmap(mapped, size); return nil }
|
||
base = bytes
|
||
byteCount = size
|
||
}
|
||
|
||
deinit {
|
||
munmap(UnsafeMutableRawPointer(mutating: base), byteCount)
|
||
}
|
||
|
||
/// Whether the list has `word` (lowercase NFC, as the builder writes them): one bisection over
|
||
/// the mapped bytes, nothing allocated for a native string.
|
||
func contains(_ word: String) -> Bool {
|
||
line(of: word) != nil
|
||
}
|
||
|
||
/// The number after `word`'s tab (a `.capitals` percent, a `.valid` cost), or nil when the
|
||
/// list lacks the word or its line has none.
|
||
func value(of word: String) -> Int? {
|
||
guard let (end, next) = line(of: word), end < next else { return nil }
|
||
var value = 0
|
||
for i in (end + 1)..<next {
|
||
let digit = Int(base[i]) - 48
|
||
guard (0...9).contains(digit) else { return nil }
|
||
value = value * 10 + digit
|
||
}
|
||
return value
|
||
}
|
||
|
||
/// Where `word`'s line ends (its tab or newline) and where its newline is.
|
||
private func line(of word: String) -> (end: Int, next: Int)? {
|
||
var word = word
|
||
return word.withUTF8 { key -> (Int, Int)? in
|
||
guard !key.isEmpty else { return nil }
|
||
var lo = 0, hi = byteCount
|
||
while lo < hi {
|
||
// The line holding the middle byte: back to its start, on to its tab or newline.
|
||
var start = (lo + hi) / 2
|
||
while start > lo, base[start - 1] != 10 { start -= 1 }
|
||
var end = start
|
||
while base[end] != 10, base[end] != 9 { end += 1 }
|
||
var next = end
|
||
while base[next] != 10 { next += 1 }
|
||
var order = 0
|
||
var k = 0
|
||
while order == 0, k < key.count, start + k < end {
|
||
order = Int(base[start + k]) - Int(key[k])
|
||
k += 1
|
||
}
|
||
if order == 0 { order = (end - start) - key.count }
|
||
if order == 0 { return (end, next) }
|
||
if order < 0 { lo = next + 1 } else { hi = start }
|
||
}
|
||
return nil
|
||
}
|
||
}
|
||
|
||
/// A language's list in the keyboard's bundle (`kind` "valid" or "capitals"), mapped once per
|
||
/// process; nil when the build has none, as `TypingDecoderModel.bundled` finds the model.
|
||
@MainActor static func bundled(_ language: String, _ kind: String = "valid") -> HexWordList? {
|
||
let name = language + "." + kind
|
||
if let cached = loaded[name] { return cached }
|
||
var url = Bundle.main.url(forResource: language, withExtension: kind)
|
||
if url == nil, let plugins = Bundle.main.builtInPlugInsURL,
|
||
let keyboard = Bundle(url: plugins.appendingPathComponent("HEXKeyboard.appex")) {
|
||
url = keyboard.url(forResource: language, withExtension: kind)
|
||
}
|
||
let list = url.flatMap { HexWordList(url: $0) }
|
||
loaded[name] = list
|
||
return list
|
||
}
|
||
|
||
@MainActor private static var loaded: [String: HexWordList?] = [:]
|
||
}
|
||
|
||
/// A language's own words: its typing model's and its `HexWordList`'s. Where Hex has both, they
|
||
/// alone say whether a spelling is a word, not the phone's dictionary, which iOS may not have
|
||
/// for the language or may have take any letters for one (iOS 27.2 simulator: `nb_NO`, `pl_PL`).
|
||
nonisolated struct HexOwnWords: Sendable {
|
||
/// A word the books write with a capital this often is a name: typed all lowercase it is no
|
||
/// word, so autocorrect gives it its capital ("paris" -> "Paris", "jean" -> "Jean"), as
|
||
/// `UITextChecker` does with a word it knows only capitalised (Android's `TypingWord`: 95).
|
||
static let nameShare = 95
|
||
|
||
let language: String
|
||
let model: TypingDecoderModel
|
||
let list: HexWordList
|
||
/// `<code>.capitals`, when the keyboard ships one: names and other words the books write with
|
||
/// a capital, with how often they do (50–100).
|
||
var capitals: HexWordList?
|
||
|
||
/// Whether `word` is one, case-folded ("Żółw" is "żółw"; lists are lowercase) and with ’ read
|
||
/// as '; a name typed all lowercase isn't ("paris"), its capitalised form is ("Paris"). The
|
||
/// list lookups allocate nothing; folding a word with a capital or a ’ copies it.
|
||
func knows(_ word: String) -> Bool {
|
||
guard !word.isEmpty else { return true }
|
||
let typedLower = !word.contains(where: \.isUppercase)
|
||
var lower = typedLower ? word : word.lowercased()
|
||
if lower.contains("\u{2019}") { lower = lower.replacingOccurrences(of: "\u{2019}", with: "'") }
|
||
let share = capitals?.value(of: lower)
|
||
if typedLower, lower.count >= 2, let share, share >= Self.nameShare { return false }
|
||
return model.index(of: lower) != nil || list.contains(lower) || share != nil
|
||
}
|
||
|
||
/// Whether `word`, typed all lowercase, is one the books capitalise only part of the time
|
||
/// (50–94%: "tom", "mary", also ordinary words or capped as such): the phone's dictionary,
|
||
/// where it works, decides whether it is a name (`SystemTextChecker.isWordValid`).
|
||
func mayBeName(_ word: String) -> Bool {
|
||
guard word.count >= 2, !word.contains(where: \.isUppercase), let share = capitals?.value(of: word) else { return false }
|
||
return share < Self.nameShare
|
||
}
|
||
}
|
||
|
||
@MainActor
|
||
extension AutocorrectEngine {
|
||
/// Gives this engine's spelling dictionary the decoder model's language's own words, when the
|
||
/// keyboard ships a list for it, or takes them away. Called whenever `decoder` changes.
|
||
func useOwnWords(_ model: TypingDecoderModel?) {
|
||
guard let system = checker as? SystemTextChecker else { return }
|
||
system.ownWords = model.flatMap { model in
|
||
HexWordList.bundled(model.language).map {
|
||
HexOwnWords(language: model.language, model: model, list: $0, capitals: HexWordList.bundled(model.language, "capitals"))
|
||
}
|
||
}
|
||
}
|
||
|
||
/// Whether this language has words to correct with: the phone's dictionary, or Hex's own
|
||
/// model for it. A missing system dictionary never switches a Hex model off.
|
||
var hasWords: Bool {
|
||
hasDictionary || decoder?.model.language == language.rawValue
|
||
}
|
||
}
|