mirror of
https://github.com/anomalyco/hex.git
synced 2026-11-22 21:23:02 +01:00
* iOS: French dictation gets French spacing before ? ! ; : and inside « »
X (Philippe): "When dictating, in some languages punctuation has a space
before, like question and exclamation marks in French. Right now they're not
inserted."
Black box first:
- Apple Speech (macOS 27.2.1, say -v Thomas and Amélie): fr-FR and fr-CH write
"ça va ?" with a plain U+0020 space; fr-CA's DictationTranscriber writes
"ça va?" (its SpeechTranscriber a plain space). None writes U+202F/U+00A0.
Hex's Polished, Casual and Notes then took every space before a mark out
(fixSpacingBeforePunctuation), so French came out "ça va?".
- Apple's Français (France) AZERTY keyboard (iOS 27.2 simulator, real taps in
Messages): "Salut" then ? gives "Salut?", "salut ?" keeps the plain space
typed, same for ! : ;. The keyboard never adds French spacing.
Change:
- FrenchSpacing, the dictation pipeline's last step (DictationPipeline.format,
after the style, lists, vocabulary spellings and Lowercase; every style but
Verbatim, which keeps the model's text). France and every other French: a
narrow no-break space (U+202F) before ? ! ; and a no-break space (U+00A0)
before : and inside « ». Canada (OQLF): none before ? ! ;, U+00A0 before :
and inside « ». Switzerland: U+202F before ? ! ; and inside « », U+00A0
before :. A translation into French is spaced as French (format already
takes the target language).
- Whatever space was there (none, several, NBSP, NNBSP, thin) becomes exactly
one; running it twice changes nothing. Only a mark that ends a word is
spaced (followed by a space, the end, another mark or a closing quote or
bracket), so 10:30, URLs and ?q=1, a mark typed against a URL or address,
emoticons (:) ;-) :D), code in backticks, "::", a mark at a line's start and
the second mark of ?! stay as they are.
- Full keyboard: unchanged, matching the system keyboard (marks go in as
tapped in every French layout).
Defaults picked (no tester answer): Swiss guillemets use the narrow space
(Guide du typographe); Verbatim is left alone.
Verified (iPhone 17 simulator, iOS 27.2, Mac Studio): xcodebuild test, 530
tests, 1 skipped (needs HEX_SPEECH_CLIPS), 0 failures, including new
FrenchSpacingTests (pipeline in every style, Canada and Switzerland, no
doubled spaces, times/URLs/emoticons/code untouched, idempotent) and
FullKeyboardEngineTests.testFrenchMarksGoInAsTyped (fr, fr-CA, fr-CH).
* iOS: bilingual autocorrect keeps English as good as alone beside French or German
X (Philippe): "Would also be great if the keyboard supported typing multiple
languages without having to actively choose which one to use at a given
time, like Gboard."
What works today: bilingual autocorrect (#2097, #2143, #2164) already pairs
any two Latin-script languages with an iOS dictionary (English with Français,
Deutsch, Italiano, Português, Polski, Norsk, Română, Español), whatever their
order: the first other chosen Latin language is the partner, its dictionary
is checked per word and the strip shows both languages' fixes and completions.
So no new pairs were needed. But the held-out English benchmark (2,000
Tatoeba sentences, Spatial autocorrect on) got much worse with a partner on:
5.75% WER alone, 7.56% with Français, 7.53% with Deutsch (7.18% with the
German Spatial experiment). Causes, measured:
- iOS's French and German dictionaries take many English typos for words
("dont", "im", "ive", "dk", "dl", "ro", "tkm"), and a word either language
knew was never changed, so "Im" and "dont" stayed (61 and 42 losses).
- They also take common English words ("the", "you", "your", "not",
"please"), so the words-before lean almost never said English.
- The partner's spelling guesses changed names English kept ("Ziri" ->
"Zuri").
Changes (AutocorrectEngine):
- languageLean: a word both dictionaries know counts, from three letters, for
the language whose typing model has it among its commonest (-ln P <= 10).
- In a sentence that leans the keyboard's language, a word the partner knows
still gets this language's sure fixes (unambiguousFixes, two-letter slips).
- Without a lean, Spatial autocorrect may still read the taps as another word,
counting the partner's word as a valid one (changed only on its valid-word
margin), except words with a sure fix, which the next word decides.
- bilingualRevision: at a sentence's start, "Im"/"Dont" become "I'm"/"Don't"
at the next word's space when that word leans English (German "Im Haus"
stays; the revision's undo is the existing one).
- A partner word without a vowel is no word (iOS abbreviations), and the
partner's guess of a name is never a fix.
- Kept: #2143's guard (workingPartner ignores a dictionary that takes every
word), #2164's layout rule (English and Spanish on the first one's keys;
other languages keep their own layouts), dictation following the keyboard.
Removed: "a word either language knows as typed is never changed" without
exception; it now has the exceptions above.
Black box (iOS 27.2 simulator): Settings offers "Add to English (US)
Keyboard" for French (France, Canada, Switzerland) ("Type in English and
French on the same keyboard": one "English & French" keyboard on QWERTY, no
switching); Portuguese, Polish, Turkish offer "Add to Existing Keyboard".
It fixed "I dont know" and "Dont do it", kept "est" and "merci" after English
words; its French data is thin in the simulator ("je pense" -> "he pence"),
so its French weighting is unverified.
Verified (iPhone 17 simulator, iOS 27.2, Mac Studio, rebased on #2180):
- Benchmark (ZZDecoderBench, not committed): English alone 5.75% (unchanged
by this change); with Français 7.56% -> 5.79% (correct words broken 125 ->
99, 113 alone); with Deutsch 7.53% -> 5.79%; Deutsch with its Spatial
model 7.18% -> 5.70%. Before rebasing (2a023f91): alone 5.24%, Français
7.07% -> 5.24%, Deutsch 7.08% -> 5.27%.
- Code-switching probe (centre taps): after "we said that", 1 of 80 French
words changes (dont -> don't) and 3 of 72 German (auch, ist, habe), none
before; at a message's start and after French or German words, none; slips
in French/German words at a message's start fixed as before (26/50, 18/40).
- Letter p95 with a partner 4.1-5.4 ms (3.9-5.1 ms before), 2.2 ms alone.
- xcodebuild test: 530 tests, 1 skipped, 0 failures, including new
AutocorrectEngineTests.testBesideAnotherLanguageEnglishStillFixesItsOwnTypos
and the existing bilingual tests (testABilingualKeyboardKeepsBothLanguagesWords,
testTheFrenchKeyboardFixesEnglishTyposWithEnglishBesideIt,
testAPartnerDictionaryThatTakesEveryWordIsIgnored).
149 lines
6.4 KiB
Swift
149 lines
6.4 KiB
Swift
import Foundation
|
||
|
||
/// French typography's spaces before `? ! ; :` and inside `« »`, the dictation pipeline's last
|
||
/// step (`DictationPipeline.format`). Apple Speech doesn't write them (black box, macOS 27.2.1:
|
||
/// fr-FR and fr-CH write a plain space before "?", fr-CA's DictationTranscriber none) and the
|
||
/// styles take out every space before a mark, so French came out as "ça va?". Apple's French
|
||
/// keyboards don't add one either (iOS 27.2 simulator), so the full keyboard types marks as you
|
||
/// tap them.
|
||
///
|
||
/// - France (and every French but Canada's and Switzerland's): a narrow no-break space
|
||
/// (U+202F) before `? ! ;`, a no-break space (U+00A0) before `:` and inside `« »`.
|
||
/// - Canada (OQLF): no space before `? ! ;`, a no-break space before `:` and inside `« »`.
|
||
/// - Switzerland (Guide du typographe): narrow no-break spaces before `? ! ;` and inside
|
||
/// `« »`, a no-break space before `:`.
|
||
///
|
||
/// Whatever space was there (none, one, several, or another kind) becomes exactly that one. A
|
||
/// mark is spaced only where it ends a word: followed by a space, a line break, the end,
|
||
/// another mark or a closing quote or bracket. So times (10:30), ratios, URLs ("https://",
|
||
/// "?q=1"), a URL or address the mark is typed against, emoticons (":)", ";-)", ":D"), code in
|
||
/// backticks, a mark at a line's start, and the second mark of "?!" stay as they are.
|
||
nonisolated enum FrenchSpacing {
|
||
enum Convention: Equatable {
|
||
case france
|
||
case canada
|
||
case switzerland
|
||
}
|
||
|
||
static let narrowNoBreakSpace: Character = "\u{202F}"
|
||
static let noBreakSpace: Character = "\u{00A0}"
|
||
|
||
/// The convention for `language` (BCP 47), or nil when it isn't French.
|
||
static func convention(for language: String?) -> Convention? {
|
||
guard let language else { return nil }
|
||
let locale = Locale(identifier: language)
|
||
guard locale.language.languageCode?.identifier == "fr" else { return nil }
|
||
switch locale.region?.identifier {
|
||
case "CA": return .canada
|
||
case "CH": return .switzerland
|
||
default: return .france
|
||
}
|
||
}
|
||
|
||
/// `text` with French spacing when `language` is French; otherwise unchanged.
|
||
static func apply(_ text: String, language: String?) -> String {
|
||
guard let convention = convention(for: language) else { return text }
|
||
return apply(text, convention: convention)
|
||
}
|
||
|
||
static func apply(_ text: String, convention: Convention) -> String {
|
||
let chars = Array(text)
|
||
guard chars.contains(where: { "?!;:«»".contains($0) }) else { return text }
|
||
let beforeQuestion: Character? = convention == .canada ? nil : narrowNoBreakSpace
|
||
let insideGuillemets = convention == .switzerland ? narrowNoBreakSpace : noBreakSpace
|
||
|
||
var out: [Character] = []
|
||
out.reserveCapacity(chars.count + 8)
|
||
// Spaces read but not yet written: a mark after them may replace them.
|
||
var pending: [Character] = []
|
||
var inCode = false
|
||
var afterOpeningGuillemet = false
|
||
|
||
func flush() {
|
||
out.append(contentsOf: pending)
|
||
pending.removeAll()
|
||
}
|
||
|
||
for (index, ch) in chars.enumerated() {
|
||
let next = index + 1 < chars.count ? chars[index + 1] : nil
|
||
if ch == "`" {
|
||
inCode.toggle()
|
||
afterOpeningGuillemet = false
|
||
flush()
|
||
out.append(ch)
|
||
continue
|
||
}
|
||
if inCode {
|
||
flush()
|
||
out.append(ch)
|
||
continue
|
||
}
|
||
if isHorizontalSpace(ch) {
|
||
pending.append(ch)
|
||
continue
|
||
}
|
||
if afterOpeningGuillemet {
|
||
afterOpeningGuillemet = false
|
||
if !ch.isNewline, ch != "»" {
|
||
pending = [insideGuillemets]
|
||
}
|
||
}
|
||
switch ch {
|
||
case "?", "!", ";", ":":
|
||
if endsWord(ch, next: next),
|
||
let space = spaceBefore(ch, out: out, pending: pending, question: beforeQuestion) {
|
||
pending.removeAll()
|
||
if let space { out.append(space) }
|
||
} else {
|
||
flush()
|
||
}
|
||
out.append(ch)
|
||
case "«":
|
||
flush()
|
||
out.append(ch)
|
||
afterOpeningGuillemet = true
|
||
case "»":
|
||
if let last = out.last, !last.isNewline, last != "«" {
|
||
pending = [insideGuillemets]
|
||
}
|
||
flush()
|
||
out.append(ch)
|
||
default:
|
||
flush()
|
||
out.append(ch)
|
||
}
|
||
}
|
||
if !afterOpeningGuillemet { flush() }
|
||
return String(out)
|
||
}
|
||
|
||
private static func isHorizontalSpace(_ ch: Character) -> Bool {
|
||
ch == " " || ch == "\t" || ch == noBreakSpace || ch == narrowNoBreakSpace || ch == "\u{2009}"
|
||
}
|
||
|
||
/// Whether `mark` ends a word here, rather than sitting inside a time, a URL or an
|
||
/// emoticon: what follows is nothing, a space, a line break, another mark or a closing
|
||
/// quote or bracket, and for `:` and `;` not an emoticon's mouth (":)", ";(").
|
||
private static func endsWord(_ mark: Character, next: Character?) -> Bool {
|
||
guard let next else { return true }
|
||
if mark == ":" || mark == ";", ")(][|".contains(next) { return false }
|
||
return next.isWhitespace || "?!.…,»”\"’')]}".contains(next)
|
||
}
|
||
|
||
/// The space to write before `mark` (`.some(nil)` for none), or nil to leave what is
|
||
/// there: at a line's start, after an opening quote or bracket or another ";" or ":"
|
||
/// ("::"), and for a mark typed
|
||
/// against a URL or an address. The second mark of "?!" goes against the first.
|
||
private static func spaceBefore(
|
||
_ mark: Character, out: [Character], pending: [Character], question: Character?
|
||
) -> Character?? {
|
||
guard let last = out.last, !last.isNewline, !"(«“[{".contains(last) else { return nil }
|
||
if "?!".contains(mark), "?!".contains(last) { return .some(nil) }
|
||
if ";:".contains(last) { return nil }
|
||
if pending.isEmpty {
|
||
let word = String(out.reversed().prefix { !$0.isWhitespace }.reversed())
|
||
if word.contains("://") || word.contains("@") || word.lowercased().hasPrefix("www.") { return nil }
|
||
}
|
||
return .some(mark == ":" ? noBreakSpace : question)
|
||
}
|
||
}
|