hex/ios/HEXMobile/AcknowledgementsView.swift
Ryan Vogel 10614acea7
iOS keyboard: Polish autocorrects with Hex's own model, word list and capitals (#2423)
* iOS keyboard: Polish autocorrects with Hex's own model and word list

Polish autocorrect today is only as good as iOS's Polish dictionary. Without Apple's word data,
that dictionary takes any letters for a word and guesses in English ("moze" -> "more"), and its
guesses rarely carry a word's marks. A TestFlight tester called it unusable. Now:
- HEXKeyboard/TypingModel/pl.hextm (150,000 words, 184,327 pairs, 3.60 MB) and pl.valid (525,084
  words, 8.29 MB), byte for byte what scripts/build-typing-model.py pl builds (#2401: NKJP
  n-grams, ParlaMint's check, PoliMorf). Bundling them is all it takes: the decoder reads Polish
  taps with the model (FullKeyboardSettings.typingModel), and HexWordList/HexOwnWords (#2404)
  answer what is a word from the model and pl.valid instead of the phone's dictionary.
- Settings › Spatial autocorrect says "English, Spanish, French and Polish only".
- The gate: typing-gate.py types Polish (QWERTY rows); accent-keys.json and critical-cases.json
  gain pl, and quality-gate.py now accepts a configured language without an Android model
  (it required equality). TypingGateReplayTests picks ł from l's long-press menu (ł has no
  decomposition to find its key).
- ios/README.md and the benchmark README give the numbers.

Gate (ios/scripts/typing-gate.py run ... pl, iPhone 17 simulator, iOS 27.2): 85.98% fewer word
errors than the raw taps (gate 70%), 0.22% of correctly typed words changed (gate 0.5%), 95.46%
of unambiguous marks restored (gate 85%), 5/5 critical cases: PASS.

Known gaps: no Polish capitals list, so a name typed in lowercase stays lowercase ("warszawa");
English + Polish bilingual typing and a real iPhone's timing are not measured.

* iOS keyboard: Polish names take their capital from Polish's own capitals list

Without a list, Hex's Polish own words (pl.hextm and pl.valid) left every name typed in
lowercase as typed: the gate's names-kept line read 0.00%, as with the phone's dictionary on the
simulator. pl.capitals (2,800 words, 36,472 bytes) is build-capitals.py's output in the format
fr.capitals and de.capitals use, read by HexWordList/HexOwnWords as theirs are.

build-capitals.py pl counts case from ParlaMint 5.0's Polish transcripts (CC BY 4.0, already one
of Polish's credited inputs; checked against its SHA-256), leaving out each sentence's first word,
which any word can start. A word PoliMorf (BSD 2-Clause) lists in lowercase is an ordinary word and
is held at 94% (listed, never capitalised on its own), as SCOWL's lowercase words are in English:
"tom" is a volume, "marszałek" a title, "polska" an adjective. Otherwise the rules are the other
languages': model words from 50%, names outside the model from 90% and once per million words.

Gate (typing-gate.py run ... pl, iPhone 17 simulator, iOS 27.2), names kept / capitals added on
the all-lowercase pass:
- phone's dictionary (no Polish files): 0.00% / 0.00% (and -1.36% errors, FAIL);
- pl.hextm + pl.valid: 0.00% / 0.00%;
- pl.hextm + pl.valid + pl.capitals: 10.65% / 0.00%, with 86.01% fewer errors, 0.22% changed,
  95.46% unambiguous marks, 5/5 critical cases (PASS).
Rebuilt on exos from the pinned inputs: SHA-256 34bfa18b..., the committed file.
2026-11-22 14:45:55 -05:00

205 lines
11 KiB
Swift
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

import SwiftUI
/// Settings › Acknowledgements: the data Hex ships, with the notices its licences ask for.
struct AcknowledgementsView: View {
@Environment(\.dismiss) private var dismiss
var body: some View {
NavigationStack {
ScrollView {
VStack(alignment: .leading, spacing: 20) {
ForEach(Self.shown, id: \.title) { entry in
VStack(alignment: .leading, spacing: 6) {
Text(entry.title)
.font(.system(size: 15, weight: .medium))
.foregroundStyle(HexTheme.textBase)
Text(entry.body)
.font(.system(size: 13))
.foregroundStyle(HexTheme.textMuted)
.textSelection(.enabled)
}
}
}
.padding(20)
.frame(maxWidth: .infinity, alignment: .leading)
}
.background(HexTheme.bgBase)
.navigationTitle("Acknowledgements")
.navigationBarTitleDisplayMode(.inline)
.toolbar {
ToolbarItem(placement: .confirmationAction) {
Button("Done") { dismiss() }
}
}
}
}
struct Entry {
let title: String
let body: String
}
/// Spatial autocorrect's word lists (`scripts/build-typing-model.py`), the keyboard's
/// never-suggest list (`OffensiveWords`), and the speech models Hex downloads (Parakeet V2,
/// and Parakeet Ultra for Experiments › Multilingual model); keep in step with ios/README.md.
static let entries: [Entry] = [
Entry(
title: "Google Books Ngram data",
body: """
Spatial autocorrect's word and word-pair frequencies, German's capitalisation \
shares and French's list of words outside its model come from the Google Books Ngram Viewer data, version 3 (2020), by Google, \
licensed under Creative Commons Attribution 3.0 (creativecommons.org/licenses/by/3.0). \
Hex sums the counts from 1980 to 2019 and keeps only words in the lists below.
"""
),
Entry(
title: "SCOWL",
body: """
English words are checked against SCOWL (Spell Checker Oriented Word Lists) and \
VarCon by Kevin Atkinson, sizes up to 70 (wordlist.aspell.net).
Copyright 2000-2018 by Kevin Atkinson
Permission to use, copy, modify, distribute and sell these word lists, the \
associated scripts, the output created from the scripts, and its documentation \
for any purpose is hereby granted without fee, provided that the above copyright \
notice appears in all copies and that both that copyright notice and this \
permission notice appear in supporting documentation. Kevin Atkinson makes no \
representations about the suitability of this array for any purpose. It is \
provided "as is" without express or implied warranty.
SCOWL's inflections come from WordNet 1.6, Copyright 1997 by Princeton \
University. All rights reserved. THIS SOFTWARE AND DATABASE IS PROVIDED "AS IS" \
AND PRINCETON UNIVERSITY MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR \
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PRINCETON UNIVERSITY MAKES NO \
REPRESENTATIONS OR WARRANTIES OF MERCHANTABILITY OR FITNESS FOR ANY PARTICULAR \
PURPOSE OR THAT THE USE OF THE LICENSED SOFTWARE, DATABASE OR DOCUMENTATION WILL \
NOT INFRINGE ANY THIRD PARTY PATENTS, COPYRIGHTS, TRADEMARKS OR OTHER RIGHTS. The \
name of Princeton University or Princeton may not be used in advertising or \
publicity pertaining to distribution of the software and/or database.
Its other sources (12Dicts, Moby Words II, and Brian Kelk's UK English Wordlist) \
are in the public domain.
"""
),
Entry(
title: "List of Dirty, Naughty, Obscene, and Otherwise Bad Words",
body: """
The keyboard's suggestion filter uses the List of Dirty, Naughty, Obscene, and \
Otherwise Bad Words (LDNOOBW), © 2012–2020 Shutterstock, Inc., and contributors: \
its English, Spanish, German and Norwegian lists, revision 5faf2ba \
(github.com/LDNOOBW/List-of-Dirty-Naughty-Obscene-and-Otherwise-Bad-Words), \
licensed under Creative Commons Attribution 4.0 International \
(creativecommons.org/licenses/by/4.0). Hex lowercased and classified the entries, \
omitted ordinary, medical and search-only terms, and added common slurs, \
inflections and spelling variants.
"""
),
Entry(
title: "Parakeet V2",
body: """
Hex dictates English with NVIDIA's Parakeet TDT 0.6B v2 \
(huggingface.co/nvidia/parakeet-tdt-0.6b-v2), licensed under Creative Commons \
Attribution 4.0 International (creativecommons.org/licenses/by/4.0), downloaded as \
FluidInference's Core ML conversion (huggingface.co/FluidInference/\
parakeet-tdt-0.6b-v2-coreml, CC BY 4.0), which stores the encoder's weights at \
reduced precision; Hex uses its files unchanged.
"""
),
Entry(
title: "Parakeet Ultra",
body: """
Experiments › Multilingual model downloads Parakeet Ultra by Moondream \
(huggingface.co/moondream/parakeet-ultra), a post-training of NVIDIA's Parakeet \
TDT 0.6B v3 (huggingface.co/nvidia/parakeet-tdt-0.6b-v3), both licensed under \
Creative Commons Attribution 4.0 International \
(creativecommons.org/licenses/by/4.0). Core ML conversion by FluidInference \
(huggingface.co/FluidInference/parakeet-ultra-coreml, revision 95eaa59), which \
stores the encoder's weights as 8-bit integers and re-exports the decoder and \
joint; Hex uses its files unchanged.
"""
),
Entry(
title: "RLA-ES",
body: """
Spanish words are checked against RLA-ES, the Spanish spelling dictionaries for \
LibreOffice by Santiago Bosio and contributors (Spain and Mexico), used under the \
Mozilla Public License 1.1 (github.com/sbosio/rla-es).
"""
),
]
/// The entries the app shows: Polish's once the keyboard bundles its model (`pl.hextm`).
static var shown: [Entry] {
entries + (keyboardBundles("pl.hextm") ? polishEntries : [])
}
private static func keyboardBundles(_ file: String) -> Bool {
guard let plugins = Bundle.main.builtInPlugInsURL else { return false }
let path = plugins.appendingPathComponent("HEXKeyboard.appex").appendingPathComponent(file).path
return FileManager.default.fileExists(atPath: path)
}
/// Polish's typing model (`scripts/build-typing-model.py pl`, its inputs in
/// `POLISH_INPUTS`); keep in step with ios/README.md.
static let polishEntries: [Entry] = [
Entry(
title: "National Corpus of Polish n-grams",
body: """
Polish word and word-pair frequencies come from the n-grams of the National Corpus \
of Polish's balanced subcorpus (NKJP; Institute of Computer Science, Polish Academy \
of Sciences; zil.ipipan.waw.pl/NKJPNGrams), made available by its authors under a \
Creative Commons Attribution licence (creativecommons.org/licenses/by/; the page \
names no version). Hex keeps the words PoliMorf \
lists, leaves out spellings typed without their marks, and stores the frequencies \
rounded.
"""
),
Entry(
title: "ParlaMint",
body: """
Which Polish spellings are missing their marks, and which words are names, is checked \
against the Polish part of \
ParlaMint 5.0, the Sejm and Senate transcripts 2015-2022 (Erjavec et al., \
Multilingual comparable corpora of parliamentary debates ParlaMint 5.0, CLARIN.SI, \
hdl.handle.net/11356/2004), licensed under Creative Commons Attribution 4.0 \
International (creativecommons.org/licenses/by/4.0).
"""
),
Entry(
title: "PoliMorf",
body: """
Polish words are checked against PoliMorf (2018-07-23; Woliński, Miłkowski, \
Ogrodniczuk, Przepiórkowski, Szałkiewicz; hdl.handle.net/11321/577).
Portions copyright © 2011 Marek Futrega, Piotr Gackiewicz, Mirosław Prywata, \
Włodzimierz Macewicz, Marcin Miłkowski
Portions copyright © 2007–2018 Zygmunt Saloni, Włodzimierz Gruszczyński, \
Marcin Woliński, Robert Wołosz, Danuta Skowrońska
Portions copyright © 2011–2018 Institute of Computer Science, Polish Academy of Sciences
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, \
are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list \
of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this \
list of conditions and the following disclaimer in the documentation and/or other \
materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY COPYRIGHT HOLDERS “AS IS” AND ANY EXPRESS OR IMPLIED \
WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF \
MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT \
SHALL COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, \
INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT \
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR \
PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, \
WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR \
OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF \
THE POSSIBILITY OF SUCH DAMAGE.
"""
),
]
}