Hugging Face's logo

desert-ant-labs
/
tongue

Text Classification
ONNX
language-identification
language-detection
langid
on-device
edge-ai
mobile
browser
offline
multilingual
Model card Files
xet
 Community
tongue / THIRD_PARTY_NOTICES.md
bkicanao's picture
bkicanao
Update repository
b529947
preview code
|
Raw
History Blame Contribute Delete
4.04 kB

Third-party notices — tongue

tongue is trained from scratch and does not derive from any third-party model. The training and evaluation corpora below are licensed by their respective projects, and those licenses apply to that data. Nothing in the Desert Ant Labs Source-Available License overrides them.

Every source used for training is CC0, CC BY, or a permissive software license. Share-alike and non-commercial sources are excluded by policy, and the corpus build enforces the exclusion and records a provenance manifest of every file kept and dropped.

Training data

Tatoeba — sentence and link exports

  • Source: tatoeba.org — downloads.tatoeba.org/exports/
  • License: CC BY 2.0 FR
  • Attribution: © Tatoeba contributors, licensed under CC BY 2.0 FR
  • Use in tongue: the primary corpus. Sentences are windowed into 1–5 and 8-word spans, and their distinct tokens supply single-word training rows. The translation-link graph is used to split train/validation by translation family so a sentence and its translations cannot straddle the split.

Common Voice — sentence collections

  • Source: common-voice/common-voice (server/data)
  • License: CC0 1.0
  • Use in tongue: additional sentence-register text, strongest for languages where Tatoeba is thin (Catalan, Basque, Galician, Welsh, Swahili, Belarusian).
  • Exclusions: files derived from Wikipedia or Europarl are not used, as their upstream sources are share-alike. The fetcher drops any file matching *wiki* or *europarl* and writes the resulting keep/drop manifest.

Wikidata Lexemes

  • Source: wikidata.org lexeme dumps
  • License: CC0 1.0
  • Use in tongue: dictionary-register vocabulary rows (single words).

Hunspell dictionaries

  • Source: wooorm/dictionaries — English, Dutch, Lithuanian, Russian, Turkish, Persian
  • License: per dictionary (MIT / BSD / Apache-2.0 and similar permissive terms; see each dictionary's own license file)
  • Use in tongue: additional clean single-word vocabulary for those six languages.

Universal Dependencies treebanks

  • Source: universaldependencies.org
  • License: CC BY 4.0 (attribution required), verified per treebank
  • Treebanks used: UD_Spanish-AnCora, UD_Catalan-AnCora, UD_Finnish-FTB, UD_Italian-MarkIT, UD_Portuguese-Porttinari
  • Use in tongue: written-register sentences (news, reviews), taken from the # text = lines. Share-alike and non-commercial UD treebanks are excluded.

Evaluation only — never used for training

  • FLORES-200 — NLLB Team et al. — CC BY-SA 4.0 — held out; used for the neutral truncation benchmark.
  • WiLI-2018 — ODC-BY 1.0 — held out; tail-language coverage check.
  • eld benchmark — nitotm/efficient-language-detector — Apache-2.0 — held out; independent generalization check.

These sets are used to measure the model and are never part of the training corpus. Leipzig / Wortschatz corpora are excluded from training in every form (news, web, wiki, frequency lists, and mirrors), because another detector's published test set is drawn from that collection and training on it would make comparisons meaningless.

Comparison baselines

Numbers reported alongside tongue were produced by running these systems on the identical rows and language subsets. They are not redistributed here.