Upload folder using huggingface_hub

Browse files

Files changed (9) hide show

README.md +241 -242
__init__.py +4 -0
config.json +67 -47
configuration_dihya.py +86 -0
export.stats.json +64 -0
model.safetensors +3 -0
modeling_dihya.py +320 -0
tokenizer.json +341 -0
tokenizer_config.json +7 -0

README.md CHANGED Viewed

@@ -7,6 +7,7 @@ language:
 - tzm
 - shy
 license: apache-2.0
 tags:
 - berber
 - amazigh
@@ -17,7 +18,6 @@ tags:
 - tamazight
 - shawiya
 - language-identification
-- metric-learning
 - conformal-prediction
 - low-resource
 pipeline_tag: text-classification
@@ -29,335 +29,334 @@ model-index:
   results:
   - task:
       type: text-classification
-      name: Berber language identification across six Latin-script varieties
     dataset:
-      type: agbalu/BerberLID-Eval
-      name: Held-out benchmark, 1,500 sentences balanced across six varieties
     metrics:
     - type: accuracy
       value: 0.8552
-      name: Overall accuracy (7 classes)
     - type: f1
       value: 0.8562
-      name: Macro-F1 over six varieties and NOT_AMAZIGH
 ---
 # Dihya-5M
-A 5.19M-parameter byte-level Conv-Transformer for **Berber Language Identification**, trained from
-scratch to distinguish six Latin-script Amazigh varieties against each other and an explicit
-`NOT_AMAZIGH` rejection class: Kabyle (`kab_Latn`), Tashelhit (`shi_Latn`), Tarifit (`rif_Latn`),
-Tamasheq (`taq_Latn`), Central Atlas Tamazight (`tzm_Latn`), and Shawiya (`shy_Latn`).
-It reaches **85.52% overall accuracy** and **0.8562 macro-F1** on a balanced, source-stratified
-held-out benchmark of 1,500 sentences — against 72.40% accuracy and 0.7023 macro-F1 for an
-unstratified baseline. The decisive gain is on **Shawiya: from 0.2222 to 0.7923 F1, a 3.6×
-improvement on the language the baseline abandons.**
-It ships with pre-calibrated **split-conformal prediction sets** (98.78% empirical coverage, mean
-set size 1.098) for pipelines where a silent misclassification is worse than a widened label.
-It is, as far as we can establish, **the first multi-dialect language identifier published with
-conformal safety guarantees for the Berber language family.**
-Named after **Dihya** (also known as *al-Kāhina*, c. 650–702 CE), Amazigh queen and military
-commander of the Aurès Mountains who unified the Berber tribes across North Africa.
-## The task
-Berber language identification is not classifying typologically distant languages. The six varieties
-here share 80–90% of their core vocabulary: *aman* (water), *argaz* (man), *taddart* (village),
-*awal* (word). A naive model that reads shared pan-Berber roots will be confidently wrong on
-anything short of a long, idiom-dense sentence.
-| Variety | Code | Diagnostic signal |
-|---|---|---|
-| Kabyle | `kab_Latn` | Spirantized stops `ḍ ṭ ṛ`, Schwa written `e`, interrogative `amek` |
-| Tashelhit | `shi_Latn` | Unspirantized stops, intensifier `bahra`, vocative `aytma` |
-| Tarifit | `rif_Latn` | Spirantized stops, interrogative `mayemmi` / `mamec`, clitics `nnecm` |
-| Tamasheq | `taq_Latn` | Explicit short vowels `ə`, Saharan particle `ahal`, root `əmərəw` |
-| Central Atlas | `tzm_Latn` | Preposition `g` (not `deg`), root `wammas`, shared Morocco phonotactics |
-| Shawiya | `shy_Latn` | Relative `mḍa`, double clitics `acek-acem`, Aurès toponym `Awras` |
-| Non-Amazigh | `NOT_AMAZIGH` | French, English, Arabic, Spanish, Latin, other |
-The three structural traps that collapse standard models on this task:
-**The long-tail imbalance trap.** Kabyle has tens of thousands of open-corpus lines; Shawiya has a
-few hundred. Under empirical risk minimisation the gradient from Kabyle overwhelms Shawiya, driving
-its recall to 22% or below.
-**The domain confound trap.** If one variety is represented entirely by Bible translations and
-another entirely by dictionary glosses, a model learns biblical syntax vs. internet vocabulary —
-not dialect phonotactics. It scores well on training data and collapses when the domain shifts.
-**The subword fragmentation trap.** BPE tokenizers trained on Kabyle fragment cognates in Tamasheq
-or Shawiya into arbitrary byte-pair shards, destroying the morphological alignment that carries the
-discriminative signal.
-## Results
-**1,500 sentences drawn from the held-out benchmark** `data/processed/bench/lid-eval.jsonl` — 250
-per variety, disjoint from train (77,756) and dev (4,093), across balanced multi-domain sources.
-| system | accuracy | macro-F1 |
-|---|---|---|
-| **Dihya-5M** | **85.52%** | **0.8562** |
-| unstratified baseline | 72.40% | 0.7023 |
-Reproduce both rows in about fifteen seconds on a laptop, no GPU:
-```bash
-make lid TASK=evaluate
-```
-**The second row is the row that makes the first one mean something.** A classifier that does nothing
-unusual achieves 72.40% by getting Tamasheq and Tarifit right and abandoning Shawiya entirely.
-Dihya-5M's contribution is the +13.12 points above that floor, and the honest way to read 0.8562
-macro-F1 is as a **uniform lift across the tail, not a drift in the head classes**.
-| label | support | P | R | F1 | baseline F1 |
-|---|---|---|---|---|---|
-| `taq_Latn` | 250 | 0.888 | 0.948 | **0.917** | 0.943 |
-| `rif_Latn` | 250 | 0.865 | 0.925 | **0.894** | 0.777 |
-| `shi_Latn` | 250 | 0.885 | 0.885 | **0.885** | 0.874 |
-| `shy_Latn` | 250 | 0.785 | 0.800 | **0.792** | 0.222 |
-| `tzm_Latn` | 250 | 0.775 | 0.784 | **0.780** | 0.751 |
-| `kab_Latn` | 250 | 0.762 | 0.754 | **0.758** | 0.648 |
-| `NOT_AMAZIGH` | 250 | 0.970 | 0.965 | **0.967** | 0.000 |
-**Tamasheq is the single class where the baseline has an edge.** The baseline's 0.943 rests on
-Tamasheq being lexically the most distinct variety — Saharan vocabulary, explicit short vowels —
-so a bag-of-words signal is already strong there. Dihya-5M's 0.917 trades one point on the easy
-class for 57 points on Shawiya.
-**Shawiya is the deciding result.** 0.2222 baseline F1 means the variety is functionally
-unclassified. 0.7923 means 80% recall on held-out text, from Group-DRO reweighting that refuses
-to let the optimiser satisfy the average by ignoring the tail.
-**The conformal prediction sets carry their own guarantee.** Calibrated on 4,093 development
-sentences at target coverage 99.0%, empirical coverage on the test set is **98.78%**, with a mean
-set size of **1.098 labels**. A sentence of ambiguous shared pan-Berber proverb receives a set of
-two or three labels; a sentence with a clear dialectal marker receives a singleton. The guarantee
-is distribution-free; it does not depend on the model's calibration being correct.
-**The checkpoint shipped is `best.pt`**, selected on the development composite score
-½(Macro-F1 + Worst-Group Recall) — ensuring the test set was not touched during model selection.
-## Intended use
-Identifying Berber varieties and filtering non-Amazigh text in uncurated data pipelines:
-- **Corpus construction**: screening web scrapes and OCR output to tag each sentence with its
-  dialect and discard non-Berber lines before they enter training corpora for downstream models.
-- **Pipeline routing**: sitting before dialect-specific models to route input to the appropriate
-  one — orthography standardisation ([`agbalu/Boulifa-48M`](https://huggingface.co/agbalu/Boulifa-48M)),
-  ASR post-processing ([`agbalu/Belaid-31M`](https://huggingface.co/agbalu/Belaid-31M)), or TTS
-  ([`agbalu/Matoub-82M`](https://huggingface.co/agbalu/Matoub-82M)).
-- **Conformal filtering**: using prediction sets rather than point predictions when a misclassified
-  dialect would corrupt a downstream corpus.
-**Not suitable for**: classifying Tifinagh or Arabic-script Berber text; making decisions about
-individuals on the basis of their regional variety; or distinguishing sub-varieties within any one
-of the six classes listed.
-## Usage
-`torch` and `agbalu.lid.Identifier`, nothing else:
 ```python
-from pathlib import Path
-from agbalu.lid.infer import Identifier
-identifier = Identifier.load(Path("artifacts/runs/dihya"))
-predictions = identifier.predict([
-    "Azul fell-awen, amek tettilim?",
-    "Mayemmi ur teccid ticemrarin nnecm?",
-    "Bonjour tout le monde.",
-])
-for pred in predictions:
-    print(pred.language, pred.confidence, pred.prediction_set)
-# kab_Latn 0.9987 ('kab_Latn',)
-# rif_Latn 1.0000 ('rif_Latn',)
-# NOT_AMAZIGH 0.9994 ('NOT_AMAZIGH',)
 ```
-Or from the command line:
-```bash
-make lid TEXT="Mayemmi ur teccid ticemrarin nnecm a gma?"
-# rif_Latn 1.0000 set=('rif_Latn',)
 ```
-**Prediction sets widen gracefully along the continuum.** A sentence of shared pan-Berber vocabulary
-without a variety-specific marker will receive a set of two or three labels rather than a forced
-singleton with spurious confidence.
-**Input is Latin script.** Bytes are the vocabulary; a Tifinagh byte sequence produces a label, but
-that label has not been evaluated and should not be trusted.
 ## Architecture
 | | |
 |---|---|
-| Parameters | **5,187,712** |
 | Encoder layers | 6 |
 | Hidden / feed-forward | 256 / 704 (SwiGLU) |
-| Attention heads / head size | 8 / 32 |
-| Positions | rotary (RoPE) |
-| Conv stem | three parallel 1D depthwise-separable convolutions, kernels 3, 5, 7 |
-| Pooling | parametric attentive pooling |
-| Classification head | cosine head with LDAM additive margins |
-| Contrastive head | 2-layer projection, 256 → 128, memory bank 2,048 |
-| Vocabulary | 258 slots (256 raw UTF-8 bytes + PAD + UNK) |
-| Dropout | 0.10 |
-**The byte vocabulary.** The orthographies that separate Kabyle from Shawiya from Tamasheq differ in
-exactly the characters where the scripts diverge — `ɛ ɣ ḥ ḍ ṣ ṭ ẓ ṛ` are two or three UTF-8 bytes
-each. A byte model sees these distinctions as a short byte sequence rather than as one token it may
-never have seen in a minority language's training shard.
-**The convolutional stem.** Three depthwise-separable 1D convolutions (kernels 3, 5, 7) run in
-parallel over the byte embeddings before the encoder's first layer. Kernel 3 covers the grapheme
-trigrams that spirantization spans; kernel 5 covers consonantal root skeletons; kernel 7 covers
-the clitic complexes that are among the clearest regional markers.
-**Attentive pooling.** Replaces uniform averaging with a learned scalar score per byte position,
-so the representation concentrates on the 10–15% of positions carrying diagnostic morphology
-rather than the shared vowel sequences that surround it.
-**LDAM margins and Group-DRO.** Both operate on the same 20 (language, domain) cells. LDAM sets
-the angular margin for each class as $m_k = C / n_k^{1/4}$, widening the decision boundary for
-low-resource varieties. Group-DRO tracks the worst-performing cell and exponentiates its loss
-weight at each step, preventing the optimiser from satisfying the average by abandoning Shawiya.
-## Training
-| | |
-|---|---|
-| training rows | 77,756 across 20 (language, domain) cells |
-| epochs | 8 |
-| checkpoint | step 2,898 of 5,176, selected on development composite score |
-| development score there | 0.9363 macro-F1, 0.8125 worst-group recall (½ = 0.8744) |
-| hardware | one A10G, ~18.3 minutes |
-Selection is on the composite score ½(Macro-F1 + Worst-Group Recall) and not on loss, and here
-that is not a formality. The final-epoch checkpoint (step 5,176) scores 97.87% dev accuracy — two
-points above step 2,898 — but its worst-group recall falls from 81.25% to 68.75% on the
-Shawiya/lexicon cell. Selecting on accuracy would have published a model that recovered Shawiya
-halfway and called it done.
-**Two-phase schedule.** The first 50% of steps (0–2,588) train on plain cross-entropy. At step
-2,588, the LDAM margins switch on at `margin_scale = 0.5` and the Group-DRO exponentiated weights
-activate. This deferred scheme is from the LDAM paper and measured here: margins from step 0 drove
-the loss to 4.01 at step 120 where the same run without them reached 0.76. The margins are useful
-once the features exist and are an obstacle before that.
-**Contrastive head.** A 2-layer projection head (256 → 128) adds a supervised contrastive loss
-(weight 0.5) alongside the classification objective. Hard negatives are drawn from a memory bank of
-2,048 embeddings, sampling *different languages in the same domain* — which forces the trunk to
-discard topical and stylistic features and learn pure dialectal phonotactics.
-## Calibration
-Split-conformal calibration was run over the full 4,093-sentence development set after model
-selection, with target marginal coverage $1 - \alpha = 0.99$.
 | | |
 |---|---|
-| calibration size | 4,093 sentences |
-| target coverage | 99.0% |
-| empirical coverage on test | **98.78%** |
-| calibration quantile ($\hat{q}$) | 0.9979 |
-| mean prediction set size | **1.098** |
-The guarantee is exchangeability, not distributional assumptions on the model: if the development
-and test sets are exchangeable draws from the same source, the coverage holds regardless of how
-miscalibrated the raw softmax scores are.
 ## Limitations
-**Dialect continuum ambiguity on shared vocabulary.** A short sentence of pan-Berber core vocabulary
-without a variety-specific marker cannot be deterministically assigned to one variety — and should
-not be. The conformal sets flag this with size > 1; calling one label correct in that case would be
-over-claiming what the text contains.
-**Tifinagh and Arabic script are out of scope.** Dihya-5M is evaluated on Latin script only. Bytes
-from a Tifinagh sentence will produce a prediction, but that prediction has not been measured and
-should not be used.
-**Short inputs lose diagnostic context.** A single word — especially a shared cognate — can arrive
-as `NOT_AMAZIGH` or as any variety. The model was fitted on natural sentences; single-word queries
-are not the input shape it was built for.
-**One language per sentence.** Code-switching between Berber varieties within a sentence is not
-handled. A sentence mixing Kabyle and French will receive a single class label, which may be wrong.
-**No safety evaluation of any kind** has been performed.
-## What was not measured
-- **No inter-annotator ceiling exists for Berber variety attribution.** On sentences of shared
-  pan-Berber vocabulary without a clear regional marker, fluent native speakers disagree — so no
-  figure here can be read as a fraction of what is attainable.
-- **No audio or ASR evaluation.** Performance was measured on clean orthographic text. Transcribed
-  speech carries recognition errors that may degrade classification on short sentences.
-- **Out-of-domain generalisation was not benchmarked** outside the six source types used in
-  `lid-eval.jsonl`.
 ## Files
-| file | size | description |
-|---|---|---|
-| `best.pt` | 20.8 MB | weights + config dict, selected on dev composite score (step 2,898) |
-| `config.json` | 1.1 KB | model architecture hyperparameters |
-| `calibration.json` | 890 B | split-conformal calibration quantile ($\hat{q} = 0.9979$) |
-| `held-out-report.json` | 4.2 KB | full evaluation breakdown on the 1,500-sentence test set |
-The checkpoint holds the model state dict and the config dict, nothing else. No optimiser state,
-no scheduler state, no training curve. Training cannot be resumed from the published file.
 ## Reproduction
 ```bash
-make modal-lid EPOCHS=8             # train on Modal A10G (~18 minutes)
-make modal-lid-pull                 # download best checkpoint to artifacts/runs/dihya/
-make lid TASK=evaluate              # reproduce the 1,500-sentence benchmark
-make lid TEXT="Mayemmi ur teccid ticemrarin nnecm?"
-make release REPO=dihya             # stage for Hub upload
 ```
-The evaluation scores Dihya-5M and the baseline **in the same pass over the same rows**, because a
-baseline computed elsewhere on a different sample is not a comparison.
 ## The name
-**Dihya** (also known as *al-Kāhina*, c. 650–702 CE) was an Amazigh queen, religious figure, and
-military commander of the Jarawa tribe, from the Aurès Mountains of what is now eastern Algeria —
-the Shawiya homeland.
-She led Berber resistance across Numidia and Ifriqiya and unified disparate Amazigh tribes across
-the same geographic continuum this model is trained to recognise: from the Atlantic coast to the
-Sahara, from Tarifit in the north to Tamasheq in the south. The language boundary the model draws
-is the one she crossed.
-The naming is homage; it implies no endorsement by anyone.
 ## Citation
 ```bibtex
 @software{agbalu_dihya_2026,
-  title  = {Dihya-5M: language identification for the Berber dialect continuum},
   author = {AƔBALU},
   year   = {2026},
   url    = {https://huggingface.co/agbalu/Dihya-5M},
-  note   = {5,187,712 parameters; 85.52% accuracy and 0.8562 macro-F1 on 1,500 held-out sentences}
 }
 ```
 ## Licence
-**Apache-2.0** on the weights and the code.
 Part of [AƔBALU](https://huggingface.co/agbalu), a Kabyle and Amazigh corpus and model collection.
-The naming is homage.

 - tzm
 - shy
 license: apache-2.0
+library_name: transformers
 tags:
 - berber
 - amazigh
 - tamazight
 - shawiya
 - language-identification
 - conformal-prediction
 - low-resource
 pipeline_tag: text-classification
   results:
   - task:
       type: text-classification
+      name: Berber language identification, six Latin-script varieties plus rejection
     dataset:
+      type: berber-lid-heldout
+      name: AƔBALU Berber LID held-out set (1,050 sentences, 150 per class)
     metrics:
     - type: accuracy
       value: 0.8552
+      name: Accuracy over 7 classes
     - type: f1
       value: 0.8562
+      name: Macro-F1 over 7 classes
 ---
 # Dihya-5M
+A 5.19M-parameter byte-level Conv-Transformer that separates six Latin-script Amazigh varieties
+from each other and from an explicit `NOT_AMAZIGH` rejection class: Kabyle (`kab_Latn`),
+Tashelhit (`shi_Latn`), Tarifit (`rif_Latn`), Tamasheq (`taq_Latn`), Central Atlas Tamazight
+(`tzm_Latn`) and Shawiya (`shy_Latn`).
+On a balanced, source-stratified held-out set of 1,050 sentences it reaches **85.52% accuracy and
+0.8562 macro-F1** over the seven classes. Scored on the six Berber classes alone — the label set
+the public identifiers can be asked about — it reaches **84.44% accuracy**, against **41.78% for
+GlotLID and 28.33% for NLLB's `lid218e`** measured in the same pass over the same rows.
+It ships with a split-conformal quantile, so it can return a *set* of labels rather than one, for
+pipelines where a silent misclassification costs more than a widened answer.
+## The task
+These are not typologically distant languages. The six varieties share much of a core lexicon —
+*aman* (water), *argaz* (man), *awal* (word) — and one Latin orthography. The evidence that
+separates them is affixal, phonotactic and clitic, and a short sentence of shared vocabulary
+often carries none of it.
+That matters beyond the benchmark. NLLB's own identifier mined 90.1% of this project's parallel
+corpus, and it labels the siblings Kabyle: **91.3% of Shawiya, 80.7% of Tarifit, 79.3% of
+Tashelhit and 73.3% of Central Atlas Tamazight**, measured here. Datasets labelled "Kabyle" or
+"Tamazight" inherit that.
+## Results
+**1,050 sentences, 150 per class**, drawn round-robin across each language's sources so that no
+class is one corpus wearing a language's name. Disjoint from train (77,756) and dev (4,093).
+### Against the public identifiers
+All three scored in one pass over the same 900 Berber rows. `NOT_AMAZIGH` is excluded here: it is
+a decision this model makes and not a language GlotLID or `lid218e` has a label for.
+| system | accuracy | classes it can name | macro-F1 over those |
+|---|---|---|---|
+| **Dihya-5M** | **84.44%** | **6 of 6** | 0.8493 |
+| GlotLID | 41.78% | 3 of 6 | 0.6837 |
+| NLLB `lid218e` | 28.33% | 2 of 6 | 0.6022 |
+The last column is three different statistics — each system's macro is taken over the classes it
+can name, so GlotLID's 0.6837 averages three classes and Dihya's 0.8493 averages six. It is
+reported that way because averaging a system over labels absent from its inventory measures the
+inventory, not the discrimination. The accuracy column is the comparable number.
+**The quantity that matters downstream is how much sibling text ends up labelled Kabyle**, since
+that is what a filter keyed on `kab` silently ingests:
+| true language | → Kabyle, Dihya | → Kabyle, GlotLID | → Kabyle, `lid218e` |
+|---|---|---|---|
+| Tashelhit | **2.0%** | 17.3% | 79.3% |
+| Tarifit | **6.7%** | 90.0% | 80.7% |
+| Tamasheq | **7.3%** | 0.0% | 6.7% |
+| Central Atlas Tamazight | **16.0%** | 66.7% | 73.3% |
+| Shawiya | **19.3%** | 96.0% | 91.3% |
+GlotLID sends no Tamasheq to Kabyle because it has a good Tamasheq label, and it is the one place
+a public system is not the weaker option — its Tamasheq F1 is 0.891 against Dihya's 0.921, close
+enough that the ordering should not be leaned on.
+### Per class, all seven
+The seven-class report, which is what `held-out-report.json` in this repository holds.
+| label | support | precision | recall | F1 |
+|---|---|---|---|---|
+| `NOT_AMAZIGH` | 150 | 0.949 | 0.987 | 0.967 |
+| `taq_Latn` | 150 | 1.000 | 0.847 | 0.917 |
+| `rif_Latn` | 150 | 0.916 | 0.873 | 0.894 |
+| `shi_Latn` | 150 | 0.811 | 0.973 | 0.885 |
+| `shy_Latn` | 150 | 0.936 | 0.687 | 0.792 |
+| `tzm_Latn` | 150 | 0.952 | 0.660 | 0.780 |
+| `kab_Latn` | 150 | 0.626 | 0.960 | 0.758 |
+**Kabyle's precision is the lowest number here, and it is the one to read first.** At 0.626 the
+model still over-assigns Kabyle — it recovers 96.0% of the Kabyle and pays for it by pulling in
+sibling text. That is the same failure direction as the systems above, an order of magnitude
+smaller. Anyone building a Kabyle-only corpus should treat a `kab_Latn` label as a filter, not as
+a verdict.
+**Recall on Tamazight and Shawiya is the weak side.** 0.660 and 0.687 mean a third of each is
+missed, mostly to Kabyle and Tashelhit.
+### Where it is worst
+Selection and reporting are on `(language, domain)` cells, not on classes, because a class average
+hides a domain. Two cells are worth naming:
+| cell | support | recall |
+|---|---|---|
+| `shy_Latn/interface` | 3 | 0.000 |
+| `tzm_Latn/interface` | 61 | 0.410 |
+**`tzm_Latn/interface` is a real defect**: software-localisation strings in Central Atlas
+Tamazight are short, formulaic, and share their register with the Tashelhit and Kabyle
+localisations they were translated alongside. Two in five are missed.
+**`shy_Latn/interface` at 0.000 is three examples**, and three examples do not measure anything.
+It is listed because it is the worst cell in the file and omitting it would make the file
+disagree with the card, not because it supports a conclusion.
+### Conformal prediction sets
+Calibrated by split conformal at target coverage 99.0%. The quantile is fitted on one half of the
+development set and coverage measured on the other, because measuring coverage on the rows that
+set the quantile reports the fit rather than the guarantee.
+| | |
+|---|---|
+| fitted on | 2,047 dev sentences |
+| coverage measured on | 2,046 held-out dev sentences |
+| target coverage | 99.0% |
+| empirical coverage | **98.78%** |
+| quantile `q̂` | 0.9979 |
+| mean set size | **1.098 labels** |
+**This is measured on development data, not on the 1,050-sentence test set.** The exchangeability
+the guarantee rests on holds between the two halves of dev; whether it holds against the test set,
+whose sources are stratified differently, is not established here.
+## Usage
 ```python
+from transformers import AutoModelForSequenceClassification
+model = AutoModelForSequenceClassification.from_pretrained(
+    "agbalu/Dihya-5M", trust_remote_code=True
+)
+for row in model.identify([
+    "Azul fell-awen, amek tettilim ass-a?",
+    "Mayemmi ur teccid ticemrarin nnecm a gma?",
+    "Bonjour tout le monde, comment allez-vous?",
+]):
+    print(row["language"], round(row["confidence"], 4), row["prediction_set"])
+# kab_Latn 0.9959 ('kab_Latn', 'shy_Latn')
+# rif_Latn 0.9998 ('rif_Latn',)
+# NOT_AMAZIGH 0.9997 ('NOT_AMAZIGH',)
 ```
+`identify` applies the logit adjustment and the conformal quantile that the numbers above were
+measured with; both travel in `config.json`. The plain `forward` returns raw cosine logits and
+does neither, so a pipeline that reads `logits.argmax()` is scoring a different classifier than
+the one this card describes.
+The tokenizer is the 256 UTF-8 byte values and is only needed for batching:
+```python
+from transformers import AutoModelForSequenceClassification, AutoTokenizer
+tokenizer = AutoTokenizer.from_pretrained("agbalu/Dihya-5M")
+model = AutoModelForSequenceClassification.from_pretrained(
+    "agbalu/Dihya-5M", trust_remote_code=True
+)
+batch = tokenizer(["Azul fell-awen"], return_tensors="pt", padding=True, truncation=True)
+logits = model(**batch).logits
 ```
+**Prediction sets widen where the text is genuinely ambiguous.** The Kabyle line above returns
+`('kab_Latn', 'shy_Latn')` — Shawiya and Kabyle are adjacent on the continuum and that greeting
+carries no marker separating them. A forced singleton there would be a guess with a confidence
+attached.
+## Intended use
+Screening and routing text in corpus pipelines: tagging web scrapes and OCR output by variety,
+discarding non-Berber lines, and routing input to variety-specific models. Use the prediction set
+rather than the argmax where a wrong label would enter a training corpus.
+**Not suitable for** Tifinagh or Arabic-script text, for distinguishing sub-varieties inside any
+of the six classes, or for any decision about a person on the basis of the variety they write.
 ## Architecture
 | | |
 |---|---|
+| Parameters, trained | 5,187,712 |
+| Parameters, published | 5,089,408 |
 | Encoder layers | 6 |
 | Hidden / feed-forward | 256 / 704 (SwiGLU) |
+| Attention heads | 8 × 32, rotary positions |
+| Conv stem | three parallel depthwise-separable 1D convolutions, kernels 3, 5, 7 |
+| Pooling | attentive (learned score per position) |
+| Head | cosine, scale 24.0 |
+| Vocabulary | 258 (256 UTF-8 bytes, `[PAD]`, `[UNK]`) |
+| Maximum input | 256 bytes |
+The published model is smaller than the trained one because the contrastive projection head
+(98,304 parameters) shapes the trunk during training and no forward pass reads it at inference.
+**Bytes, not characters or subwords.** The orthographies differ in exactly the characters where
+they diverge — `ɣ ɛ ḥ ḍ ṣ ṭ ẓ ṛ ṯ` are two or three UTF-8 bytes each. A byte vocabulary sees those
+as a short sequence rather than as one token a minority variety's training shard may never
+contain. Truncation is on bytes and may cut a character in half, which is deliberate: text arrives
+already truncated at inference.
+**Three kernel widths** because the evidence sits at three scales — a grapheme cluster (`ţ` against
+`t`), an affix (the `u-`/`w-` annexation), and a clitic chain.
+## Training
 | | |
 |---|---|
+| training rows | 77,756 across 20 `(language, domain)` cells |
+| dev rows | 4,093 |
+| schedule | 8 epochs, 5,176 steps; batch 12 cells × 10 rows |
+| published checkpoint | step 2,898, selected on dev |
+| dev score there | 0.9363 macro-F1, 0.8125 worst-group recall |
+| hardware | one A10G |
+Three objectives address three separate failures of an earlier build that read 98.26% on a dev
+split drawn from the training sources and 76.27% on held-out text:
+**LDAM margins** for the long tail — `m_k = C / n_k^(1/4)`, from
+[Cao et al., NeurIPS 2019](https://arxiv.org/abs/1906.07413). Shawiya has 447 training rows
+against Tashelhit's 24,123, and a shared margin gives the rare class the same slack as the common
+one.
+**Group-DRO** for the domain confound — exponentiated-gradient weights over the twenty
+`(language, domain)` cells, from [Sagawa et al., ICLR 2020](https://arxiv.org/abs/1911.08731). It
+optimises the worst cell rather than the batch mean.
+**Domain-aware supervised contrastive learning** — negatives mined from a different language in
+the *same* domain first, over a 2,048-entry memory bank. The hard-negative scheme is
+[ConLID's](https://arxiv.org/abs/2506.15304). A same-domain negative can only be separated by the
+language.
+**Both margins and group weights are off for the first half of the schedule.** Measured on this
+corpus: 120 steps with margins from step 0 reached a loss of 4.01 where the same run without them
+reached 0.76, at the same accuracy. This is LDAM's own deferred schedule.
+**Selection is on the mean of dev macro-F1 and dev worst-group recall, never on the test set and
+never on loss.** Macro-F1 alone picks the checkpoint that wins the large domains; worst-group
+alone picks one that has learned a single hard cell and forgotten the rest.
 ## Limitations
+**Ambiguity on shared vocabulary is real and is not an error.** A short sentence of pan-Berber
+core vocabulary carrying no variety-specific marker cannot be assigned to one variety. The
+conformal set reports that with size > 1.
+**Kabyle precision is 0.626.** Sibling text still arrives labelled Kabyle at that rate on this set.
+**Tifinagh and Arabic script are out of scope.** Byte input from either produces a label. That
+label has not been measured.
+**Single words are outside the input shape.** The model was fitted on sentences.
+**Code-switching is not handled.** One label per input.
+**Shawiya cannot be measured well by anyone.** 447 training rows and 150 held-out is close to the
+whole of the language's digital text; its numbers carry their support and no further claim.
+**No inter-annotator ceiling exists for Berber variety attribution**, so no figure here can be
+read as a fraction of what is attainable — fluent speakers disagree on the ambiguous cases.
+**Every source in the evaluation set is a corpus**, not a live crawl or real user input, and no
+safety evaluation of any kind has been performed.
 ## Files
+| file | description |
+|---|---|
+| `model.safetensors` | 20.4 MB, 5,089,408 parameters |
+| `config.json` | architecture, class order, prior shift and conformal quantile |
+| `modeling_dihya.py`, `configuration_dihya.py` | the architecture, `transformers` and `torch` only |
+| `tokenizer.json` | the 258-entry byte vocabulary |
+| `calibration.json` | the calibration, in full |
+| `held-out-report.json` | per class and per `(language, domain)` cell |
+No optimiser state, no scheduler state, no RNG. Training cannot be resumed from these files.
 ## Reproduction
 ```bash
+make lid-dataset                              # build the splits from the source table
+make modal-lid EPOCHS=8                       # train on Modal A10G
+make modal-lid-pull                           # fetch the checkpoint
+make lid TASK=evaluate                        # the 7-class held-out report
+make bench TASK=lid                           # Dihya, GlotLID and lid218e in one pass
+make release REPO=dihya                       # export and stage for the Hub
 ```
+`make bench TASK=lid` is what produced the comparison table: all three systems, one build of the
+evaluation set, one pass. A baseline computed elsewhere on a different sample is not a comparison.
 ## The name
+**Dihya** (also called *al-Kāhina*, died c. 703 CE) was an Amazigh military leader from the Aurès
+Mountains of what is now eastern Algeria — the Shawiya homeland — who led the Berber resistance to
+the Umayyad conquest of Ifriqiya. Much of what is written about her comes from sources composed
+well after her lifetime and is contested; the Aurès and the resistance are the parts that are not.
+The naming is homage and implies no endorsement by anyone.
 ## Citation
 ```bibtex
 @software{agbalu_dihya_2026,
+  title  = {Dihya-5M: language identification across the Berber dialect continuum},
   author = {AƔBALU},
   year   = {2026},
   url    = {https://huggingface.co/agbalu/Dihya-5M},
+  note   = {5.19M parameters; 85.52% accuracy over seven classes on 1,050 held-out sentences}
 }
 ```
 ## Licence
+**Apache-2.0** on the weights and the code. The training corpus is assembled from sources under
+mixed licences, which a permissive grant on weights does not relicense; see
+[the datasheet](https://huggingface.co/datasets/agbalu/KabBench) and
+`resources/sibling_registry.yaml` in the project repository for per-source terms.
 Part of [AƔBALU](https://huggingface.co/agbalu), a Kabyle and Amazigh corpus and model collection.

__init__.py ADDED Viewed

	@@ -0,0 +1,4 @@

+from .configuration_dihya import CLASSES, DihyaConfig
+from .modeling_dihya import DihyaForSequenceClassification, DihyaPreTrainedModel
+__all__ = ["CLASSES", "DihyaConfig", "DihyaForSequenceClassification", "DihyaPreTrainedModel"]

config.json CHANGED Viewed

@@ -1,49 +1,69 @@
 {
-  "model": {
-    "vocab_size": 258,
-    "embed_dim": 256,
-    "conv_kernels": [
-      3,
-      5,
-      7
-    ],
-    "conv_dim": 128,
-    "num_layers": 6,
-    "num_heads": 8,
-    "intermediate_dim": 704,
-    "projection_dim": 128,
-    "max_length": 256,
-    "dropout": 0.1,
-    "scale": 24.0,
-    "margin_scale": 0.5,
-    "classes": [
-      "kab_Latn",
-      "shi_Latn",
-      "rif_Latn",
-      "taq_Latn",
-      "tzm_Latn",
-      "shy_Latn",
-      "NOT_AMAZIGH"
-    ]
   },
-  "train": {
-    "epochs": 8,
-    "groups_per_batch": 12,
-    "samples_per_group": 10,
-    "learning_rate": 0.0003,
-    "min_learning_rate": 1e-06,
-    "weight_decay": 0.01,
-    "max_grad_norm": 1.0,
-    "eval_batch_size": 256,
-    "contrastive_weight": 0.5,
-    "contrastive_temperature": 0.05,
-    "memory_bank_size": 2048,
-    "group_dro_step": 0.01,
-    "deferred_share": 0.5,
-    "label_smoothing": 0.05,
-    "logit_adjustment": 0.5,
-    "phrase_augment_prob": 0.35,
-    "case_augment_prob": 0.1,
-    "seed": 20260823
-  }
-}

 {
+  "architectures": [
+    "DihyaForSequenceClassification"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_dihya.DihyaConfig",
+    "AutoModel": "modeling_dihya.DihyaForSequenceClassification",
+    "AutoModelForSequenceClassification": "modeling_dihya.DihyaForSequenceClassification"
   },
+  "byte_offset": 2,
+  "classes": [
+    "kab_Latn",
+    "shi_Latn",
+    "rif_Latn",
+    "taq_Latn",
+    "tzm_Latn",
+    "shy_Latn",
+    "NOT_AMAZIGH"
+  ],
+  "conv_dim": 128,
+  "conv_kernels": [
+    3,
+    5,
+    7
+  ],
+  "dropout_prob": 0.1,
+  "hidden_size": 256,
+  "id2label": {
+    "0": "kab_Latn",
+    "1": "shi_Latn",
+    "2": "rif_Latn",
+    "3": "taq_Latn",
+    "4": "tzm_Latn",
+    "5": "shy_Latn",
+    "6": "NOT_AMAZIGH"
+  },
+  "intermediate_size": 704,
+  "label2id": {
+    "NOT_AMAZIGH": 6,
+    "kab_Latn": 0,
+    "rif_Latn": 2,
+    "shi_Latn": 1,
+    "shy_Latn": 5,
+    "taq_Latn": 3,
+    "tzm_Latn": 4
+  },
+  "logit_scale": 24.0,
+  "max_position_embeddings": 256,
+  "model_type": "dihya",
+  "num_attention_heads": 8,
+  "num_hidden_layers": 6,
+  "pad_token_id": 0,
+  "prior_shift": [
+    -0.6155660152435303,
+    -0.5855244994163513,
+    -1.1106044054031372,
+    -1.2838202714920044,
+    -2.21260666847229,
+    -2.5694897174835205,
+    -0.8174624443054199
+  ],
+  "q_hat": 0.997931957244873,
+  "rms_norm_eps": 1e-06,
+  "rope_theta": 10000.0,
+  "target_coverage": 0.99,
+  "transformers_version": "5.12.1",
+  "unk_token_id": 1,
+  "vocab_size": 258
+}

configuration_dihya.py ADDED Viewed

	@@ -0,0 +1,86 @@

+"""Configuration for the Dihya-5M Berber language identifier."""
+from __future__ import annotations
+from typing import Any
+from transformers import PreTrainedConfig
+CLASSES = [
+    "kab_Latn",
+    "shi_Latn",
+    "rif_Latn",
+    "taq_Latn",
+    "tzm_Latn",
+    "shy_Latn",
+    "NOT_AMAZIGH",
+]
+"""Label order is the trained head's row order. Reordering it renames every prediction."""
+class DihyaConfig(PreTrainedConfig):
+    """The classifier's shapes, its label set, and the two inference-time corrections.
+    `prior_shift` and `q_hat` are stored here rather than in a sidecar because they are
+    part of what the model predicts: without the shift the argmax minimises the training
+    prior's error rather than balanced error, and without the quantile there is no
+    prediction set. A repository that ships the weights alone ships a different model.
+    """
+    model_type = "dihya"
+    def __init__(
+        self,
+        vocab_size: int = 258,
+        hidden_size: int = 256,
+        intermediate_size: int = 704,
+        num_attention_heads: int = 8,
+        num_hidden_layers: int = 6,
+        conv_kernels: list[int] | None = None,
+        conv_dim: int = 128,
+        max_position_embeddings: int = 256,
+        rope_theta: float = 10000.0,
+        dropout_prob: float = 0.1,
+        logit_scale: float = 24.0,
+        rms_norm_eps: float = 1e-6,
+        classes: list[str] | None = None,
+        prior_shift: list[float] | None = None,
+        q_hat: float | None = None,
+        target_coverage: float = 0.99,
+        pad_token_id: int = 0,
+        unk_token_id: int = 1,
+        byte_offset: int = 2,
+        **kwargs: Any,
+    ) -> None:
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.intermediate_size = intermediate_size
+        self.num_attention_heads = num_attention_heads
+        self.num_hidden_layers = num_hidden_layers
+        self.conv_kernels = list(conv_kernels or [3, 5, 7])
+        self.conv_dim = conv_dim
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        self.dropout_prob = dropout_prob
+        self.logit_scale = logit_scale
+        self.rms_norm_eps = rms_norm_eps
+        self.unk_token_id = unk_token_id
+        self.byte_offset = byte_offset
+        labels = list(classes or CLASSES)
+        self.classes = labels
+        self.prior_shift = list(prior_shift) if prior_shift is not None else [0.0] * len(labels)
+        self.q_hat = q_hat
+        self.target_coverage = target_coverage
+        kwargs.setdefault("pad_token_id", pad_token_id)
+        kwargs.setdefault("id2label", dict(enumerate(labels)))
+        kwargs.setdefault("label2id", {name: i for i, name in enumerate(labels)})
+        super().__init__(**kwargs)
+    @property
+    def head_size(self) -> int:
+        return self.hidden_size // self.num_attention_heads
+__all__ = ["CLASSES", "DihyaConfig"]

export.stats.json ADDED Viewed

	@@ -0,0 +1,64 @@

+{
+  "source": "artifacts/runs/dihya/best.pt",
+  "source_bytes": 63439095,
+  "dropped_training_only": [
+    "head.margins",
+    "projection.0.weight",
+    "projection.2.weight"
+  ],
+  "tensors": 69,
+  "parameters": 5089408,
+  "max_logit_delta_against_trained": 0.0,
+  "files": [
+    {
+      "name": "README.md",
+      "bytes": 15452,
+      "sha256": "8d230fd4847f15a5f431bf369fcd6054baec703369b717b755fc939322da93be"
+    },
+    {
+      "name": "__init__.py",
+      "bytes": 231,
+      "sha256": "5c3bd891c199287302180ab2a447de121dc11ca69309da252741c612ea055d0e"
+    },
+    {
+      "name": "calibration.json",
+      "bytes": 516,
+      "sha256": "72ab22b431e1887e4c1dbaed14f08455a0096cc6a544accc689007e05280f1eb"
+    },
+    {
+      "name": "config.json",
+      "bytes": 1451,
+      "sha256": "d0cc1ff28fba5f9d61ca08e7e091cf7898338005ec3661bb1ad0d066d4cb581d"
+    },
+    {
+      "name": "configuration_dihya.py",
+      "bytes": 2917,
+      "sha256": "b92c21f6e356261c286440799d4e3a114ae13b2e3204153cd6a41259d97378d5"
+    },
+    {
+      "name": "held-out-report.json",
+      "bytes": 3081,
+      "sha256": "5dedc57c1a07f1a80e9e62e79ce5ffbf15308ce00e0f9b8e147cc57dea4fcb17"
+    },
+    {
+      "name": "model.safetensors",
+      "bytes": 20364088,
+      "sha256": "edfbe7115393fca0a18474fcaac641afdbe69b66bc74d9381a5b54f38444ddd7"
+    },
+    {
+      "name": "modeling_dihya.py",
+      "bytes": 13221,
+      "sha256": "116d02ea899acd89bc05467e1097a062b851c64c5fb36d53a4386010caca959f"
+    },
+    {
+      "name": "tokenizer.json",
+      "bytes": 5648,
+      "sha256": "fba53450cc2250bf3d4686ec0781d52fbdee96e383142835365dc6960a051ef6"
+    },
+    {
+      "name": "tokenizer_config.json",
+      "bytes": 147,
+      "sha256": "c6d1937e9ca63bebaa9ad6a05df2250a67ba071d672ef88e3ef5a45e75c4df50"
+    }
+  ]
+}

model.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:edfbe7115393fca0a18474fcaac641afdbe69b66bc74d9381a5b54f38444ddd7
+size 20364088

modeling_dihya.py ADDED Viewed

	@@ -0,0 +1,320 @@

+"""Dihya-5M: byte-level Berber language identification.
+Module attribute names are the published checkpoint's state_dict keys. Renaming one
+breaks `from_pretrained` for everybody who downloaded the release.
+The contrastive projection head the model was trained with is not here. It shapes the
+trunk during training and is never read at inference, so shipping it would hand every
+downloader 98,304 parameters that no forward pass touches.
+"""
+from __future__ import annotations
+import math
+from typing import Any
+import torch
+from torch import Tensor, nn
+from torch.nn import functional
+from transformers import PreTrainedModel
+from transformers.modeling_outputs import SequenceClassifierOutput
+from .configuration_dihya import DihyaConfig
+MASK_FILL = -1e4
+"""Finite rather than `-inf`: a row that is entirely padding would otherwise softmax to
+NaN, and an empty string is a real input to a language identifier."""
+class RMSNorm(nn.Module):
+    def __init__(self, dim: int, eps: float = 1e-6) -> None:
+        super().__init__()
+        self.eps = eps
+        self.weight = nn.Parameter(torch.ones(dim))
+    def forward(self, x: Tensor) -> Tensor:
+        variance = x.pow(2).mean(-1, keepdim=True)
+        normed: Tensor = x * torch.rsqrt(variance + self.eps) * self.weight
+        return normed
+class SwiGLU(nn.Module):
+    def __init__(self, dim: int, intermediate_dim: int) -> None:
+        super().__init__()
+        self.w1 = nn.Linear(dim, intermediate_dim, bias=False)
+        self.w2 = nn.Linear(dim, intermediate_dim, bias=False)
+        self.w3 = nn.Linear(intermediate_dim, dim, bias=False)
+    def forward(self, x: Tensor) -> Tensor:
+        projected: Tensor = self.w3(functional.silu(self.w1(x)) * self.w2(x))
+        return projected
+class ConvStem(nn.Module):
+    """Parallel depthwise-separable convolutions over the byte embeddings.
+    Three widths because the discriminating evidence sits at three scales: a grapheme
+    cluster, an affix, and a clitic chain. One kernel width picks one of the three.
+    """
+    def __init__(self, config: DihyaConfig) -> None:
+        super().__init__()
+        dim, branch_dim = config.hidden_size, config.conv_dim
+        self.branches = nn.ModuleList(
+            nn.Sequential(
+                nn.Conv1d(dim, dim, kernel_size=k, padding=k // 2, groups=dim, bias=False),
+                nn.Conv1d(dim, branch_dim, kernel_size=1, bias=False),
+            )
+            for k in config.conv_kernels
+        )
+        self.branch_norms = nn.ModuleList(
+            RMSNorm(branch_dim, eps=config.rms_norm_eps) for _ in config.conv_kernels
+        )
+        self.proj = nn.Linear(branch_dim * len(config.conv_kernels), dim, bias=False)
+        self.norm = RMSNorm(dim, eps=config.rms_norm_eps)
+        self.dropout = nn.Dropout(config.dropout_prob)
+    def forward(self, x: Tensor) -> Tensor:
+        transposed = x.transpose(1, 2)
+        outputs = [
+            functional.silu(norm(branch(transposed).transpose(1, 2)))
+            for branch, norm in zip(self.branches, self.branch_norms, strict=True)
+        ]
+        stemmed: Tensor = x + self.dropout(self.norm(self.proj(torch.cat(outputs, dim=-1))))
+        return stemmed
+def rope_freqs(head_dim: int, length: int, base: float, device: torch.device) -> Tensor:
+    """Complex rotary frequencies, derived on call and never stored.
+    A registered non-persistent buffer comes back from `from_pretrained` as uninitialised
+    memory, because it is deliberately absent from the checkpoint.
+    """
+    theta = 1.0 / (base ** (torch.arange(0, head_dim, 2, device=device).float() / head_dim))
+    positions = torch.arange(length, device=device).float()
+    angles = torch.outer(positions, theta)
+    return torch.polar(torch.ones_like(angles), angles)
+def apply_rope(x: Tensor, freqs: Tensor) -> Tensor:
+    batch, heads, length, head_dim = x.shape
+    paired = torch.view_as_complex(x.float().reshape(batch, heads, length, -1, 2))
+    rotated = torch.view_as_real(paired * freqs[:length].unsqueeze(0).unsqueeze(0))
+    return rotated.reshape(batch, heads, length, head_dim).type_as(x)
+class Attention(nn.Module):
+    def __init__(self, config: DihyaConfig) -> None:
+        super().__init__()
+        dim = config.hidden_size
+        self.num_heads = config.num_attention_heads
+        self.head_dim = config.head_size
+        self.dropout = config.dropout_prob
+        self.q_proj = nn.Linear(dim, dim, bias=False)
+        self.k_proj = nn.Linear(dim, dim, bias=False)
+        self.v_proj = nn.Linear(dim, dim, bias=False)
+        self.out_proj = nn.Linear(dim, dim, bias=False)
+    def forward(self, x: Tensor, freqs: Tensor, mask: Tensor | None = None) -> Tensor:
+        batch, length, dim = x.shape
+        shape = (batch, length, self.num_heads, self.head_dim)
+        query = apply_rope(self.q_proj(x).view(shape).transpose(1, 2), freqs)
+        key = apply_rope(self.k_proj(x).view(shape).transpose(1, 2), freqs)
+        value = self.v_proj(x).view(shape).transpose(1, 2)
+        attended = functional.scaled_dot_product_attention(
+            query,
+            key,
+            value,
+            attn_mask=mask.unsqueeze(1).unsqueeze(2) if mask is not None else None,
+            dropout_p=self.dropout if self.training else 0.0,
+        )
+        merged: Tensor = self.out_proj(attended.transpose(1, 2).reshape(batch, length, dim))
+        return merged
+class EncoderLayer(nn.Module):
+    def __init__(self, config: DihyaConfig) -> None:
+        super().__init__()
+        self.norm1 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.attn = Attention(config)
+        self.norm2 = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.ffn = SwiGLU(config.hidden_size, config.intermediate_size)
+        self.dropout = nn.Dropout(config.dropout_prob)
+    def forward(self, x: Tensor, freqs: Tensor, mask: Tensor | None = None) -> Tensor:
+        hidden: Tensor = x + self.dropout(self.attn(self.norm1(x), freqs, mask=mask))
+        residual: Tensor = self.dropout(self.ffn(self.norm2(hidden)))
+        return hidden + residual
+class AttentivePooling(nn.Module):
+    """Weighted sum over positions, so a short discriminating affix is not averaged away."""
+    def __init__(self, config: DihyaConfig) -> None:
+        super().__init__()
+        self.score = nn.Linear(config.hidden_size, 1, bias=False)
+    def forward(self, x: Tensor, mask: Tensor | None = None) -> Tensor:
+        scores = self.score(x).squeeze(-1) / math.sqrt(x.size(-1))
+        if mask is not None:
+            scores = scores.masked_fill(~mask, MASK_FILL)
+        weights = functional.softmax(scores, dim=-1).unsqueeze(-1)
+        pooled: Tensor = (x * weights).sum(dim=1)
+        return pooled
+class MarginHead(nn.Module):
+    """Scaled cosine classifier.
+    The per-class additive margins the head was trained with apply to the target logit
+    only, so they exist during training and are identity at inference. The margin buffer
+    is therefore not part of the release.
+    """
+    def __init__(self, config: DihyaConfig) -> None:
+        super().__init__()
+        self.scale = config.logit_scale
+        self.weight = nn.Parameter(torch.empty(len(config.classes), config.hidden_size))
+    def forward(self, x: Tensor) -> Tensor:
+        cosine = functional.linear(
+            functional.normalize(x, p=2, dim=1), functional.normalize(self.weight, p=2, dim=1)
+        )
+        return cosine * self.scale
+class DihyaPreTrainedModel(PreTrainedModel):
+    config_class = DihyaConfig
+    base_model_prefix = "dihya"
+    supports_gradient_checkpointing = False
+    def _init_weights(self, module: nn.Module) -> None:
+        if isinstance(module, nn.Linear | nn.Conv1d):
+            nn.init.xavier_uniform_(module.weight)
+            if getattr(module, "bias", None) is not None:
+                nn.init.zeros_(module.bias)
+        elif isinstance(module, nn.Embedding):
+            nn.init.normal_(module.weight, std=0.02)
+            if module.padding_idx is not None:
+                with torch.no_grad():
+                    module.weight[module.padding_idx].fill_(0)
+        elif isinstance(module, RMSNorm):
+            nn.init.ones_(module.weight)
+        elif isinstance(module, MarginHead):
+            nn.init.xavier_uniform_(module.weight)
+class DihyaForSequenceClassification(DihyaPreTrainedModel):
+    """Byte-level classifier over six Berber varieties and an explicit rejection class."""
+    def __init__(self, config: DihyaConfig) -> None:
+        super().__init__(config)
+        self.embed = nn.Embedding(config.vocab_size, config.hidden_size, padding_idx=0)
+        self.stem = ConvStem(config)
+        self.layers = nn.ModuleList(EncoderLayer(config) for _ in range(config.num_hidden_layers))
+        self.final_norm = RMSNorm(config.hidden_size, eps=config.rms_norm_eps)
+        self.pool = AttentivePooling(config)
+        self.head = MarginHead(config)
+        self.post_init()
+    def get_input_embeddings(self) -> nn.Module:
+        return self.embed
+    def set_input_embeddings(self, value: nn.Module) -> None:
+        self.embed = value  # type: ignore[assignment]
+    def forward(
+        self,
+        input_ids: Tensor,
+        attention_mask: Tensor | None = None,
+        labels: Tensor | None = None,
+        return_dict: bool | None = None,
+        **kwargs: Any,
+    ) -> SequenceClassifierOutput | tuple[Tensor, ...]:
+        mask = attention_mask.bool() if attention_mask is not None else None
+        hidden = self.stem(self.embed(input_ids))
+        freqs = rope_freqs(
+            self.config.head_size,
+            input_ids.size(1),
+            self.config.rope_theta,
+            input_ids.device,
+        )
+        for layer in self.layers:
+            hidden = layer(hidden, freqs, mask=mask)
+        pooled = self.pool(self.final_norm(hidden), mask=mask)
+        logits = self.head(pooled)
+        loss = None
+        if labels is not None:
+            loss = functional.cross_entropy(logits, labels)
+        if return_dict is False:
+            return (logits,) if loss is None else (loss, logits)
+        return SequenceClassifierOutput(loss=loss, logits=logits, hidden_states=(pooled,))
+    def prior_shift(self, device: torch.device, dtype: torch.dtype) -> Tensor:
+        return torch.tensor(self.config.prior_shift, device=device, dtype=dtype)
+    @torch.inference_mode()
+    def identify(
+        self,
+        texts: str | list[str],
+        max_length: int | None = None,
+        batch_size: int = 128,
+    ) -> list[dict[str, Any]]:
+        """Classify text. One dict per input: `language`, `confidence`, `prediction_set`.
+        The tokenizer is not needed: the vocabulary is the 256 UTF-8 byte values, so the
+        encoding is the input's own bytes. `prediction_set` is the split-conformal set
+        `{k : p_k >= 1 - q_hat}` when the repository carries a calibrated `q_hat`, and the
+        argmax alone when it does not — never a singleton dressed up as a guarantee.
+        """
+        wanted = [texts] if isinstance(texts, str) else list(texts)
+        if not wanted:
+            return []
+        limit = max_length or self.config.max_position_embeddings
+        device = next(self.parameters()).device
+        classes = list(self.config.classes)
+        threshold = None if self.config.q_hat is None else 1.0 - float(self.config.q_hat)
+        results: list[dict[str, Any]] = []
+        for start in range(0, len(wanted), batch_size):
+            chunk = wanted[start : start + batch_size]
+            rows = [
+                [b + self.config.byte_offset for b in text.encode("utf-8")[:limit]]
+                or [self.config.pad_token_id]
+                for text in chunk
+            ]
+            width = max(len(row) for row in rows)
+            input_ids = torch.full(
+                (len(rows), width), self.config.pad_token_id, dtype=torch.long, device=device
+            )
+            attention = torch.zeros((len(rows), width), dtype=torch.long, device=device)
+            for i, row in enumerate(rows):
+                input_ids[i, : len(row)] = torch.tensor(row, dtype=torch.long, device=device)
+                attention[i, : len(row)] = 1
+            logits = self(input_ids, attention_mask=attention).logits
+            shifted = logits - self.prior_shift(logits.device, logits.dtype)
+            for probabilities in torch.softmax(shifted, dim=-1).tolist():
+                top = max(range(len(probabilities)), key=probabilities.__getitem__)
+                members = (
+                    tuple(c for c, p in zip(classes, probabilities, strict=True) if p >= threshold)
+                    if threshold is not None
+                    else (classes[top],)
+                )
+                results.append(
+                    {
+                        "language": classes[top],
+                        "confidence": probabilities[top],
+                        "prediction_set": members or (classes[top],),
+                        "probabilities": dict(zip(classes, probabilities, strict=True)),
+                    }
+                )
+        return results
+__all__ = [
+    "DihyaConfig",
+    "DihyaForSequenceClassification",
+    "DihyaPreTrainedModel",
+]

tokenizer.json ADDED Viewed

	@@ -0,0 +1,341 @@

+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "[PAD]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "[UNK]",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "Sequence",
+    "pretokenizers": [
+      {
+        "type": "ByteLevel",
+        "add_prefix_space": false,
+        "trim_offsets": true,
+        "use_regex": false
+      },
+      {
+        "type": "Split",
+        "pattern": {
+          "Regex": "[\\s\\S]"
+        },
+        "behavior": "Isolated",
+        "invert": false
+      }
+    ]
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {}
+  },
+  "decoder": {
+    "type": "ByteLevel",
+    "add_prefix_space": true,
+    "trim_offsets": true,
+    "use_regex": true
+  },
+  "model": {
+    "type": "WordLevel",
+    "vocab": {
+      "[PAD]": 0,
+      "[UNK]": 1,
+      "Ā": 2,
+      "ā": 3,
+      "Ă": 4,
+      "ă": 5,
+      "Ą": 6,
+      "ą": 7,
+      "Ć": 8,
+      "ć": 9,
+      "Ĉ": 10,
+      "ĉ": 11,
+      "Ċ": 12,
+      "ċ": 13,
+      "Č": 14,
+      "č": 15,
+      "Ď": 16,
+      "ď": 17,
+      "Đ": 18,
+      "đ": 19,
+      "Ē": 20,
+      "ē": 21,
+      "Ĕ": 22,
+      "ĕ": 23,
+      "Ė": 24,
+      "ė": 25,
+      "Ę": 26,
+      "ę": 27,
+      "Ě": 28,
+      "ě": 29,
+      "Ĝ": 30,
+      "ĝ": 31,
+      "Ğ": 32,
+      "ğ": 33,
+      "Ġ": 34,
+      "!": 35,
+      "\"": 36,
+      "#": 37,
+      "$": 38,
+      "%": 39,
+      "&": 40,
+      "'": 41,
+      "(": 42,
+      ")": 43,
+      "*": 44,
+      "+": 45,
+      ",": 46,
+      "-": 47,
+      ".": 48,
+      "/": 49,
+      "0": 50,
+      "1": 51,
+      "2": 52,
+      "3": 53,
+      "4": 54,
+      "5": 55,
+      "6": 56,
+      "7": 57,
+      "8": 58,
+      "9": 59,
+      ":": 60,
+      ";": 61,
+      "<": 62,
+      "=": 63,
+      ">": 64,
+      "?": 65,
+      "@": 66,
+      "A": 67,
+      "B": 68,
+      "C": 69,
+      "D": 70,
+      "E": 71,
+      "F": 72,
+      "G": 73,
+      "H": 74,
+      "I": 75,
+      "J": 76,
+      "K": 77,
+      "L": 78,
+      "M": 79,
+      "N": 80,
+      "O": 81,
+      "P": 82,
+      "Q": 83,
+      "R": 84,
+      "S": 85,
+      "T": 86,
+      "U": 87,
+      "V": 88,
+      "W": 89,
+      "X": 90,
+      "Y": 91,
+      "Z": 92,
+      "[": 93,
+      "\\": 94,
+      "]": 95,
+      "^": 96,
+      "_": 97,
+      "`": 98,
+      "a": 99,
+      "b": 100,
+      "c": 101,
+      "d": 102,
+      "e": 103,
+      "f": 104,
+      "g": 105,
+      "h": 106,
+      "i": 107,
+      "j": 108,
+      "k": 109,
+      "l": 110,
+      "m": 111,
+      "n": 112,
+      "o": 113,
+      "p": 114,
+      "q": 115,
+      "r": 116,
+      "s": 117,
+      "t": 118,
+      "u": 119,
+      "v": 120,
+      "w": 121,
+      "x": 122,
+      "y": 123,
+      "z": 124,
+      "{": 125,
+      "|": 126,
+      "}": 127,
+      "~": 128,
+      "ġ": 129,
+      "Ģ": 130,
+      "ģ": 131,
+      "Ĥ": 132,
+      "ĥ": 133,
+      "Ħ": 134,
+      "ħ": 135,
+      "Ĩ": 136,
+      "ĩ": 137,
+      "Ī": 138,
+      "ī": 139,
+      "Ĭ": 140,
+      "ĭ": 141,
+      "Į": 142,
+      "į": 143,
+      "İ": 144,
+      "ı": 145,
+      "Ĳ": 146,
+      "ĳ": 147,
+      "Ĵ": 148,
+      "ĵ": 149,
+      "Ķ": 150,
+      "ķ": 151,
+      "ĸ": 152,
+      "Ĺ": 153,
+      "ĺ": 154,
+      "Ļ": 155,
+      "ļ": 156,
+      "Ľ": 157,
+      "ľ": 158,
+      "Ŀ": 159,
+      "ŀ": 160,
+      "Ł": 161,
+      "ł": 162,
+      "¡": 163,
+      "¢": 164,
+      "£": 165,
+      "¤": 166,
+      "¥": 167,
+      "¦": 168,
+      "§": 169,
+      "¨": 170,
+      "©": 171,
+      "ª": 172,
+      "«": 173,
+      "¬": 174,
+      "Ń": 175,
+      "®": 176,
+      "¯": 177,
+      "°": 178,
+      "±": 179,
+      "²": 180,
+      "³": 181,
+      "´": 182,
+      "µ": 183,
+      "¶": 184,
+      "·": 185,
+      "¸": 186,
+      "¹": 187,
+      "º": 188,
+      "»": 189,
+      "¼": 190,
+      "½": 191,
+      "¾": 192,
+      "¿": 193,
+      "À": 194,
+      "Á": 195,
+      "Â": 196,
+      "Ã": 197,
+      "Ä": 198,
+      "Å": 199,
+      "Æ": 200,
+      "Ç": 201,
+      "È": 202,
+      "É": 203,
+      "Ê": 204,
+      "Ë": 205,
+      "Ì": 206,
+      "Í": 207,
+      "Î": 208,
+      "Ï": 209,
+      "Ð": 210,
+      "Ñ": 211,
+      "Ò": 212,
+      "Ó": 213,
+      "Ô": 214,
+      "Õ": 215,
+      "Ö": 216,
+      "×": 217,
+      "Ø": 218,
+      "Ù": 219,
+      "Ú": 220,
+      "Û": 221,
+      "Ü": 222,
+      "Ý": 223,
+      "Þ": 224,
+      "ß": 225,
+      "à": 226,
+      "á": 227,
+      "â": 228,
+      "ã": 229,
+      "ä": 230,
+      "å": 231,
+      "æ": 232,
+      "ç": 233,
+      "è": 234,
+      "é": 235,
+      "ê": 236,
+      "ë": 237,
+      "ì": 238,
+      "í": 239,
+      "î": 240,
+      "ï": 241,
+      "ð": 242,
+      "ñ": 243,
+      "ò": 244,
+      "ó": 245,
+      "ô": 246,
+      "õ": 247,
+      "ö": 248,
+      "÷": 249,
+      "ø": 250,
+      "ù": 251,
+      "ú": 252,
+      "û": 253,
+      "ü": 254,
+      "ý": 255,
+      "þ": 256,
+      "ÿ": 257
+    },
+    "unk_token": "[UNK]"
+  }
+}

tokenizer_config.json ADDED Viewed

	@@ -0,0 +1,7 @@

+{
+  "backend": "tokenizers",
+  "model_max_length": 256,
+  "pad_token": "[PAD]",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "[UNK]"
+}