Upload laya-triage model

README.md

@@ -32,49 +32,64 @@ metrics:
 
 Classifies GitHub issues written in any language as **bug**, **feature**, **question** or **docs**. A fine-tune of [Laya multilingual](https://huggingface.co/convaiinnovations/laya-multilingual) (mmBERT-base) used by the [laya-triage](https://github.com/elnachto/laya-triage) GitHub Action for non-English issues, next to the English model [laya-triage-en](https://huggingface.co/elnachto/laya-triage-en).
 
+This is the v1.1 model: the v1.0 multilingual model, fine-tuned on real recent issues from active repositories.
+
 ## Results
 
-The same 500 NLBSE'23 validation issues, machine-translated with NLLB-200 into 13 languages. Accuracy (±3 points per language):
-
-| Language | Laya base | Jev (TypeSafe, hosted) | **laya-triage** |
-|---|---|---|---|
-| English | 89.2% | 88.2% | **90.6%** |
-| German | 74.6% | 85.8% | **87.4%** |
-| Vietnamese | 71.2% | 84.4% | **86.6%** |
-| Chinese | 68.2% | 83.8% | **86.2%** |
-| Japanese | 67.8% | 84.4% | **86.0%** |
-| Turkish | 68.2% | 85.0% | **85.8%** |
-| Indonesian | 73.8% | 86.0% | 85.6% |
-| Spanish | 73.8% | 85.6% | 85.2% |
-| Hindi | 66.2% | 85.4% | 85.0% |
-| Portuguese | 74.0% | 86.6% | 84.8% |
-| Russian | 70.8% | 85.8% | 84.6% |
-| Korean | 64.2% | 83.8% | **84.4%** |
-| French | 71.6% | 85.2% | 84.2% |
-| Arabic | 67.4% | 84.8% | 84.0% |
-
-laya-triage and Jev are within noise of each other across languages; both are far ahead of the untuned base.
-
-Translations can flatter a model trained on translations, so we also checked real issues: on 367 non-English issues opened in 2026 (never seen, written by people, not translated) accuracy went from **47.1% to 65.7%** without class priors, and to 61.0% with the priors the action applies. That set is balanced on purpose, which penalizes priors; on typical repositories the priors help.
+### Real non-English issues
+
+Issues written by people, not translated, that the router sent to this model. No repository in these sets was used for training.
+
+| Set | v1.0 | **v1.1** |
+|---|---|---|
+| Recent issues from 288 active repositories (1,031 routed here), adapted to each repository | 74.0% | **77.6%** |
+| Issues opened in 2026 (367 routed here), priors set to the set's balanced mix | 65.7% | **71.1%** |
+
+### 14 languages
+
+The same 500 NLBSE'23 validation issues, machine-translated with NLLB-200. Accuracy of the whole system (router + both models) with the natural class mix, ±3 points per language:
+
+| Language | Laya base | Jev (TypeSafe, hosted) | laya-triage v1.0 | **laya-triage v1.1** |
+|---|---|---|---|---|
+| English | 89.2% | 88.2% | 90.6% | **92.4%** |
+| Vietnamese | 71.2% | 84.4% | 86.6% | **87.6%** |
+| German | 74.6% | 85.8% | 87.4% | **86.4%** |
+| Turkish | 68.2% | 85.0% | 85.8% | **86.4%** |
+| Portuguese | 74.0% | 86.6% | 84.8% | 86.4% |
+| Indonesian | 73.8% | 86.0% | 85.6% | **86.2%** |
+| Spanish | 73.8% | 85.6% | 85.2% | **86.2%** |
+| Russian | 70.8% | 85.8% | 84.6% | 85.6% |
+| French | 71.6% | 85.2% | 84.2% | **85.4%** |
+| Hindi | 66.2% | 85.4% | 85.0% | 85.2% |
+| Arabic | 67.4% | 84.8% | 84.0% | **85.0%** |
+| Chinese | 68.2% | 83.8% | 86.2% | **84.8%** |
+| Japanese | 67.8% | 84.4% | 86.0% | **84.8%** |
+| Korean | 64.2% | 83.8% | 84.4% | **84.8%** |
+
+v1.1 is ahead of Jev in 11 of 14 languages (in bold); most gaps between the two are within noise. Average over the 13 non-English languages: 85.8% for v1.1, 85.4% for v1.0 and 85.1% for Jev.
 
 ## How to use
 
 Use it through the Laya router together with the English model, with the exact training question. See [laya-triage-en](https://huggingface.co/elnachto/laya-triage-en#how-to-use) for the code; this checkpoint is the `"multilingual"` entry.
 
+### Class balance
+
+v1.1 was trained on bug 53.2%, feature 30.8%, question 11.0% and docs 5.0%. The mix is stored in `rl_agent_config.json` under `laya_triage.mezcla_base`. To match a repository's own mix, multiply the probabilities by your class frequencies divided by these and renormalize. The action does this automatically with `class-priors`.
+
 ## Training
 
 - Base: `convaiinnovations/laya-multilingual` (mmBERT-base, 322M parameters).
-- Data: 2,000 NLBSE'23 training issues (500 per class) translated into 13 languages with NLLB-200 distilled 600M, plus the English originals and 10,000 extra English issues: 35,200 examples. Code blocks, stack traces and error messages were left untranslated, as in real issues.
-- Validation: 200 issues held out in all 14 languages (no issue appears in both splits in any language).
-- One epoch, same recipe as the English model; the second epoch overfit and was discarded.
-- Calibration: temperature 1.203 (the base model was overconfident), ECE 0.077 → 0.053.
+- v1.0: 2,000 NLBSE'23 training issues (500 per class) translated into 13 languages with NLLB-200 distilled 600M, plus the English originals and 10,000 extra English issues: 35,200 examples. Code blocks, stack traces and error messages were left untranslated, as in real issues.
+- v1.1: v1.0 fine-tuned on 97,593 issues, 47,593 recent issues from active repositories labeled by a maintainer plus 50,000 NLBSE'23 issues so it keeps what it already knew. Repositories used for validation and testing were excluded. Validation accuracy on recent issues from unseen repositories went from 65.1% to 72.2%.
+- Same recipe as the English model: AdamW, lr 2.5e-5 encoder / 1e-4 head, label smoothing 0.1, bf16.
+- Calibration: temperature 1.013, ECE 0.037 on recent validation issues.
 
 ## Limitations
 
-- Machine-translated training data: real issues mix languages, code and English error messages more than translations do.
-- 13 languages measured; others are supported by the base model but not evaluated.
-- Trained on balanced classes, so it expects class priors: multiply the probabilities by the priors stored in `rl_agent_config.json` (`laya_triage.priores`: bug 0.526, feature 0.370, question 0.060, docs 0.044) and renormalize, as the action does.
+- Most non-English issues in real repositories are in Chinese; the other languages are measured mainly on translations, which are cleaner than real issues that mix languages, code and English error messages.
+- 14 languages measured; others are supported by the base model but not evaluated.
+- **question** and **docs** remain the hardest classes, as in the English model.
 
 ## Credits
 
-Base model by [Convai Innovations](https://huggingface.co/convaiinnovations/laya-multilingual) (Apache-2.0). Translation with [NLLB-200](https://huggingface.co/facebook/nllb-200-distilled-600M). Data from the [NLBSE'23 tool competition](https://github.com/nlbse2023/issue-report-classification). Built by [elnachto](https://github.com/elnachto).
+Base model by [Convai Innovations](https://huggingface.co/convaiinnovations/laya-multilingual) (Apache-2.0). Translation with [NLLB-200](https://huggingface.co/facebook/nllb-200-distilled-600M). Data from the [NLBSE'23 tool competition](https://github.com/nlbse2023/issue-report-classification) and public GitHub issues. Built by [elnachto](https://github.com/elnachto).

model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:67429ad745be450ce8b539e135affe5b8057692f84278c1cbd081070f0f78bd9
+oid sha256:be222fb925b5d9d1c18bad3268004f76f4b97c75f24cae0b9d12382878275111
 size 643835692

rl_agent_config.json

@@ -11,7 +11,7 @@
   "amp_dtype": "bf16",
   "model_name": "rl-agent",
   "temperature": [
-    1.2034,
+    1.0133,
     1.0,
     1.0
   ],
@@ -26,14 +26,14 @@
     "variante": "sin_other",
     "epoca": 1,
     "validacion": {
-      "exactitud": 0.6886,
-      "f1_macro": 0.6783
+      "exactitud": 0.7223,
+      "f1_macro": 0.6368
     },
-    "priores": {
-      "bug": 0.526,
-      "feature": 0.37,
-      "question": 0.06,
-      "docs": 0.044
+    "mezcla_base": {
+      "bug": 0.532,
+      "feature": 0.308,
+      "question": 0.11,
+      "docs": 0.05
     }
   }
 }