Statim Decide Multilingual Base 0.7.0

DATA_LICENSES.md

@@ -200,6 +200,124 @@ Licence as recorded per row by tasksource; source names follow its build manifes
 
 Filter statistics of this build: rows 2,500,000, not commercial 1,124,896, audit excluded 489,942, not permissive 219,847, too long 6,489, test duplicate 7.
 
+### Mixture v6 sources (111 sources, 534,231 items, at most 6200 per source)
+
+Licence checked at the source for every entry (registry `tools/finetune/sources/v6-keep.json`, review notes and attribution in `tools/finetune/sources/v6-research.md`). Every text that occurs in an evaluation suite was removed first (862,438 banned texts).
+
+| Source | Category | Licence | Languages | Items |
+|---|---|---|---|---:|
+| [3nesdeniz/agentic-prompt-injection-5k](https://huggingface.co/datasets/3nesdeniz/agentic-prompt-injection-5k) | prompt-injection | CC-BY-4.0 | en | 6,200 |
+| [3nesdeniz/turkish-conversation-prompt-injection](https://huggingface.co/datasets/3nesdeniz/turkish-conversation-prompt-injection) | prompt-injection | CC-BY-4.0 | tr | 530 |
+| [Adilbai/kz-gov-complaints-data-kz-ru](https://huggingface.co/datasets/Adilbai/kz-gov-complaints-data-kz-ru) | complaint, sentiment | Apache-2.0 | ru, kk | 1,200 |
+| [adiprog14/lingrow-support-tickets](https://huggingface.co/datasets/adiprog14/lingrow-support-tickets) | complaint | MIT | en | 6,200 |
+| [ai4bharat/IndicSentiment](https://huggingface.co/datasets/ai4bharat/IndicSentiment) | sentiment | CC0-1.0 (AI4Bharat/IndicBERT README; HF card has none) | en, hi, bn, mr, ta, te, ur, gu +6 | 6,200 |
+| [alaminxpro/university-students-complaints](https://huggingface.co/datasets/alaminxpro/university-students-complaints) | complaint | CC-BY-4.0 | en | 546 |
+| [allenai/prosocial-dialog](https://huggingface.co/datasets/allenai/prosocial-dialog) | safety-moderation | CC-BY-4.0 | en | 6,200 |
+| [alusci/sms-otp-spam-dataset](https://huggingface.co/datasets/alusci/sms-otp-spam-dataset) | spam-sms | MIT (templated; low value) | en | 6,200 |
+| [amyrmahdy/decima-synthetic-decisions](https://huggingface.co/datasets/amyrmahdy/decima-synthetic-decisions) | typed-decisions | cc-by-4.0 (card; fully synthetic, teacher Gemma-4-26B-A4B-it, Apache-2.0 model card) | en, fa, ar, ru | 6,200 |
+| [ankitkupadhyay/XNLI](https://huggingface.co/datasets/ankitkupadhyay/XNLI) | nli | apache-2.0 (card); content inherits MNLI/OANC terms | ar, bg, de, el, en, es, fr, hi +7 | 6,200 |
+| [Anthropic/hh-rlhf](https://huggingface.co/datasets/Anthropic/hh-rlhf) | safety-moderation | MIT | en | 6,200 |
+| [AshenFdo/synthetic_blood_request_urgency_dataset](https://huggingface.co/datasets/AshenFdo/synthetic_blood_request_urgency_dataset) | urgency | mit | en | 2,500 |
+| [Avature/Job-Title-Similarity](https://huggingface.co/datasets/Avature/Job-Title-Similarity) | similarity | apache-2.0 | de, en, es, fr, it, ja, nl, pl +3 | 4,563 |
+| [BEE-spoke-data/consumer-finance-complaints](https://huggingface.co/datasets/BEE-spoke-data/consumer-finance-complaints) | complaint | CC0-1.0 card; CFPB US federal data, narratives published with consumer opt-in consent | en | 6,200 |
+| [boun-tabi/nli_tr](https://huggingface.co/datasets/boun-tabi/nli_tr) | nli | same terms as MultiNLI (GitHub boun-tabi/NLI-TR README) | tr | 6,200 |
+| [brighter-dataset/BRIGHTER-emotion-categories](https://huggingface.co/datasets/brighter-dataset/BRIGHTER-emotion-categories) | emotion | cc-by-4.0 | hi | 3,629 |
+| [brighter-dataset/BRIGHTER-emotion-categories](https://huggingface.co/datasets/brighter-dataset/BRIGHTER-emotion-categories) | emotion | cc-by-4.0 | mr | 3,768 |
+| [clips/VaccinChatNL](https://huggingface.co/datasets/clips/VaccinChatNL) | intent-dialogue-act | CC-BY-4.0 | nl | 6,200 |
+| [cngchis/Support-Ticket-Router-12K-Cleaned](https://huggingface.co/datasets/cngchis/Support-Ticket-Router-12K-Cleaned) | complaint | Apache-2.0 | en | 6,200 |
+| [CohereForAI/aya_redteaming](https://huggingface.co/datasets/CohereForAI/aya_redteaming) | safety-moderation | Apache-2.0 | en, fr, es, ru, ar, hi, sr, tl | 494 |
+| [CohereLabs/aya_dataset](https://huggingface.co/datasets/CohereLabs/aya_dataset) | language-id | apache-2.0 | 65 incl. ar, de, en, fr, hi, it, ja, nl, pl, pt, ru, es, tr, zh | 6,200 |
+| [community-datasets/re_dial](https://huggingface.co/datasets/community-datasets/re_dial) | sentiment | CC-BY-4.0 | en | 6,200 |
+| [community-datasets/tapaco](https://huggingface.co/datasets/community-datasets/tapaco) | similarity | cc-by-2.0 (Tatoeba CC-BY 2.0 FR) | en, de, fr, es, it, pt, nl, pl +6 | 6,200 |
+| [Console-AI/IT-helpdesk-synthetic-tickets](https://huggingface.co/datasets/Console-AI/IT-helpdesk-synthetic-tickets) | complaint, urgency | MIT | en | 1,000 |
+| ConvLab/crosswoz (github thu-coai/CrossWOZ) | intent-dialogue-act | Apache-2.0 | zh | 6,200 |
+| [ddrg/super_eurlex](https://huggingface.co/datasets/ddrg/super_eurlex) | topic | MIT card; EUR-Lex reuse authorised incl. commercial with attribution (Decision 2011/833/EU) | bg, cs, da, de, el, en, es, et +16 | 6,200 |
+| [declare-lab/CategoricalHarmfulQA](https://huggingface.co/datasets/declare-lab/CategoricalHarmfulQA) | safety-moderation | Apache-2.0 | en, zh, vi | 550 |
+| [dell-research-harvard/headlines-semantic-similarity](https://huggingface.co/datasets/dell-research-harvard/headlines-semantic-similarity) | similarity | cc-by-2.0 (off-copyright US newspapers) | en | 6,200 |
+| [dhruv0808/indic_sentiment_analyzer](https://huggingface.co/datasets/dhruv0808/indic_sentiment_analyzer) | sentiment | CC-BY-4.0 | en, hi, te, ta, kn, or, bn, gu +4 | 6,200 |
+| [dvgodoy/CUAD_v1_Contract_Understanding_clause_classification](https://huggingface.co/datasets/dvgodoy/CUAD_v1_Contract_Understanding_clause_classification) | topic | CC-BY-4.0 | en | 6,200 |
+| [E3-JSI/synthetic-multi-pii-ner-v1](https://huggingface.co/datasets/E3-JSI/synthetic-multi-pii-ner-v1) | pii | mit | en, fr, de, el, nl, it, sl | 2,971 |
+| [elvanalabs/sarcasm-statements-90](https://huggingface.co/datasets/elvanalabs/sarcasm-statements-90) | sarcasm | mit | en | 90 |
+| [Fumika/Wikinews-multilingual](https://huggingface.co/datasets/Fumika/Wikinews-multilingual) | topic | CC-BY-2.5 (Wikinews) | en, es, fr, de, pt, pl, it, zh +25 | 6,200 |
+| [gfissore/arxiv-abstracts-2021](https://huggingface.co/datasets/gfissore/arxiv-abstracts-2021) | topic | CC0-1.0 (arXiv metadata) | en | 6,200 |
+| [asappresearch/abcd](https://github.com/asappresearch/abcd) | complaint | MIT (GitHub LICENSE) | en | 6,200 |
+| [bvidgen/Dynamically-Generated-Hate-Speech-Dataset (v0.2.3.csv; NOT tasksource/dynahate mirror tagged gpl)](https://github.com/bvidgen/Dynamically-Generated-Hate-Speech-Dataset) | toxicity-hate | CC-BY-4.0 (upstream README) | en | 6,200 |
+| [HLTCHKUST/BiToD (mirror DeepPavlov/BiToD)](https://github.com/HLTCHKUST/BiToD) | intent-dialogue-act | Apache-2.0 | en, zh | 6,200 |
+| [PolyAI-LDN/task-specific-datasets/nlupp](https://github.com/PolyAI-LDN/task-specific-datasets) | intent-dialogue-act | CC-BY-4.0 | en | 705 |
+| [wwbp/empathic_reactions](https://github.com/wwbp/empathic_reactions) | emotion | cc-by-4.0 | en | 3,719 |
+| [GoktugD/turkish-formality-rewrite-500k](https://huggingface.co/datasets/GoktugD/turkish-formality-rewrite-500k) | formality | cc0-1.0 | tr | 6,200 |
+| [GoktugD/turkish-intent-classification-1m](https://huggingface.co/datasets/GoktugD/turkish-intent-classification-1m) | intent-dialogue-act | CC0-1.0 (template-generated) | tr | 6,200 |
+| [GoktugD/turkish-nli-constructed-1.5m](https://huggingface.co/datasets/GoktugD/turkish-nli-constructed-1.5m) | nli | cc0-1.0 | tr | 6,200 |
+| [google-research-datasets/poem_sentiment](https://huggingface.co/datasets/google-research-datasets/poem_sentiment) | sentiment | CC-BY-4.0 | en | 892 |
+| google-research-datasets/taskmaster1/2/3 (github Taskmaster TM-1..TM-4) | intent-dialogue-act | CC-BY-4.0 | en | 6,200 |
+| [gretelai/gretel-pii-masking-en-v1](https://huggingface.co/datasets/gretelai/gretel-pii-masking-en-v1) | pii | apache-2.0 | en | 6,200 |
+| [gretelai/synthetic_pii_finance_multilingual](https://huggingface.co/datasets/gretelai/synthetic_pii_finance_multilingual) | pii | apache-2.0 | en, fr, de, nl, es, it, sv | 6,200 |
+| [hblim/customer-complaints](https://huggingface.co/datasets/hblim/customer-complaints) | complaint | MIT | en | 1,260 |
+| [Helsinki-NLP/tatoeba](https://huggingface.co/datasets/Helsinki-NLP/tatoeba) | language-id | cc-by-2.0 | 300+ incl. all priority | 6,200 |
+| [Helsinki-NLP/tatoeba](https://huggingface.co/datasets/Helsinki-NLP/tatoeba) | similarity | cc-by-2.0 (Tatoeba CC-BY 2.0 FR) | en, de, fr, es, it, pt, nl, pl +6 | 6,200 |
+| [ibm-research/AttaQ](https://huggingface.co/datasets/ibm-research/AttaQ) | safety-moderation | MIT | en | 1,402 |
+| [IDinsight/urgency_detection_maternal_health_synthetic](https://huggingface.co/datasets/IDinsight/urgency_detection_maternal_health_synthetic) | urgency | mit | en | 6,200 |
+| jagoldz/gahd (filter via GitHub jagol/gahd gahd_disaggregated.csv) | toxicity-hate | CC-BY-4.0 | de | 5,441 |
+| [jmccardle/pulse-sofroniew-emotion-concept-texts](https://huggingface.co/datasets/jmccardle/pulse-sofroniew-emotion-concept-texts) | emotion | cc-by-4.0 | en | 6,200 |
+| [joelniklaus/covid19_emergency_event](https://huggingface.co/datasets/joelniklaus/covid19_emergency_event) | topic | CC0-1.0 | en, fr, hu, it, nb, nl, pl | 1,202 |
+| [joelniklaus/german_argument_mining](https://huggingface.co/datasets/joelniklaus/german_argument_mining) | argument-mining | cc-by-4.0 | de | 6,200 |
+| [Johnson8187/Chinese_Multi-Emotion_Dialogue_Dataset](https://huggingface.co/datasets/Johnson8187/Chinese_Multi-Emotion_Dialogue_Dataset) | emotion | mit | zh | 6,200 |
+| [JusteLeo/French-emotion](https://huggingface.co/datasets/JusteLeo/French-emotion) | emotion | mit | fr | 6,200 |
+| [kchawla123/casino](https://huggingface.co/datasets/kchawla123/casino) | intent-dialogue-act | CC-BY-4.0 | en | 3,643 |
+| [Kenshiii/synthetic-product-reviews](https://huggingface.co/datasets/Kenshiii/synthetic-product-reviews) | sentiment | CC-BY-4.0 | en | 687 |
+| [KhiredNetworks/synthetic-product-reviews](https://huggingface.co/datasets/KhiredNetworks/synthetic-product-reviews) | sentiment | MIT | en | 6,200 |
+| [leonvanbokhorst/synthetic-complaints-v2](https://huggingface.co/datasets/leonvanbokhorst/synthetic-complaints-v2) | complaint, sentiment | MIT | en | 6,200 |
+| [liri-uzh/cfpb-complaints-mini](https://huggingface.co/datasets/liri-uzh/cfpb-complaints-mini) | complaint | CC0-1.0 (CFPB public domain) | en | 6,200 |
+| [llm-for-emotion/Cultural-Emo](https://huggingface.co/datasets/llm-for-emotion/Cultural-Emo) | emotion | mit | ar, de, en, hi, es | 3,999 |
+| [lyon-nlp/clustering-hal-s2s](https://huggingface.co/datasets/lyon-nlp/clustering-hal-s2s) | topic | Apache-2.0 card; HAL metadata CC0 | fr | 6,200 |
+| [masakhane/InjongoIntent](https://huggingface.co/datasets/masakhane/InjongoIntent) | intent-dialogue-act | Apache-2.0 | en, am, ee, ha, ig, rw, ln, lg +9 | 6,200 |
+| [matsuxr/JaGovFaqs-22k](https://huggingface.co/datasets/matsuxr/JaGovFaqs-22k) | similarity | cc-by-4.0 (Japanese government copyright policy) | ja | 6,200 |
+| [maximoss/mnli-nineeleven-fr](https://huggingface.co/datasets/maximoss/mnli-nineeleven-fr) | nli | bsd-2-clause | fr | 3,988 |
+| [MoritzLaurer/synthetic_zeroshot_mixtral_v0.1](https://huggingface.co/datasets/MoritzLaurer/synthetic_zeroshot_mixtral_v0.1) | nli | apache-2.0 (Mixtral-8x7B outputs) | en | 6,200 |
+| [mteb/toxic_conversations_50k](https://huggingface.co/datasets/mteb/toxic_conversations_50k) | toxicity-hate | CC-BY-4.0 (Civil Comments text CC0) | en | 6,200 |
+| [NABA-AI/LUB-Saudi-Arabic-Intent](https://huggingface.co/datasets/NABA-AI/LUB-Saudi-Arabic-Intent) | intent-dialogue-act | CC-BY-4.0 (synthetic) | ar | 1,500 |
+| [NagaYu/deference-keigo-corpus](https://huggingface.co/datasets/NagaYu/deference-keigo-corpus) | formality | cc-by-4.0 | ja | 3,186 |
+| [napsternxg/wands](https://huggingface.co/datasets/napsternxg/wands) | similarity | MIT (wayfair/WANDS) | en | 6,200 |
+| [NortheasternUniversity/big_patent](https://huggingface.co/datasets/NortheasternUniversity/big_patent) | topic | CC-BY-4.0 | en | 6,200 |
+| [Novora/Tri-Class-Sentiment-Synthetic](https://huggingface.co/datasets/Novora/Tri-Class-Sentiment-Synthetic) | sentiment | CC0-1.0 | en | 6,200 |
+| [nvidia/Aegis-AI-Content-Safety-Dataset-1.0](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-1.0) | safety-moderation | CC-BY-4.0 | en | 6,200 |
+| [nvidia/Aegis-AI-Content-Safety-Dataset-2.0](https://huggingface.co/datasets/nvidia/Aegis-AI-Content-Safety-Dataset-2.0) | safety-moderation | CC-BY-4.0 | en | 6,200 |
+| [nvidia/CantTalkAboutThis-Topic-Control-Dataset](https://huggingface.co/datasets/nvidia/CantTalkAboutThis-Topic-Control-Dataset) | safety-topic-control | CC-BY-4.0 | en | 1,073 |
+| [nvidia/Nemotron-PII](https://huggingface.co/datasets/nvidia/Nemotron-PII) | pii | cc-by-4.0 | en | 6,200 |
+| [nvidia/Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1](https://huggingface.co/datasets/nvidia/Nemotron-RL-Agentic-Indirect-Prompt-Injection-v1) | prompt-injection (indirect) | CC-BY-4.0 | en | 1,220 |
+| [nyu-mll/multi_nli](https://huggingface.co/datasets/nyu-mll/multi_nli) | nli | OANC licence (permissive, commercial OK) per MNLI paper/card; fiction genre mixed incl. CC-BY-SA-3.0 | en | 6,200 |
+| [OpenAssistant/oasst2](https://huggingface.co/datasets/OpenAssistant/oasst2) | toxicity-moderation | Apache-2.0 | en, es, ru, zh, de, fr, pt, it +4 | 6,200 |
+| [OpenSafetyLab/Salad-Data](https://huggingface.co/datasets/OpenSafetyLab/Salad-Data) | safety-moderation | Apache-2.0 | en | 6,200 |
+| [OrSabbach/food-delivery-support-tickets](https://huggingface.co/datasets/OrSabbach/food-delivery-support-tickets) | complaint | MIT | en | 6,200 |
+| [pacoreyes/StanceSentences](https://huggingface.co/datasets/pacoreyes/StanceSentences) | stance | apache-2.0 | en | 972 |
+| pfb30/multi_woz_v22 (github budzianowski/multiwoz) | intent-dialogue-act | MIT (upstream); Apache-2.0 (card) | en | 6,200 |
+| [PolyAI/minds14](https://huggingface.co/datasets/PolyAI/minds14) | complaint | CC-BY-4.0 | cs, de, en, es, fr, it, ko, nl +4 | 6,200 |
+| [Process-Venue/IntentClassification_Dataset_for_AI_Assistant_Prompt_Routing_Hindi](https://huggingface.co/datasets/Process-Venue/IntentClassification_Dataset_for_AI_Assistant_Prompt_Routing_Hindi) | intent-dialogue-act | Apache-2.0 (text provenance undocumented) | hi | 4,998 |
+| [reshabhs/SPML_Chatbot_Prompt_Injection](https://huggingface.co/datasets/reshabhs/SPML_Chatbot_Prompt_Injection) | prompt-injection | MIT | en | 6,200 |
+| [RichardSakaguchiMS/brazilian-customer-service-conversations](https://huggingface.co/datasets/RichardSakaguchiMS/brazilian-customer-service-conversations) | complaint, sentiment | Apache-2.0 | pt | 1,510 |
+| [s2pidape/support-ticket-dataset](https://huggingface.co/datasets/s2pidape/support-ticket-dataset) | complaint | CC-BY-4.0 | en | 6,200 |
+| [shreyaspullehf/emotion-dataset-20-emotions](https://huggingface.co/datasets/shreyaspullehf/emotion-dataset-20-emotions) | emotion | mit | en | 6,200 |
+| [sileod/attempto-nli](https://huggingface.co/datasets/sileod/attempto-nli) | nli | apache-2.0 | en | 6,200 |
+| [SINAI/ALIA-es-discriminative-stance-detection](https://huggingface.co/datasets/SINAI/ALIA-es-discriminative-stance-detection) | stance | cc-by-4.0 | es | 2,850 |
+| [stjiris/IRIS_sts](https://huggingface.co/datasets/stjiris/IRIS_sts) | similarity | mit | pt | 3,334 |
+| [sutro/synthetic-product-reviews-20k](https://huggingface.co/datasets/sutro/synthetic-product-reviews-20k) | sentiment | MIT | en | 6,200 |
+| [sweatSmile/sarcastic-dataset](https://huggingface.co/datasets/sweatSmile/sarcastic-dataset) | sarcasm | mit | en | 1,440 |
+| [takehika/wanli-ja-nli](https://huggingface.co/datasets/takehika/wanli-ja-nli) | nli | cc-by-4.0 | ja | 6,200 |
+| [tanaos/synthetic-emotion-detection-dataset-v1](https://huggingface.co/datasets/tanaos/synthetic-emotion-detection-dataset-v1) | emotion | mit | en | 6,200 |
+| [tanaos/synthetic-sentiment-analysis-dataset-v1](https://huggingface.co/datasets/tanaos/synthetic-sentiment-analysis-dataset-v1) | sentiment | MIT | en | 6,200 |
+| [tasksource/esci](https://huggingface.co/datasets/tasksource/esci) | similarity | apache-2.0 (amazon-science/esci-data) | en, es, ja | 6,200 |
+| [tasksource/help-desk-tickets](https://huggingface.co/datasets/tasksource/help-desk-tickets) | complaint, urgency | CC-BY-4.0 (Mendeley btm76zndnt v3) | en, mixed | 357 |
+| [tasksource/it-support-tickets](https://huggingface.co/datasets/tasksource/it-support-tickets) | complaint | CC-BY-4.0 (Zenodo 7648117) | en, de, pt, es | 1,568 |
+| [theatticusproject/cuad-qa](https://huggingface.co/datasets/theatticusproject/cuad-qa) | reading-comprehension | CC-BY-4.0 | en | 6,200 |
+| [theatticusproject/maud](https://huggingface.co/datasets/theatticusproject/maud) | reading-comprehension | CC-BY-4.0 | en | 6,200 |
+| [uoe-nlp/multi3-nlu](https://huggingface.co/datasets/uoe-nlp/multi3-nlu) | intent-dialogue-act | CC-BY-4.0 | am, mr, tr, es | 5,636 |
+| [urchade/synthetic-pii-ner-mistral-v1](https://huggingface.co/datasets/urchade/synthetic-pii-ner-mistral-v1) | pii | apache-2.0 | en, fr, it, de, es | 6,200 |
+| [vic35get/nhtsa_complaints_dataset](https://huggingface.co/datasets/vic35get/nhtsa_complaints_dataset) | complaint | Apache-2.0 card; NHTSA US federal data | en | 6,200 |
+| [Wismut/nym-pii-multilingual-data](https://huggingface.co/datasets/Wismut/nym-pii-multilingual-data) | pii | mit | en, de, fr, es, it, pt, nl, pl +14 | 6,200 |
+| [WorkInTheDark/FairytaleQA](https://huggingface.co/datasets/WorkInTheDark/FairytaleQA) | reading-comprehension | Apache-2.0 | en | 6,200 |
+| [YiMeng-SYSU/chinese-logic-sentiment-dataset](https://huggingface.co/datasets/YiMeng-SYSU/chinese-logic-sentiment-dataset) | sentiment | Apache-2.0 | zh | 2,176 |
+| [3609356 (ClaimBuster)](https://zenodo.org/records/3609356) | claim-detection | cc-by-4.0 | en | 1,032 |
+
 ## Used only for evaluation (never trained on)
 
 | Dataset | Licence |

README.md

@@ -47,90 +47,90 @@ model-index:
     dataset:
       name: typed-decisions (test split, first 2,000 decisions; its train split is
         replay data)
-      type: typed-decisions
+      type: LocalLLaMA/typed-decisions
     metrics:
     - type: accuracy
-      value: 0.7585
+      value: 0.763
   - task:
       type: text-classification
     dataset:
       name: Banking77 (test split, first 2,000 rows, all 77 intents in one question)
-      type: banking77
+      type: PolyAI/banking77
     metrics:
     - type: accuracy
-      value: 0.9035
+      value: 0.914
   - task:
       type: text-classification
     dataset:
       name: MASSIVE intents (mean over 12 languages, 150 seeded stratified test rows
         each)
-      type: massive_intents
+      type: AmazonScience/massive
     metrics:
     - type: accuracy
-      value: 0.7717
+      value: 0.7995
   - task:
       type: text-classification
     dataset:
       name: AG News (zero-shot (never trained on), first 2,000 test rows)
-      type: ag_news
+      type: fancyzhx/ag_news
     metrics:
     - type: accuracy
-      value: 0.9315
+      value: 0.9295
   - task:
       type: text-classification
     dataset:
       name: DAIR Emotion (zero-shot, first 2,000 test rows)
-      type: dair_emotion
+      type: dair-ai/emotion
     metrics:
     - type: accuracy
-      value: 0.5265
+      value: 0.504
   - task:
       type: text-classification
     dataset:
       name: HWU64 intents (English, 150 rows; rows overlapping MASSIVE removed)
-      type: hwu64_intents
+      type: hwu64
     metrics:
     - type: accuracy
-      value: 0.82
+      value: 0.7867
   - task:
       type: text-classification
     dataset:
       name: SIB-200 topics (zero-shot, mean over 4 languages, 150 rows each)
-      type: sib-200_topics
+      type: Davlan/sib200
     metrics:
     - type: accuracy
-      value: 0.7217
+      value: 0.7817
   - task:
       type: text-classification
     dataset:
       name: Sentiment (zero-shot, mean over 12 languages, 150 rows each)
-      type: sentiment
+      type: tyqiangz/multilingual-sentiments
     metrics:
     - type: accuracy
-      value: 0.5933
+      value: 0.605
   - task:
       type: text-classification
     dataset:
       name: HateCheck (zero-shot, mean over 11 languages, 150 rows each)
-      type: hatecheck
+      type: mteb/multi-hatecheck
     metrics:
     - type: accuracy
-      value: 0.6467
+      value: 0.6358
   - task:
       type: text-classification
     dataset:
       name: Belebele reading (zero-shot, mean over 4 languages, 150 rows each)
-      type: belebele_reading
+      type: facebook/belebele
     metrics:
     - type: accuracy
-      value: 0.31
+      value: 0.2583
 ---
 
 # Statim Decide Multilingual Base
 
 A decision model for [Statim](https://github.com/BEKO2210/statim), the native C++ engine for typed decisions: ask any text a
 **choice**, a **score** or a **yes/no** question and get calibrated answers from one forward pass, on
-CPU or GPU, without Python at runtime. Version **0.4.0**, fine-tuned from
+CPU or GPU, without Python at runtime. Version **0.7.0**, fine-tuned from
 [`convaiinnovations/laya-multilingual`](https://huggingface.co/convaiinnovations/laya-multilingual) (mmBERT-base encoder).
 
 <video controls preload="none" width="100%" poster="https://beko2210.github.io/statim/images/film-16x9.webp" src="https://beko2210.github.io/statim/video/statim-flagship-60s-16x9.mp4"></video>
@@ -170,84 +170,140 @@ Statim's parity tests; q8_0 is smaller and faster on CPU with slightly different
 ## Evaluation
 
 Measured by Statim's no-harm gate ([`tools/finetune/gate.py`](https://github.com/BEKO2210/statim/blob/main/tools/finetune/gate.py))
-on held-out test data the model selection never looked at. Against the checkpoint it was trained from, on 54 held-out suites: **18 significant gains, 36 within noise, 0 regressions** (two combined binomial standard errors).
+on held-out test data the model selection never looked at. Against the checkpoint it was trained from, on 89 held-out suites: **23 significant gains, 65 within noise, 0 regressions** (gains: more than two combined binomial standard errors; regressions: significant after Holm-Bonferroni over all suites; 1 nominal drop beyond two standard errors did not stay significant).
 
 | Suite | Role | This model | Base checkpoint | Protocol |
 |---|---|---|---|---|
-| typed-decisions | trained | **0.7585** | 0.3510 | test split, first 2,000 decisions; its train split is replay data |
-| Banking77 | trained | **0.9035** | 0.5175 | test split, first 2,000 rows, all 77 intents in one question |
-| MASSIVE intents | trained | **0.7717** | 0.3400 | mean over 12 languages, 150 seeded stratified test rows each |
-| AG News | held out | **0.9315** | 0.9380 | zero-shot (never trained on), first 2,000 test rows |
-| DAIR Emotion | held out | **0.5265** | 0.5320 | zero-shot, first 2,000 test rows |
-| HWU64 intents | held out | **0.8200** | 0.5000 | English, 150 rows; rows overlapping MASSIVE removed |
-| SIB-200 topics | held out | **0.7217** | 0.7484 | zero-shot, mean over 4 languages, 150 rows each |
-| Sentiment | held out | **0.5933** | 0.5595 | zero-shot, mean over 12 languages, 150 rows each |
-| HateCheck | held out | **0.6467** | 0.6303 | zero-shot, mean over 11 languages, 150 rows each |
-| Belebele reading | held out | **0.3100** | 0.2467 | zero-shot, mean over 4 languages, 150 rows each |
+| typed-decisions | trained | **0.7630** | 0.7585 | test split, first 2,000 decisions; its train split is replay data |
+| Banking77 | trained | **0.9140** | 0.9035 | test split, first 2,000 rows, all 77 intents in one question |
+| MASSIVE intents | trained | **0.7995** | 0.7717 | mean over 12 languages, 150 seeded stratified test rows each |
+| AG News | held out | **0.9295** | 0.9315 | zero-shot (never trained on), first 2,000 test rows |
+| DAIR Emotion | held out | **0.5040** | 0.5265 | zero-shot, first 2,000 test rows |
+| HWU64 intents | held out | **0.7867** | 0.8200 | English, 150 rows; rows overlapping MASSIVE removed |
+| SIB-200 topics | held out | **0.7817** | 0.7217 | zero-shot, mean over 4 languages, 150 rows each |
+| Sentiment | held out | **0.6050** | 0.5933 | zero-shot, mean over 12 languages, 150 rows each |
+| HateCheck | held out | **0.6358** | 0.6467 | zero-shot, mean over 11 languages, 150 rows each |
+| Belebele reading | held out | **0.2583** | 0.3100 | zero-shot, mean over 4 languages, 150 rows each |
+
+### Decision categories
+
+One held-out suite per decision category, built from splits of the training sources that the mixture never loads; any text that also occurs in the training mixture is dropped. 150 items per language, macro over languages.
+
+| Category | Languages | This model | Base checkpoint |
+|---|---|---|---|
+| complaint | en | **0.767** | 0.607 |
+| emotion | de, en, es, fr, hi, zh | **0.586** | 0.520 |
+| fact check | en | **0.313** | 0.260 |
+| formality | ja, tr | **0.773** | 0.503 |
+| intent | en, nl, tr | **0.753** | 0.516 |
+| nli | en, ja, tr | **0.747** | 0.704 |
+| pii | ar, de, en, es, fr, it, ja, nl, ru, sv, zh | **0.856** | 0.595 |
+| reading | en | **0.927** | 0.560 |
+| safety | en | **0.727** | 0.600 |
+| sentiment | en, zh | **0.800** | 0.740 |
+| similarity | pt | **0.833** | 0.627 |
+| stance | en | **0.893** | 0.660 |
+| topic | en | **0.607** | 0.267 |
+| urgency | en | **0.893** | 0.660 |
 
 Published systems under the same protocol, for orientation: typed-decisions meraGPT 0.768,
 laya-typed-decisions 0.766, Jev 0.727; AG News zero-shot Laya 0.950, GPT-3 (CARP) 0.926, Jev 0.881;
-Banking77 supervised MPNet 0.941; MASSIVE XLM-R base 0.857 (full train set). Sources:
+Banking77 supervised MPNet 0.941; MASSIVE XLM-R base 0.857 over 12 languages (full train set). Sources:
 [docs/ROADMAP.md](https://github.com/BEKO2210/statim/blob/main/docs/ROADMAP.md).
 
-<details><summary>All 54 held-out suites</summary>
+<details><summary>All 89 held-out suites</summary>
 
 | Suite | Accuracy | Rows |
 |---|---|---|
-| `amazon_massive_intent/ar` | 0.6600 | 150 |
-| `amazon_massive_intent/de` | 0.7667 | 150 |
-| `amazon_massive_intent/en` | 0.8133 | 150 |
-| `amazon_massive_intent/es` | 0.7733 | 150 |
-| `amazon_massive_intent/fr` | 0.7600 | 150 |
-| `amazon_massive_intent/hi` | 0.7600 | 150 |
-| `amazon_massive_intent/it` | 0.7667 | 150 |
-| `amazon_massive_intent/ja` | 0.7800 | 150 |
+| `amazon_massive_intent/ar` | 0.7067 | 150 |
+| `amazon_massive_intent/de` | 0.7867 | 150 |
+| `amazon_massive_intent/en` | 0.8267 | 150 |
+| `amazon_massive_intent/es` | 0.8133 | 150 |
+| `amazon_massive_intent/fr` | 0.8200 | 150 |
+| `amazon_massive_intent/hi` | 0.7867 | 150 |
+| `amazon_massive_intent/it` | 0.8200 | 150 |
+| `amazon_massive_intent/ja` | 0.8267 | 150 |
 | `amazon_massive_intent/pl` | 0.8000 | 150 |
-| `amazon_massive_intent/ru` | 0.8200 | 150 |
-| `amazon_massive_intent/tr` | 0.7600 | 150 |
-| `amazon_massive_intent/zh-CN` | 0.8000 | 150 |
-| `belebele/ar` | 0.2467 | 150 |
-| `belebele/de` | 0.3400 | 150 |
-| `belebele/en` | 0.3467 | 150 |
-| `belebele/hi` | 0.3067 | 150 |
-| `farstail/fa` | 0.6333 | 150 |
-| `go_emotions/en` | 0.4667 | 150 |
-| `hwu64/en` | 0.8200 | 150 |
-| `indonli/id` | 0.7000 | 150 |
-| `multi_hatecheck/ar` | 0.5800 | 150 |
-| `multi_hatecheck/de` | 0.6667 | 150 |
-| `multi_hatecheck/en` | 0.6333 | 150 |
-| `multi_hatecheck/es` | 0.6733 | 150 |
-| `multi_hatecheck/fr` | 0.6667 | 150 |
-| `multi_hatecheck/hi` | 0.5600 | 150 |
-| `multi_hatecheck/it` | 0.6867 | 150 |
-| `multi_hatecheck/nl` | 0.6000 | 150 |
-| `multi_hatecheck/pl` | 0.6400 | 150 |
-| `multi_hatecheck/pt` | 0.7067 | 150 |
-| `multi_hatecheck/zh` | 0.7000 | 150 |
-| `multilingual_sentiments/ar` | 0.5533 | 150 |
-| `multilingual_sentiments/de` | 0.5600 | 150 |
-| `multilingual_sentiments/en` | 0.6400 | 150 |
-| `multilingual_sentiments/es` | 0.5467 | 150 |
-| `multilingual_sentiments/fr` | 0.5533 | 150 |
-| `multilingual_sentiments/hi` | 0.4600 | 150 |
-| `multilingual_sentiments/id` | 0.7133 | 150 |
-| `multilingual_sentiments/it` | 0.6000 | 150 |
-| `multilingual_sentiments/ja` | 0.6733 | 150 |
-| `multilingual_sentiments/ms` | 0.5267 | 150 |
-| `multilingual_sentiments/pt` | 0.6600 | 150 |
-| `multilingual_sentiments/zh` | 0.6333 | 150 |
-| `semrel/ar` | 0.2400 | 150 |
-| `semrel/en` | 0.2200 | 150 |
-| `semrel/hi` | 0.2200 | 150 |
-| `sib200/ar` | 0.7067 | 150 |
-| `sib200/de` | 0.7267 | 150 |
-| `sib200/en` | 0.7467 | 150 |
+| `amazon_massive_intent/ru` | 0.8333 | 150 |
+| `amazon_massive_intent/tr` | 0.7867 | 150 |
+| `amazon_massive_intent/zh-CN` | 0.7867 | 150 |
+| `belebele/ar` | 0.2933 | 150 |
+| `belebele/de` | 0.2133 | 150 |
+| `belebele/en` | 0.2467 | 150 |
+| `belebele/hi` | 0.2800 | 150 |
+| `categories:complaint/en` | 0.7667 | 150 |
+| `categories:emotion/de` | 0.4533 | 150 |
+| `categories:emotion/en` | 0.5733 | 150 |
+| `categories:emotion/es` | 0.5733 | 150 |
+| `categories:emotion/fr` | 0.6600 | 150 |
+| `categories:emotion/hi` | 0.7133 | 150 |
+| `categories:emotion/zh` | 0.5400 | 150 |
+| `categories:fact_check/en` | 0.3133 | 150 |
+| `categories:formality/ja` | 0.5467 | 150 |
+| `categories:formality/tr` | 1.0000 | 150 |
+| `categories:intent/en` | 0.8200 | 150 |
+| `categories:intent/nl` | 0.4400 | 150 |
+| `categories:intent/tr` | 1.0000 | 150 |
+| `categories:nli/en` | 0.8067 | 150 |
+| `categories:nli/ja` | 0.6333 | 150 |
+| `categories:nli/tr` | 0.8000 | 150 |
+| `categories:pii/ar` | 0.8467 | 150 |
+| `categories:pii/de` | 0.8400 | 150 |
+| `categories:pii/en` | 0.8933 | 150 |
+| `categories:pii/es` | 0.8533 | 150 |
+| `categories:pii/fr` | 0.8933 | 150 |
+| `categories:pii/it` | 0.8400 | 150 |
+| `categories:pii/ja` | 0.8467 | 150 |
+| `categories:pii/nl` | 0.7600 | 150 |
+| `categories:pii/ru` | 0.9067 | 150 |
+| `categories:pii/sv` | 0.8333 | 150 |
+| `categories:pii/zh` | 0.9000 | 150 |
+| `categories:reading/en` | 0.9267 | 150 |
+| `categories:safety/en` | 0.7267 | 150 |
+| `categories:sentiment/en` | 0.8000 | 150 |
+| `categories:sentiment/zh` | 0.8000 | 150 |
+| `categories:similarity/pt` | 0.8333 | 150 |
+| `categories:stance/en` | 0.8933 | 150 |
+| `categories:topic/en` | 0.6067 | 150 |
+| `categories:urgency/en` | 0.8933 | 150 |
+| `farstail/fa` | 0.7000 | 150 |
+| `go_emotions/en` | 0.4733 | 150 |
+| `hwu64/en` | 0.7867 | 150 |
+| `indonli/id` | 0.6667 | 150 |
+| `multi_hatecheck/ar` | 0.5733 | 150 |
+| `multi_hatecheck/de` | 0.6867 | 150 |
+| `multi_hatecheck/en` | 0.6133 | 150 |
+| `multi_hatecheck/es` | 0.6200 | 150 |
+| `multi_hatecheck/fr` | 0.6867 | 150 |
+| `multi_hatecheck/hi` | 0.5733 | 150 |
+| `multi_hatecheck/it` | 0.6600 | 150 |
+| `multi_hatecheck/nl` | 0.6533 | 150 |
+| `multi_hatecheck/pl` | 0.6333 | 150 |
+| `multi_hatecheck/pt` | 0.6467 | 150 |
+| `multi_hatecheck/zh` | 0.6467 | 150 |
+| `multilingual_sentiments/ar` | 0.5600 | 150 |
+| `multilingual_sentiments/de` | 0.5800 | 150 |
+| `multilingual_sentiments/en` | 0.6933 | 150 |
+| `multilingual_sentiments/es` | 0.5267 | 150 |
+| `multilingual_sentiments/fr` | 0.5867 | 150 |
+| `multilingual_sentiments/hi` | 0.5333 | 150 |
+| `multilingual_sentiments/id` | 0.7467 | 150 |
+| `multilingual_sentiments/it` | 0.5667 | 150 |
+| `multilingual_sentiments/ja` | 0.6600 | 150 |
+| `multilingual_sentiments/ms` | 0.5333 | 150 |
+| `multilingual_sentiments/pt` | 0.6333 | 150 |
+| `multilingual_sentiments/zh` | 0.6400 | 150 |
+| `semrel/ar` | 0.2467 | 150 |
+| `semrel/en` | 0.2000 | 150 |
+| `semrel/hi` | 0.2267 | 150 |
+| `sib200/ar` | 0.7867 | 150 |
+| `sib200/de` | 0.8000 | 150 |
+| `sib200/en` | 0.8333 | 150 |
 | `sib200/hi` | 0.7067 | 150 |
-| `test/ag_news` | 0.9315 | 2000 |
-| `test/banking77` | 0.9035 | 2000 |
-| `test/emotion` | 0.5265 | 2000 |
-| `test/typed_decisions` | 0.7585 | 2000 |
+| `test/ag_news` | 0.9295 | 2000 |
+| `test/banking77` | 0.9140 | 2000 |
+| `test/emotion` | 0.5040 | 2000 |
+| `test/typed_decisions` | 0.7630 | 2000 |
 
 </details>
 
@@ -255,8 +311,8 @@ Reproduce these numbers: [REPRODUCE.md](https://github.com/BEKO2210/statim/blob/
 
 ## Training
 
-Multi-task fine-tuning with `train_multitask.py --clean` from checkpoint `laya-multilingual`, best
-epoch `19/raw` selected on validation data only. Training data: only sources whose
+Multi-task fine-tuning with `train_multitask.py --clean` from checkpoint `laya-multilingual-big1`, best
+epoch `12/raw` selected on validation data only. Training data: only sources whose
 licence permits commercial use and imposes no ShareAlike or copyleft terms (Banking77, MASSIVE,
 typed-decisions replay, a licence-audited tasksource mixture, Nemotron-Safety-Guard, IndicGuard,
 MINDS-14, SNIPS), every one listed with its licence in

SHA256SUMS

@@ -1,12 +1,12 @@
-5c0aa16064949ae42cb1db6f563c81e14e6838483608ad1d679d418e7d864fc3  DATA_LICENSES.md
+a8e5e0bb5dfdd8d831f2d1bdfafdb1ddfecb1fc3e662ef21929ef8511389a075  DATA_LICENSES.md
 1106ca74eec5e589cec780e4f19f5218ea84ceb6b0d23594d7d9b5b30a2b2322  LICENSE-MODEL.md
 2416135f4479640b6237a07ab323585cd7eaee91041dd4208846d22d17aceb61  NOTICE
-4a28b309bfa966a9ff2fe53ed4b9e04cc9da6ef9da1e26dbca8d759c9730c41f  README.md
+a12f4a971a7fdbfd73657f8e8a575072d3355bbd992f31aa06fedbeba9546bcd  README.md
 83f6916d13ef0f556ac461f28308dc2bffa7ebeadee8ec9e2db5812020ea5bb4  checkpoint/encoder/config.json
-e34dc66c238dc1d3606ce12c24e7deb853e9eb2319c76c99f725af274bf88f3f  checkpoint/model.safetensors
-ae3b1cc7d74023d2d7641e7518855b17c869800bcf7552514eef5bd60f4898eb  checkpoint/rl_agent_config.json
+c44425f14ac9d55508f73a6f371e4e2802ed59e646287c3abbec10b653a19840  checkpoint/model.safetensors
+9b889caa1ae60f0d6e343a04799f88985d705799e5dfa7e05accbc639c131841  checkpoint/rl_agent_config.json
 609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f  checkpoint/tokenizer/tokenizer.json
 6c6b2d8e3c84ce0e671c129cd6b374b235d6f9863042a5836358d00a89bbb5a1  checkpoint/tokenizer/tokenizer_config.json
-ba88393c5f94e51c7e69d5e73ed18b583afd6d4e420247d6dfc86be3b3a817b5  evaluation/eval.json
-965d77854b2a2ce8384842120042b5768cd094166f965ad79a09f4b03d81b4a6  statim-decide-multilingual-base-f32.gguf
-39fa79fdc381a21cc324909b810d0cfb414f474656fa7994d561118b3fbc73fd  statim-decide-multilingual-base-q8_0.gguf
+d9cd02f468711f0fe0b3d7e528e095fe954bbbd6ea18b68764d96b3da84f6a58  evaluation/eval.json
+03349c619a05c78d94217c5c8279317a8ea9abc4a696d90480a370ea3a402eeb  statim-decide-multilingual-base-f32.gguf
+96fb3971656ee6b596cb0c108aff4bbe1306d87707ffe9ae5f364a52b919ff1b  statim-decide-multilingual-base-q8_0.gguf

checkpoint/model.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e34dc66c238dc1d3606ce12c24e7deb853e9eb2319c76c99f725af274bf88f3f
+oid sha256:c44425f14ac9d55508f73a6f371e4e2802ed59e646287c3abbec10b653a19840
 size 643835524

checkpoint/rl_agent_config.json

@@ -9,20 +9,19 @@
   },
   "cost_wrong_act": 3.0,
   "amp_dtype": "bf16",
-  "model_name": "laya-multilingual-big1",
+  "model_name": "laya-multilingual-multitask",
   "temperature": [
-    1.758754849433899,
-    1.2504303455352783,
-    1.3224705457687378
+    1.3157309293746948,
+    1.0678483247756958,
+    1.254639983177185
   ],
   "temperature_by_options": {
-    "choice:11+": 1.813578724861145,
-    "noul:2": 1.3224705457687378,
-    "choice:3-5": 1.3400647640228271,
-    "score:3-5": 1.3033926486968994,
-    "choice:2": 1.6684647798538208,
-    "choice:6-10": 1.2673249244689941,
-    "score:2": 1.3971556425094604
+    "choice:11+": 1.4175829887390137,
+    "choice:3-5": 1.1190439462661743,
+    "score:3-5": 1.0678483247756958,
+    "noul:2": 1.254639983177185,
+    "choice:6-10": 1.068841576576233,
+    "choice:2": 1.3097771406173706
   },
   "training": {
     "updates": 15987,
@@ -33,442 +32,304 @@
   },
   "fine_tuned": true,
   "training_multitask": {
-    "base": "laya-multilingual",
-    "best_epoch": "19/raw",
+    "base": "laya-multilingual-big1",
+    "best_epoch": "12/raw",
     "dev_before": {
-      "banking77": 0.464,
-      "massive": 0.3017,
-      "mixture": 0.475
+      "banking77": 0.898,
+      "massive": 0.8467,
+      "mixture": 0.4875
     },
     "log": [
       {
         "epoch": 1,
         "weights": "raw",
-        "train_ce": 1.6029,
-        "banking77": 0.752,
-        "massive": 0.6267,
-        "mixture": 0.52,
-        "mean": 0.6329,
-        "seconds": 1090
+        "train_ce": 0.9343,
+        "banking77": 0.858,
+        "massive": 0.825,
+        "mixture": 0.5793,
+        "mean": 0.7541,
+        "seconds": 1676
       },
       {
         "epoch": 1,
         "weights": "ema",
-        "train_ce": 1.6029,
-        "banking77": 0.614,
-        "massive": 0.4817,
-        "mixture": 0.488,
-        "mean": 0.5279,
-        "seconds": 1092
+        "train_ce": 0.9343,
+        "banking77": 0.904,
+        "massive": 0.8417,
+        "mixture": 0.5545,
+        "mean": 0.7667,
+        "seconds": 1687
       },
       {
         "epoch": 2,
         "weights": "raw",
-        "train_ce": 1.1416,
-        "banking77": 0.772,
-        "massive": 0.675,
-        "mixture": 0.529,
-        "mean": 0.6587,
-        "seconds": 2165
+        "train_ce": 0.9273,
+        "banking77": 0.882,
+        "massive": 0.8217,
+        "mixture": 0.6067,
+        "mean": 0.7701,
+        "seconds": 3372
       },
       {
         "epoch": 2,
         "weights": "ema",
-        "train_ce": 1.1416,
-        "banking77": 0.774,
-        "massive": 0.64,
-        "mixture": 0.533,
-        "mean": 0.649,
-        "seconds": 2166
+        "train_ce": 0.9273,
+        "banking77": 0.904,
+        "massive": 0.84,
+        "mixture": 0.6015,
+        "mean": 0.7818,
+        "seconds": 3374
       },
       {
         "epoch": 3,
         "weights": "raw",
-        "train_ce": 1.031,
-        "banking77": 0.806,
-        "massive": 0.715,
-        "mixture": 0.545,
-        "mean": 0.6887,
-        "seconds": 3241
+        "train_ce": 0.8975,
+        "banking77": 0.882,
+        "massive": 0.825,
+        "mixture": 0.6352,
+        "mean": 0.7807,
+        "seconds": 5049
       },
       {
         "epoch": 3,
         "weights": "ema",
-        "train_ce": 1.031,
-        "banking77": 0.818,
-        "massive": 0.6917,
-        "mixture": 0.537,
-        "mean": 0.6822,
-        "seconds": 3242
+        "train_ce": 0.8975,
+        "banking77": 0.902,
+        "massive": 0.8567,
+        "mixture": 0.6312,
+        "mean": 0.7966,
+        "seconds": 5049
       },
       {
         "epoch": 4,
         "weights": "raw",
-        "train_ce": 0.952,
-        "banking77": 0.816,
-        "massive": 0.7367,
-        "mixture": 0.536,
-        "mean": 0.6962,
-        "seconds": 4324
+        "train_ce": 0.8468,
+        "banking77": 0.894,
+        "massive": 0.8333,
+        "mixture": 0.6432,
+        "mean": 0.7902,
+        "seconds": 6725
       },
       {
         "epoch": 4,
         "weights": "ema",
-        "train_ce": 0.952,
-        "banking77": 0.854,
-        "massive": 0.74,
-        "mixture": 0.551,
-        "mean": 0.715,
-        "seconds": 4326
+        "train_ce": 0.8468,
+        "banking77": 0.902,
+        "massive": 0.8433,
+        "mixture": 0.6495,
+        "mean": 0.7983,
+        "seconds": 6725
       },
       {
         "epoch": 5,
         "weights": "raw",
-        "train_ce": 0.9832,
-        "banking77": 0.836,
-        "massive": 0.74,
-        "mixture": 0.573,
-        "mean": 0.7163,
-        "seconds": 5404
+        "train_ce": 0.8105,
+        "banking77": 0.89,
+        "massive": 0.8367,
+        "mixture": 0.6505,
+        "mean": 0.7924,
+        "seconds": 8391
       },
       {
         "epoch": 5,
         "weights": "ema",
-        "train_ce": 0.9832,
-        "banking77": 0.848,
-        "massive": 0.7517,
-        "mixture": 0.545,
-        "mean": 0.7149,
-        "seconds": 5406
+        "train_ce": 0.8105,
+        "banking77": 0.898,
+        "massive": 0.85,
+        "mixture": 0.6588,
+        "mean": 0.8023,
+        "seconds": 8391
       },
       {
         "epoch": 6,
         "weights": "raw",
-        "train_ce": 0.8739,
-        "banking77": 0.856,
-        "massive": 0.7517,
-        "mixture": 0.561,
-        "mean": 0.7229,
-        "seconds": 6476
+        "train_ce": 0.7723,
+        "banking77": 0.894,
+        "massive": 0.845,
+        "mixture": 0.6663,
+        "mean": 0.8018,
+        "seconds": 10062
       },
       {
         "epoch": 6,
         "weights": "ema",
-        "train_ce": 0.8739,
-        "banking77": 0.874,
-        "massive": 0.7633,
-        "mixture": 0.571,
-        "mean": 0.7361,
-        "seconds": 6477
+        "train_ce": 0.7723,
+        "banking77": 0.894,
+        "massive": 0.845,
+        "mixture": 0.6657,
+        "mean": 0.8016,
+        "seconds": 10062
       },
       {
         "epoch": 7,
         "weights": "raw",
-        "train_ce": 0.9355,
-        "banking77": 0.834,
-        "massive": 0.7633,
-        "mixture": 0.564,
-        "mean": 0.7204,
-        "seconds": 7550
+        "train_ce": 0.7471,
+        "banking77": 0.904,
+        "massive": 0.8333,
+        "mixture": 0.6703,
+        "mean": 0.8025,
+        "seconds": 11743
       },
       {
         "epoch": 7,
         "weights": "ema",
-        "train_ce": 0.9355,
-        "banking77": 0.872,
-        "massive": 0.7667,
-        "mixture": 0.568,
-        "mean": 0.7356,
-        "seconds": 7550
+        "train_ce": 0.7471,
+        "banking77": 0.894,
+        "massive": 0.8433,
+        "mixture": 0.6698,
+        "mean": 0.8024,
+        "seconds": 11745
       },
       {
         "epoch": 8,
         "weights": "raw",
-        "train_ce": 0.8557,
-        "banking77": 0.866,
-        "massive": 0.765,
-        "mixture": 0.573,
-        "mean": 0.7347,
-        "seconds": 8627
+        "train_ce": 0.7179,
+        "banking77": 0.89,
+        "massive": 0.84,
+        "mixture": 0.6815,
+        "mean": 0.8038,
+        "seconds": 13428
       },
       {
         "epoch": 8,
         "weights": "ema",
-        "train_ce": 0.8557,
-        "banking77": 0.886,
-        "massive": 0.77,
-        "mixture": 0.575,
-        "mean": 0.7437,
-        "seconds": 8627
+        "train_ce": 0.7179,
+        "banking77": 0.898,
+        "massive": 0.84,
+        "mixture": 0.6773,
+        "mean": 0.8051,
+        "seconds": 13431
       },
       {
         "epoch": 9,
         "weights": "raw",
-        "train_ce": 0.8353,
-        "banking77": 0.882,
-        "massive": 0.7633,
-        "mixture": 0.564,
-        "mean": 0.7364,
-        "seconds": 9700
+        "train_ce": 0.6946,
+        "banking77": 0.894,
+        "massive": 0.835,
+        "mixture": 0.6877,
+        "mean": 0.8056,
+        "seconds": 15113
       },
       {
         "epoch": 9,
         "weights": "ema",
-        "train_ce": 0.8353,
-        "banking77": 0.9,
-        "massive": 0.7883,
-        "mixture": 0.576,
-        "mean": 0.7548,
-        "seconds": 9700
+        "train_ce": 0.6946,
+        "banking77": 0.89,
+        "massive": 0.84,
+        "mixture": 0.6862,
+        "mean": 0.8054,
+        "seconds": 15114
       },
       {
         "epoch": 10,
         "weights": "raw",
-        "train_ce": 0.7674,
-        "banking77": 0.868,
-        "massive": 0.7783,
-        "mixture": 0.569,
-        "mean": 0.7384,
-        "seconds": 10771
+        "train_ce": 0.6722,
+        "banking77": 0.894,
+        "massive": 0.8383,
+        "mixture": 0.6893,
+        "mean": 0.8072,
+        "seconds": 16795
       },
       {
         "epoch": 10,
         "weights": "ema",
-        "train_ce": 0.7674,
-        "banking77": 0.902,
-        "massive": 0.7867,
-        "mixture": 0.576,
-        "mean": 0.7549,
-        "seconds": 10771
-      },
-      {
-        "epoch": 11,
-        "weights": "raw",
-        "train_ce": 0.7038,
+        "train_ce": 0.6722,
         "banking77": 0.892,
-        "massive": 0.7817,
-        "mixture": 0.578,
-        "mean": 0.7506,
-        "seconds": 11841
+        "massive": 0.84,
+        "mixture": 0.6905,
+        "mean": 0.8075,
+        "seconds": 16796
       },
       {
         "epoch": 11,
-        "weights": "ema",
-        "train_ce": 0.7038,
-        "banking77": 0.896,
-        "massive": 0.7917,
-        "mixture": 0.566,
-        "mean": 0.7512,
-        "seconds": 11841
-      },
-      {
-        "epoch": 12,
         "weights": "raw",
-        "train_ce": 0.6528,
-        "banking77": 0.878,
-        "massive": 0.7883,
-        "mixture": 0.588,
-        "mean": 0.7514,
-        "seconds": 12915
-      },
-      {
-        "epoch": 12,
-        "weights": "ema",
-        "train_ce": 0.6528,
+        "train_ce": 0.6562,
         "banking77": 0.894,
-        "massive": 0.795,
-        "mixture": 0.572,
-        "mean": 0.7537,
-        "seconds": 12915
+        "massive": 0.8467,
+        "mixture": 0.6938,
+        "mean": 0.8115,
+        "seconds": 18472
       },
       {
-        "epoch": 13,
-        "weights": "raw",
-        "train_ce": 0.6207,
-        "banking77": 0.882,
-        "massive": 0.8083,
-        "mixture": 0.582,
-        "mean": 0.7574,
-        "seconds": 13985
-      },
-      {
-        "epoch": 13,
+        "epoch": 11,
         "weights": "ema",
-        "train_ce": 0.6207,
-        "banking77": 0.888,
-        "massive": 0.795,
-        "mixture": 0.574,
-        "mean": 0.7523,
-        "seconds": 13987
-      },
-      {
-        "epoch": 14,
-        "weights": "raw",
-        "train_ce": 0.5874,
+        "train_ce": 0.6562,
         "banking77": 0.894,
-        "massive": 0.7917,
-        "mixture": 0.585,
-        "mean": 0.7569,
-        "seconds": 15061
-      },
-      {
-        "epoch": 14,
-        "weights": "ema",
-        "train_ce": 0.5874,
-        "banking77": 0.892,
-        "massive": 0.7983,
-        "mixture": 0.572,
-        "mean": 0.7541,
-        "seconds": 15061
+        "massive": 0.8433,
+        "mixture": 0.6932,
+        "mean": 0.8102,
+        "seconds": 18473
       },
       {
-        "epoch": 15,
-        "weights": "raw",
-        "train_ce": 0.5589,
-        "banking77": 0.9,
-        "massive": 0.8017,
-        "mixture": 0.58,
-        "mean": 0.7606,
-        "seconds": 16133
-      },
-      {
-        "epoch": 15,
-        "weights": "ema",
-        "train_ce": 0.5589,
-        "banking77": 0.898,
-        "massive": 0.81,
-        "mixture": 0.582,
-        "mean": 0.7633,
-        "seconds": 16135
-      },
-      {
-        "epoch": 16,
-        "weights": "raw",
-        "train_ce": 0.536,
-        "banking77": 0.898,
-        "massive": 0.795,
-        "mixture": 0.589,
-        "mean": 0.7607,
-        "seconds": 17201
-      },
-      {
-        "epoch": 16,
-        "weights": "ema",
-        "train_ce": 0.536,
-        "banking77": 0.902,
-        "massive": 0.81,
-        "mixture": 0.577,
-        "mean": 0.763,
-        "seconds": 17201
-      },
-      {
-        "epoch": 17,
-        "weights": "raw",
-        "train_ce": 0.5204,
-        "banking77": 0.9,
-        "massive": 0.8083,
-        "mixture": 0.589,
-        "mean": 0.7658,
-        "seconds": 18271
-      },
-      {
-        "epoch": 17,
-        "weights": "ema",
-        "train_ce": 0.5204,
-        "banking77": 0.902,
-        "massive": 0.8083,
-        "mixture": 0.59,
-        "mean": 0.7668,
-        "seconds": 18273
-      },
-      {
-        "epoch": 18,
-        "weights": "raw",
-        "train_ce": 0.514,
-        "banking77": 0.902,
-        "massive": 0.805,
-        "mixture": 0.592,
-        "mean": 0.7663,
-        "seconds": 19354
-      },
-      {
-        "epoch": 18,
-        "weights": "ema",
-        "train_ce": 0.514,
-        "banking77": 0.9,
-        "massive": 0.805,
-        "mixture": 0.592,
-        "mean": 0.7657,
-        "seconds": 19354
-      },
-      {
-        "epoch": 19,
-        "weights": "raw",
-        "train_ce": 0.4997,
-        "banking77": 0.898,
-        "massive": 0.8133,
-        "mixture": 0.599,
-        "mean": 0.7701,
-        "seconds": 20426
-      },
-      {
-        "epoch": 19,
-        "weights": "ema",
-        "train_ce": 0.4997,
-        "banking77": 0.902,
-        "massive": 0.81,
-        "mixture": 0.593,
-        "mean": 0.7683,
-        "seconds": 20428
-      },
-      {
-        "epoch": 20,
+        "epoch": 12,
         "weights": "raw",
-        "train_ce": 0.4975,
+        "train_ce": 0.6488,
         "banking77": 0.896,
-        "massive": 0.81,
-        "mixture": 0.594,
-        "mean": 0.7667,
-        "seconds": 21505
+        "massive": 0.8483,
+        "mixture": 0.6968,
+        "mean": 0.8137,
+        "seconds": 20158
       },
       {
-        "epoch": 20,
+        "epoch": 12,
         "weights": "ema",
-        "train_ce": 0.4975,
-        "banking77": 0.898,
-        "massive": 0.81,
-        "mixture": 0.596,
-        "mean": 0.768,
-        "seconds": 21505
+        "train_ce": 0.6488,
+        "banking77": 0.896,
+        "massive": 0.8467,
+        "mixture": 0.696,
+        "mean": 0.8129,
+        "seconds": 20159
       }
     ],
     "counts": {
       "banking77": 9493,
-      "massive": 102000,
+      "massive": 560371,
       "typed": 5400,
-      "mixture": 187561,
-      "distill": 6000
+      "mixture:safety": 77867,
+      "mixture:intent": 53461,
+      "mixture": 191609,
+      "mixture:complaint": 55918,
+      "mixture:emotion": 41770,
+      "mixture:similarity": 50686,
+      "mixture:sentiment": 46531,
+      "mixture:other": 97480,
+      "mixture:topic": 38020,
+      "mixture:reading": 18443,
+      "mixture:nli": 46891,
+      "distill": 24000
     },
-    "distill": 6000,
-    "massive_per_lang": 2000,
+    "distill": 24000,
+    "massive_per_lang": 11000,
+    "massive_langs": null,
     "sentiment_per_lang": 1200,
     "lr": [
       2.5e-05,
       0.0001
     ],
-    "epochs": 20,
+    "epochs": 12,
     "seed": 20260926,
     "budget": {
       "banking77": 12000,
       "massive": 16000,
-      "mixture": 20000,
+      "mixture": 24000,
       "typed": 4000,
-      "distill": 3000
+      "distill": 9000,
+      "mixture:safety": 2975,
+      "mixture:intent": 2465,
+      "mixture:complaint": 2521,
+      "mixture:emotion": 2179,
+      "mixture:similarity": 2400,
+      "mixture:sentiment": 2299,
+      "mixture:other": 3328,
+      "mixture:topic": 2078,
+      "mixture:reading": 1448,
+      "mixture:nli": 2308
     },
     "warmup": 0.06,
     "ema": 0.999,
     "patience": 3,
-    "mixture": "mixture-v5.jsonl.gz",
+    "optim": "adamw",
+    "mixture": "mixture-v8.jsonl.gz",
     "clean": true
   }
 }

evaluation/eval.json

@@ -1,282 +1,509 @@
 {
- "model": "models/laya-multilingual-big1",
+ "model": "models/laya-multilingual-v9",
  "validation": {
   "banking77": 0.898,
-  "massive": 0.81,
-  "sentiment": 0.58,
-  "emotion": 0.49,
-  "ag_news": 0.924
+  "massive": 0.8133,
+  "sentiment": 0.5667,
+  "emotion": 0.484,
+  "ag_news": 0.908
  },
  "heldout": {
   "test/ag_news": {
-   "acc": 0.9315,
+   "acc": 0.9295,
    "n": 2000,
-   "ece": 0.0088
+   "ece": 0.0282
   },
   "test/emotion": {
-   "acc": 0.5265,
+   "acc": 0.504,
    "n": 2000,
-   "ece": 0.2127
+   "ece": 0.2148
   },
   "test/banking77": {
-   "acc": 0.9035,
+   "acc": 0.914,
    "n": 2000,
-   "ece": 0.085
+   "ece": 0.0796
   },
   "test/typed_decisions": {
-   "acc": 0.7585,
+   "acc": 0.763,
    "n": 2000,
    "ece": null
   },
   "amazon_massive_intent/de": {
-   "acc": 0.7667,
+   "acc": 0.7867,
    "n": 150,
-   "ece": 0.1134
+   "ece": 0.0726
   },
   "amazon_massive_intent/en": {
-   "acc": 0.8133,
+   "acc": 0.8267,
    "n": 150,
-   "ece": 0.0751
+   "ece": 0.0544
   },
   "amazon_massive_intent/fr": {
-   "acc": 0.76,
+   "acc": 0.82,
    "n": 150,
-   "ece": 0.0894
+   "ece": 0.0611
   },
   "amazon_massive_intent/es": {
-   "acc": 0.7733,
+   "acc": 0.8133,
    "n": 150,
-   "ece": 0.1043
+   "ece": 0.0746
   },
   "amazon_massive_intent/it": {
-   "acc": 0.7667,
+   "acc": 0.82,
    "n": 150,
-   "ece": 0.1365
+   "ece": 0.0877
   },
   "amazon_massive_intent/tr": {
-   "acc": 0.76,
+   "acc": 0.7867,
    "n": 150,
-   "ece": 0.1207
+   "ece": 0.0614
   },
   "amazon_massive_intent/pl": {
    "acc": 0.8,
    "n": 150,
-   "ece": 0.106
+   "ece": 0.0585
   },
   "amazon_massive_intent/ru": {
-   "acc": 0.82,
+   "acc": 0.8333,
    "n": 150,
-   "ece": 0.119
+   "ece": 0.0718
   },
   "amazon_massive_intent/ja": {
-   "acc": 0.78,
+   "acc": 0.8267,
    "n": 150,
-   "ece": 0.0907
+   "ece": 0.0875
   },
   "amazon_massive_intent/zh-CN": {
-   "acc": 0.8,
+   "acc": 0.7867,
    "n": 150,
-   "ece": 0.1124
+   "ece": 0.0666
   },
   "amazon_massive_intent/ar": {
-   "acc": 0.66,
+   "acc": 0.7067,
    "n": 150,
-   "ece": 0.1182
+   "ece": 0.0606
   },
   "amazon_massive_intent/hi": {
-   "acc": 0.76,
+   "acc": 0.7867,
    "n": 150,
-   "ece": 0.1192
+   "ece": 0.0609
   },
   "multilingual_sentiments/ar": {
-   "acc": 0.5533,
+   "acc": 0.56,
    "n": 150,
-   "ece": 0.1422
+   "ece": 0.1499
   },
   "multilingual_sentiments/zh": {
-   "acc": 0.6333,
+   "acc": 0.64,
    "n": 150,
-   "ece": 0.1794
+   "ece": 0.1621
   },
   "multilingual_sentiments/en": {
-   "acc": 0.64,
+   "acc": 0.6933,
    "n": 150,
-   "ece": 0.1111
+   "ece": 0.0874
   },
   "multilingual_sentiments/fr": {
-   "acc": 0.5533,
+   "acc": 0.5867,
    "n": 150,
-   "ece": 0.1315
+   "ece": 0.1192
   },
   "multilingual_sentiments/de": {
-   "acc": 0.56,
+   "acc": 0.58,
    "n": 150,
-   "ece": 0.1269
+   "ece": 0.1361
   },
   "multilingual_sentiments/hi": {
-   "acc": 0.46,
+   "acc": 0.5333,
    "n": 150,
-   "ece": 0.2051
+   "ece": 0.1885
   },
   "multilingual_sentiments/id": {
-   "acc": 0.7133,
+   "acc": 0.7467,
    "n": 150,
-   "ece": 0.1191
+   "ece": 0.0827
   },
   "multilingual_sentiments/it": {
-   "acc": 0.6,
+   "acc": 0.5667,
    "n": 150,
-   "ece": 0.1432
+   "ece": 0.1605
   },
   "multilingual_sentiments/ja": {
-   "acc": 0.6733,
+   "acc": 0.66,
    "n": 150,
-   "ece": 0.1392
+   "ece": 0.1772
   },
   "multilingual_sentiments/ms": {
-   "acc": 0.5267,
+   "acc": 0.5333,
    "n": 150,
-   "ece": 0.189
+   "ece": 0.197
   },
   "multilingual_sentiments/pt": {
-   "acc": 0.66,
+   "acc": 0.6333,
    "n": 150,
-   "ece": 0.0952
+   "ece": 0.1496
   },
   "multilingual_sentiments/es": {
-   "acc": 0.5467,
+   "acc": 0.5267,
    "n": 150,
    "ece": 0.2224
   },
   "go_emotions/en": {
-   "acc": 0.4667,
+   "acc": 0.4733,
    "n": 150,
-   "ece": 0.1269
+   "ece": 0.1692
   },
   "multi_hatecheck/en": {
-   "acc": 0.6333,
+   "acc": 0.6133,
    "n": 150,
-   "ece": 0.0572
+   "ece": 0.108
   },
   "multi_hatecheck/de": {
-   "acc": 0.6667,
+   "acc": 0.6867,
    "n": 150,
-   "ece": 0.1041
+   "ece": 0.0632
   },
   "multi_hatecheck/fr": {
-   "acc": 0.6667,
+   "acc": 0.6867,
    "n": 150,
-   "ece": 0.0257
+   "ece": 0.1177
   },
   "multi_hatecheck/es": {
-   "acc": 0.6733,
+   "acc": 0.62,
    "n": 150,
-   "ece": 0.038
+   "ece": 0.0724
   },
   "multi_hatecheck/it": {
-   "acc": 0.6867,
+   "acc": 0.66,
    "n": 150,
-   "ece": 0.0648
+   "ece": 0.0974
   },
   "multi_hatecheck/nl": {
-   "acc": 0.6,
+   "acc": 0.6533,
    "n": 150,
-   "ece": 0.1074
+   "ece": 0.0851
   },
   "multi_hatecheck/pl": {
-   "acc": 0.64,
+   "acc": 0.6333,
    "n": 150,
-   "ece": 0.0847
+   "ece": 0.0867
   },
   "multi_hatecheck/pt": {
-   "acc": 0.7067,
+   "acc": 0.6467,
    "n": 150,
-   "ece": 0.0865
+   "ece": 0.0782
   },
   "multi_hatecheck/zh": {
-   "acc": 0.7,
+   "acc": 0.6467,
    "n": 150,
-   "ece": 0.0713
+   "ece": 0.0846
   },
   "multi_hatecheck/ar": {
-   "acc": 0.58,
+   "acc": 0.5733,
    "n": 150,
-   "ece": 0.085
+   "ece": 0.0855
   },
   "multi_hatecheck/hi": {
-   "acc": 0.56,
+   "acc": 0.5733,
    "n": 150,
-   "ece": 0.0877
+   "ece": 0.1385
   },
   "sib200/en": {
-   "acc": 0.7467,
+   "acc": 0.8333,
    "n": 150,
-   "ece": 0.0713
+   "ece": 0.0795
   },
   "sib200/de": {
-   "acc": 0.7267,
+   "acc": 0.8,
    "n": 150,
-   "ece": 0.0726
+   "ece": 0.0728
   },
   "sib200/ar": {
-   "acc": 0.7067,
+   "acc": 0.7867,
    "n": 150,
-   "ece": 0.0872
+   "ece": 0.1031
   },
   "sib200/hi": {
    "acc": 0.7067,
    "n": 150,
-   "ece": 0.0964
+   "ece": 0.1364
   },
   "hwu64/en": {
-   "acc": 0.82,
+   "acc": 0.7867,
    "n": 150,
-   "ece": 0.0925
+   "ece": 0.0727
   },
   "indonli/id": {
-   "acc": 0.7,
+   "acc": 0.6667,
    "n": 150,
-   "ece": 0.1418
+   "ece": 0.1434
   },
   "farstail/fa": {
-   "acc": 0.6333,
+   "acc": 0.7,
    "n": 150,
-   "ece": 0.1371
+   "ece": 0.1228
   },
   "belebele/en": {
-   "acc": 0.3467,
+   "acc": 0.2467,
    "n": 150,
-   "ece": 0.1656
+   "ece": 0.3155
   },
   "belebele/de": {
-   "acc": 0.34,
+   "acc": 0.2133,
    "n": 150,
-   "ece": 0.1268
+   "ece": 0.3476
   },
   "belebele/ar": {
-   "acc": 0.2467,
+   "acc": 0.2933,
    "n": 150,
-   "ece": 0.1813
+   "ece": 0.247
   },
   "belebele/hi": {
-   "acc": 0.3067,
+   "acc": 0.28,
    "n": 150,
-   "ece": 0.1124
+   "ece": 0.3109
   },
   "semrel/en": {
-   "acc": 0.22,
+   "acc": 0.2,
    "n": 150,
-   "ece": 0.1619
+   "ece": 0.2866
   },
   "semrel/ar": {
-   "acc": 0.24,
+   "acc": 0.2467,
    "n": 150,
-   "ece": 0.1085
+   "ece": 0.2724
   },
   "semrel/hi": {
-   "acc": 0.22,
+   "acc": 0.2267,
+   "n": 150,
+   "ece": 0.2519
+  },
+  "categories:sentiment/en": {
+   "acc": 0.8,
+   "n": 150,
+   "ece": 0.0704,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:sentiment/zh": {
+   "acc": 0.8,
+   "n": 150,
+   "ece": 0.0818,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:emotion/de": {
+   "acc": 0.4533,
+   "n": 150,
+   "ece": 0.0991,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:emotion/en": {
+   "acc": 0.5733,
+   "n": 150,
+   "ece": 0.1161,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:emotion/es": {
+   "acc": 0.5733,
+   "n": 150,
+   "ece": 0.0795,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:emotion/fr": {
+   "acc": 0.66,
    "n": 150,
-   "ece": 0.1612
+   "ece": 0.0923,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:emotion/hi": {
+   "acc": 0.7133,
+   "n": 150,
+   "ece": 0.1035,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:emotion/zh": {
+   "acc": 0.54,
+   "n": 150,
+   "ece": 0.0978,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:complaint/en": {
+   "acc": 0.7667,
+   "n": 150,
+   "ece": 0.0751,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:nli/en": {
+   "acc": 0.8067,
+   "n": 150,
+   "ece": 0.0613,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:nli/ja": {
+   "acc": 0.6333,
+   "n": 150,
+   "ece": 0.12,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:nli/tr": {
+   "acc": 0.8,
+   "n": 150,
+   "ece": 0.0736,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:safety/en": {
+   "acc": 0.7267,
+   "n": 150,
+   "ece": 0.0738,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:reading/en": {
+   "acc": 0.9267,
+   "n": 150,
+   "ece": 0.0317,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:similarity/pt": {
+   "acc": 0.8333,
+   "n": 150,
+   "ece": 0.0741,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:topic/en": {
+   "acc": 0.6067,
+   "n": 150,
+   "ece": 0.1069,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:intent/en": {
+   "acc": 0.82,
+   "n": 150,
+   "ece": 0.1027,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:intent/nl": {
+   "acc": 0.44,
+   "n": 150,
+   "ece": 0.1407,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:intent/tr": {
+   "acc": 1.0,
+   "n": 150,
+   "ece": 0.0014,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:stance/en": {
+   "acc": 0.8933,
+   "n": 150,
+   "ece": 0.0921,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:formality/ja": {
+   "acc": 0.5467,
+   "n": 150,
+   "ece": 0.0616,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:formality/tr": {
+   "acc": 1.0,
+   "n": 150,
+   "ece": 0.0667,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:urgency/en": {
+   "acc": 0.8933,
+   "n": 150,
+   "ece": 0.1511,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:fact_check/en": {
+   "acc": 0.3133,
+   "n": 150,
+   "ece": 0.3145,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/ar": {
+   "acc": 0.8467,
+   "n": 150,
+   "ece": 0.0645,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/de": {
+   "acc": 0.84,
+   "n": 150,
+   "ece": 0.0387,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/en": {
+   "acc": 0.8933,
+   "n": 150,
+   "ece": 0.0967,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/es": {
+   "acc": 0.8533,
+   "n": 150,
+   "ece": 0.06,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/fr": {
+   "acc": 0.8933,
+   "n": 150,
+   "ece": 0.0387,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/it": {
+   "acc": 0.84,
+   "n": 150,
+   "ece": 0.0638,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/ja": {
+   "acc": 0.8467,
+   "n": 150,
+   "ece": 0.0413,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/nl": {
+   "acc": 0.76,
+   "n": 150,
+   "ece": 0.1127,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/ru": {
+   "acc": 0.9067,
+   "n": 150,
+   "ece": 0.0725,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/sv": {
+   "acc": 0.8333,
+   "n": 150,
+   "ece": 0.0326,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
+  },
+  "categories:pii/zh": {
+   "acc": 0.9,
+   "n": 150,
+   "ece": 0.0608,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6"
   }
- }
+ },
+ "reported": {
+  "categories:emotion/pt": {
+   "acc": 0.5067,
+   "n": 150,
+   "ece": 0.1138,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6",
+   "note": "no v6 training source has emotion labels in Portuguese: zero-shot"
+  },
+  "categories:emotion/ru": {
+   "acc": 0.6933,
+   "n": 150,
+   "ece": 0.0784,
+   "pool": "e0c48b24784ef0c9+b8a87e85f3bf72b6",
+   "note": "no v6 training source has emotion labels in Russian: zero-shot"
+  }
+ },
+ "mixture": "/home/belkis/Schreibtisch/My WorkSpace/statim/data/mixture-v8.jsonl.gz"
 }

statim-decide-multilingual-base-f32.gguf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:965d77854b2a2ce8384842120042b5768cd094166f965ad79a09f4b03d81b4a6
-size 908307072
+oid sha256:03349c619a05c78d94217c5c8279317a8ea9abc4a696d90480a370ea3a402eeb
+size 908307040

statim-decide-multilingual-base-q8_0.gguf

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:39fa79fdc381a21cc324909b810d0cfb414f474656fa7994d561118b3fbc73fd
-size 356674176
+oid sha256:96fb3971656ee6b596cb0c108aff4bbe1306d87707ffe9ae5f364a52b919ff1b
+size 356674144