Publish Laya PT/ES NLI checkpoint

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer/tokenizer.json filter=lfs diff=lfs merge=lfs -text

ATTRIBUTIONS.md

@@ -0,0 +1,10 @@
+# Attributions
+
+This checkpoint derives from `convaiinnovations/laya-multilingual`, revision `052592a15d198d9ad47da779604259b10b47b7aa`. Its Hugging Face model card declares the Apache License 2.0.
+
+Portuguese training and evaluation data came from InferBR by Lucas Bencke and contributors. The pinned source revision is `b6c5e0b27e52477bbc5433d130b0a0e825cf74dc`. InferBR is licensed under the MIT License.
+
+Spanish training and evaluation data came from InferES by Venelin Kovatchev and contributors. The pinned source revision is `c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2`. InferES is licensed under Creative Commons Attribution 4.0 International.
+
+No source dataset rows, source audio, or third-party model documentation are redistributed in this repository.
+

README.md

@@ -0,0 +1,177 @@
+---
+license: apache-2.0
+library_name: laya
+base_model: convaiinnovations/laya-multilingual
+pipeline_tag: text-classification
+language:
+  - pt
+  - es
+datasets:
+  - venelin/inferes
+tags:
+  - laya
+  - mmbert
+  - natural-language-inference
+  - portuguese
+  - spanish
+  - calibrated-decisions
+  - rlcd
+  - audio-adapter-compatible
+model-index:
+  - name: Laya PT/ES NLI
+    results:
+      - task:
+          type: text-classification
+          name: Three-way natural language inference
+        dataset:
+          type: custom
+          name: Fixed InferBR and InferES test sample
+          config: 256 Portuguese and 256 Spanish examples, seed 42
+          split: test
+        metrics:
+          - type: accuracy
+            value: 0.8671875
+            name: Accuracy
+          - type: f1
+            value: 0.8669693326424834
+            name: Macro F1
+          - type: accuracy
+            value: 0.8673734610123119
+            name: Balanced accuracy
+---
+
+# Laya PT/ES NLI
+
+This checkpoint fine-tunes `convaiinnovations/laya-multilingual` for three-way natural language inference in Portuguese and Spanish. It predicts `contradiction`, `entailment`, or `neutral` in one non-autoregressive forward pass.
+
+The checkpoint keeps the original 322M-parameter Laya architecture. It adds no inference module. It also preserves exact `input_ids` and `inputs_embeds` parity, which is required by the separate audio-adapter experiment.
+
+## Use
+
+```bash
+pip install laya
+```
+
+```python
+import laya
+
+agent = laya.load("YOUR_NAMESPACE/laya-pt-es-nli")
+questions = {
+    "relation": {
+        "type": "choice",
+        "instructions": "Qual é a relação lógica entre o estado e a hipótese: Um animal está dormindo.",
+        "criteria": {
+            "contradiction": "o estado torna a hipótese falsa",
+            "entailment": "o estado confirma a hipótese",
+            "neutral": "o estado não confirma nem contradiz a hipótese",
+        },
+    }
+}
+result = agent.predict({"state": "Um gato dorme no sofá."}, questions)
+print(result["answers"]["relation"])
+```
+
+Use localized instructions and option descriptions. The training and evaluation label order is `contradiction`, `entailment`, `neutral`.
+
+## Evaluation
+
+The main comparison uses the same 512 held-out examples for every target. It contains 256 InferBR examples and 256 InferES examples. Sampling uses seed 42. The sample-ID digest is `da18f9ec812db66ceb731299b2986d629121334b0565d0d1ea721b25e396d25e`. Local results and latency come from `four-model-nli-r2`. TypeSafe results come from `typesafe-comparison-r1` on the same sample IDs.
+
+| Model | PT macro F1 | ES macro F1 | Overall macro F1 | Balanced accuracy | Brier | ECE | p50 | p95 | Peak VRAM |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| **Laya PT/ES NLI** | **0.9295** | **0.8028** | **0.8670** | **0.8674** | 0.2185 | 0.0611 | 31.7 ms | 34.0 ms | 0.66 GiB |
+| Laya Multilingual base | 0.6054 | 0.5150 | 0.5622 | 0.5583 | 0.5926 | 0.1191 | 31.2 ms | 35.0 ms | 0.66 GiB |
+| OpenJEV v2 | 0.9248 | 0.7943 | 0.8601 | 0.8591 | **0.2057** | 0.0515 | 120.4 ms | 143.5 ms | 9.21 GiB |
+| TypeSafe JEV API | 0.9532 | 0.8366 | 0.8950 | 0.8944 | 0.1518 | **0.0188** | 585.6 ms | 668.5 ms | not available |
+
+ECE uses ten equal-width bins and top-label confidence. Brier is the sum across the three classes. Laya and TypeSafe use localized typed-choice prompts. OpenJEV uses its native premise and hypothesis template.
+
+The complete sealed test contains 3,315 examples. This checkpoint has 0.8657 macro F1 overall, 0.9324 in Portuguese, and 0.7914 in Spanish. Its option-order agreement is 0.9741. The fitted choice temperature is 6.737636.
+
+## Latency and memory
+
+The local benchmark ran on a Modal `A100-40GB` worker. The image used Python 3.12, PyTorch 2.13.0, and Transformers 5.14.1. Each latency value measures one warm batch-size-one call. It excludes model load time.
+
+| Model | p50 | p95 | Peak VRAM | Load time |
+|---|---:|---:|---:|---:|
+| **Laya PT/ES NLI** | **31.7 ms** | **34.0 ms** | **0.66 GiB** | 54.8 s |
+| Laya Multilingual base | 31.2 ms | 35.0 ms | 0.66 GiB | 41.5 s |
+| OpenJEV v2 | 120.4 ms | 143.5 ms | 9.21 GiB | 23.7 s |
+| TypeSafe JEV API | 585.6 ms | 668.5 ms | not available | not applicable |
+
+TypeSafe latency includes the remote network. It is not an apples-to-apples local-model latency result. Local measurements use 64 examples after two warm-up calls.
+
+## Training
+
+Training updated the complete encoder and decision head. It did not use LoRA or an adapter. The objective was RLCD plus cross-entropy. RLCD used four perturbations, log plus spherical reward, and a linear noise schedule from 0.4 to 0.1.
+
+The run trained independent 4-, 8-, and 16-epoch candidates from the same pinned base revision. Validation selected the 8-epoch checkpoint. The selection key was mean PT/ES macro F1, then weakest-language macro F1, then negative NLL.
+
+The selected run used these values:
+
+- batch size: 16
+- gradient accumulation: 4
+- encoder learning rate: 2.5e-5
+- head learning rate: 1e-4
+- AdamW weight decay: 0.01
+- cosine learning-rate schedule, minimum 1e-6
+- gradient clip: 1.0
+- BF16 autocast and saved BF16 weights
+- gradient checkpointing with non-reentrant execution
+- seed: 42
+- train, validation, and sealed test sizes: 13,971, 1,275, and 3,315
+
+Portuguese and Spanish had equal sampling weight. Each epoch balanced all six language-label buckets. The fitted validation temperature was saved in `rl_agent_config.json`.
+
+A Spanish-focused continuation tested 1, 2, and 4 additional epochs with a 2:1 Spanish-to-Portuguese sampling weight. No candidate passed the fixed promotion gate. This release therefore uses the original 8-epoch checkpoint.
+
+The exact run command was:
+
+```bash
+uv run modal run modal_laya_finetune.py::app.train_laya_multilingual \
+  --run laya-nli-pt-es-r1 \
+  --epoch-budgets 4,8,16 \
+  --batch-size 16 \
+  --gradient-accumulation 4
+```
+
+The comparison command was:
+
+```bash
+uv run modal run modal_benchmarks.py::app.run_nli_benchmark \
+  --run four-model-nli-r1
+```
+
+## Data and license
+
+The Portuguese source is [InferBR](https://github.com/lbencke/InferBR) at revision `b6c5e0b27e52477bbc5433d130b0a0e825cf74dc`, licensed under MIT. The Spanish source is [InferES](https://huggingface.co/datasets/venelin/inferes) at revision `c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2`, licensed under CC BY 4.0.
+
+InferES provides train and test files. The run removed train/test duplicate pairs. It then made a deterministic 10% per-class validation split from the remaining train data with seed 42. InferBR rows marked as low quality were excluded. All splits were deduplicated and checked for pair overlap.
+
+The base checkpoint declares Apache 2.0. This derivative is released under Apache 2.0. InferES attribution remains required. See `ATTRIBUTIONS.md` and `provenance.json`. This repository does not redistribute any training dataset or audio.
+
+## Audio compatibility
+
+The compatibility gate used 768 pairs from 256 MLS and Common Voice clips. Direct embedding lookup and split-state splice matched the token-ID path exactly in FP32 and BF16. Maximum logit, marker-state, and action-logit differences were zero. Label and layout agreement were 1.0.
+
+This is not an audio model. It contains no audio encoder or trained audio adapter. The compatibility result only confirms that an external adapter can enter through `inputs_embeds` without changing Laya's decision path.
+
+## Limitations
+
+- Spanish is materially weaker than Portuguese on both validation and test data.
+- Calibration is domain-specific. Refit temperature on a held-out target-domain set before using probabilities as risk scores.
+- The 512-example comparison has no confidence intervals and does not establish superiority over OpenJEV or TypeSafe.
+- The checkpoint was trained only for three-way NLI. Other Laya question types were not retrained or validated.
+- The local latency results apply to the stated software and GPU. CPU and other GPU results can differ.
+- Dataset artifacts can contain annotation errors and social biases from their source corpora.
+- Do not use this model as the sole basis for medical, legal, financial, or safety-critical decisions.
+
+## Files
+
+- `model.safetensors`: full fine-tuned model weights.
+- `encoder/` and `tokenizer/`: pinned model configuration and tokenizer.
+- `rl_agent_config.json`: Laya runtime configuration and calibrated temperature.
+- `training.json`: training configuration, validation metrics, history, and source manifest.
+- `evaluation_results.json`: compact sealed-test and comparison results.
+- `provenance.json`: base revision, dataset revisions, runtime, and compatibility gates.
+- `SHA256SUMS`: release integrity manifest.

SHA256SUMS

@@ -0,0 +1,11 @@
+cf3d6c3d6fcddd7b9087fa011c8c181bd67ec0d3f81c8dfc79e6b816edf7e10b  ATTRIBUTIONS.md
+f16153096e4e0e5b74162c36c5d7b8b59f50f8cf699f354ad36997fd56a9aac0  README.md
+212544c8358591ff7ed19a4a80cca59340cbd594729c8698dee9cd5eea5f253d  encoder/config.json
+ec1602d9559791fd5c66472a2ccd116d453d5b2771bd53f564c9934abf0d7d6b  evaluation_results.json
+94c5ade6106cba72e07987d6a2fb977e7e7d1532f23e54f6ec2bfa7bbfe719be  model.safetensors
+328300408c2332c1aa42a0bfaee9e6480f2a1b2e31948cf2cf0f39d30a4fab7b  provenance.json
+9438ec88dab246eca6b84c11cb89cdb4447b4828e1705d725fcf88ce1cda01f0  publish.py
+b86ac53abcb1eeeaf90feae206e395f043b1b089e4d0f526200728b140040b0d  rl_agent_config.json
+609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f  tokenizer/tokenizer.json
+cb01c24cdc73e85640cb6bcde2469cc4b05b7c184664cb57f6ccfc7b5a45d779  tokenizer/tokenizer_config.json
+f256ccb3930b045a5202e3afba5d0537cd133b494f8ee3788d81c26e6a9e7688  training.json

encoder/config.json

@@ -0,0 +1,79 @@
+{
+  "architectures": [
+    "ModernBertForMaskedLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "classifier_activation": "gelu",
+  "classifier_bias": false,
+  "classifier_dropout": 0.0,
+  "classifier_pooling": "mean",
+  "cls_token_id": 1,
+  "decoder_bias": true,
+  "deterministic_flash_attn": false,
+  "dtype": "float32",
+  "embedding_dropout": 0.0,
+  "eos_token_id": 1,
+  "global_attn_every_n_layers": 3,
+  "gradient_checkpointing": false,
+  "hidden_activation": "gelu",
+  "hidden_size": 768,
+  "initializer_cutoff_factor": 2.0,
+  "initializer_range": 0.02,
+  "intermediate_size": 1152,
+  "layer_norm_eps": 1e-05,
+  "layer_types": [
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention",
+    "sliding_attention",
+    "sliding_attention",
+    "full_attention"
+  ],
+  "local_attention": 128,
+  "mask_token_id": 4,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "mlp_dropout": 0.0,
+  "model_type": "modernbert",
+  "norm_bias": false,
+  "norm_eps": 1e-05,
+  "num_attention_heads": 12,
+  "num_hidden_layers": 22,
+  "pad_token_id": 0,
+  "position_embedding_type": "sans_pos",
+  "repad_logits_with_grad": false,
+  "rope_parameters": {
+    "full_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    },
+    "sliding_attention": {
+      "rope_theta": 160000,
+      "rope_type": "default"
+    }
+  },
+  "sep_token_id": 1,
+  "sparse_pred_ignore_index": -100,
+  "sparse_prediction": false,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.14.1",
+  "vocab_size": 256000
+}

evaluation_results.json

@@ -0,0 +1,33 @@
+{
+  "schema_version": 1,
+  "label_order": ["contradiction", "entailment", "neutral"],
+  "sealed_test": {
+    "n_examples": 3315,
+    "temperature": 6.737636089324951,
+    "accuracy": 0.8663650155067444,
+    "macro_f1": 0.8657161857755229,
+    "nll": 0.4384844899177551,
+    "brier": 0.22188593447208405,
+    "ece": 0.061342481945757754,
+    "option_order_agreement": 0.9740573167800903,
+    "by_language": {
+      "pt": {"n_examples": 1705, "accuracy": 0.9325513243675232, "macro_f1": 0.9324053374823894, "nll": 0.2746226489543915, "brier": 0.12065495550632477, "ece": 0.05137685628142208},
+      "es": {"n_examples": 1610, "accuracy": 0.7962732911109924, "macro_f1": 0.7914361746136098, "nll": 0.6120151281356812, "brier": 0.3290901780128479, "ece": 0.09335547377122566}
+    }
+  },
+  "fixed_512_comparison": {
+    "local_run": "four-model-nli-r2",
+    "typesafe_run": "typesafe-comparison-r1",
+    "seed": 42,
+    "per_language": 256,
+    "sample_sha256": "da18f9ec812db66ceb731299b2986d629121334b0565d0d1ea721b25e396d25e",
+    "latency_examples": 64,
+    "warmup_calls": 2,
+    "models": {
+      "laya_pt_es": {"accuracy": 0.8671875, "balanced_accuracy": 0.8673734610123119, "macro_f1": 0.8669693326424834, "pt_macro_f1": 0.9295275148432717, "es_macro_f1": 0.8027949622344871, "brier": 0.2185381066577995, "ece": 0.061121337406802964, "option_order_agreement": 0.9921875, "p50_s": 0.03165481199999931, "p95_s": 0.034017716150002285, "peak_vram_bytes": 707878400},
+      "laya_multilingual": {"accuracy": 0.55859375, "balanced_accuracy": 0.5583447332421341, "macro_f1": 0.5621945405613227, "pt_macro_f1": 0.6054199410298268, "es_macro_f1": 0.5150492462700995, "brier": 0.5925970264745448, "ece": 0.11910045781405645, "option_order_agreement": 0.830078125, "p50_s": 0.031237861000001033, "p95_s": 0.035027764200007985, "peak_vram_bytes": 704986624},
+      "openjev_v2": {"accuracy": 0.859375, "balanced_accuracy": 0.8591427268581852, "macro_f1": 0.8600981307482297, "pt_macro_f1": 0.9248010163075886, "es_macro_f1": 0.7942985329442359, "brier": 0.20570788629959827, "ece": 0.05154871946433563, "p50_s": 0.12035475150000252, "p95_s": 0.14351390009999243, "peak_vram_bytes": 9890091008},
+      "typesafe_jev": {"accuracy": 0.89453125, "balanced_accuracy": 0.8944140446876426, "macro_f1": 0.895036364601582, "pt_macro_f1": 0.9531952992402649, "es_macro_f1": 0.8365977013371774, "brier": 0.151828515625, "ece": 0.018828125000000022, "option_order_agreement": 0.986328125, "p50_s": 0.5855838539999354, "p95_s": 0.6685402502499754, "network_included": true}
+    }
+  }
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:94c5ade6106cba72e07987d6a2fb977e7e7d1532f23e54f6ec2bfa7bbfe719be
+size 643835692

provenance.json

@@ -0,0 +1,18 @@
+{
+  "schema_version": 1,
+  "run": "laya-nli-pt-es-r1",
+  "selected_candidate_epochs": 8,
+  "candidate_epoch_budgets": [4, 8, 16],
+  "spanish_continuation": {"candidate_additional_epochs": [1, 2, 4], "selected_additional_epochs": 0, "promotion": false},
+  "base_model": {"repo_id": "convaiinnovations/laya-multilingual", "revision": "052592a15d198d9ad47da779604259b10b47b7aa", "declared_license": "apache-2.0"},
+  "training_runtime": {"provider": "Modal", "gpu_request": "A100-40GB", "python": "3.12", "torch": "2.13.0", "transformers": "5.14.1", "accelerate": "1.14.0", "peft": "0.20.0"},
+  "split_sizes": {"train": 13971, "validation": 1275, "test": 3315},
+  "dataset_assets": [
+    {"dataset": "InferBR", "revision": "b6c5e0b27e52477bbc5433d130b0a0e825cf74dc", "license": "MIT", "files": {"LICENSE": "b1cd6172bf8061abcbd7917e9246ddeded210339482e802e3413b74cc7e0de38", "train.csv": "ccce8f72fae244729833bc2a2bf86bce7cff18d8383c07ec4389b1cd453b85f4", "val.csv": "45a855012940750694b37616dfc346325aadff7e661a75ddabc8b90a4375d3aa", "test.csv": "50d16e4c6d951349c188547f54c73f5699e8be0abb88780b55d3666a755dc32d"}},
+    {"dataset": "InferES", "revision": "c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2", "license": "CC-BY-4.0", "files": {"README.md": "c590a3ed872bde10e3f9f81e6963644d45587a422bb6c985a0e5aa03ee69c59e", "train.csv": "6e7fcef798b3d37522588dbcb5c88e574139aae4cb334fc163168744f12bbe7c", "test.csv": "faa73fb8567b7664cf03f5f01cf4aae1e0c940a64dc57ad8b36d1fcaeaeffc1a"}}
+  ],
+  "audio_compatibility_gate": {"n_clips": 256, "n_pairs": 768, "triplet_manifest_sha256": "f6f20be86141370fc0e12ad8ec674718210e5a60e5d421e3a302b2cd7205d0b3", "dtypes": ["float32", "bfloat16"], "max_abs_logits": 0.0, "max_abs_marker_states": 0.0, "max_abs_act_logits": 0.0, "label_agreement": 1.0, "layout_agreement": 1.0, "passed": true},
+  "redistributed_training_data": false,
+  "redistributed_audio": false
+}
+

publish.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""Validate and publish this bundle without storing or printing HF_TOKEN."""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+from pathlib import Path
+
+
+BUNDLE = Path(__file__).resolve().parent
+REQUIRED = {
+    "README.md",
+    "ATTRIBUTIONS.md",
+    "SHA256SUMS",
+    "encoder/config.json",
+    "evaluation_results.json",
+    "model.safetensors",
+    "provenance.json",
+    "rl_agent_config.json",
+    "tokenizer/tokenizer.json",
+    "tokenizer/tokenizer_config.json",
+    "training.json",
+}
+
+
+def digest(path: Path) -> str:
+    value = hashlib.sha256()
+    with path.open("rb") as handle:
+        for chunk in iter(lambda: handle.read(1024 * 1024), b""):
+            value.update(chunk)
+    return value.hexdigest()
+
+
+def validate() -> None:
+    missing = sorted(name for name in REQUIRED if not (BUNDLE / name).is_file())
+    if missing:
+        raise SystemExit(f"Missing release files: {', '.join(missing)}")
+    for name in ("encoder/config.json", "evaluation_results.json", "provenance.json", "rl_agent_config.json", "tokenizer/tokenizer_config.json", "training.json"):
+        with (BUNDLE / name).open(encoding="utf-8") as handle:
+            json.load(handle)
+    lines = (BUNDLE / "SHA256SUMS").read_text(encoding="utf-8").splitlines()
+    expected = {}
+    for line in lines:
+        checksum, name = line.split("  ", 1)
+        expected[name] = checksum
+    for name, checksum in expected.items():
+        path = BUNDLE / name
+        if not path.is_file() or digest(path) != checksum:
+            raise SystemExit(f"Checksum mismatch: {name}")
+    if not (BUNDLE / "README.md").read_text(encoding="utf-8").startswith("---\n"):
+        raise SystemExit("README.md has no Hugging Face YAML metadata")
+    print(f"Validated {len(expected)} checksums in {BUNDLE}")
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser()
+    parser.add_argument("--check-only", action="store_true")
+    parser.add_argument("--repo-id", help="Defaults to <authenticated-user>/laya-pt-es-nli")
+    args = parser.parse_args()
+    validate()
+    if args.check_only:
+        return
+    token = os.environ.get("HF_TOKEN")
+    if not token:
+        raise SystemExit("HF_TOKEN is required for upload")
+    from huggingface_hub import HfApi
+
+    api = HfApi(token=token)
+    identity = api.whoami()
+    owner = str(identity["name"])
+    repo_id = args.repo_id or f"{owner}/laya-pt-es-nli"
+    requested_owner, separator, name = repo_id.partition("/")
+    allowed = {owner, *(str(org["name"]) for org in identity.get("orgs", []))}
+    if not separator or name != "laya-pt-es-nli" or requested_owner not in allowed:
+        raise SystemExit("repo-id must be <authenticated-user-or-org>/laya-pt-es-nli")
+    api.create_repo(repo_id=repo_id, repo_type="model", private=False, exist_ok=True)
+    api.upload_folder(
+        repo_id=repo_id,
+        repo_type="model",
+        folder_path=BUNDLE,
+        commit_message="Publish Laya PT/ES NLI checkpoint",
+        ignore_patterns=["__pycache__/**", "*.pyc"],
+    )
+    print(f"Published https://huggingface.co/{repo_id}")
+
+
+if __name__ == "__main__":
+    main()

rl_agent_config.json

@@ -0,0 +1,27 @@
+{
+  "encoder": "jhu-clsp/mmBERT-base",
+  "head_layers": 2,
+  "max_len": 1024,
+  "head_max_len": 256,
+  "max_prefixes": 6,
+  "act_costs": {
+    "escalate": 0.5
+  },
+  "cost_wrong_act": 3.0,
+  "amp_dtype": "bf16",
+  "model_name": "rl-agent",
+  "temperature": [
+    6.737636089324951,
+    1.0,
+    1.0
+  ],
+  "temperature_by_options": {},
+  "training": {
+    "updates": 15987,
+    "epochs_completed": 4,
+    "hours": 4.97,
+    "world_size": 1,
+    "fine_tuned_from_checkpoint": false
+  },
+  "fine_tuned": true
+}

tokenizer/tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:609d8f4c067cd3950f88594c5a802616cea245823836ef5848ee4fc40aab5b6f
+size 34363188

tokenizer/tokenizer_config.json

@@ -0,0 +1,23 @@
+{
+  "bos_token": "<bos>",
+  "clean_up_tokenization_spaces": false,
+  "cls_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": {
+    "extra_0": "<start_of_turn>",
+    "extra_1": "<end_of_turn>"
+  },
+  "local_files_only": false,
+  "mask_token": "<mask>",
+  "model_input_names": [
+    "input_ids",
+    "attention_mask"
+  ],
+  "model_max_length": 8192,
+  "pad_token": "<pad>",
+  "padding_side": "right",
+  "sep_token": "<eos>",
+  "spaces_between_special_tokens": false,
+  "tokenizer_class": "PreTrainedTokenizerFast",
+  "unk_token": "<unk>"
+}

training.json

@@ -0,0 +1,258 @@
+{
+  "config": {
+    "epochs": 8,
+    "batch_size": 16,
+    "gradient_accumulation": 4,
+    "encoder_lr": 2.5e-05,
+    "head_lr": 0.0001,
+    "weight_decay": 0.01,
+    "group_size": 4,
+    "sigma_start": 0.4,
+    "sigma_end": 0.1,
+    "spherical_weight": 0.75,
+    "cross_entropy_weight": 1.0,
+    "grad_clip": 1.0,
+    "seed": 42,
+    "max_steps": null
+  },
+  "validation": {
+    "temperature": 6.737636089324951,
+    "overall": {
+      "n_examples": 1275,
+      "accuracy": 0.869019627571106,
+      "macro_f1": 0.8684708751770857,
+      "per_class": [
+        {
+          "precision": 0.8637469586374696,
+          "recall": 0.8352941176470589,
+          "f1": 0.8492822966507177
+        },
+        {
+          "precision": 0.8669950738916257,
+          "recall": 0.8669950738916257,
+          "f1": 0.8669950738916257
+        },
+        {
+          "precision": 0.8755458515283843,
+          "recall": 0.9031531531531531,
+          "f1": 0.8891352549889135
+        }
+      ],
+      "confusion": [
+        [
+          355,
+          29,
+          41
+        ],
+        [
+          38,
+          352,
+          16
+        ],
+        [
+          18,
+          25,
+          401
+        ]
+      ],
+      "nll": 0.4360364079475403,
+      "brier": 0.2183937281370163,
+      "ece": 0.04733535635750741
+    },
+    "by_language": {
+      "pt": {
+        "n_examples": 633,
+        "accuracy": 0.9431279897689819,
+        "macro_f1": 0.9430095871700123,
+        "per_class": [
+          {
+            "precision": 0.943127962085308,
+            "recall": 0.943127962085308,
+            "f1": 0.943127962085308
+          },
+          {
+            "precision": 0.9490740740740741,
+            "recall": 0.9534883720930233,
+            "f1": 0.951276102088167
+          },
+          {
+            "precision": 0.9368932038834952,
+            "recall": 0.9323671497584541,
+            "f1": 0.9346246973365618
+          }
+        ],
+        "confusion": [
+          [
+            199,
+            3,
+            9
+          ],
+          [
+            6,
+            205,
+            4
+          ],
+          [
+            6,
+            8,
+            193
+          ]
+        ],
+        "nll": 0.25479501485824585,
+        "brier": 0.10705043375492096,
+        "ece": 0.058824770618230104
+      },
+      "es": {
+        "n_examples": 642,
+        "accuracy": 0.795950174331665,
+        "macro_f1": 0.7919974927113839,
+        "per_class": [
+          {
+            "precision": 0.78,
+            "recall": 0.7289719626168224,
+            "f1": 0.7536231884057971
+          },
+          {
+            "precision": 0.7736842105263158,
+            "recall": 0.7696335078534031,
+            "f1": 0.7716535433070866
+          },
+          {
+            "precision": 0.8253968253968254,
+            "recall": 0.8776371308016878,
+            "f1": 0.8507157464212678
+          }
+        ],
+        "confusion": [
+          [
+            156,
+            26,
+            32
+          ],
+          [
+            32,
+            147,
+            12
+          ],
+          [
+            12,
+            17,
+            208
+          ]
+        ],
+        "nll": 0.6147370338439941,
+        "brier": 0.3281761407852173,
+        "ece": 0.07957642641849816
+      }
+    }
+  },
+  "history": [
+    {
+      "epoch": 1.0,
+      "sigma": 0.4,
+      "loss": 0.3529272115014159,
+      "ce_loss": 0.4184872030187943,
+      "reward": 0.2136233496819178
+    },
+    {
+      "epoch": 2.0,
+      "sigma": 0.35714285714285715,
+      "loss": 0.22856930547368215,
+      "ce_loss": 0.2664471471820005,
+      "reward": 0.45009025061397756
+    },
+    {
+      "epoch": 3.0,
+      "sigma": 0.3142857142857143,
+      "loss": 0.44020914059830374,
+      "ce_loss": 0.4539933671216609,
+      "reward": 0.4695609428467806
+    },
+    {
+      "epoch": 4.0,
+      "sigma": 0.27142857142857146,
+      "loss": 0.3634551393018673,
+      "ce_loss": 0.3668573111245656,
+      "reward": 0.5148424091975824
+    },
+    {
+      "epoch": 5.0,
+      "sigma": 0.22857142857142856,
+      "loss": 0.2925769820513702,
+      "ce_loss": 0.29792355096248074,
+      "reward": 0.5659681633840992
+    },
+    {
+      "epoch": 6.0,
+      "sigma": 0.1857142857142857,
+      "loss": 0.21010468557679843,
+      "ce_loss": 0.2128874539987309,
+      "reward": 0.6190930930116952
+    },
+    {
+      "epoch": 7.0,
+      "sigma": 0.14285714285714285,
+      "loss": 0.16850203610472006,
+      "ce_loss": 0.16884397977840113,
+      "reward": 0.6464318029771307
+    },
+    {
+      "epoch": 8.0,
+      "sigma": 0.09999999999999998,
+      "loss": 0.12187420110390038,
+      "ce_loss": 0.12067060629254318,
+      "reward": 0.6735549877755358
+    }
+  ],
+  "datasets": [
+    {
+      "name": "inferbr-LICENSE",
+      "url": "https://raw.githubusercontent.com/lbencke/InferBR/b6c5e0b27e52477bbc5433d130b0a0e825cf74dc/LICENSE",
+      "sha256": "b1cd6172bf8061abcbd7917e9246ddeded210339482e802e3413b74cc7e0de38",
+      "license": "MIT",
+      "revision": "b6c5e0b27e52477bbc5433d130b0a0e825cf74dc"
+    },
+    {
+      "name": "inferbr-train.csv",
+      "url": "https://raw.githubusercontent.com/lbencke/InferBR/b6c5e0b27e52477bbc5433d130b0a0e825cf74dc/train.csv",
+      "sha256": "ccce8f72fae244729833bc2a2bf86bce7cff18d8383c07ec4389b1cd453b85f4",
+      "license": "MIT",
+      "revision": "b6c5e0b27e52477bbc5433d130b0a0e825cf74dc"
+    },
+    {
+      "name": "inferbr-validation.csv",
+      "url": "https://raw.githubusercontent.com/lbencke/InferBR/b6c5e0b27e52477bbc5433d130b0a0e825cf74dc/val.csv",
+      "sha256": "45a855012940750694b37616dfc346325aadff7e661a75ddabc8b90a4375d3aa",
+      "license": "MIT",
+      "revision": "b6c5e0b27e52477bbc5433d130b0a0e825cf74dc"
+    },
+    {
+      "name": "inferbr-test.csv",
+      "url": "https://raw.githubusercontent.com/lbencke/InferBR/b6c5e0b27e52477bbc5433d130b0a0e825cf74dc/test.csv",
+      "sha256": "50d16e4c6d951349c188547f54c73f5699e8be0abb88780b55d3666a755dc32d",
+      "license": "MIT",
+      "revision": "b6c5e0b27e52477bbc5433d130b0a0e825cf74dc"
+    },
+    {
+      "name": "inferes-README.md",
+      "url": "https://huggingface.co/datasets/venelin/inferes/resolve/c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2/README.md",
+      "sha256": "c590a3ed872bde10e3f9f81e6963644d45587a422bb6c985a0e5aa03ee69c59e",
+      "license": "CC-BY-4.0",
+      "revision": "c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2"
+    },
+    {
+      "name": "inferes-train.csv",
+      "url": "https://huggingface.co/datasets/venelin/inferes/resolve/c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2/train.csv",
+      "sha256": "6e7fcef798b3d37522588dbcb5c88e574139aae4cb334fc163168744f12bbe7c",
+      "license": "CC-BY-4.0",
+      "revision": "c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2"
+    },
+    {
+      "name": "inferes-test.csv",
+      "url": "https://huggingface.co/datasets/venelin/inferes/resolve/c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2/test.csv",
+      "sha256": "faa73fb8567b7664cf03f5f01cf4aae1e0c940a64dc57ad8b36d1fcaeaeffc1a",
+      "license": "CC-BY-4.0",
+      "revision": "c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2"
+    }
+  ]
+}