README.md

---
license: apache-2.0
library_name: laya
base_model: convaiinnovations/laya-multilingual
pipeline_tag: text-classification
language:
  - pt
  - es
datasets:
  - venelin/inferes
tags:
  - laya
  - mmbert
  - natural-language-inference
  - portuguese
  - spanish
  - calibrated-decisions
  - rlcd
  - audio-adapter-compatible
model-index:
  - name: Laya PT/ES NLI
    results:
      - task:
          type: text-classification
          name: Three-way natural language inference
        dataset:
          type: custom
          name: Fixed InferBR and InferES test sample
          config: 256 Portuguese and 256 Spanish examples, seed 42
          split: test
        metrics:
          - type: accuracy
            value: 0.8671875
            name: Accuracy
          - type: f1
            value: 0.8669693326424834
            name: Macro F1
          - type: accuracy
            value: 0.8673734610123119
            name: Balanced accuracy
---

# Laya PT/ES NLI

This checkpoint fine-tunes `convaiinnovations/laya-multilingual` for three-way natural language inference in Portuguese and Spanish. It predicts `contradiction`, `entailment`, or `neutral` in one non-autoregressive forward pass.

The checkpoint keeps the original 322M-parameter Laya architecture. It adds no inference module. It also preserves exact `input_ids` and `inputs_embeds` parity, which is required by the separate audio-adapter experiment.

## Use

```bash
pip install laya
```

```python
import laya

agent = laya.load("frankyy03/laya-pt-es-nli")
questions = {
    "relation": {
        "type": "choice",
        "instructions": "Qual é a relação lógica entre o estado e a hipótese: Um animal está dormindo.",
        "criteria": {
            "contradiction": "o estado torna a hipótese falsa",
            "entailment": "o estado confirma a hipótese",
            "neutral": "o estado não confirma nem contradiz a hipótese",
        },
    }
}
result = agent.predict({"state": "Um gato dorme no sofá."}, questions)
print(result["answers"]["relation"])
```

Use localized instructions and option descriptions. The training and evaluation label order is `contradiction`, `entailment`, `neutral`.

## Evaluation

The main comparison uses the same 512 held-out examples for every target. It contains 256 InferBR examples and 256 InferES examples. Sampling uses seed 42. The sample-ID digest is `da18f9ec812db66ceb731299b2986d629121334b0565d0d1ea721b25e396d25e`. Local results and latency come from `four-model-nli-r2`. TypeSafe results come from `typesafe-comparison-r1` on the same sample IDs.

| Model | PT macro F1 | ES macro F1 | Overall macro F1 | Balanced accuracy | Brier | ECE | p50 | p95 | Peak VRAM |
|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|
| **Laya PT/ES NLI** | **0.9295** | **0.8028** | **0.8670** | **0.8674** | 0.2185 | 0.0611 | 31.7 ms | 34.0 ms | 0.66 GiB |
| Laya Multilingual base | 0.6054 | 0.5150 | 0.5622 | 0.5583 | 0.5926 | 0.1191 | 31.2 ms | 35.0 ms | 0.66 GiB |
| OpenJEV v2 | 0.9248 | 0.7943 | 0.8601 | 0.8591 | **0.2057** | 0.0515 | 120.4 ms | 143.5 ms | 9.21 GiB |
| TypeSafe JEV API | 0.9532 | 0.8366 | 0.8950 | 0.8944 | 0.1518 | **0.0188** | 585.6 ms | 668.5 ms | not available |

ECE uses ten equal-width bins and top-label confidence. Brier is the sum across the three classes. Laya and TypeSafe use localized typed-choice prompts. OpenJEV uses its native premise and hypothesis template.

The complete sealed test contains 3,315 examples. This checkpoint has 0.8657 macro F1 overall, 0.9324 in Portuguese, and 0.7914 in Spanish. Its option-order agreement is 0.9741. The fitted choice temperature is 6.737636.

## Latency and memory

The local benchmark ran on a Modal `A100-40GB` worker. The image used Python 3.12, PyTorch 2.13.0, and Transformers 5.14.1. Each latency value measures one warm batch-size-one call. It excludes model load time.

| Model | p50 | p95 | Peak VRAM | Load time |
|---|---:|---:|---:|---:|
| **Laya PT/ES NLI** | **31.7 ms** | **34.0 ms** | **0.66 GiB** | 54.8 s |
| Laya Multilingual base | 31.2 ms | 35.0 ms | 0.66 GiB | 41.5 s |
| OpenJEV v2 | 120.4 ms | 143.5 ms | 9.21 GiB | 23.7 s |
| TypeSafe JEV API | 585.6 ms | 668.5 ms | not available | not applicable |

TypeSafe latency includes the remote network. It is not an apples-to-apples local-model latency result. Local measurements use 64 examples after two warm-up calls.

## Training

Training updated the complete encoder and decision head. It did not use LoRA or an adapter. The objective was RLCD plus cross-entropy. RLCD used four perturbations, log plus spherical reward, and a linear noise schedule from 0.4 to 0.1.

The run trained independent 4-, 8-, and 16-epoch candidates from the same pinned base revision. Validation selected the 8-epoch checkpoint. The selection key was mean PT/ES macro F1, then weakest-language macro F1, then negative NLL.

The selected run used these values:

- batch size: 16
- gradient accumulation: 4
- encoder learning rate: 2.5e-5
- head learning rate: 1e-4
- AdamW weight decay: 0.01
- cosine learning-rate schedule, minimum 1e-6
- gradient clip: 1.0
- BF16 autocast and saved BF16 weights
- gradient checkpointing with non-reentrant execution
- seed: 42
- train, validation, and sealed test sizes: 13,971, 1,275, and 3,315

Portuguese and Spanish had equal sampling weight. Each epoch balanced all six language-label buckets. The fitted validation temperature was saved in `rl_agent_config.json`.

A Spanish-focused continuation tested 1, 2, and 4 additional epochs with a 2:1 Spanish-to-Portuguese sampling weight. No candidate passed the fixed promotion gate. This release therefore uses the original 8-epoch checkpoint.

## Data and license

The Portuguese source is [InferBR](https://github.com/lbencke/InferBR) at revision `b6c5e0b27e52477bbc5433d130b0a0e825cf74dc`, licensed under MIT. The Spanish source is [InferES](https://huggingface.co/datasets/venelin/inferes) at revision `c371a1915e6902b40182b2ae83c5ec7fe5e6cbd2`, licensed under CC BY 4.0.

InferES provides train and test files. The run removed train/test duplicate pairs. It then made a deterministic 10% per-class validation split from the remaining train data with seed 42. InferBR rows marked as low quality were excluded. All splits were deduplicated and checked for pair overlap.

The base checkpoint declares Apache 2.0. This derivative is released under Apache 2.0. InferES attribution remains required. See `ATTRIBUTIONS.md` and `provenance.json`. This repository does not redistribute any training dataset or audio.

## Limitations

- Spanish is materially weaker than Portuguese on both validation and test data.
- Calibration is domain-specific. Refit temperature on a held-out target-domain set before using probabilities as risk scores.
- The 512-example comparison has no confidence intervals and does not establish superiority over OpenJEV or TypeSafe.
- The checkpoint was trained only for three-way NLI. Other Laya question types were not retrained or validated.
- The local latency results apply to the stated software and GPU. CPU and other GPU results can differ.
- Dataset artifacts can contain annotation errors and social biases from their source corpora.

## Files

- `model.safetensors`: full fine-tuned model weights.
- `encoder/` and `tokenizer/`: pinned model configuration and tokenizer.
- `rl_agent_config.json`: Laya runtime configuration and calibrated temperature.
- `training.json`: training configuration, validation metrics, history, and source manifest.
- `evaluation_results.json`: compact sealed-test and comparison results.
- `provenance.json`: base revision, dataset revisions, runtime, and compatibility gates.
- `SHA256SUMS`: release integrity manifest.