# SwissLegalEvals — Results Report

**Generated:** 2026-07-28 · **Models completed:** 15 / 15 · **Profile:** `default` (full, uncapped — 55,861 samples/model)

Benchmarks: **SLDS** (landmark-decision summarization), **SwiLTra-Bench** (legal translation: `sdst`, `slt`, `sscprt`), **LEXam** (open questions `lexam_oq` + multiple-choice `lexam_mcq_*`).

---

## Scoring key

| Family | What it measures | Metric | Scale |
|--------|------------------|--------|-------|
| `slds` | Landmark decision summarization | LLM-judge (DeepSeek-V4-Pro) | 0–100 |
| `sdst` | Decision-summary translation (text level) | LLM-judge (gpt-4o-mini) | 0–100 |
| `slt` | Law translation (paragraph level) | LLM-judge (gpt-4o-mini) | 0–100 |
| `sscprt` | Supreme-court press-release translation | LLM-judge (gpt-4o-mini) | 0–100 |
| `lexam_oq` | Legal open questions | LLM-judge (DeepSeek-R1) | 0–100 |
| `lexam_mcq_{4,8,16}_idk` | Legal MCQ with "I don't know" option | `trad_score` (plain accuracy) | 0–1 (shown ×100) |

`trad_score` is plain accuracy (1 if correct, else 0; picking IDK scores 0 like any wrong answer). The separate `idk_score` is the calibration metric (+1 correct, 0 for IDK, −1 for a wrong/unparseable answer) — that is the one that rewards abstaining when uncertain and penalizes confident wrong answers. The LEXam MCQ table reports `idk_score (trad_score)`.

`translation_avg` = mean of `sdst`, `slt`, `sscprt`. `mcq_avg` = mean of the three MCQ `trad_score`s (×100). `overall` = mean of `slds`, `lexam_oq`, `translation_avg`, `mcq_avg`.

---

## Overall ranking

Composite over the four task groups. **Hy-MT2-30B ran translation only** (SwiLTra-Bench), so its overall is translation-only and not directly comparable.

| Rank | Model | Overall | Translation | SLDS | LEXam OQ | MCQ avg |
|------|-------|--------:|------------:|-----:|---------:|--------:|
| —  | Hy-MT2-30B-A3B *(translation only)* | (61.94) | 61.94 | – | – | – |
| 1  | NVIDIA-Nemotron-3-Ultra-550B | **59.03** | 63.42 | 55.13 | 66.45 | 51.14 |
| 2  | Kimi-K2.6 | **58.64** | 65.34 | 49.81 | 63.87 | 55.55 |
| 3  | DeepSeek-V4-Pro | **58.29** | 64.22 | 50.49 | 63.64 | 54.81 |
| 4  | DeepSeek-V4-Flash | **57.29** | 61.28 | 50.97 | 63.06 | 53.87 |
| 5  | MiniMax-M3 | **56.91** | 63.98 | 49.55 | 63.39 | 50.71 |
| 6  | GLM-5.2 | **55.46** | 66.24 | 52.27 | 58.77 | 44.55 |
| 7  | gemma-4-31B-it | **54.77** | 57.18 | 48.04 | 57.82 | 56.03 |
| 8  | gpt-oss-120b | **51.44** | 58.48 | 46.27 | 55.61 | 45.40 |
| 9  | Mistral-Medium-3.5-128B | **48.11** | 58.69 | 33.97 | 52.81 | 46.97 |
| 10 | Qwen3.5-35B-A3B | **42.77** | 42.53 | 46.73 | 43.77 | 38.06 |
| 11 | Apertus-v1.5-70B | **40.22** | 65.21 | 37.99 | 43.67 | 14.02 |
| 12 | Llama-3.3-70B-Instruct | **33.00** | 60.77 | 16.10 | 40.20 | 14.92 |
| 13 | Olmo-3.1-32B-Think | **26.04** | 20.79 | 18.39 | 38.52 | 26.45 |
| 14 | LFM2.5-8B-A1B | **24.75** | 37.75 | 11.08 | 24.51 | 25.66 |

---

## Translation (SwiLTra-Bench) — all 15 models

| Rank | Model | Translation avg | sdst | slt | sscprt |
|------|-------|----------------:|-----:|----:|-------:|
| 1  | GLM-5.2 | **66.24** | 71.01 | 67.25 | 60.47 |
| 2  | Kimi-K2.6 | **65.34** | 70.11 | 66.34 | 59.55 |
| 3  | Apertus-v1.5-70B | **65.21** | 71.68 | 67.06 | 56.88 |
| 4  | DeepSeek-V4-Pro | **64.22** | 69.22 | 62.23 | 61.22 |
| 5  | MiniMax-M3 | **63.98** | 66.99 | 65.83 | 59.11 |
| 6  | NVIDIA-Nemotron-3-Ultra-550B | **63.42** | 69.56 | 63.66 | 57.02 |
| 7  | Hy-MT2-30B-A3B | **61.94** | 66.60 | 63.45 | 55.76 |
| 8  | DeepSeek-V4-Flash | **61.28** | 65.70 | 63.92 | 54.23 |
| 9  | Llama-3.3-70B-Instruct | **60.77** | 68.12 | 62.94 | 51.26 |
| 10 | Mistral-Medium-3.5-128B | **58.69** | 65.48 | 58.47 | 52.12 |
| 11 | gpt-oss-120b | **58.48** | 63.80 | 59.47 | 52.17 |
| 12 | gemma-4-31B-it | **57.18** | 63.47 | 48.93 | 59.16 |
| 13 | Qwen3.5-35B-A3B | **42.53** | 45.50 | 30.61 | 51.48 |
| 14 | LFM2.5-8B-A1B | **37.75** | 41.97 | 40.98 | 30.30 |
| 15 | Olmo-3.1-32B-Think | **20.79** | 23.97 | 28.80 | 9.61 |

---

## LEXam multiple-choice (`idk_score (trad_score)` ×100)

| Model | mcq_4 | mcq_8 | mcq_16 | avg |
|-------|------:|------:|-------:|----:|
| DeepSeek-V4-Pro | 32.18 (62.42) | 12.72 (54.63) | -4.33 (47.37) | **13.52 (54.81)** |
| gemma-4-31B-it | 33.66 (66.40) | 11.25 (55.45) | -7.54 (46.23) | **12.45 (56.03)** |
| Kimi-K2.6 | 37.85 (67.85) | 10.80 (54.84) | -11.46 (43.97) | **12.40 (55.55)** |
| DeepSeek-V4-Flash | 30.49 (62.05) | 9.07 (52.95) | -6.00 (46.60) | **11.19 (53.87)** |
| NVIDIA-Nemotron-3-Ultra-550B | 33.59 (63.06) | 7.16 (48.67) | -15.29 (41.68) | **8.49 (51.14)** |
| MiniMax-M3 | 28.34 (62.42) | 7.70 (50.06) | -20.21 (39.66) | **5.28 (50.71)** |
| Mistral-Medium-3.5-128B | 19.16 (57.54) | -5.04 (46.02) | -24.81 (37.35) | **-3.56 (46.97)** |
| gpt-oss-120b | 18.71 (58.82) | -16.08 (41.58) | -28.18 (35.79) | **-8.51 (45.40)** |
| GLM-5.2 | 2.88 (50.76) | -4.90 (44.90) | -23.98 (38.01) | **-8.67 (44.55)** |
| Qwen3.5-35B-A3B | 8.49 (52.91) | -12.58 (32.00) | -41.37 (29.28) | **-15.15 (38.06)** |
| Olmo-3.1-32B-Think | 1.76 (32.32) | -27.87 (25.30) | -51.35 (21.75) | **-25.82 (26.45)** |
| LFM2.5-8B-A1B | -9.97 (32.89) | -35.61 (26.57) | -61.34 (17.52) | **-35.64 (25.66)** |
| Apertus-v1.5-70B | -48.21 (24.54) | -74.82 (11.62) | -88.17 (5.91) | **-70.40 (14.02)** |
| Llama-3.3-70B-Instruct | -48.22 (25.89) | -75.51 (12.22) | -86.71 (6.65) | **-70.15 (14.92)** |

(Hy-MT2-30B did not run LEXam.)

---

## Notes & caveats

- **Different scales:** judge families are 0–100; MCQ is a 0–1 `trad_score` (plain accuracy — IDK counts as wrong). The composite rescales MCQ to 0–100 — treat `overall` as an indicative aggregate, not a normalized benchmark score.
- **Judge self-preference:** SLDS is judged by DeepSeek-V4-Pro, so DeepSeek models' SLDS scores may carry mild self-preference bias.
- **MCQ difficulty scales with option count:** every model drops from mcq_4 → mcq_16, as expected.
- **Top tier is tight:** Nemotron, Kimi, both DeepSeek variants and MiniMax-M3 sit within ~2.5 points overall. Nemotron leads SLDS and LEXam OQ; GLM-5.2 leads translation.
- **GLM-5.2 is the best translator** (66.24 avg, top on `slt`) but Apertus posts the highest `sdst` (71.68). GLM's overall is dragged down by weak MCQ and the lowest LEXam-OQ of that tier — a strong-translation / weaker-exam profile.
- **Apertus-v1.5-70B** is a near-frontier translator (65.21, third overall; best `sdst`) whose MCQ collapses to 14.02, same failure mode as Llama-3.3-70B.
- **Mistral-Medium-3.5-128B** lands mid-pack at 48.11: solid translation (58.69) and MCQ (46.97) but weak SLDS (33.97).
- **MiniMax-M3** lands mid-frontier: strong translation (63.98, near the top) but middling MCQ.
- **Llama-3.3-70B-Instruct** is a translation specialist: it scores 60.77 on translation but only 16.10 on SLDS and 14.92 on MCQ.
- **Small open models lag hard** on summarization (`slds`) and press-release translation (`sscprt`) — LFM2.5-8B and Olmo are near-floor on `slds`/`sscprt`.

## Provenance

- Source: `results/results/**/results_*.json` (latest run per model).
- Regenerate: `./.venv/bin/python -m swiss_legal_evals.aggregate` then `... -m swiss_legal_evals.plot`.
- Per-family bar charts: `plots/*.html`. Tidy tables: `results/summary_long.csv`, `results/summary_family_mean.csv`, `results/summary.csv`.

### Not included
- **qwen3.5-397b** — paused (never run to completion).