Model card: measured results, reproduction, known issues

README.md

@@ -0,0 +1,359 @@
+---
+license: apache-2.0
+base_model: nex-agi/Nex-N2.5-mini
+base_model_relation: quantized
+pipeline_tag: image-text-to-text
+library_name: gguf
+tags:
+  - gguf
+  - llama.cpp
+  - rocm
+  - amd
+  - rocmfp4
+  - rocmfpx
+  - strix-halo
+  - amd-strix-halo
+  - gfx1151
+  - ryzen-ai-max
+  - ryzen-ai-max-395
+  - radeon-8060s
+  - moe
+  - reasoning
+  - multimodal
+  - vision
+  - nex
+  - qwen3.5
+  - quantized
+---
+
+# Nex-N2.5-mini — ROCmFP4 for AMD Strix Halo (gfx1151)
+
+ROCmFP4 / ROCmFPX quantizations of **[nex-agi/Nex-N2.5-mini](https://huggingface.co/nex-agi/Nex-N2.5-mini)** —
+35,107,181,936 parameters (BF16), 40-layer Qwen3.5 MoE (30 Gated DeltaNet linear-attention + 10 full-attention layers), 256 routed experts / 8 active, 262,144-token context, text + image — built and measured on an AMD Ryzen AI
+Max+ 395 (Radeon 8060S, `gfx1151`). Upstream publishes no GGUF.
+
+- Another public ROCmFP4 build of this model exists — [julianmb/Nex-N2.5-mini-ROCmFP4-GGUF](https://huggingface.co/julianmb/Nex-N2.5-mini-ROCmFP4-GGUF): its `Nex-N2.5-mini-ROCmFP4-STRIX_LEAN.gguf` stores `output.weight` as `Q4_0_ROCMFP4_FAST` and carries no imatrix metadata. Every tier here keeps `output.weight` at `Q6_K`, and the imatrix builds are a separate repo.
+- **Vision projector included.**
+- **No MTP head.** `mtp_num_hidden_layers: 1` is declared in `config.json`, but the checkpoint ships no `mtp.*` weights (the converted BF16 GGUF reads back 0 `nextn` tensors). There is no speculative
+  decoding on these files.
+- Importance-matrix builds of the same three 4-bit tiers: **[kingjones777/Nex-N2.5-mini-ROCmFP4-imatrix-GGUF](https://huggingface.co/kingjones777/Nex-N2.5-mini-ROCmFP4-imatrix-GGUF)**.
+
+## Which file should I use?
+
+Ryzen AI Max+ 395 (MAX-1), ROCm 7.2.4, unpatched `llama-server` at `d3ca537` (see [Quick start](#quick-start)), `-c 65536`, one request at a time (`--parallel 1`), greedy (`temp 0`, `top_k 1`), `ignore_eos` so every arm generates exactly 256 tokens after a code prompt of 7,094–7,102 tokens (the first 30,000 characters of `convert_hf_to_gguf.py` plus an instruction), a unique nonce per request and `cache_prompt: false` (`cache_n = 0` asserted on every timed request), 1 warm-up then the median of 3. Decode numbers are the server's own `predicted_per_second`. Box iced: no other model loaded.
+
+| File | ftype | Size | BPW⁴ | KLD vs BF16 ↓² | Same top-1 ↑ | PPL (× BF16) | TG ROCm0 | TG Vulkan0 | PP ROCm0 |
+| --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | 106 | 17.46 GiB | 4.32 | 0.1044 ± 0.0014 | 86.66 % | 6.4740 ± 0.0798 (×1.0393) | 63.94 | 68.16 | 1158 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-COHERENT.gguf` | 102 | 18.48 GiB | 4.58 | 0.0971 ± 0.0013 | 87.29 % | 6.5617 ± 0.0812 (×1.0534) | 61.74 | 68.00 | 1187 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-FAST.gguf` | 103 | 17.37 GiB | 4.30 | 0.1088 ± 0.0014 | 86.41 % | 6.5498 ± 0.0809 (×1.0515) | 63.31 | 68.62 | 1157 |
+| *BF16 reference* | 32 | 64.61 GiB³ | 16.01 | 0 | 100 % | 6.2290 ± 0.0753¹ | — | — | — |
+
+TG = decode tokens/s after the 7,094–7,102-token code prompt, no draft head. PP = prefill tokens/s on ROCm0.
+¹ The BF16 PPL shown is the paired base every "×" ratio is computed against (averaged over the same scored tokens in the KL-divergence runs). The standalone BF16 run's own summary line reads 6.2303 ± 0.0754.
+² Quality columns: see *Where the quality numbers come from* below.
+³ BF16 conversion of the checkpoint; not published.
+⁴ BPW as printed by `llama-quantize`: bits per weight over the 34,660,610,688 weights in each GGUF. The 35,107,181,936-parameter count above also includes the 446,571,248-weight vision tower, which ships in the projector file.
+
+**Start with `STRIX_LEAN`; take `COHERENT` if quality matters more than speed.** `STRIX_LEAN` decodes 3.6 % faster than `COHERENT` on ROCm0 (63.94 vs 61.74 tok/s; per-run ranges 63.56–63.96 and 61.74–61.80) and within 0.2 % of it on Vulkan0 (68.16 vs 68.00), and prefills within 2.4 % of it on ROCm0 (1158 vs 1187 tok/s). `COHERENT`'s KLD is 7.1 % lower (3.8σ) for 1051 MiB more.
+
+`FAST` does not buy a clear speed gain here: `FAST` decodes within 1.0 % of `STRIX_LEAN` on ROCm0 (63.31 vs 63.94 tok/s; per-run ranges 63.14–63.38 and 63.56–63.96) and within 0.7 % of it on Vulkan0 (68.62 vs 68.16), and prefills within 0.1 % of it on ROCm0 (1157 vs 1158 tok/s); its KLD is higher than `STRIX_LEAN`'s (0.1088 vs 0.1044, +4.2 %, 2.2σ).
+
+KLD order (lower is closer to BF16): `COHERENT` 0.0971, `STRIX_LEAN` 0.1044 (+7.6 % vs `COHERENT`), `FAST` 0.1088 (+12.1 % vs `COHERENT`). `STRIX_LEAN` is 1051 MiB smaller than `COHERENT`; `FAST` is 91 MiB smaller than `STRIX_LEAN`. Speed gaps below 3.0 % (decode) and 3.0 % (prefill) are called a tie: the larger of 3 % and the widest gap measured between files that do identical work per token (each imatrix file and its standard twin: decode 1.0 %, prefill 1.1 %).
+
+**The [imatrix build](https://huggingface.co/kingjones777/Nex-N2.5-mini-ROCmFP4-imatrix-GGUF) is measurably closer to BF16 at the same size:** `STRIX_LEAN` 0.1044 → 0.0852 (-18.5 %, 10.0σ); `COHERENT` 0.0971 → 0.0769 (-20.7 %, 11.3σ); `FAST` 0.1088 → 0.0890 (-18.2 %, 10.2σ).
+
+Quality is graded against the **BF16 GGUF** (reference logits computed on the CPU) on a **held-out** corpus (wikitext-2 *test*, `-c 2048`, 40 chunks × 1,023 scored tokens each — the second half of every window, less its first token — = 40,920), never on the imatrix calibration text. **KLD** is the per-token KL divergence of each quant's next-token distribution from BF16's on the same tokens — far more sensitive than perplexity.
+
+**Where the quality numbers come from.** Measured directly on these files, against BF16 logits computed **on the CPU** in the same session (first-window perplexity 5.6964; Vulkan0 gave 5.6953 for the same window). Every file was graded on ROCm0 (the columns above) and again on Vulkan0. The STRIX_LEAN grade was run twice on each backend and every per-chunk row matched: 40 of 40 on ROCm0, 40 of 40 on Vulkan0. Why not the GPU for the reference: see [Known issues](#known-issues-and-limits).
+
+**Same files, same reference, graded on each backend** — KLD ROCm0 / Vulkan0: STRIX_LEAN 0.1044 / 0.1044 (0.0 %, 0.0σ); COHERENT 0.0971 / 0.0972 (+0.1 %, 0.0σ); FAST 0.1088 / 0.1088 (0.0 %, 0.0σ); imatrix STRIX_LEAN 0.0852 / 0.0836 (-1.9 %, 0.9σ); imatrix COHERENT 0.0769 / 0.0768 (-0.1 %, 0.1σ); imatrix FAST 0.0890 / 0.0891 (+0.2 %, 0.1σ). The two backends agree within noise on every file.
+
+## Quick start
+
+**1. Download**
+
+```bash
+hf download kingjones777/Nex-N2.5-mini-ROCmFP4-GGUF --local-dir ~/models/nex
+```
+
+**2. Build `llama-server`** — ROCmFPX at the measured commit (ROCm and Vulkan
+prerequisites: the project's [build guide](https://github.com/charlie12345/ROCmFPX/blob/d3ca537/docs/build.md)). No patch.
+
+```bash
+git clone https://github.com/charlie12345/ROCmFPX.git && cd ROCmFPX
+git checkout d3ca537
+HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
+cmake -S . -B build -DCMAKE_BUILD_TYPE=Release \
+  -DGGML_HIP=ON -DGGML_VULKAN=ON -DGPU_TARGETS=gfx1151 \
+  -DGGML_HIP_GRAPHS=ON -DGGML_HIP_NO_VMM=ON -DLLAMA_CURL=OFF
+cmake --build build --target llama-server -j
+```
+
+(The CMake options of the measured build are listed in [Reproduction](#reproduction).)
+
+`d3ca537` is also in the history of the official [ROCmFPX/ROCmFPX](https://github.com/ROCmFPX/ROCmFPX) repository.
+
+**3. Serve**
+
+```bash
+env LD_LIBRARY_PATH=$PWD/build/bin:/opt/rocm/lib HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1 \
+build/bin/llama-server \
+  -m ~/models/nex/Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf \
+  --mmproj ~/models/nex/mmproj-Nex-N2.5-mini-BF16.gguf \
+  --chat-template-file ~/models/nex/chat_template_enable_thinking.jinja --reasoning off \
+  -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -dev ROCm0 \
+  -c 65536 --host 127.0.0.1 --port 8080
+```
+
+(`LD_LIBRARY_PATH` avoids a soname clash on machines that also have a Vulkan-only llama.cpp build.) The exact measured
+argv is in [Reproduction](#reproduction).
+
+**4. Call** — upstream sampling. Thinking is off unless the request sets `"enable_thinking": true` (as here; drop that
+line for a direct answer):
+
+```bash
+curl http://127.0.0.1:8080/v1/chat/completions \
+  -H 'Content-Type: application/json' \
+  -d '{
+    "messages": [{"role": "user", "content": "Hello"}],
+    "temperature": 0.7,
+    "top_p": 0.95,
+    "top_k": 40,
+    "chat_template_kwargs": {"enable_thinking": true}
+  }'
+```
+
+| Flag | Why |
+| --- | --- |
+| `--chat-template-file …/chat_template_enable_thinking.jinja` | The model's own template plus one line (see [Reasoning controls](#reasoning-controls)). Without it llama-server leaves the reasoning in `content` and thinking-on tool calls fail ([measured](#tool-calling)). |
+| `--reasoning off` | Thinking stays off unless a request passes `"enable_thinking": true`. |
+| `--jinja` | Already on by default in this build; keep it on — the reasoning controls (`chat_template_kwargs`) and tool calling rely on the Jinja chat template. |
+| `-fit off` | Autofit reads `MemAvailable` on integrated GPUs and can silently shrink context or push tensors to CPU. |
+| `-cram <MiB>` | Not set above (default 8 GiB of host RAM for saved prompts). Set it on a shared box — see [Known issues](#known-issues-and-limits). |
+| `--mmproj` | Loads the 27-layer vision tower. Drop the flag for text-only. |
+
+Requires a llama.cpp build with ROCmFP4 / ROCmFPX tensor-type support; stock llama.cpp rejects these tensor types.
+
+## Reasoning controls
+
+The model's own chat template switches thinking with `chat_template_kwargs.reasoning_effort` and ignores
+`enable_thinking`:
+
+| `reasoning_effort` | What the stock template emits |
+| --- | --- |
+| `"none"` | empty `<think>\n\n</think>` (no thinking) |
+| `"high"` | opens `<think>\n` (always think) |
+| `"medium"`, unset, or anything else | opens `<think>` and lets the model decide (adaptive; upstream default is `"medium"`) |
+
+llama-server decides how to split reasoning from the answer by rendering the template with `enable_thinking` on and
+off. With this template both renders are the same, so it never extracts the reasoning ([measured](#tool-calling)).
+`chat_template_enable_thinking.jinja` in this repo is the model's `chat_template.jinja` (sha256
+`f1753536417ee87cded4bc5017354eb2123c079d20c634403a3d808b2ba3cc5b`) with one line added at the top (file sha256 `9183c7ba8510fb9628edd2265a4cad8c02b3aecec3cbdf65620f47494e75836d`):
+
+```jinja
+{%- if reasoning_effort is not defined and enable_thinking is defined %}{%- set reasoning_effort = 'high' if enable_thinking else 'none' %}{%- endif %}
+```
+
+Serve it with `--chat-template-file` and `--reasoning off`.
+
+Measured on the standard STRIX_LEAN file with the included template file and `--reasoning off` (greedy probes: a direct question, a one-word instruction and a tool request, each with the tool schema attached; tool suite: 3 passes):
+
+| request | thinking | reasoning ends up in | measured |
+| --- | --- | --- | --- |
+| no `chat_template_kwargs` | off (the server default with `--reasoning off`) | — | 0 of 3 replies with think tags in `content` |
+| `"enable_thinking": true` | on | `reasoning_content` | tool suite with thinking on: 19/21 (a pass requires no think tags in `content`) |
+| `"enable_thinking": false` | off | — | 0 of 3 replies with think tags in `content` |
+| `"reasoning_effort": "none"` | off | — | 0 of 3 replies with think tags in `content` |
+| `"reasoning_effort": "high"` | on | **`content`** — do not use | 3 of 3 replies with think tags in `content` |
+| `"reasoning_effort": "medium"` | adaptive | **`content`** — do not use | 3 of 3 replies with think tags in `content` |
+
+So: switch thinking with `enable_thinking` only. Thinking-off tool checks: 21/21.
+
+Upstream serving (SGLang) uses `--reasoning-parser qwen3 --tool-call-parser qwen3_coder`. Recommended sampling:
+temperature 0.7, top_p 0.95, top_k 40.
+
+Earlier assistant turns are re-rendered **with** their reasoning (contexts grow faster than with templates that drop
+it). With thinking on and a small `max_tokens`, the whole budget can go to reasoning and `content` comes back empty —
+raise `max_tokens` before concluding the model is broken.
+
+Tool calls use the XML-style `<tool_call><function=…><parameter=…>` format, which llama.cpp parses natively
+through the Jinja chat template (on by default).
+
+## Speed
+
+| File | Backend | Workload | Decode tok/s (min–max) | Prefill tok/s |
+| --- | --- | --- | ---: | ---: |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | ROCm0 | code | 63.94 (63.56–63.96) | 1158 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | Vulkan0 | code | 68.16 (68.10–68.21) | 1006 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-COHERENT.gguf` | ROCm0 | code | 61.74 (61.74–61.80) | 1187 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-COHERENT.gguf` | Vulkan0 | code | 68.00 (67.88–68.09) | 1001 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-FAST.gguf` | ROCm0 | code | 63.31 (63.14–63.38) | 1157 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-FAST.gguf` | Vulkan0 | code | 68.62 (68.60–68.67) | 999 |
+| `Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | ROCm0 | code | 63.29 (63.19–63.41) | 1145 |
+| `Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | Vulkan0 | code | 67.77 (67.74–67.86) | 995 |
+| `Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-COHERENT.gguf` | ROCm0 | code | 61.56 (61.53–61.63) | 1184 |
+| `Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-COHERENT.gguf` | Vulkan0 | code | 67.81 (67.77–67.85) | 999 |
+| `Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-FAST.gguf` | ROCm0 | code | 63.03 (62.55–63.11) | 1154 |
+| `Nex-N2.5-mini-imatrix-Q4_0-ROCmFP4-FAST.gguf` | Vulkan0 | code | 68.44 (68.42–68.56) | 994 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | ROCm0 | prose | 63.06 (62.94–63.09) | 1139 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | Vulkan0 | prose | 67.34 (67.31–67.35) | 978 |
+
+`STRIX_LEAN`: ROCm0 decodes 6.2 % slower than Vulkan0 (63.94 vs 68.16 tok/s; ranges 63.56–63.96 / 68.10–68.21) and prefills 15.1 % faster than it (1158 vs 1006 tok/s). `COHERENT`: ROCm0 decodes 9.2 % slower than Vulkan0 (61.74 vs 68.00 tok/s; ranges 61.74–61.80 / 67.88–68.09) and prefills 18.5 % faster than it (1187 vs 1001 tok/s). `FAST`: ROCm0 decodes 7.7 % slower than Vulkan0 (63.31 vs 68.62 tok/s; ranges 63.14–63.38 / 68.60–68.67) and prefills 15.8 % faster than it (1157 vs 999 tok/s). STRIX_LEAN workload range on ROCm0: code 63.94 tok/s vs prose 63.06 (prose decodes within 1.4 % of code); Vulkan0 code 68.16 vs prose 67.34 (prose decodes within 1.2 % of code). Prompt lengths: code 7,094–7,102 tokens (the first 30,000 characters of `convert_hf_to_gguf.py` plus an instruction), prose 7,453–7,457 tokens (the first 34,000 characters of wikitext-2 *train* plus a writing instruction). Speed gaps below 3.0 % (decode) and 3.0 % (prefill) are called a tie: the larger of 3 % and the widest gap measured between files that do identical work per token (each imatrix file and its standard twin: decode 1.0 %, prefill 1.1 %).
+
+## Prompt caching
+
+Measured: pairs of requests that share a long code prefix and differ only in the closing instruction. The second
+request of each pair runs warm (`cache_prompt: true`, resuming from what the first one left) and then cold
+(`cache_prompt: false`), and the two replies are compared byte for byte. Every prompt is padded to one token length so
+warm and cold see identical chunking.
+
+| server | second-request prompt tokens reused | processed | warm reply = cold reply |
+| --- | ---: | ---: | :---: |
+| d3ca537, unpatched | **6,011** of 7,039 (all 5 pairs) | 1,028 | 5/5 |
+
+In 5 request pairs sharing a long prefix at one fixed prompt length of 7,039 tokens, every second request resumed from the checkpoint the first one left 1,028 tokens before its end — **6,011 tokens reused (85 %), 1,028 processed** — median prefill **1.0 s instead of 6.2 s** cold (6.1× faster). Each warm reply was byte-identical to a cold run of the same prompt in **5/5** exchanges.
+
+llama-server processes the last `n_ubatch` + 4 tokens of every prompt as two batches so it can checkpoint there (1,024 + 4 = 1,028 tokens with the `-ub 1024` used in these measurements, where the server default is `-ub 512`; [upstream PR #20288](https://github.com/ggml-org/llama.cpp/pull/20288)). A turn that resumes from a checkpoint left by a prompt of a *different* length therefore splits its tail differently from a cold run, and float rounding can flip a greedy token.
+
+## Tool calling
+
+The template emits the XML-style `<tool_call><function=…><parameter=…>` format, which llama.cpp parses natively
+through the Jinja chat template (on by default). Suite run through `llama-server`, at the checkpoint's recommended
+sampling (temperature 0.7, top-p 0.95, top-k 40):
+
+**40/42** over three passes with the quick-start configuration, **6/14** with the stock template, run on `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf`. Quick start = the included template file + `--reasoning off`, thinking switched with `enable_thinking`; stock = the model's own template, thinking switched with `reasoning_effort` (`high` / `none`). A check passes only with a native `tool_calls` entry carrying the right arguments and no raw XML or think tags left in `content`.
+
+| check | quick start, thinking ON | quick start, thinking OFF | stock template, thinking ON | stock template, thinking OFF |
+| --- | :---: | :---: | :---: | :---: |
+| multi-arg | 3/3 | 3/3 | ❌ | ✅ |
+| nested-object | 2/3 | 3/3 | ❌ | ❌ |
+| enum | 3/3 | 3/3 | ❌ | ✅ |
+| correct-decline | 3/3 | 3/3 | ❌ | ✅ |
+| multi-turn | 3/3 | 3/3 | ❌ | ✅ |
+| streaming | 3/3 | 3/3 | ❌ | ✅ |
+| parallel | 2/3 | 3/3 | ❌ | ✅ |
+
+**Stock chat template: 6/14.** Every thinking-on check failed. Re-run with the raw replies kept, 7 of 7 thinking-on replies carried the reasoning and a `</think>` in `content`, and 0 had any `reasoning_content`. llama-server builds its reasoning parser by rendering the template with `enable_thinking` on and off; this template ignores `enable_thinking` (it switches on `reasoning_effort`), so the parser finds no reasoning markers and extracts nothing. Server switches did not help (`--reasoning-format deepseek`: 3 of 3 replies still had reasoning in `content`; `--chat-template-kwargs` with `reasoning_effort`: 3 of 3 replies still had reasoning in `content`; `--reasoning on`: 3 of 3 replies still had reasoning in `content`). With thinking off, `nested-object` failed on an HTTP 500 — see [Known issues](#known-issues-and-limits).
+
+**With the included `chat_template_enable_thinking.jinja` and `--reasoning off` (the quick start): 13/14, 13/14, 14/14 over three passes of the same suite (40/42)** — thinking off 21/21, thinking on 19/21; the misses were `nested-object` with thinking on (pass 1), `parallel` with thinking on (pass 2). A pass requires a native `tool_calls` entry with the right arguments and no think tags in `content`. Each check is a single sample at the recommended temperature 0.7.
+
+## Vision
+
+`mmproj-Nex-N2.5-mini-BF16.gguf` is the 27-layer vision tower (width 1152),
+loaded with `--mmproj`. Its attention follows the server's `-fa` setting, so both settings were checked.
+
+Probe: a synthetic 448×448 image with a red circle and a blue square (a model that ignores the image cannot name both), sent to `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` with `--mmproj`, temperature 0. Pass = the reply names every expected term (`red`, `blue`, `circle`, `square`).
+
+| | `-fa on` | `-fa off` |
+| --- | :---: | :---: |
+| STRIX_LEAN + projector | ✅ 4/4 terms | ✅ 4/4 terms |
+
+✅ **Images work with `-fa on` and `-fa off`.** The `-fa on` image test was repeated with the included template file and `--reasoning off` (the quick start): ✅ passed.
+
+Reply (`-fa on`):
+
+> The image shows two shapes: a red circle on the left and a blue square on the right.
+
+## Memory
+
+Measured footprint (drop in `MemAvailable`) of STRIX_LEAN with the vision projector, q8_0 KV cache, `-cram 512`, one slot, no draft head:
+
+| context | after load | after one request (30,000-character code prompt) |
+| ---: | ---: | ---: |
+| 65,536 | 21.11 GiB | 21.29 GiB |
+| 262,144 | 24.36 GiB | 24.52 GiB |
+
+No row was decode-benchmarked beyond that one request. Nothing beyond these rows was measured.
+
+## Quantization methodology
+
+```bash
+# 1. convert: text model and the vision projector (the checkpoint has no mtp.* tensors)
+python convert_hf_to_gguf.py hf --outtype bf16 --model-name Nex-N2.5-mini --outfile Nex-N2.5-mini-BF16.gguf
+python convert_hf_to_gguf.py hf --outtype bf16 --mmproj --model-name Nex-N2.5-mini --outfile mmproj-Nex-N2.5-mini-BF16.gguf
+
+# 2. quantize from BF16 only; the LM head is forced up on every tier and read back by exact tensor name
+llama-quantize --output-tensor-type q6_K                            Nex-N2.5-mini-BF16.gguf OUT Q4_0_ROCMFP4_STRIX_LEAN 16
+llama-quantize --output-tensor-type q6_K --token-embedding-type q6_K Nex-N2.5-mini-BF16.gguf OUT Q4_0_ROCMFP4_COHERENT 16
+llama-quantize --output-tensor-type q6_K                            Nex-N2.5-mini-BF16.gguf OUT Q4_0_ROCMFP4_FAST 16
+
+# 3. BF16 reference logits on the CPU only (this build's ROCm0 path computes the BF16 MoE wrong — Known issues)
+llama-perplexity -m Nex-N2.5-mini-BF16.gguf -f wikitext-2-raw/wiki.test.raw -c 2048 -b 2048 --chunks 40 --kl-divergence-base bf16.kld \
+  -dev none -ngl 0 --no-op-offload -t 16
+# 4. grade each shipped file against those logits, on each GPU backend
+llama-perplexity -m OUT --kl-divergence-base bf16.kld --kl-divergence -c 2048 -b 2048 -ngl 999 -fa on -dio -dev ROCm0
+llama-perplexity -m OUT --kl-divergence-base bf16.kld --kl-divergence -c 2048 -b 2048 -ngl 999 -fa on -dio -dev Vulkan0
+```
+
+Receipts (the built file is the receipt — exact tensor names, never a substring match; `recipe/logs/`):
+
+| File | `output.weight` | `token_embd.weight` | tensors |
+| --- | --- | --- | ---: |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | Q6_K | Q5_K | 733 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-COHERENT.gguf` | Q6_K | Q6_K | 733 |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-FAST.gguf` | Q6_K | Q4_0_ROCMFP4_FAST | 733 |
+
+`tie_word_embeddings` is false, so the output head is a real tensor and `--output-tensor-type q6_K` does real work.
+All three tiers pin `output.weight` to `q6_K`; COHERENT also pins `token_embd.weight` to `q6_K`, while STRIX_LEAN and
+FAST keep their tier's own embedding type (shown in the receipts).
+
+## Reproduction
+
+```
+server   : https://github.com/charlie12345/ROCmFPX @ d3ca537
+           unpatched; build dir /opt/llama-rocm/rocmfpx-724/build-hipvk, Release, Unix Makefiles, GGML_HIP=ON GGML_VULKAN=ON
+           GGML_HIP_GRAPHS=ON GGML_HIP_NO_VMM=ON GGML_NATIVE=ON AMDGPU_TARGETS=gfx1151 LLAMA_CURL=OFF
+           CMAKE_HIP_COMPILER=/opt/rocm-7.2.4/lib/llvm/bin/clang
+           sha256 llama-quantize   1559189968a30781f837e4d25ca3cdf26c8eb0351023675406535eddcbe0db13
+           sha256 llama-imatrix    bfdce5c21a01bdc8043d69335b74a7b68b8335ae989991a1e10e21d5faeaeb8e
+           sha256 llama-perplexity c2175925bb0c61df89ff787845d8e03a35403f684be52719ac2987b8ca636a53
+           sha256 llama-server     d077b98f3f04df4b482a5826c8922e39cda3aa8d2e2b2fbe1870f36d245cebe7
+source   : nex-agi/Nex-N2.5-mini  revision 87420286149d9cce9bd46cd335ef9bda33c37c1b
+model    : Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf (the argv below; every file was measured the same way)
+argv     : /opt/llama-rocm/rocmfpx-724/build-hipvk/bin/llama-server -m /mnt/models/nex-n2.5-mini/out/Nex-N2.5-mini-Q4_0_ROCMFP4_STRIX_LEAN.gguf -dev ROCm0 -ngl 999 -fa on -dio --jinja -fit off --parallel 1 -c 65536 -b 2048 -ub 1024 --host 127.0.0.1 --port 18600 --no-webui
+template : the quick-start tool-suite and image rows add --chat-template-file chat_template_enable_thinking.jinja
+           --reasoning off to this argv (recipe/pipeline/run_tools_roff.sh -> nex_tools_tpl.py; their server logs
+           read "chat template, thinking = 0"); the speed rows use the stock template
+env      : LD_LIBRARY_PATH=<build>/bin:/opt/rocm-7.2.4/lib
+           HSA_OVERRIDE_GFX_VERSION=11.5.1 GGML_HIP_ENABLE_UNIFIED_MEMORY=1
+box      : aimax — AMD Ryzen AI Max+ 395 / Radeon 8060S (gfx1151), 124 GiB, GTT 131072 MiB,
+           kernel 6.17.6-061706-generic, ROCm 7.2.4
+protocol : 7,094–7,102-token code prompt, 256 generated tokens, temp 0 / top_k 1, ignore_eos, cache_prompt false,
+           1 warm-up + median of 3, no co-resident models (box iced)
+measured : 2026-09-16 to 2026-09-17, by the pipeline in recipe/ (every raw number in recipe/results_summary.json and recipe/raw/)
+```
+
+## Files
+
+| File | Size | sha256 |
+| --- | ---: | --- |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-STRIX_LEAN.gguf` | 17.46 GiB | `4437c3d8871acf04e4be8a9b773cf69697e5fc4b7b41fae41261f246ee5add9e` |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-COHERENT.gguf` | 18.48 GiB | `43673ce7b05f6d3924f6268e7e3102bab8fc4f91771505ef5734beaddacaa2f1` |
+| `Nex-N2.5-mini-Q4_0-ROCmFP4-FAST.gguf` | 17.37 GiB | `156b516ecfc0f8e400b165b7d42d959a0bd902814bb6949ca040c991494886d1` |
+| `mmproj-Nex-N2.5-mini-BF16.gguf` | 0.84 GiB | `a80a82977e4787a838a612316c79a6533c1b098fcac19372cc78929120162a4e` |
+| `chat_template_enable_thinking.jinja` | 7.7 KiB | `9183c7ba8510fb9628edd2265a4cad8c02b3aecec3cbdf65620f47494e75836d` |
+
+`SHA256SUMS` covers every model file and the chat template file. `recipe/` holds the measurement pipeline (`recipe/pipeline/`), raw per-run
+results (`recipe/raw/`), build and receipt logs (`recipe/logs/`), and `results_summary.json` with every measured value
+on this card. Architecture facts (layer counts, vocabulary, vision depth) come from the checkpoint's `config.json` at
+revision `87420286149d9cce9bd46cd335ef9bda33c37c1b`.
+
+## Known issues and limits
+
+- **Do not run the BF16 GGUF of this model on ROCm0 with this build.** `d3ca537` computes it wrong on that path: wikitext-2 perplexity 105.9 on ROCm0 vs 6.23 on the CPU (first window 139.1 vs 5.70; also wrong with `-fa off`). The 4-bit files are not affected — their ROCm0 grades are in the table — and the BF16 file is not published here; the quality reference was computed on the CPU instead.
+- **llama-server rejects a tool call whose required arguments are not in the order the schema lists them** — HTTP 500, *The model produced output that does not match the expected peg-native format*: this build's parser for the XML tool-call format expects required arguments in definition order. On the standard STRIX_LEAN file with the stock template and thinking off, the `nested-object` request (three required arguments) hit it in the tool suite and in 2 of 4 repeats; the model had written a well-formed call with the arguments reordered. Be ready to retry on this error.
+- **With the included template, do not set `reasoning_effort` to `high` or `medium`** — the reasoning goes back into `content` (6 of 6 probe replies). `enable_thinking: true` is the way to turn thinking on.
+- **With thinking on, a very short answer can stay inside the think block.** Served as in the quick start, `Reply with the single word: ready` (no tools, `enable_thinking: true`, temperature 0) came back with the word in `reasoning_content` and an empty `content` on 2 of 2 FAST files tested. With thinking off the same request returned `ready` in `content`.
+- **No MTP head.** The checkpoint's `config.json` declares `mtp_num_hidden_layers: 1`, but the weights contain **no** `mtp.*` tensors (1,026 tensors total). There is no multi-token-prediction head and no speculative decoding on these files — do not pass `--spec-type draft-mtp`.
+- **The stock chat template ignores `enable_thinking`, and llama-server cannot separate its reasoning** (see [Tool calling](#tool-calling)). Serve with the included `chat_template_enable_thinking.jinja` and `--reasoning off`, and switch thinking per request with `enable_thinking` — see [Reasoning controls](#reasoning-controls).
+- **Earlier assistant turns are re-rendered with their reasoning**, so multi-turn contexts grow faster than with templates that drop it.
+- **`llama-server`'s host-RAM prompt cache defaults to 8 GiB** (`-cram 8192`). On a shared box, set `-cram` explicitly.
+- Measured on Linux only (Ryzen AI Max+ 395, ROCm 7.2.4, unpatched `d3ca537`).
+- **Not measured:** decode beyond a 7,457-token prompt (a 262,144-token context was loaded in the memory test, not benchmarked), long-context quality, video input, concurrency above 1, and task-level accuracy. Perplexity/KLD measure next-token fidelity to BF16 on prose, not reasoning or code correctness.
+
+## License and attribution
+
+Apache-2.0, inherited from the base model. Weights and architecture: **Nex-AGI**
+([nex-agi/Nex-N2.5-mini](https://huggingface.co/nex-agi/Nex-N2.5-mini)). ROCmFP4 / ROCmFPX quantization format and
+runtime: the ROCmFPX project. Quantization and measurements: kingjones777.