Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,209 @@
+---
+license: mit
+base_model: ornith-ai/Ornith-1.5-397B
+base_model_relation: quantized
+quantized_by: Lna-Lab
+library_name: gguf
+pipeline_tag: text-generation
+tags:
+  - gguf
+  - imatrix
+  - iq3_xxs
+  - moe
+  - qwen3_5_moe
+  - llama.cpp
+  - agentic-coding
+language:
+  - en
+  - zh
+  - ja
+---
+
+# Ornith-1.5-397B — IQ3_XXS GGUF
+
+An **IQ3_XXS** quantization of [ornith-ai/Ornith-1.5-397B](https://huggingface.co/ornith-ai/Ornith-1.5-397B).
+
+The official [Ornith-1.5-397B-GGUF](https://huggingface.co/ornith-ai/Ornith-1.5-397B-GGUF) repository
+stops at **Q4_K_M (224.08 GiB)**. That does not fit in 192 GB of VRAM. This one does.
+
+| | size | BPW |
+|---|---|---|
+| official Q4_K_M | 224.08 GiB | 4.86 |
+| **this IQ3_XXS** | **142.68 GiB** | **3.09** |
+
+Split into 4 files of ≤45 GB. Point `llama.cpp` at `-00001-of-00004` and it loads all four.
+
+---
+
+## Files
+
+| file | bytes |
+|---|---|
+| `Ornith-1.5-397B-IQ3_XXS-00001-of-00004.gguf` | 44,460,551,168 |
+| `Ornith-1.5-397B-IQ3_XXS-00002-of-00004.gguf` | 44,665,664,288 |
+| `Ornith-1.5-397B-IQ3_XXS-00003-of-00004.gguf` | 44,695,059,584 |
+| `Ornith-1.5-397B-IQ3_XXS-00004-of-00004.gguf` | 19,373,124,448 |
+
+Total 1098 tensors, 153,194,399,488 bytes across 4 files (142.67 GiB; the unsplit file is 153,194,399,008 B — the difference is per-split headers). See `SHA256SUMS.txt`.
+
+For vision, use `mmproj-Ornith-1.5-397B-BF16.gguf` from the
+[official GGUF repo](https://huggingface.co/ornith-ai/Ornith-1.5-397B-GGUF) (not mirrored here).
+
+---
+
+## How it was made
+
+Source was the **official Q8_0 GGUF** (392.56 GiB, 8.51 BPW), not the BF16 checkpoint.
+That means `--allow-requantize` was used — this is a re-quantization of an already-quantized
+tensor set. Q8_0 is close to lossless, but this is stated plainly so you can weigh it.
+
+```
+llama-quantize --allow-requantize \
+  --imatrix <imatrix.gguf> \
+  --token-embedding-type q5_K \
+  Ornith-1.5-397B-Q8_0.gguf Ornith-1.5-397B-IQ3_XXS.gguf IQ3_XXS 64
+
+llama-gguf-split --split --split-max-size 45G \
+  Ornith-1.5-397B-IQ3_XXS.gguf Ornith-1.5-397B-IQ3_XXS
+```
+
+`--token-embedding-type q5_K` overrides the IQ3_XXS default (`iq3_s`) for `token_embd`.
+With a 248,320-token vocabulary carrying CJK, the extra ~250 MiB is worth it.
+
+Quantization took **31m40s** on a Threadripper PRO 9985WX (64 cores, 64 threads).
+
+### About the importance matrix
+
+**The imatrix is not ours and is not mirrored here.** We used
+[`unsloth/Qwen3.5-397B-A17B-GGUF`](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF)'s
+`imatrix_unsloth.gguf_file` (80 chunks × 11264 tokens).
+
+This works because **Ornith-1.5-397B is a light fine-tune of Qwen/Qwen3.5-397B-A17B**:
+
+- the 1371 non-MTP tensor names are **identical sets** (set difference is empty)
+- the vision tower is **bit-identical** (frozen), as are `linear_attn.A_log` and `dt_bias`
+- the language trunk has cosine similarity **0.9993–0.99999** (relative L2 of 1–4%)
+- the safetensors `total_size` differs by exactly 13,191,153,536 B — precisely the MTP head
+
+We verified name compatibility before quantizing: **765 of 765 imatrix entries match tensors in
+the Ornith Q8_0** (100%). The 180 quantizable tensors without imatrix coverage are norms and
+`ssm_conv1d`, which are not quantized anyway.
+
+Notably, 765 is the same `quantize.imatrix.entries_count` recorded in the official Ornith GGUF
+headers — the official build used the same number of entries.
+
+If you want a purpose-built imatrix, compute one against this model directly. We did not,
+and we say so rather than implying otherwise.
+
+---
+
+## Measured
+
+Pure CPU, Threadripper PRO 9985WX, 64 threads, `-dev none`:
+
+| | prefill | decode |
+|---|---|---|
+| Q8_0 (reference) | 41.0–41.8 t/s | 9.7–9.8 t/s |
+| **IQ3_XXS** | **33.9–34.6 t/s** | **13.0–13.1 t/s** |
+
+Same prompt (three summer haiku, different kigo, one line each), `temp 0.8`, thinking off:
+
+**Q8_0** —
+```
+金魚売り通り過ぎていく水の音
+青トマトかじれば夏の朝の味
+夕立やアスファルト跳ねる子らの声
+```
+
+**IQ3_XXS** —
+```
+夏日や池の鯉ゆく水草かげ
+夏炉や炉の灰に眠る火の粉かな
+夏空や雲の切れ間より富士の山
+```
+
+Both hold 5-7-5 and use three distinct summer kigo. Q8_0 reaches for more modern imagery,
+IQ3_XXS sits closer to classical form. Neither is broken.
+
+Perplexity has **not** been measured. Stated as missing rather than guessed at.
+
+### Why not IQ2
+
+We also baked IQ2_XXS (97.65 GiB, 2.12 BPW) and **do not recommend it**. It answers factual
+questions correctly ("日本の首都は東京です") but cannot carry out multi-step generation — asked
+for haiku it emits bullet-point glossaries of season words, and at `temp 0.8` it degenerates into
+repetition with stray tokens. At 2.12 BPW this model does not survive. It is not published here.
+
+IQ3_XXS is, in our measurements, the floor.
+
+---
+
+## Usage
+
+```bash
+llama-server -m Ornith-1.5-397B-IQ3_XXS-00001-of-00004.gguf \
+  -c 32768 --threads 64
+```
+
+**Ornith is a reasoning model and it thinks at length.** With `-n 1500` it had not finished
+deliberating. For direct answers:
+
+```
+--chat-template-kwargs '{"enable_thinking":false}'
+```
+
+If you keep thinking on, budget generously (the 35B sibling needed ≥6500 tokens) and strip
+everything before `</think>` before parsing code out of a response — otherwise you will grade
+the model's scratch work instead of its answer.
+
+### ⚠️ GPU offload does not work yet on SM 12.0
+
+On 12× RTX PRO 2000 Blackwell (SM 12.0, CUDA 13.2) this model **crashes on GPU**:
+
+```
+ggml_cuda_compute_forward: SOFT_MAX failed
+CUDA error: invalid argument
+```
+
+Isolated by bisecting `-ngl`:
+
+- `-ngl 1` (layer 59, a **full_attention** layer) → runs
+- `-ngl 2` (adds layer 58, a **linear_attention** layer) → crashes
+
+So it is the linear-attention (gated delta net) path. `-fa on` does not help
+(`flash_attn = enabled` is logged and SOFT_MAX is still reached), nor does `--no-warmup`,
+nor `-ub 1 -b 1`. Reproduced on both a 2026-08-10 build and on master at `d59d455`
+(174 commits newer). CPU inference is unaffected.
+
+Separately, llama.cpp **misclassifies Blackwell as an integrated GPU** because
+`cudaDeviceProp.integrated` is non-zero (the driver API correctly reports 0 for the same device).
+Only the first "iGPU" is kept, so `-sm`/`-ts` silently do nothing and everything piles onto
+device 0. Upstream [#26901](https://github.com/ggml-org/llama.cpp/issues/26901), open since
+2026-08-11. Work around it by naming devices explicitly:
+
+```
+-dev CUDA0,CUDA1,CUDA2,CUDA3,CUDA4,CUDA5,CUDA6,CUDA7,CUDA8,CUDA9,CUDA10,CUDA11
+-ts 4.5,5,5,5,5,5,5,5,5,5,5,6.5
+```
+
+That does distribute the layers correctly (verified in the load log) — the SOFT_MAX crash is a
+separate, unresolved problem.
+
+### Note for anyone re-converting from safetensors
+
+`config.json` declares `mtp_num_hidden_layers=1`, but **there is not a single MTP tensor in the
+checkpoint** (1371 tensors, 0 MTP) or in the official GGUF (1098 tensors, 0 nextn). The 35B-A3B
+sibling does ship 785 of them; the 397B does not, in either 1.0 or 1.5.
+
+Convert with `--no-mtp`. Without it you get a GGUF declaring `block_count=61` with an empty
+`blk.60`, and llama.cpp fails at load with a missing-tensor error.
+
+---
+
+## Attribution
+
+- Base model: [ornith-ai/Ornith-1.5-397B](https://huggingface.co/ornith-ai/Ornith-1.5-397B) — MIT. All credit for the model belongs to its authors.
+- Importance matrix: [unsloth/Qwen3.5-397B-A17B-GGUF](https://huggingface.co/unsloth/Qwen3.5-397B-A17B-GGUF).
+- Tooling: [llama.cpp](https://github.com/ggml-org/llama.cpp).
+
+This repository contributes quantized weights and the measurements above. Nothing else.