kingjones777/Leanstral-1.5-119B-A6B-ROCmFP4-STRIX_LEAN-GGUF

Leanstral-1.5-119B-A6B — ROCmFP4 STRIX_LEAN (text-only GGUF) — AMD Ryzen AI Max+ 395 / Strix Halo / gfx1151

ROCmFP4 Strix-Lean quant of mistralai/Leanstral-1.5-119B-A6B for AMD Radeon / Ryzen AI Max+ (gfx1151) inference via the public charlie12345/ROCmFPX llama.cpp fork.

Field	Value
Base model	mistralai/Leanstral-1.5-119B-A6B
License	Apache-2.0 (derivative redistributable)
Architecture	deepseek2 (DeepSeek-V3 MLA + MoE)
MoE	128 routed experts / 4 active + 1 shared (~6.5B active / 119B total)
This file	Text-only LM GGUF (vision / Pixtral projector dropped)
Quant	Q4_0_ROCMFP4_STRIX_LEAN
Size	~59.0 GiB (60420 MiB reported by quantizer, 4.26 BPW)
Source weights	Community BF16 sahilchachra/Leanstral-1.5-119B-A6B-BF16 (dequantized from Mistral FP8)

Mistral ships Leanstral FP8-native without HF-format weights or an official GGUF. This quant is converted from the community BF16 transformers dump, then quantized with ROCmFP4 Strix-Lean kernels.

Files

File	Notes
Leanstral-1.5-119B-A6B-Q4_0_ROCMFP4_STRIX_LEAN.gguf	Text-only deepseek2 GGUF
SHA256SUMS	Basename-only checksums

Text-only note

The base model includes a Pixtral vision tower + multimodal projector. This GGUF is language-model only (vision tensors omitted at convert time). It is not a multimodal / vision checkpoint. A text-only deepseek2 GGUF of this model family is a known-good pattern (e.g. public Q4_K_M text-only releases).

Runtime (Ryzen AI Max+ 395 / gfx1151)

export LD_LIBRARY_PATH=/path/to/ROCmFPX/build/bin:/opt/rocm/lib
export HSA_OVERRIDE_GFX_VERSION=11.5.1
export GGML_HIP_ENABLE_UNIFIED_MEMORY=1

./llama-server \
  --host 127.0.0.1 --port 8099 \
  -ngl 999 -fa on -dio --no-warmup --jinja \
  --ctx-size 65536 \
  --cache-type-k q8_0 --cache-type-v q8_0 \
  --parallel 1 \
  --temp 1.0 --top-p 0.95 --min-p 0.05 \
  --model Leanstral-1.5-119B-A6B-Q4_0_ROCMFP4_STRIX_LEAN.gguf

Critical flags

• -dio is mandatory on large unified-memory loads. The mmap path can hang around ~120GB RSS with GTT stuck at 0.
• Use a ROCmFP4-capable build of charlie12345/ROCmFPX (or equivalent) that implements Q4_0_ROCMFP4_STRIX_LEAN.
• Load one ~60GB model at a time on 128GB unified memory.

Convert / quantize recipe

# 1) Convert HF BF16 → F16 GGUF (text-only / deepseek2)
#    Config must resolve as DeepseekV3ForCausalLM text path.
#    MoE experts ship pre-stacked as gate_up_proj / down_proj;
#    split gate_up as (n_exp, 2*n_ff, n_embd) → gate + up.
python convert_hf_to_gguf.py /path/to/leanstral-bf16 \
  --outtype f16 \
  --outfile Leanstral-1.5-119B-A6B-F16.gguf

# 2) Quantize Strix-Lean
./llama-quantize \
  Leanstral-1.5-119B-A6B-F16.gguf \
  Leanstral-1.5-119B-A6B-Q4_0_ROCMFP4_STRIX_LEAN.gguf \
  Q4_0_ROCMFP4_STRIX_LEAN 8

Verify: general.architecture == deepseek2.

A/B — ROCmFP4 STRIX_LEAN vs Q4_K_M (same machine / flags)

Hardware: Ryzen AI Max+ 395 (gfx1151), ROCm, 128GB unified memory.
Server flags identical: -ngl 999 -fa on -dio --no-warmup --jinja --ctx-size 65536 --cache-type-k q8_0 --cache-type-v q8_0 --parallel 1.
Baseline: public-style Q4_K_M text-only GGUF (~68 GiB). Gen cap 256 tokens.

Prompt size (actual tokens)	Metric	STRIX_LEAN (this)	Q4_K_M	Δ
~7.6K (7616)	Prefill tok/s	449.9	439.2	+2.4%
~7.6K (7616)	Decode tok/s	37.41	36.85	+1.5%
~23.1K (23081)	Prefill tok/s	184.3	182.2	+1.2%
~23.1K (23081)	Decode tok/s	23.70	25.83	−8.2%
—	File size	~59 GiB	~68 GiB	~−13%

Lean 4 quality spot-check

Both quants produce coherent Lean 4 for:

1. ∀ n : Nat, n + 0 = n — both emit valid Lean (induction / simp / Mathlib-style).
2. Harder: commutativity of Nat addition — both produce structured induction sketches; neither was formally typechecked here.

Did we lose anything?

• Size: clear win for STRIX_LEAN.
• Decode: essentially flat at 8K context; **8% slower at ~23K** vs Q4_K_M on this run.
• Prefill: slight STRIX_LEAN edge (not the usual −12/−14% tradeoff seen on some other models).
• Quality: no obvious collapse; treat as comparable for Lean-agent use, with Q4_K_M slightly more concise on the easy lemma in this spot-check.

If your workload is decode-heavy at long context, keep Q4_K_M. If you want a smaller footprint on Strix unified memory with comparable short-context decode, this quant is a reasonable option.

SHA256

See SHA256SUMS (basename only).

Links

• Base: https://huggingface.co/mistralai/Leanstral-1.5-119B-A6B
• BF16 source used for convert: https://huggingface.co/sahilchachra/Leanstral-1.5-119B-A6B-BF16
• Runtime fork: https://github.com/charlie12345/ROCmFPX

Acknowledgements

This build would not exist without the work below. Please star and follow these projects — the quantisation format used here is their engineering, not mine.

ROCmFPX — maintained by charlie12345 / caf The ROCmFP4 / ROCmFPX tensor formats (ggml types 100–106) exist only in this fork. Every ROCmFP4 file in this repository was produced with its llama-quantize, and runs on its runtime. The fork also credits collaborators ciru-ai, Tom Turney, PlunderStruck and Aydan S., and acknowledges AMD for hardware support. Licensed MIT, based on upstream llama.cpp.

llama.cpp — ggml-org and contributors The inference engine, GGUF format and conversion tooling everything here is built on.

AMD ROCm The compute platform these builds target — ROCm 7.2.4 on gfx1151 / Radeon 8060S.

Base model authors — see base_model in the metadata above; all model weights, licences and capabilities are theirs. This repository contributes quantisation and measurement only.

If you use these files, please credit ROCmFPX alongside this repository.