Upload BUILD.md with huggingface_hub

BUILD.md

@@ -0,0 +1,73 @@
+# Building the runtime (ROCmFPX / llama.cpp) yourself
+
+This GGUF uses the **ROCmFPX** quant family (`Q2_0_ROCMFPX`), which needs a matching
+`llama-server` / `llama-quantize`. Build it from the pinned source so your binary understands
+the 2-bit ROCmFP codebook and the `nextn` (MTP) tensors.
+
+## Pinned source
+
+- Repo:   https://github.com/charlie12345/ROCmFPX.git
+- Commit: `b2f5829db8beefc22b49481247d180a48b06793a` (b2f5829)
+
+Building any other commit is not guaranteed to read `Q2_0_ROCMFPX`.
+
+## Quick build (AMD, HIP — MI300X / gfx942, adapt `AMDGPU_TARGETS` for your card)
+
+```bash
+git clone https://github.com/charlie12345/ROCmFPX.git
+cd ROCmFPX && git checkout b2f5829
+
+export ROCM_PATH=/opt/rocm            # e.g. /opt/rocm-7.2.4 on some installs
+cmake -S . -B build \
+  -DGGML_HIP=ON \
+  -DAMDGPU_TARGETS=gfx942 \           # RX 9060 XT = gfx1200 (RDNA4); MI300X = gfx942
+  -DCMAKE_BUILD_TYPE=Release \
+  -DCMAKE_PREFIX_PATH="$ROCM_PATH"
+cmake --build build -j --target llama-server llama-quantize llama-cli
+# -> build/bin/llama-server, build/bin/llama-quantize
+```
+
+For a consumer RDNA4 card (RX 9060 XT) you can also build the **Vulkan** backend
+(`-DGGML_VULKAN=ON` instead of `-DGGML_HIP=ON`) if you prefer it over ROCm/HIP.
+
+The `build_rocmfpx.sh` in this repo is the exact script used to build the binaries that
+produced this GGUF (HIP, gfx942). Edit `AMDGPU_TARGETS` for your GPU.
+
+## The quant recipe (how this 12GB build was made)
+
+Pure 2-bit-everywhere collapses this model; pure 3.5-bit is coherent but 19GB (won't fit 16GB).
+The fix is **role-aware**: 2-bit experts (the bulk) + Q6 attention/embeddings/shared-experts.
+Reproduce from an f16 GGUF (`convert_hf_to_gguf.py ... --outtype f16`) with:
+
+```bash
+./build/bin/llama-quantize \
+  --token-embedding-type Q6_0_ROCMFPX \
+  --output-tensor-type  Q6_0_ROCMFPX \
+  --tensor-type attn_q=Q6_0_ROCMFPX  --tensor-type attn_k=Q6_0_ROCMFPX \
+  --tensor-type attn_v=Q6_0_ROCMFPX  --tensor-type attn_qkv=Q6_0_ROCMFPX \
+  --tensor-type attn_output=Q6_0_ROCMFPX --tensor-type attn_gate=Q6_0_ROCMFPX \
+  --tensor-type ffn_gate_shexp=Q6_0_ROCMFPX --tensor-type ffn_up_shexp=Q6_0_ROCMFPX \
+  --tensor-type ffn_down_shexp=Q6_0_ROCMFPX \
+  model-f16.gguf model-Q2KXL_ROCMFPX.gguf Q2_0_ROCMFPX
+# base Q2_0_ROCMFPX -> the ffn_*_exps (256-expert) tensors go 2-bit; overrides keep the rest at Q6.
+```
+
+## Serve it (OpenAI-compatible API on :8080)
+
+```bash
+./build/bin/llama-server \
+  -m Qwen3.8-Distill-35B-A3B-Coder-Abliterated-Q2_ROCMFPX.gguf \
+  --host 127.0.0.1 --port 8080 \
+  -ngl 99 -c 16384 -fa on --jinja \
+  --alias qwen38-distill-a3b
+```
+
+On a 16GB card, context and concurrency share one KV pool — pick one:
+single-stream long context (`-c 32768 -np 1`) **or** many short sessions (`-c 8192 -np 8`).
+
+## Note on MTP
+
+The `nextn` (MTP) tensors are present in this GGUF (block 40). Whether they are used for
+speculative decoding depends on your `llama-server` build's runtime support — check its startup
+log for a draft/nextn line. If not auto-used, the model still runs correctly as a standard
+single-token decoder.