Zhongzhu/OSCAR-LLAMACPP-Qwen3-4B-Thinking-2507-INT2-KV

Qwen3 4B Thinking 2507 - OSCAR INT2 KV cache (GGUF)

Qwen3-4B-Thinking-2507 in GGUF, packaged for the OSCAR ~2-bit (INT2) KV cache fork of llama.cpp.

This repository contains a rotation-baked GGUF plus the raw calibrated rotation matrices. The model is intended for OSCAR INT2 KV-cache inference: the per-layer rotation is already embedded in the GGUF, so the KV cache can be quantized to 2 bits while retaining near-f16 quality.

Files

file	contents	size	what it is for
qwen3-4b-rot-kv.gguf	Qwen3 4B Thinking GGUF with OSCAR rotation baked in	2.3 GB	Main model for OSCAR INT2 KV-cache inference.
k_rotation_qqt_r_h_pbr.pt	raw calibrated K rotation	2.3 MB	Use with a compatible base GGUF if you want to bake the rotation yourself.
v_rotation_sst_r_h_pbr.pt	raw calibrated V rotation	2.3 MB	Use with a compatible base GGUF if you want to bake the rotation yourself.

"OSCAR rotation" is an in-graph, post-RoPE orthogonal transform applied to K and V so that the KV cache quantizes well at 2-bit. The -rot-kv GGUF already contains the rotation tensors; no runtime rotation file is needed for normal inference.

Run

Use the OSCAR llama.cpp fork built with the backend you need (for example Metal on Apple Silicon).

LLAMA_KV_FUSED_FA=1 LLAMA_KV_NO_HADAMARD=1 LLAMA_KV_CLIP_RATIO=0.96 \
LLAMA_KV_HP_SINK=512 LLAMA_KV_HP_RECENT=2048 \
./build/bin/llama-server -m qwen3-4b-rot-kv.gguf \
  -fa on -ngl 99 -c 16384 \
  --cache-type-k q2_0 --cache-type-v q2_0 \
  --host 127.0.0.1 --port 8080

env var	meaning	value
LLAMA_KV_FUSED_FA	fused INT2+f16 flash-attention kernels, fast path when available	1
LLAMA_KV_NO_HADAMARD	rotation is in-graph, so skip the in-quant Hadamard	1
LLAMA_KV_CLIP_RATIO	per-row outlier clip before quant	0.96
LLAMA_KV_HP_SINK / LLAMA_KV_HP_RECENT	tokens kept high-precision (first/last)	512 / 2048

--cache-type-v f16 keeps V high-precision, which may improve quality at the cost of more memory.

Bake your own rotation

If you have a compatible base Qwen3-4B-Thinking-2507 GGUF, you can bake the included rotation matrices into it with the OSCAR export script:

python3 oscar-rotation/export_rot_kv_gguf.py \
  --base    qwen3-4b-base.gguf \
  --rot-dir . \
  --out     qwen3-4b-rot-kv.gguf