Hugging Face's logo

sh111111111111111
/
Qwen3-4B-Instruct-2507-BitClass2-GGUF

Text Generation
GGUF
English
Chinese
quantized
mixed-precision
bitclass
qwen3
imatrix
conversational
Model card Files
xet
 Community
sh111111111111111 commited on
Commit
1a2dec6
·
0 Parent(s):

Duplicate from sh111111111111111/Qwen3-4B-Instruct-2507-ShapeLearn2-GGUF_old

Browse files
Files changed (7) hide show
  1. .gitattributes +40 -0
  2. Qwen3-4B-Instruct-2507-Q3_K_S.gguf +3 -0
  3. Qwen3-4B-Instruct-2507-Q4_K_M.gguf +3 -0
  4. Qwen3-4B-Instruct-2507-Q5_K_M.gguf +3 -0
  5. Qwen3-4B-Instruct-2507-Q6_K.gguf +3 -0
  6. Qwen3-4B-Instruct-2507-Q8_0.gguf +3 -0
  7. README.md +75 -0
.gitattributes ADDED
@@ -0,0 +1,40 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ *.7z filter=lfs diff=lfs merge=lfs -text
2
+ *.arrow filter=lfs diff=lfs merge=lfs -text
3
+ *.bin filter=lfs diff=lfs merge=lfs -text
4
+ *.bz2 filter=lfs diff=lfs merge=lfs -text
5
+ *.ckpt filter=lfs diff=lfs merge=lfs -text
6
+ *.ftz filter=lfs diff=lfs merge=lfs -text
7
+ *.gz filter=lfs diff=lfs merge=lfs -text
8
+ *.h5 filter=lfs diff=lfs merge=lfs -text
9
+ *.joblib filter=lfs diff=lfs merge=lfs -text
10
+ *.lfs.* filter=lfs diff=lfs merge=lfs -text
11
+ *.mlmodel filter=lfs diff=lfs merge=lfs -text
12
+ *.model filter=lfs diff=lfs merge=lfs -text
13
+ *.msgpack filter=lfs diff=lfs merge=lfs -text
14
+ *.npy filter=lfs diff=lfs merge=lfs -text
15
+ *.npz filter=lfs diff=lfs merge=lfs -text
16
+ *.onnx filter=lfs diff=lfs merge=lfs -text
17
+ *.ot filter=lfs diff=lfs merge=lfs -text
18
+ *.parquet filter=lfs diff=lfs merge=lfs -text
19
+ *.pb filter=lfs diff=lfs merge=lfs -text
20
+ *.pickle filter=lfs diff=lfs merge=lfs -text
21
+ *.pkl filter=lfs diff=lfs merge=lfs -text
22
+ *.pt filter=lfs diff=lfs merge=lfs -text
23
+ *.pth filter=lfs diff=lfs merge=lfs -text
24
+ *.rar filter=lfs diff=lfs merge=lfs -text
25
+ *.safetensors filter=lfs diff=lfs merge=lfs -text
26
+ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
27
+ *.tar.* filter=lfs diff=lfs merge=lfs -text
28
+ *.tar filter=lfs diff=lfs merge=lfs -text
29
+ *.tflite filter=lfs diff=lfs merge=lfs -text
30
+ *.tgz filter=lfs diff=lfs merge=lfs -text
31
+ *.wasm filter=lfs diff=lfs merge=lfs -text
32
+ *.xz filter=lfs diff=lfs merge=lfs -text
33
+ *.zip filter=lfs diff=lfs merge=lfs -text
34
+ *.zst filter=lfs diff=lfs merge=lfs -text
35
+ *tfevents* filter=lfs diff=lfs merge=lfs -text
36
+ Qwen3-4B-Instruct-2507-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
37
+ Qwen3-4B-Instruct-2507-Q5_K_M.gguf filter=lfs diff=lfs merge=lfs -text
38
+ Qwen3-4B-Instruct-2507-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
39
+ Qwen3-4B-Instruct-2507-Q3_K_S.gguf filter=lfs diff=lfs merge=lfs -text
40
+ Qwen3-4B-Instruct-2507-Q6_K.gguf filter=lfs diff=lfs merge=lfs -text
Qwen3-4B-Instruct-2507-Q3_K_S.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:83ac55108915c8cead0e0eda9f7931b300b27d6c23e6ee8b26db87754521903c
3
+ size 1624344704
Qwen3-4B-Instruct-2507-Q4_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:76e5ae3aebe3fd7bd8317bbf9280d294ec59c3fc840a7f47b0f07b9feb3c96ff
3
+ size 2346866304
Qwen3-4B-Instruct-2507-Q5_K_M.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:32726f25ada6a130cdc375165e21367b52e852cd204c8f6b344ee86d008c54c3
3
+ size 2596988544
Qwen3-4B-Instruct-2507-Q6_K.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e567e777735b915f2357b34841482e3908edbe41657058b3bd85853dc1598d50
3
+ size 2933728384
Qwen3-4B-Instruct-2507-Q8_0.gguf ADDED
@@ -0,0 +1,3 @@
 
 
 
 
1
+ version https://git-lfs.github.com/spec/v1
2
+ oid sha256:e9263a0c1433b00699206b03b427010ebdb3bccb8cf7ea1d9011a527a2e823be
3
+ size 4280403296
README.md ADDED
@@ -0,0 +1,75 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ ---
2
+ language: [en, zh]
3
+ license: apache-2.0
4
+ library_name: gguf
5
+ base_model: Qwen/Qwen3-4B-Instruct-2507
6
+ tags: [quantized, gguf, mixed-precision, shapelearn, qwen3]
7
+ pipeline_tag: text-generation
8
+ ---
9
+
10
+ # Qwen3-4B-Instruct-2507 — ShapeLearn Mixed-Precision GGUF
11
+
12
+ Mixed-precision GGUF quantizations of [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507)
13
+ using **Hessian-informed per-tensor bit allocation**. Each tensor group receives
14
+ the precision level that minimizes quality loss for its measured sensitivity —
15
+ more bits where they matter, fewer where they don't.
16
+
17
+ ## Available Quantizations
18
+
19
+ | File | BPW | Size | PPL ↓ | tok/s | Use Case |
20
+ |---|---|---|---|---|---|
21
+ | [`Qwen3-4B-Instruct-2507-Q8_0.gguf`](./Qwen3-4B-Instruct-2507-Q8_0.gguf) | 8.5 | 3.99 GB | 2.651 | 11.4 | Near-lossless reference |
22
+ | [`Qwen3-4B-Instruct-2507-Q6_K.gguf`](./Qwen3-4B-Instruct-2507-Q6_K.gguf) | 5.8 | 2.73 GB | 2.888 | 13.6 | High quality, moderate size |
23
+ | [`Qwen3-4B-Instruct-2507-Q5_K_M.gguf`](./Qwen3-4B-Instruct-2507-Q5_K_M.gguf) | 5.2 | 2.42 GB | 2.971 | 14.3 | Balanced quality and size |
24
+ | [`Qwen3-4B-Instruct-2507-Q4_K_M.gguf`](./Qwen3-4B-Instruct-2507-Q4_K_M.gguf) | 4.7 | 2.19 GB | 2.978 | 14.1 | Best quality-to-size ratio |
25
+ | [`Qwen3-4B-Instruct-2507-Q3_K_S.gguf`](./Qwen3-4B-Instruct-2507-Q3_K_S.gguf) | 3.2 | 1.51 GB | 3.214 | 18.9 | Maximum compression |
26
+
27
+ **Recommended:** Q4_K_M for the best quality-to-size ratio (PPL 2.978 at just 2.19 GB).
28
+ Q3_K_S for maximum compression. Q6_K for high quality.
29
+
30
+ ## How It Works
31
+
32
+ Standard quantization applies one precision level uniformly across all tensors.
33
+ ShapeLearn uses **Hessian-based sensitivity analysis** (H_diag = mean(X²) per layer)
34
+ to identify which tensors lose the most quality when quantized, then solves an
35
+ LP-optimal knapsack allocation: minimize Σ(sensitivity × quantization_error)
36
+ subject to total size ≤ target. Sensitive tensors get higher precision,
37
+ insensitive ones get lower precision, at the same total file size.
38
+
39
+ Within each suffix group, the fractional BPW planner further varies types
40
+ per-layer using blended imatrix + Hessian scores, so late attention layers
41
+ (most sensitive) get higher precision than middle layers (least sensitive).
42
+
43
+ ## Key Sensitivity Findings (Qwen3-4B)
44
+
45
+ - **Late attention layers (29-35) are most sensitive** — blk.34 k/v score 1.0
46
+ - **down_proj is the most sensitive MLP tensor** — projects back to residual stream
47
+ - **gate_proj/up_proj are least sensitive** — safe to quantize aggressively
48
+ - **K > V for attention weight sensitivity** — k_proj averages 0.66 vs v_proj 0.50
49
+
50
+ ## Usage
51
+
52
+ ```bash
53
+ # Download
54
+ huggingface-cli download sh111111111111111/Qwen3-4B-Instruct-2507-ShapeLearn2-GGUF \
55
+ Qwen3-4B-Instruct-2507-Q4_K_M.gguf --local-dir .
56
+
57
+ # Chat with llama.cpp
58
+ llama-cli -m Qwen3-4B-Instruct-2507-Q4_K_M.gguf -cnv
59
+
60
+ # Serve via API
61
+ llama-server -m Qwen3-4B-Instruct-2507-Q4_K_M.gguf --port 8080
62
+
63
+ # Ollama
64
+ ollama run hf.co/sh111111111111111/Qwen3-4B-Instruct-2507-ShapeLearn2-GGUF:Qwen3-4B-Instruct-2507-Q4_K_M.gguf
65
+ ```
66
+
67
+ ## Benchmark Details
68
+
69
+ All benchmarks run on NVIDIA GB10 ATOM (128GB unified memory, aarch64).
70
+ llama.cpp commit 406f4e3. PPL via `llama-perplexity` (2 chunks, 851 context).
71
+ tok/s via `llama-bench` (tg128, ngl=999).
72
+
73
+ ## License
74
+
75
+ Apache 2.0, inherited from [Qwen3-4B-Instruct-2507](https://huggingface.co/Qwen/Qwen3-4B-Instruct-2507).