Albert-yz9yt commited on
Commit
ceddbde
Β·
verified Β·
1 Parent(s): d9e3479

Add minimum hardware requirements section

Browse files
Files changed (1) hide show
  1. README.md +33 -1
README.md CHANGED
@@ -85,7 +85,39 @@ This is fundamentally different from a **reasoning model** (like Apex), which ex
85
 
86
  ---
87
 
88
- ## πŸ“Š Tooling Benchmark β€” BugTraceAI Ultra Bench v1.0
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
89
 
90
  Benchmarked on 2026-05-11 at temperature 0.1 and 0.3.
91
 
 
85
 
86
  ---
87
 
88
+ ## οΏ½ Minimum Hardware Requirements
89
+
90
+ Getting a 27B model running well on consumer hardware is not trivial β€” it requires careful quantization. The IMatrix-guided Q4_K_S used here preserves quality in the most critical weight layers, so you get near-F16 performance at a fraction of the VRAM cost.
91
+
92
+ **Q4_K_S β€” 15 GB (Recommended)**
93
+
94
+ - Minimum: **RTX 3090 (24 GB VRAM)** β€” full GPU offload, fast inference
95
+ - RTX 4090 (24 GB) β€” same, slightly faster
96
+ - RTX 4080 (16 GB) β€” runs with reduced context (2048–4096)
97
+ - A4000 (16 GB) β€” workstation-grade, solid for pipelines
98
+ - 2Γ— RTX 3060 (12 GB) β€” split layers across GPUs with `-ts` flag
99
+ - CPU fallback: 64 GB+ RAM β€” slower but fully functional
100
+
101
+ **Q6_K β€” 21 GB (High Fidelity)**
102
+
103
+ - Minimum: **RTX 3090 / A5000 (24 GB VRAM)** β€” tight fit, recommended 4096 ctx
104
+ - A6000 (48 GB) β€” comfortable full offload
105
+ - H100 / A100 (80 GB) β€” server-grade, full context at speed
106
+
107
+ **Practical tip for llama-server:**
108
+ ```bash
109
+ # RTX 3090/4090 β€” full GPU offload
110
+ ./llama-server -m model.gguf -ngl 99 -c 4096 --port 8080
111
+
112
+ # RTX 4080 16GB β€” partial offload
113
+ ./llama-server -m model.gguf -ngl 28 -c 2048 --port 8080
114
+ ```
115
+
116
+ > The fact that this model runs on a single consumer GPU is the result of significant quantization work β€” IMatrix calibration on a domain-specific security corpus ensures the quality loss is minimal where it matters most.
117
+
118
+ ---
119
+
120
+ ## οΏ½πŸ“Š Tooling Benchmark β€” BugTraceAI Ultra Bench v1.0
121
 
122
  Benchmarked on 2026-05-11 at temperature 0.1 and 0.3.
123