danielhanchen commited on
Commit
e76cc18
Β·
verified Β·
1 Parent(s): 17bf31a

Add UD-Q4_K_XL run instructions (build llama.cpp from PR #25165)

Browse files
Files changed (1) hide show
  1. README.md +47 -0
README.md CHANGED
@@ -156,6 +156,53 @@ cd llama.cpp && cmake -B build && cmake --build build -j
156
  --spec-type draft-dflash --spec-draft-n-max 15 -fa on --jinja --port 8000
157
  ```
158
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
159
  ## Controlling reasoning
160
 
161
  Laguna S 2.1 has native reasoning support and works best with *preserved thinking*:
 
156
  --spec-type draft-dflash --spec-draft-n-max 15 -fa on --jinja --port 8000
157
  ```
158
 
159
+ ### Running Unsloth's `UD-Q4_K_XL` with llama.cpp (PR #25165)
160
+
161
+ The GGUFs in this repo are Unsloth [Dynamic 2.0](https://docs.unsloth.ai/basics/unsloth-dynamic-2.0-ggufs)
162
+ quants (imatrix calibrated). Laguna support is not in a tagged llama.cpp release
163
+ yet, so build llama.cpp from
164
+ [ggml-org/llama.cpp#25165](https://github.com/ggml-org/llama.cpp/pull/25165):
165
+
166
+ ```shell
167
+ git clone https://github.com/ggml-org/llama.cpp
168
+ cd llama.cpp
169
+ gh pr checkout 25165
170
+ # build with CUDA (drop -DGGML_CUDA=ON for a CPU-only build)
171
+ cmake -B build -DGGML_CUDA=ON
172
+ cmake --build build -j --config Release --target llama-cli llama-server
173
+ cd ..
174
+ ```
175
+
176
+ Download the `UD-Q4_K_XL` shards (~40GB, split into 3 files):
177
+
178
+ ```shell
179
+ huggingface-cli download unsloth/Laguna-S-2.1-GGUF \
180
+ --include "UD-Q4_K_XL/*" \
181
+ --local-dir Laguna-S-2.1-GGUF
182
+ ```
183
+
184
+ Serve it with `llama-server` (pass the first shard; the remaining shards load
185
+ automatically):
186
+
187
+ ```shell
188
+ ./llama.cpp/build/bin/llama-server \
189
+ --model Laguna-S-2.1-GGUF/UD-Q4_K_XL/Laguna-S-2.1-UD-Q4_K_XL-00001-of-00003.gguf \
190
+ --jinja -fa on -ngl 99 --ctx-size 16384 --port 8000
191
+ ```
192
+
193
+ Or run a one-off generation with `llama-cli`:
194
+
195
+ ```shell
196
+ ./llama.cpp/build/bin/llama-cli \
197
+ --model Laguna-S-2.1-GGUF/UD-Q4_K_XL/Laguna-S-2.1-UD-Q4_K_XL-00001-of-00003.gguf \
198
+ --jinja -ngl 99 -p "Write a Flappy Bird game in Python."
199
+ ```
200
+
201
+ > [!NOTE]
202
+ > `-ngl 99` offloads all layers to GPU; lower it (or drop it) if you run out of
203
+ > VRAM. `UD-Q4_K_XL` is roughly 40GB, so it fits on a single 48GB+ GPU or splits
204
+ > across several GPUs.
205
+
206
  ## Controlling reasoning
207
 
208
  Laguna S 2.1 has native reasoning support and works best with *preserved thinking*: