1bitLabs/PrismML-Bonsai2-27B-V100-CUDA

PrismML Bonsai 2 27B — V100 CUDA (sm_70)

Community CUDA build so Ternary Bonsai 2 27B runs with real low-bit kernels on NVIDIA Tesla V100 (compute capability 7.0 / Volta).

Download-and-go: grab the tarball (or bin/), grab PrismML weights, run llama-server. No compile required.

Honest speed receipts (Tesla V100 32GB)

Build	Long decode	Notes
Stock Prism CUDA prebuild	~13 tok/s	fatbin had sm_120 kernels — wrong arch for V100
This rebuild CMAKE_CUDA_ARCHITECTURES=70	~46.07 tok/s long / ~46.52 tok/s html	rebench after sm_70 rebuild
Flag-tune baseline mean	~47.55 tok/s	-ngl 99 -c 8192 -np 1
Q4 baseline (same V100)	~33 tok/s	same card, different model/quant

Do not expect 70–80 tok/s for Bonsai 2 on V100 — that later result was Bonsai 1 + DSpark, not this package.

Recommended flags from our tune: -ngl 99 -c 8192 -np 1. Flash-attn (-fa) on hurt in our A/B; leave it off unless you re-measure.

See receipts/ for JSON / markdown excerpts from the lab runs.

Download and go

1. Weights (not in this repo — ~6.8GB GGUF stays on PrismML):

hf download prism-ml/Ternary-Bonsai-2-27B-gguf Ternary-Bonsai-2-27B-PQ2_0.gguf --local-dir ./models

2. Binaries from this repo:

# tarball
hf download <THIS_REPO> bonsai2-v100-cuda-sm70.tar.gz --local-dir .
tar -xzf bonsai2-v100-cuda-sm70.tar.gz
# or use the unpacked bin/cuda-v100/ tree if present

3. Serve:

export LD_LIBRARY_PATH="$PWD/bin/cuda-v100:/usr/local/cuda/lib64:$LD_LIBRARY_PATH"
./bin/cuda-v100/llama-server \
  -m ./models/Ternary-Bonsai-2-27B-PQ2_0.gguf \
  --host 0.0.0.0 --port 8080 \
  -ngl 99 -c 8192 -np 1 \
  --jinja --alias teacher27b-bonsai2 --reasoning-budget 0

What’s inside

• bonsai2-v100-cuda-sm70.tar.gz — slim package: bin/cuda-v100/llama-server + libggml*.so* + libllama.so* (not the GGUF)
• bin/cuda-v100/ — same binaries unpacked (optional convenience)
• RECIPE.md — rebuild commands
• receipts/ — measured tok/s before vs after on Tesla V100 32GB
• SHA256SUMS — checksum for the tarball

Why this exists

PrismML’s Linux CUDA prebuilt binaries target newer arches (we observed sm_120 only in libggml-cuda). On a V100 that falls off the fast path — ternary stops “flying” and can look slower than a plain Q4 (~13 vs ~33 tok/s in our A/B).

This repo ships a rebuild of the PrismML llama.cpp fork (prism branch) with -DCMAKE_CUDA_ARCHITECTURES=70, plus the exact recipe and speed receipts from our lab.

Weights stay on PrismML’s HF repos (Apache 2.0). We redistribute binaries we compiled from their public fork + build instructions.

Rebuild yourself

See RECIPE.md, or:

git clone -b prism https://github.com/PrismML-Eng/llama.cpp
cd llama.cpp
cmake -B build -G Ninja -DGGML_CUDA=ON \
  -DCMAKE_CUDA_ARCHITECTURES=70 \
  -DCMAKE_BUILD_TYPE=Release \
  -DCMAKE_CUDA_COMPILER=/usr/local/cuda/bin/nvcc
cmake --build build -j "$(nproc)"

Or Prism’s demo helper:

git clone https://github.com/PrismML-Eng/Bonsai-demo.git
cd Bonsai-demo
./scripts/build_cuda_linux.sh --cuda-path /usr/local/cuda-12.8 --archs "70" --output cuda-v100

Credits

• Model & fork: PrismML / PrismML-Eng/llama.cpp (prism branch)
• Lab build + receipts: 1bitLabs / STACKS (Nathan Bowdish)

License

Model weights: Apache 2.0 (PrismML).
llama.cpp fork: follow upstream / PrismML license on that repo.
This packaging (recipe + receipts + binaries we built): Apache 2.0.