prism-ml

/

Ternary-Bonsai-2-27B-gguf

Like 2.57k

Follow

Prism ML 4.16k

Predispositions:
() You MUST build; the pre-built executables are not working, they use CPU only (even if you remove the CUDA+HIPs and the script reaches Vulkan)
() Vulkan is faster, as always

Tools that are always helpful:
Git, CMake, and Ninja (Ninja comes with VS; also you may install it separately). Ask your loved one where to get them and how to install, it is straightforward.

For running first time:
git clone https://github.com/PrismML-Eng/llama.cpp.git

Or for updating over Ternary 1:
go to the directory of llama.cpp (the main one) and do
git pull

To run HIPs you need the workstation driver, the game adrenaline driver contains only Vulkan. HIPs maybe have better pp, but Vulkan have better tg (or it was the opposite? don't remember already, as summarized speed Vulkan is better)

Important notice:
Vulkan is multiplatform, you have my encouragement to try it with Intel videocards too!

I try everytime to enable HTTPS, but it doesn't work (with the main llama.cpp is the same), however, here is what I do:
install OpenSSL

Preparation to building:
under PowerShell (under Command prompt you have to use Path):
copy C:\Program Files\OpenSSL-Win64\lib\VC\x64\MD*.* C:\Program Files\OpenSSL-Win64\lib - this is only the 1st time after installing OpenSSL, just to make the libraries 'visible' to the compiler
$env:OPENSSL_ROOT_DIR = "C:\Program Files\OpenSSL-Win64"
$env:GGML_VULKAN_FORCE_COOPMAT="1" - this enables the new matrix cores in RDNA3/4, but also 'may' have effect on RDNA1/2, at least doesn't hurt for sure
$env:GGML_VK_SUBALLOCATION_BLOCK_SIZE="4294967296" - the default was maybe 512MB, setting to 4GB ... have some good effect, yes

cmake -S . -B build.Vulkan -G Ninja -DGGML_VULKAN=ON -DGGML_VULKAN_USE_COOPMAT=ON -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++  -DOPENSSL_ROOT_DIR="C:\Program Files\OpenSSL-Win64" -DGGML_OPENMP=OFF -DLLAMA_BUILD_BORINGSSL=ON -DLLAMA_BUILD_TESTS=OFF

with the current fork of PrismML, the tests=OFF (last parameter) is mandatory, without it the compilation break and can't finish !
() -B build.Vulkan is the directory, where the compiled will go, use whatever you like (or need) here
() you may notice that I duplicate the enviroinment variables as command parameters, and also added BoringSSL for the HTTPS/SSL, and yet no success. Hope for you it will be better, I have no more ideas than to install and include them (as shown in the command).

Build:
cmake --build build.Vulkan --config Release --

after that, as usual in the directory there will be subdir bin\ where all the .exe files reside.

Again why to use this?
Coz the 'official' information is not fully true, just like it was for Ternary1. It works wonderfully, the pre-built executables are unusable for AMD (both ROCm and Vulkan, they have CPU offload only), they generate their own directory tree, and their command prompt is very basic. I think most of the team have no access to AMD hardware.

Here is what you may find useful as starting command for 16GB AMD Radeon card:
.\build.vulkan\bin\llama-server.exe -m "Ternary-Bonsai-2-27B-PQ2_0" -ngl 99 -fa on -c 262144 --host 0.0.0.0 -t 16 --dynatemp-range 0.15 --top-p 0.34 --top-k 12 --min-p 0.45 --repeat-penalty 1.12 --presence-penalty 0.0 -ctk q5_1 -ctv q5_1 --spec-type ngram-mod,ngram-map-k4v -np 1 --spec-draft-n-max 2 --spec-ngram-mod-n-match 20 --spec-ngram-mod-n-min 36 --spec-ngram-mod-n-max 68 --spec-ngram-map-k4v-size-n 10 --cache-ram 4096 --chat-template-file .\Qwen-3.8\Qwen-sharp-chat_template.jinja --reasoning-effort medium --perf --slot-save-path .\cache\ -b 8192 -ub 448 -cms 3172 -ctxcp 56 --lookup-cache-dynamic .\cache\n-gram.cache -lm dio

() --chat-template-file .\Qwen-3.8\Qwen-sharp-chat_template.jinja I'm using the sharp jinja template from peculiar-ragdoll https://huggingface.co/peculiar-ragdoll/Qwen-Sharp-Chat-Templates/tree/main
() -ub 448 is due to AMD RDNA 2 architecture (RX 6xxx cards), no idea how it behaves on RDNA3/4 (it will have benefit to be 768 or even 1024)
() -t 16 is to use 16 CPU threads if anything goes there, not actually needed
() I'm yet to experiment with MTPs, for Ternary1 DSpark was not working, despite it was paired with it.

In my case (RDNA2 with 16GB VRAM) from the dense QWEN 3.8 27B I can use Q3-K-XXS only, this baby here gives 6x pp and slightly lower tg. For some reason currently ngram is not working, only 5% approval, so still on the path to fix it. While with the dense model it was ~83% approval (same topics).
With this default 256K context my vram is 14.4 GB from 16 GB used, CPU usage is most of the time 1% to 3% (despite it shows 0.7 GB shared RAM usage).

So, again: all is working. The model is truly, visibly better than the Q3_XXS dense.
Their efforts (PrismML team) are monumental and their success is much, much more than that!

Thank you very much!
With the official builds:
https://github.com/PrismML-Eng/llama.cpp/releases
Everything I did, whether Vulkan or Rocm, was running on RAM+CPU and the model was not loaded on the gpu.
But with your method and manual build, I was finally able to run this model on my system.
Windows : 11.
Graphics card : amd 6800.

I built it as Vulkan and the final folder size of the engine was 1.34 GB.
I used it with the program :
https://github.com/unslothai/unsloth#-get-started

I also built llama.cpp with ROCm successfully on my RX 6800, and the ROCm backend works correctly.

Initially, I suspected that Unsloth Studio might not be using the ROCm backend properly because I was getting only around 30 tokens/s inside Studio. However, after checking the actual llama-server.exe process and device detection, I found that the issue was actually related to the runtime/build configuration.

My custom engine initially contained a Vulkan build, so Unsloth was launching:

Vulkan0: AMD Radeon RX 6800

After removing that build and setting the engine to use my actual ROCm build, Unsloth correctly reports:

ROCm0: AMD Radeon RX 6800

and the model now runs at around 37–38 tokens/s in Unsloth Studio.

For comparison, when I run the same ROCm engine directly, without Unsloth Studio:

.\build\bin\llama-server.exe `
  -m "C:\Users\Admin\.lmstudio\models\normal\Ternary-Bonsai-2-27B\Ternary-Bonsai-2-27B-PQ2_0_2.gguf" `
  --device ROCm0 `
  --host 127.0.0.1 `
  --port 8080

I get around 44.61 tokens/s with:

• Context: 115,200
• Slots: 4
• n_ctx_slot = 115200
• kv_unified = true
• Backend: ROCm
• GPU: AMD Radeon RX 6800
• Model: Ternary Bonsai 2 27B PQ2_0

So the ROCm backend itself is working correctly in Unsloth. The remaining performance difference appears to come from the additional runtime parameters/configuration used by Unsloth Studio rather than incorrect ROCm detection.

In other words:

Native ROCm llama.cpp: ~44.6 tok/s
Unsloth Studio using the same ROCm engine: ~37–38 tok/s

The Studio configuration still needs some tuning if the goal is to match the native llama.cpp performance.

photo of native ui :

photo of unsloth setting with my rocm engine :

I don't use the Unsloth studio - only llama.cpp directly. And with RX6950XT never saw tg over 17-18t/s ... so your speed is amazing! Win10 if that matters. Also had to revert to 2026.3 Adrenaline as 2026.8 is worse. And 2025.xx is with slower Vulkan.

However, if you continue to have such impressive speeds with ROCm and keep it for work, you may try
@set HSA_ENABLE_SDMA_GANG=1
@set TENSILE_SOLUTION_SELECTION_METHOD=2
and then start the server (llama or unsloth or other)
for me tg have about 8-9% improvement with these, the 2nd one have effect, the 1st have no measurable effect.
Hope for you it is again better ^_^

Maybe I'm putting too many parameters and they 'fight' instead of helping. Have to try simple command, just like yours.
And have to check if there is new ROCm, my last attempts with it were 2 months ago maybe.

GL ~~~