llmfan46

/

gemma-4-31B-it-uncensored-heretic-GGUF

Like 187

Interestingly, the llmfan46/gemma-4-31B-it-qat-q4_0-uncensored-heretic-GGUF doesn't perform as well for RP

It was done with a different uncensoring run and the model is not exactly the same either, look at the benchmarks:

MMLU test results:

gemma-4-31B-it:

============================================================

Total questions: 7021

Correct: 6073

Accuracy: 0.8650 (86.50%)

Parse failures: 52

============================================================

gemma-4-31B-it-uncensored-heretic:

============================================================

Total questions: 7021

Correct: 6031

Accuracy: 0.8590 (85.90%)

Parse failures: 37

============================================================

gemma-4-31B-it-qat-q4_0:

============================================================

Total questions: 7021

Correct: 6050

Accuracy: 0.8617 (86.17%)

Parse failures: 61

============================================================

gemma-4-31B-it-qat-q4_0-uncensored-heretic:

============================================================

Total questions: 7021

Correct: 5930

Accuracy: 0.8446 (84.46%)

Parse failures: 26

============================================================

it seems like QAT might have affected this

Yeah, seems to be even worse for the 123B version as according to benchmarks the vanilla QAT of the 12B is much worse than the original non-QAT version. So just use the original model, no need for the QAT q4_0 version since the original performs better.

Could this be related to the model? I noticed in another thread ('Previous version') that other users are complaining about invalidated context cache. Do you think there is a connection?

I have no idea, this model was released months ago and everybody seems to be happy with it for months, my writing finetune Ortenzya was made using my own gemma-4-31B-it-uncensored-heretic version and I am actually using it right now and there is no issue, could this be a recent issue with llama.cpp? Since this:

https://github.com/ggml-org/llama.cpp/issues/25301

Was only submitted 45 minutes ago?

Are you using direct llama.cpp (llama-ui) or LM Studio?

If you are using direct llama.cpp, try these:

If you know the model fits entirely on VRAM and you want the model in Instruct Mode and you want 8192 tokens of context length:

D:\llama-cpp-repo\build\bin\Release\llama-server.exe ^
-m "D:\Models\llmfan46\gemma-4-31B-it-uncensored-heretic-GGUF\gemma-4-31B-it-uncensored-heretic-Q4_K_M.gguf" ^
-ngl 999 ^
-c 8192 ^
-np 1 ^
-fa on ^
--reasoning off ^
--swa-full ^
--host 127.0.0.1 ^
--port 8080

--swa-full ^ removes the lag that can happen from time to time due to prompt reprocessing and --reasoning off ^ disables reasoning.

If you know the model doesn't fit entirely on VRAM and spills over on RAM and you want 8192 tokens of context length:

D:\llama-cpp-repo\build\bin\Release\llama-server.exe ^
-m "D:\Models\llmfan46\gemma-4-31B-it-uncensored-heretic-GGUF\gemma-4-31B-it-uncensored-heretic-BF16.gguf" ^
-c 8192 ^
-np 1 ^
-fa on ^
--jinja ^
--cache-reuse 1 ^
--swa-full ^
--host 127.0.0.1 ^
--port 8080

--cache-reuse 1 ^ and --swa-full ^ again takes care of the lag even when the model is not fully resident in VRAM.

Was only submitted 45 minutes ago?

Yes, that is my ticket.

If you are using direct llama.cpp, try these...

I'm using this command:
llama-server.exe --port 8085 -ctk q4_0 -ctv q4_0 -fa on -np 1 --threads 8 --cpu-strict 1 -ngl 999 --ctx-size 70000 --jinja --temp 1.0 --top-p 0.95 --top-k 64 -m "C:\[AI]\models\gemma-4-31B-it-uncensored-heretic-Q4_K_M.gguf" --offline --reasoning off --mmproj "C:\[AI]\models\gemma-4-uncensored-31B-it-mmproj-BF16.gguf" --no-mmproj-offload --spec-type draft-mtp --spec-draft-n-max 2 --model-draft "C:\[AI]\models\gemma-4-31b-it-qat-q4_0-assistant.gguf"
I have already tested this without MTP, but the situation remains the same.

Regarding the Ortenzya finetune: I tried it, but it performs worse than the original model in my native language.

As for the suggested flags:
--swa-full - requires too much VRAM to be usable in my case (I have 4090)
--cache-reuse 1 - does not seem to resolve the issue.

Yes, that is my ticket.

Oh okay got it, wasn't sure.

Regarding the Ortenzya finetune: I tried it, but it performs worse than the original model in my native language.

Yeah it was mostly made to improve english and japanese.

As for the suggested flags:
--swa-full - requires too much VRAM to be usable in my case (I have 4090)
--cache-reuse 1 - does not seem to resolve the issue.

So I dunno then, sorry about that.

Maybe try on LM Studio, see if it fixes the issue?

Maybe try on LM Studio, see if it fixes the issue?

But it also uses llama.cpp inside, no?

update: - cache size limit reached, removing oldest entry (size = 720.546 MiB) - I wish there is a way to increase cache size limit via parameter...

But it also uses llama.cpp inside, no?

Well yes, but maybe it uses an older version of llama.cpp and/or it uses it's own custom version of llama.cpp? So if you want you can try and it see if the issues appears.

There is also Unsloth Studio if you want to try too:

https://unsloth.ai/docs/new/studio/install

But it's not as practical as LM Studio.

Thanks. To be honest, I tried to use different llama.cpp forks, like atomic-llama or beellama - they have the same issue.
Maybe I should ask some AI agent (like hermes) to dig into llama.cpp code and find a way to increase this cache size somehow...
Anyway, thanks for your help!

Did you try "ik_llama.cpp"?

If not, maybe it could help? You can use ik_llama.cpp directly or use Oobabooga Text Generation Web UI with the ik_llama option, see if it helps?

This is Oobabooga Text Generation Web UI Model loading settings.

Here is the list of loaders you can select from:

llama.cpp
ik_llama.cpp (leave it on llama.cpp and check where it says "ik" under "Use ik_llama.cpp instead of upstream llama.cpp.")
Transformers
ExLlamav3_HF
ExLlamav3
TensorRT-LLM

There are a few options that maybe could help? Like:

"cache-type
Valid options: llama.cpp - fp16, q8_0, q4_0; ExLlamaV3 - fp16, q2 to q8. For ExLlamaV3, you can type custom combinations for separate k/v bits (e.g. q4_q8)."

And/or:

"Activate StreamingLLM to avoid re-evaluating the entire prompt when old messages are removed.
streaming-llm"

Thanks for the ik_llama.cpp tip, I didn't know it existed.

@Kelheor

Have you had time to give LM Studio a try?

ik_llama.cpp: No Windows 11 binaries available.
Oobabooga: I'm already using SillyTavern as my frontend.

Have you had time to give LM Studio a try?

Not yet.

ik_llama.cpp: No Windows 11 binaries available.
Oobabooga: I'm already using SillyTavern as my frontend.

Have you had time to give LM Studio a try?

Not yet.

Can you try LM Studio if it fixes the issue? If it doesn't, try the newest chat_template.jinja that you can find here:

https://huggingface.co/llmfan46/gemma-4-31B-it-uncensored-heretic/blob/main/chat_template.jinja

Inside LM Studio go into:

LLMs>Inference>Prompt Template

Copy the full lines from:

https://huggingface.co/llmfan46/gemma-4-31B-it-uncensored-heretic/blob/main/chat_template.jinja

And paste it inside "Prompt Template", then load the model and try to see if it fixes the issue.

See:

With LM Studio you can edit the chat_template.jinja without having to redo the GGUF.

ik_llama.cpp: No Windows 11 binaries available.

Well that is wrong because it works on my setup no problem, both through Oobabooga and direct ik_llama.cpp

Well that is wrong because it works on my setup no problem, both through Oobabooga and direct ik_llama.cpp

Where I can download it? I can't see them on https://github.com/ikawrakow/ik_llama.cpp

Well that is wrong because it works on my setup no problem, both through Oobabooga and direct ik_llama.cpp

Where I can download it? I can't see them on https://github.com/ikawrakow/ik_llama.cpp

Well you need to clone the repo, there are no precompiled packages as you can see,

To clone it, see command line:

https://github.com/ikawrakow/ik_llama.cpp.git

But this sort of thing is more for advanced users, if you are not an advanced user then you need packages for more practicality, see here:

https://github.com/Thireus/ik_llama.cpp/releases/

Ok, I tried Windows build for ik_llama.cpp - it doesn't support MTA, it also seems to run with --swa-full by default - it works really slow, since I don't have enough VRAM.

I used DeepSeek V4 Pro with OpenCode to analyze llama.cpp sources.

Here what it found:

The real issue is in the Gemma4 SWA checkpoint system. I found a known bug marker in the code:
server-context.cpp:2363-2365:
// [TAG_CHECKPOINTS_FIX_POS_MIN]
// TODO: here we incorrectly determine that the saved checkpoint data covers
// the [pos_min, pos_max] range - this is not true for SWA models
Gemma4 uses llama_kv_cache_iswa (interleaved SWA + global attention). The SWA layers have a limited window (~4k-8k tokens) and overwrite old entries. Checkpoints are supposed to save/restore SWA state for rollback, but the pos_min recorded at checkpoint creation time is queried from kv_swa->seq_pos_min() which may not accurately reflect the range the checkpoint data actually covers.
When this mismatch causes the checkpoint validity check at line 3315 to fail:
return cur.pos_min < pos_min_thold || cur.pos_min == 0;
...all checkpoints are rejected, do_reset = true, and full reprocessing happens every request from that point on.

https://github.com/ggml-org/llama.cpp/pull/24411 - it seems some kind of fix are prepared but still not in master branch (because I can't see a fix in cloned repo).
cur.update_pos(slot.prompt.n_tokens() - n_tokens_cur, pos_min, pos_max); -> cur.update_pos(slot.prompt.n_tokens() - n_tokens_cur, pos_max - n_swa, pos_max);

But DeepSeek telling me that the fix is not perfect:

The proposed change is a step in the right direction but needs a small adjustment. Let me trace the issue through:
What pos_min currently is:
llama_memory_seq_pos_min() for ISWA memory returns only kv_swa->seq_pos_min() (llama-kv-cache-iswa.cpp:142-144). This is the min position still alive in the SWA ring buffer -- roughly pos_max - n_swa + 1.
What pos_max - n_swa computes: one position BEFORE the actual min. This makes the checkpoint appear to cover one more position than it actually does, which is safe for the lookup.
The real fix should be at the checkpoint LOOKUP, not creation. The issue is at server-context.cpp:3315:
return cur.pos_min < pos_min_thold || cur.pos_min == 0;
The pos_min recorded at checkpoint creation time gets compared against pos_min_thold at evaluation time. If pos_min from the ISWA query is ever slightly inflated (the TAG comment says it's incorrect for SWA models), this condition fails and no checkpoint matches, causing do_reset = true.
Your proposed fix at creation time is equivalent to lowering the recorded pos_min, making the lookup more lenient. That works. But a better fix is to also make the lookup itself use the theoretical window:
At server-context.cpp:3554-3555, change:
const auto pos_min = llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id);
const auto pos_max = llama_memory_seq_pos_max(llama_get_memory(ctx_tgt), slot.id);
to:
const auto pos_min_raw = llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id);
const auto pos_max = llama_memory_seq_pos_max(llama_get_memory(ctx_tgt), slot.id);
const auto pos_min = n_swa > 0 ? std::max(0, pos_max - n_swa + 1) : pos_min_raw;
This computes the theoretical SWA window start (pos_max - n_swa + 1) for SWA models, falling back to the raw value otherwise. Then use pos_min in the checkpoint update.
The key insight: Gemma4's SWA cache has exactly n_swa cells per sequence. The oldest position it can hold is exactly pos_max - n_swa + 1. If the query returns a different (higher) value, it's a tracking error. Computing it from pos_max - n_swa gives the correct answer.

So the proposed change by DeepSeek is:
const auto pos_min = llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id); ->
const auto pos_min = n_swa > 0 ? std::max<int64_t>(0, pos_max - n_swa + 1) : llama_memory_seq_pos_min(llama_get_memory(ctx_tgt), slot.id);
at Edit tools\server\server-context.cpp

However, I can't check it, since I can't compile from sources...
So maybe it could be wrong and to fix that I need more powerful AI, like Fable 5... Anyway, it seems I can't find a solution for this issue at this moment.

Try LM Studio, you can easily download and install right away:

https://lmstudio.ai/

Just download the exe, install and try loading the model see if the same issue happens, if it does, try the newest chat_template.jinja that I mentioned earlier.

I am using the model right now on LM Studio and encountering no issues.

You're right. I downloaded LM Studio. It downloaded llama.cpp-win-x86_64-nvidia-cuda12-avx2-2.23.1 server. I tested it inside LM Studio with your jinja template. Then I copied it to separate directory and uninstalled LM Studio (since I don't need it). And this server works without any cache issues, at least for now. And it work with or without your jinja template, so it's not a root cause.

I believe I've found the cause of the issue.

I was using the flag --alias "unsloth/Qwen" and had the model id set to "unsloth/Qwen" in SillyTavern. For some reason, when restarting a conversation, the context was being truncated to only 46k tokens.

However, when I setup LM Studio, I switched the model id in Silly Tavern to llmfan46/gemma-4-31B-it-uncensored-heretic-Q4_K_M, and the server suddenly loaded 67k tokens for the same conversation, and the cache issue completely disappeared.

I've confirmed that this also works in the vanilla llama-server without needing LM Studio.

Cool, glad we were able to resolve the issue.

Yeah, it just a miracle coincidence that we found a reason this way, since I never thought that model id inside SillyTavern can affect KV cache system in llama.cpp server...