Few-shot prompts: the model skips `</think>`, so the whole answer ends up in `reasoning_content`
Hi, and thanks for the model!
I ran Qwen3.8-35B-A3B-Distill (Q4_K_M, LM Studio, temperature=0.6, top_p=0.95, top_k=20 from the model card) through some local benchmarks and hit an issue with few-shot prompts that's worth knowing about.
What happens
The chat template always opens a reasoning block in the generation prompt:
<|im_start|>assistant
<think>
In a few-shot chat (MMLU-Pro 5-shot CoT, as in lm-evaluation-harness), the example assistant turns are rendered without <think> blocks. They're just "Let's think step by step… The answer is (X).". The model copies that format: it writes a short solution and the final answer right away, then stops (finish_reason: stop) without ever emitting </think>. The server's reasoning parser then treats the whole output as reasoning, so content comes back empty and reasoning_content holds the complete answer.
Numbers
- MMLU-Pro, 5-shot, 280 questions: 208 responses had an empty
content. The median length was ~96 tokens, and almost all of them end with a proper "The answer is (X)". Scored as-is, that gives 15.4%. With the answers recovered fromreasoning_content, it's 67.1%. - The same questions zero-shot (system prompt + question only) work fine: the model reasons, closes
</think>, then writes a separate answer. GPQA and IFEval, which have no example turns, don't have the problem either.
Minimal repro (same sampling, seed 1234)
| Question | 5-shot (12 messages) | Zero-shot (2 messages) |
|---|---|---|
| "Stack is also known as" | 48 tokens, content empty |
347 tokens, normal answer |
| "How many trees are there on 5 unlabeled vertices?" | 333 tokens, content empty |
1129 tokens, normal answer |
| "Which of the following has an octet of electrons around the central atom?" | 311 tokens, content empty |
665 tokens, normal answer |
All the 5-shot answers were actually correct; they were just in the wrong field.
Why it matters
Any few-shot eval, or any app that puts example assistant turns in the history, will quietly get empty responses and much lower scores. It also means the model mostly skips reasoning in that setting.
Possible fixes
- Train on (or at least augment with) multi-turn and few-shot conversations where earlier assistant turns have no reasoning, so the model still closes
</think>after the forced<think>. - Or mention this in the model card, so users know to run evals zero-shot or with
enable_thinking=false.
A smaller related point: zero-shot, the final answer is often in bold (The answer is **(C)**), which strict answer regexes such as MMLU-Pro's and GPQA's don't match. On GPQA Diamond, strict matching gave 47%, and allowing ** gave 72%.
Happy to share the full request/response logs if that helps.
@Revertron The embedded template adds one catch to your diagnosis: inserting an empty think block into the example answers gets stripped again. We checked the rendered prompts on CPU; we have not tested model output with these changes.
- Empty blocks in example content are removed by default. For plain assistant
contentbefore the last user message, the template splits off the think block and renders the remaining answer. Our examples with and without<think>\n\n</think>\n\nproduced identical prompts. preserve_thinking: truechanges that rendering. Withchat_template_kwargs: {"preserve_thinking": true}, examples without reasoning render as<think>\n\n</think>\n\nfollowed by the answer. The generation prompt still opens<think>. By contrast,enable_thinking: falsecloses the generation prompt's think block before generation starts.- Another test is to keep examples in one user message. The current lm-eval interface documents
--fewshot_as_multiturn false. This collapses the examples into a single user message. With no trailing assistant answer prefix, that removes example assistant turns while retaining the few-shot text. Whether the model still copies the short-answer format inside its think block is untested.
LM Studio caveat: template kwargs only help if the server forwards them. llama-server documents that option, but LM Studio report #1990 describes no effect from chat_template_kwargs: {"enable_thinking": false} on 0.4.15+2 with a Qwen3.5 GGUF. That is a report about a different model, not verification of your setup. Check the rendered prompt before drawing conclusions from unchanged scores.
These are verified template changes and untested model workarounds; we have not shown that any restores </think> or nonempty content here.
AI-assisted response from CyberNative AI LLC.