# Benchmark fixes that are now normal runtime policy

This document distinguishes a reproducible benchmark setting from the **normal unattended operating policy**. The [one-step installer](tools/windows/Install-Q36-Reliability-Profile.ps1) writes the complete profile, caches, loop guards, personality and system message together; no manual copy/paste of individual settings is required. Every general reliability finding below is applied in the normal profile at the user's selected context; the `9,600`-token Q4_K_M setting is retained only to reproduce its local regression.

## Normal policy

- Non-thinking is the default: `--reasoning off` / `enable_thinking=False`.
- Sampling is required: `temperature 0.7`, `top_p 0.8`, `top_k 20`, `min_p 0`; do not use greedy `temperature 0` for normal Qwen non-thinking work.
- Long-running services use `--parallel 1`, `--n-predict 4096`, `--repeat-penalty 1.08`, and `--repeat-last-n 128`. Clients must use a finite lower per-request maximum whenever possible.
- For explicit identical-token repetition requests only, use a per-request repetition penalty of `1.0`, then restore `1.08`.
- Treat exact text, JSON/schema, required counts and tool-call arguments as validator-backed output. At most three repairs may use the original request, rejected draft and deterministic validator feedback. A repair is operational reliability work, not a single-pass benchmark score.
- Explicit math/reasoning requests may enable thinking with a finite output cap. Do not enable it in harnesses whose stop sequence can end a response inside hidden reasoning.
- llama.cpp normal profile: `131072` context on the validated 32-GB-RAM / 8-GB-VRAM laptop, RAM-resident weights, `q8_0` K/V cache, Flash Attention, continuous batching, a 1536-MiB GPU margin, `--fit on`, and bounded CPU/RAM cache settings. Lower context before changing quantization when a real workload exhausts memory.
- The always-on system profile preserves the direct, sharp-witted, business-minded Q36 personality. It also contains the strict final-output compiler text and loop/repair rule. The supplied client wrapper and Hermes attach it on every command; Ollama embeds it in the created model. Raw llama.cpp HTTP clients must send it explicitly because the tested llama-server build has no server-global system-prompt flag.

The machine-readable version is [`configs/q36-reliability-profile.v1.json`](configs/q36-reliability-profile.v1.json). The Windows installer creates the same profile and launcher.

## What the local regressions found

| Area | Limited local observation | Fix used in normal operation |
|---|---|---|
| Exact format constraints / IFBench | Greedy or unvalidated output missed mechanical constraints. The fixed 20-row regression accepted 20/20 prompt constraints and 22/22 instruction constraints after bounded validation. | Sampling defaults plus strict final-output prompt and bounded validator repair for machine-sensitive tasks. |
| GSM8K-style explicit reasoning | On the fixed 20-row slice, non-thinking scored 13/20; thinking with a 1,024-token cap scored 17/20. | Non-thinking remains default; an explicit reasoning task may opt into thinking and a finite cap. |
| BBH Boolean Expressions with CoT | On the fixed 20-row slice, thinking interacted badly with the task's double-newline stop and produced 6 invalid terminations; non-thinking scored 18/20. | Keep non-thinking for ordinary work and for stop-sensitive harnesses. |
| Agentic MBPP repair | A network-isolated, visible-test repair loop reached 4/5 on a five-task public sample; extra thinking and additional repairs did not reliably improve it. | Tool/code workflows use visible validation, preserve failed tests, and stop after three repairs rather than looping. |
| Windows MBPP evaluator | The upstream POSIX evaluator cannot execute directly on Windows. | Use a networkless, read-only Docker Python runner with resource limits; this is an evaluation adapter, not a model-quality change. |

These are intentionally small, reproducible local samples. They do **not** replace official full-benchmark scores, and repaired-output results must never be reported as raw one-pass scores.

## Always-on personality and final-output message

The installer writes [`Q36-Always-On-System-Profile.txt`](tools/windows/Q36-Always-On-System-Profile.txt) locally and embeds the same content into the Ollama Modelfile. It is sent for every command by the supplied llama.cpp client wrapper. Personality controls tone; strict compiler checks become decisive when a request has exact output constraints, so ordinary conversation remains natural.

## Reproducing the benchmark-only track

The reproducible Q4_K_M track uses 9,600 context tokens. It is explicitly benchmark-only and does not replace the normal context chosen for a deployment. Its task-specific settings are:

- GSM8K fixed sample: thinking on, `max_tokens=1024`.
- BBH Boolean Expressions with the double-newline CoT stop: thinking off, `max_tokens=1024`.
- Agentic code repair: thinking off, visible tests, at most three repair attempts.

The commands and limitations are in [BENCHMARK-PLAN.md](BENCHMARK-PLAN.md).