decider-4b v2.1 GGUF: Q4_K_M, Q8_0, BF16 + decide_gguf.py

.gitattributes

@@ -33,3 +33,7 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+decider-4b-v2.1-BF16.gguf filter=lfs diff=lfs merge=lfs -text
+decider-4b-v2.1-Q4_K_M.gguf filter=lfs diff=lfs merge=lfs -text
+decider-4b-v2.1-Q8_0.gguf filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,88 @@
+---
+license: apache-2.0
+base_model: Mapika/decider-4b
+base_model_relation: quantized
+language: [en]
+pipeline_tag: text-classification
+tags: [gguf, llama.cpp, decision-model, calibrated, structured-output, one-pass]
+---
+
+# decider-4b GGUF
+
+GGUF files of [Mapika/decider-4b](https://huggingface.co/Mapika/decider-4b) v2.1 (Hub main `eb5fbdf`), for llama.cpp. decider-4b
+does not generate text. It reads a state and one or more questions, each with an explicit option list, and returns a probability
+for every option from one forward pass. See the [decider-4b card](https://huggingface.co/Mapika/decider-4b) for what the model
+is, how it was trained, and where it is weak.
+
+| file | size | use |
+|---|---|---|
+| `decider-4b-v2.1-Q4_K_M.gguf` | 2.7 GB | smallest; about 0.2 points lower in-task accuracy (table below) |
+| `decider-4b-v2.1-Q8_0.gguf` | 4.5 GB | same quality as the bf16 weights |
+| `decider-4b-v2.1-BF16.gguf` | 8.4 GB | unquantized, for making other quantizations |
+
+The tokenizer, `decider_config.json` (temperatures) and `decide_gguf.py` (the readout on llama.cpp) are in this repository too.
+
+## This is not a chat model
+
+Loading a file in `llama-cli`, `llama-server`, Ollama or LM Studio gives you a text model that continues prompts. That is not how
+decider-4b is used, and its generated text is not its answer. The answer is read from the logits of the option-letter tokens at
+each answer slot of a prompt built by `decider.prompt`, divided by the fitted temperature. `decide_gguf.py` does this with
+llama-cpp-python.
+
+## Usage
+
+```
+pip install decider-ai==1.5.0 llama-cpp-python      # llama-cpp-python 0.3.35 or newer
+# GPU: CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python   (Apple Silicon: -DGGML_METAL=on)
+hf download Mapika/decider-4b-GGUF --local-dir decider-4b-gguf \
+  --include "decider-4b-v2.1-Q4_K_M.gguf" "*.json" "*.jinja" "decide_gguf.py"
+cd decider-4b-gguf
+```
+
+```python
+from decide_gguf import GGUFDecider
+
+d = GGUFDecider("decider-4b-v2.1-Q4_K_M.gguf")      # n_gpu_layers=-1 (all on the GPU if the build has one), n_threads=...
+d.decide("My card was charged twice for the same purchase.",
+         [{"question": "Which department should handle this?", "options": ["billing", "technical", "sales"]},
+          {"question": "How urgent is this?", "options": ["low", "medium", "high"]}])
+# [{'choice': 'billing', 'confidence': 0.85, 'probs': {...}}, {'choice': 'medium', 'confidence': 0.43, 'probs': {...}}]
+```
+
+`decide_gguf.py` covers `decide()` (choice questions). The full API of the package (`system_one` with score and yes/no answers,
+the HTTP server) on GGUF files is not in a released decider-ai yet.
+
+On 8 CPU threads (server CPU), Q4_K_M takes 0.3 to 0.7 s for a request of 40 to 120 tokens; Q8_0 is about 20% slower.
+
+## Measured quality
+
+The regression set of the decider-4b card (95 tasks, 144,226 questions, 67 in-task and 28 held-out tasks) at the shipped
+temperature 1.099, read through llama.cpp (CUDA build, one prompt per decode) and compared row by row with the bf16 weights in
+PyTorch. Accuracy, NLL and ECE are means over tasks.
+
+| | in-task acc / NLL / ECE | held-out acc / NLL / ECE | same answer as bf16 PyTorch |
+|---|---|---|---|
+| bf16 weights, PyTorch | 0.8308 / 0.4145 / 0.0308 | 0.7838 / 0.5703 / 0.0781 | |
+| BF16 GGUF | 0.8308 / 0.4145 / 0.0308 | 0.7837 / 0.5700 / 0.0782 | 99.45% |
+| Q8_0 | 0.8310 / 0.4145 / 0.0309 | 0.7829 / 0.5699 / 0.0778 | 99.31% |
+| Q4_K_M | 0.8288 / 0.4194 / 0.0324 | 0.7834 / 0.5691 / 0.0733 | 97.06% |
+
+The BF16 GGUF differs from PyTorch only on near-ties (median probability difference 0.001). Q8_0 is equal to the bf16 weights
+within that noise; its tasks move up on 31 and down on 39, by at most 0.9 points. Q4_K_M is 0.2 points lower on in-task accuracy
+with slightly higher NLL; held-out accuracy is unchanged. It is lower than bf16 on 54 tasks and higher on 36; the largest drop is
+fin_phrasebank (−3.5 points), then medmcqa and mmlu (−1.3). In Q4_K_M the embedding matrix, which is also the output matrix that
+holds the option-letter rows, is stored in Q6_K.
+
+## Notes
+
+- Score one prompt per `llama_decode`, as `decide_gguf.py` does. With several prompts in one decode (as separate sequences),
+  llama.cpp in September 2026 gives probabilities that change with the other prompts in the batch, by up to 0.02 in BF16 and 0.16
+  in Q4_K_M on this model. One prompt per decode gives the same numbers on every run.
+- CPU and GPU builds give slightly different probabilities on the same file (for example 0.846 and 0.830 for "billing" above,
+  against 0.844 in PyTorch).
+- Conversion: llama.cpp `c9064dded` (2026-09-27), `convert_hf_to_gguf.py --no-mtp --outtype bf16`, then `llama-quantize` to
+  Q8_0 and Q4_K_M. `--no-mtp` is required: the checkpoint has no multi-token-prediction weights, but its config declares one
+  MTP layer, and without the flag the converter writes a file that llama.cpp cannot load.
+- The measurements above use the CUDA build. The CPU and Metal builds were not run over the regression set.
+
+License: Apache-2.0, as decider-4b and its base model Qwen/Qwen3.5-4B-Base.

chat_template.jinja

@@ -0,0 +1,154 @@
+{%- set image_count = namespace(value=0) %}
+{%- set video_count = namespace(value=0) %}
+{%- macro render_content(content, do_vision_count, is_system_content=false) %}
+    {%- if content is string %}
+        {{- content }}
+    {%- elif content is iterable and content is not mapping %}
+        {%- for item in content %}
+            {%- if 'image' in item or 'image_url' in item or item.type == 'image' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain images.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set image_count.value = image_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Picture ' ~ image_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|image_pad|><|vision_end|>' }}
+            {%- elif 'video' in item or item.type == 'video' %}
+                {%- if is_system_content %}
+                    {{- raise_exception('System message cannot contain videos.') }}
+                {%- endif %}
+                {%- if do_vision_count %}
+                    {%- set video_count.value = video_count.value + 1 %}
+                {%- endif %}
+                {%- if add_vision_id %}
+                    {{- 'Video ' ~ video_count.value ~ ': ' }}
+                {%- endif %}
+                {{- '<|vision_start|><|video_pad|><|vision_end|>' }}
+            {%- elif 'text' in item %}
+                {{- item.text }}
+            {%- else %}
+                {{- raise_exception('Unexpected item type in content.') }}
+            {%- endif %}
+        {%- endfor %}
+    {%- elif content is none or content is undefined %}
+        {{- '' }}
+    {%- else %}
+        {{- raise_exception('Unexpected content type.') }}
+    {%- endif %}
+{%- endmacro %}
+{%- if not messages %}
+    {{- raise_exception('No messages provided.') }}
+{%- endif %}
+{%- if tools and tools is iterable and tools is not mapping %}
+    {{- '<|im_start|>system\n' }}
+    {{- "# Tools\n\nYou have access to the following functions:\n\n<tools>" }}
+    {%- for tool in tools %}
+        {{- "\n" }}
+        {{- tool | tojson }}
+    {%- endfor %}
+    {{- "\n</tools>" }}
+    {{- '\n\nIf you choose to call a function ONLY reply in the following format with NO suffix:\n\n<tool_call>\n<function=example_function_name>\n<parameter=example_parameter_1>\nvalue_1\n</parameter>\n<parameter=example_parameter_2>\nThis is the value for the second parameter\nthat can span\nmultiple lines\n</parameter>\n</function>\n</tool_call>\n\n<IMPORTANT>\nReminder:\n- Function calls MUST follow the specified format: an inner <function=...></function> block must be nested within <tool_call></tool_call> XML tags\n- Required parameters MUST be specified\n- You may provide optional reasoning for your function call in natural language BEFORE the function call, but NOT after\n- If there is no function call available, answer the question like normal with your current knowledge and do not tell the user about function calls\n</IMPORTANT>' }}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {%- if content %}
+            {{- '\n\n' + content }}
+        {%- endif %}
+    {%- endif %}
+    {{- '<|im_end|>\n' }}
+{%- else %}
+    {%- if messages[0].role == 'system' %}
+        {%- set content = render_content(messages[0].content, false, true)|trim %}
+        {{- '<|im_start|>system\n' + content + '<|im_end|>\n' }}
+    {%- endif %}
+{%- endif %}
+{%- set ns = namespace(multi_step_tool=true, last_query_index=messages|length - 1) %}
+{%- for message in messages[::-1] %}
+    {%- set index = (messages|length - 1) - loop.index0 %}
+    {%- if ns.multi_step_tool and message.role == "user" %}
+        {%- set content = render_content(message.content, false)|trim %}
+        {%- if not(content.startswith('<tool_response>') and content.endswith('</tool_response>')) %}
+            {%- set ns.multi_step_tool = false %}
+            {%- set ns.last_query_index = index %}
+        {%- endif %}
+    {%- endif %}
+{%- endfor %}
+{%- if ns.multi_step_tool %}
+    {{- raise_exception('No user query found in messages.') }}
+{%- endif %}
+{%- for message in messages %}
+    {%- set content = render_content(message.content, true)|trim %}
+    {%- if message.role == "system" %}
+        {%- if not loop.first %}
+            {{- raise_exception('System message must be at the beginning.') }}
+        {%- endif %}
+    {%- elif message.role == "user" %}
+        {{- '<|im_start|>' + message.role + '\n' + content + '<|im_end|>' + '\n' }}
+    {%- elif message.role == "assistant" %}
+        {%- set reasoning_content = '' %}
+        {%- if message.reasoning_content is string %}
+            {%- set reasoning_content = message.reasoning_content %}
+        {%- else %}
+            {%- if '</think>' in content %}
+                {%- set reasoning_content = content.split('</think>')[0].rstrip('\n').split('<think>')[-1].lstrip('\n') %}
+                {%- set content = content.split('</think>')[-1].lstrip('\n') %}
+            {%- endif %}
+        {%- endif %}
+        {%- set reasoning_content = reasoning_content|trim %}
+        {%- if loop.index0 > ns.last_query_index %}
+            {{- '<|im_start|>' + message.role + '\n<think>\n' + reasoning_content + '\n</think>\n\n' + content }}
+        {%- else %}
+            {{- '<|im_start|>' + message.role + '\n' + content }}
+        {%- endif %}
+        {%- if message.tool_calls and message.tool_calls is iterable and message.tool_calls is not mapping %}
+            {%- for tool_call in message.tool_calls %}
+                {%- if tool_call.function is defined %}
+                    {%- set tool_call = tool_call.function %}
+                {%- endif %}
+                {%- if loop.first %}
+                    {%- if content|trim %}
+                        {{- '\n\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- else %}
+                        {{- '<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                    {%- endif %}
+                {%- else %}
+                    {{- '\n<tool_call>\n<function=' + tool_call.name + '>\n' }}
+                {%- endif %}
+                {%- if tool_call.arguments is defined %}
+                    {%- for args_name, args_value in tool_call.arguments|items %}
+                        {{- '<parameter=' + args_name + '>\n' }}
+                        {%- set args_value = args_value | tojson | safe if args_value is mapping or (args_value is sequence and args_value is not string) else args_value | string %}
+                        {{- args_value }}
+                        {{- '\n</parameter>\n' }}
+                    {%- endfor %}
+                {%- endif %}
+                {{- '</function>\n</tool_call>' }}
+            {%- endfor %}
+        {%- endif %}
+        {{- '<|im_end|>\n' }}
+    {%- elif message.role == "tool" %}
+        {%- if loop.previtem and loop.previtem.role != "tool" %}
+            {{- '<|im_start|>user' }}
+        {%- endif %}
+        {{- '\n<tool_response>\n' }}
+        {{- content }}
+        {{- '\n</tool_response>' }}
+        {%- if not loop.last and loop.nextitem.role != "tool" %}
+            {{- '<|im_end|>\n' }}
+        {%- elif loop.last %}
+            {{- '<|im_end|>\n' }}
+        {%- endif %}
+    {%- else %}
+        {{- raise_exception('Unexpected message role.') }}
+    {%- endif %}
+{%- endfor %}
+{%- if add_generation_prompt %}
+    {{- '<|im_start|>assistant\n' }}
+    {%- if enable_thinking is defined and enable_thinking is false %}
+        {{- '<think>\n\n</think>\n\n' }}
+    {%- else %}
+        {{- '<think>\n' }}
+    {%- endif %}
+{%- endif %}

decide_gguf.py

@@ -0,0 +1,78 @@
+"""decider-4b GGUF on llama.cpp: typed decisions with calibrated probabilities, no torch forward pass.
+
+Needs `pip install decider-ai==1.5.0 llama-cpp-python` (llama-cpp-python 0.3.35 or newer; build it with
+CMAKE_ARGS="-DGGML_CUDA=on" or "-DGGML_METAL=on" for a GPU).  The prompt is built by decider.prompt with the HF tokenizer
+shipped in this repository, llama.cpp runs the rows, and the answer is the softmax over the option-letter logits at each
+answer slot, divided by the temperatures in decider_config.json.  One prompt per llama_decode (see the model card for why).
+
+    from decide_gguf import GGUFDecider
+    d = GGUFDecider("decider-4b-v2.1-Q4_K_M.gguf")          # tokenizer and decider_config.json from the same folder
+    d.decide("My card was charged twice for the same purchase.",
+             [{"question": "Which department should handle this?", "options": ["billing", "technical", "sales"]}])
+"""
+import ctypes, json, os
+
+import numpy as np
+import llama_cpp as L
+from transformers import AutoTokenizer
+from decider.infer import Example, Q, _NoShuffle
+from decider.prompt import MAX_OPTIONS, build, letter_ids
+from decider import temperature as TT
+
+_QUIET = L.llama_log_callback(lambda level, text, data: None)
+
+
+class GGUFDecider:
+    def __init__(self, gguf_path, folder=None, n_ctx=32768, n_gpu_layers=-1, n_threads=None, verbose=False):
+        folder = folder or os.path.dirname(os.path.abspath(gguf_path))
+        cfg = json.load(open(os.path.join(folder, "decider_config.json")))
+        (self.T, self.T_by_type), _ = TT.from_config(cfg)
+        self.name = "decider-" + str(cfg.get("version", "dev"))
+        self.tok = AutoTokenizer.from_pretrained(folder)
+        self.letters = np.asarray(letter_ids(self.tok))
+        if not verbose:
+            L.llama_log_set(_QUIET, ctypes.c_void_p(0))
+        L.llama_backend_init()
+        mp = L.llama_model_default_params(); mp.n_gpu_layers = n_gpu_layers
+        self.model = L.llama_model_load_from_file(os.fsencode(gguf_path), mp)
+        if not self.model:
+            raise RuntimeError(f"llama.cpp could not load {gguf_path}")
+        cp = L.llama_context_default_params()
+        cp.n_ctx = n_ctx; cp.n_batch = n_ctx; cp.n_ubatch = min(2048, n_ctx); cp.n_seq_max = 1
+        if n_threads:
+            cp.n_threads = cp.n_threads_batch = n_threads
+        self.ctx = L.llama_init_from_model(self.model, cp)
+        self.n_ctx = n_ctx
+        self.n_vocab = L.llama_vocab_n_tokens(L.llama_model_get_vocab(self.model))
+        self.batch = L.llama_batch_init(n_ctx, 0, 1)
+
+    def _slot_logits(self, ids, slots):
+        if len(ids) > self.n_ctx:
+            raise ValueError(f"prompt of {len(ids)} tokens exceeds n_ctx {self.n_ctx}")
+        L.llama_memory_clear(L.llama_get_memory(self.ctx), True)
+        b, want = self.batch, set(slots)
+        for i, t in enumerate(ids):
+            b.token[i] = t; b.pos[i] = i; b.n_seq_id[i] = 1; b.seq_id[i][0] = 0; b.logits[i] = i in want
+        b.n_tokens = len(ids)
+        if L.llama_decode(self.ctx, b) != 0:
+            raise RuntimeError("llama_decode failed")
+        rows = []
+        for s in slots:
+            p = ctypes.cast(L.llama_get_logits_ith(self.ctx, s), ctypes.POINTER(ctypes.c_float))
+            rows.append(np.ctypeslib.as_array(p, shape=(self.n_vocab,))[self.letters].astype(np.float64))
+        return rows
+
+    def decide(self, context, questions, max_ctx_tokens=1536):
+        """questions: [{"question": str, "options": [str, ...]}, ...] (2..255 options).
+        -> [{"choice", "confidence", "probs"}, ...], one per question, as decider.infer.Decider.decide returns them."""
+        for q in questions:
+            assert 2 <= len(q["options"]) <= MAX_OPTIONS, f"2..{MAX_OPTIONS} options required"
+        item = build(Example(context, [Q(q["question"], list(q["options"]), 0) for q in questions]), self.tok, _NoShuffle(),
+                     max_options=MAX_OPTIONS, max_ctx_tokens=max_ctx_tokens)
+        T = TT.for_types(self.T, self.T_by_type, ["choice"] * len(questions))
+        out = []
+        for q, lg, n, t in zip(questions, self._slot_logits(item["ids"], item["slots"]), item["nopts"], T):
+            z = lg[:n] / t; p = np.exp(z - z.max()); p /= p.sum()
+            j = int(p.argmax())
+            out.append(dict(choice=q["options"][j], confidence=float(p[j]), probs=dict(zip(q["options"], p.tolist()))))
+        return out

decider-4b-v2.1-BF16.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:a432ef364d83b62b70910f61bd08c2639e05b014d37285188779d974782896d8
+size 8424393760

decider-4b-v2.1-Q4_K_M.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c7083fcfc93f650cd66caeade4a3840ef9eeeeb1e80920907c554f70619d7c56
+size 2708804640

decider-4b-v2.1-Q8_0.gguf

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:74614659ab849bc0ec4f0431e1f84ae8bc11fe30b42c3a235a8b0049d36cbc3d
+size 4482403360

decider_config.json

@@ -0,0 +1,20 @@
+{
+ "temperature": 1.099,
+ "temperature_by_type": {
+  "choice": 1.11,
+  "noul": 1.56,
+  "score": 1.287
+ },
+ "neutralize_none": false,
+ "version": "4b-v2.1",
+ "base": "Mapika/decider-4b v1 + LoRA (merged); v1 is Qwen/Qwen3.5-4B-Base + one supervised pass over mixture v2",
+ "layout": "plain",
+ "max_options": 255,
+ "max_state_tokens": 32768,
+ "schema_first": false,
+ "schema_first_trained": false,
+ "isolated_levels": true,
+ "release_date": "2026-09-24",
+ "requires": "decider-ai>=1.4.0 for temperature_by_type; older versions serve every answer at temperature",
+ "stage": "decider-4b v1 + LoRA rank 64 (alpha 128) on attention and MLP, LR 1e-4, 2 epochs (1,518 steps of 65,536 tokens) over v2's 29,325-row mix in the plain state-first layout (generated decision families with code-computed answers, Qwen3.6-27B-written document questions kept when two independent answers agreed, human-labelled public sets, replay of v1's mixture v2), with the replay rows trained toward v1's own answer distribution (KL to v1) instead of their labels, merged into the bf16 weights; no RL stage; temperature fitted by NLL on 61 in-task regression tasks (the 67 in-task tasks without banking77, clinc_oos, mmlu, arc, winogrande, hellaswag); temperature_by_type fitted with decider.calibrate.fit_by_type on the same regression rows plus our own validation rows (choice, noul and score answers)"
+}

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:06b9509352d2af50381ab2247e083b80d32d5c0aba91c272ca9ff729b6a0e523
+size 19989325

tokenizer_config.json

@@ -0,0 +1,32 @@
+{
+  "add_prefix_space": false,
+  "audio_bos_token": "<|audio_start|>",
+  "audio_eos_token": "<|audio_end|>",
+  "audio_token": "<|audio_pad|>",
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "image_token": "<|image_pad|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 262144,
+  "model_specific_special_tokens": {
+    "audio_bos_token": "<|audio_start|>",
+    "audio_eos_token": "<|audio_end|>",
+    "audio_token": "<|audio_pad|>",
+    "image_token": "<|image_pad|>",
+    "video_token": "<|video_pad|>",
+    "vision_bos_token": "<|vision_start|>",
+    "vision_eos_token": "<|vision_end|>"
+  },
+  "pad_token": "<|endoftext|>",
+  "pretokenize_regex": "(?i:'s|'t|'re|'ve|'m|'ll|'d)|[^\\r\\n\\p{L}\\p{N}]?[\\p{L}\\p{M}]+|\\p{N}| ?[^\\s\\p{L}\\p{M}\\p{N}]+[\\r\\n]*|\\s*[\\r\\n]+|\\s+(?!\\S)|\\s+",
+  "split_special_tokens": false,
+  "tokenizer_class": "Qwen2Tokenizer",
+  "unk_token": null,
+  "video_token": "<|video_pad|>",
+  "vision_bos_token": "<|vision_start|>",
+  "vision_eos_token": "<|vision_end|>"
+}