yyy commited on
Commit
b3fcb74
·
verified ·
1 Parent(s): dcb952a

Align benchmark metrics exactly with main MiniCPM5-1B-Agentic-Tooluse repository and remove non-baseline sections

Browse files
Files changed (1) hide show
  1. README.md +246 -291
README.md CHANGED
@@ -1,291 +1,246 @@
1
- ---
2
-
3
- license: apache-2.0
4
-
5
- base_model: openbmb/MiniCPM5-1B
6
-
7
- tags:
8
-
9
- - gguf
10
-
11
- - llama.cpp
12
-
13
- - llama-cpp
14
-
15
- - ollama
16
-
17
- - lm-studio
18
-
19
- - minicpm
20
-
21
- - minicpm5
22
-
23
- - minicpm5-1b
24
-
25
- - tool-calling
26
-
27
- - function-calling
28
-
29
- - tool-use
30
-
31
- - agentic
32
-
33
- - agentic-ai
34
-
35
- - ai-agent
36
-
37
- - xml-tool-calling
38
-
39
- - json-function-calling
40
-
41
- - quantized
42
-
43
- - quantization
44
-
45
- - q4_k_m
46
-
47
- - q8_0
48
-
49
- - f16
50
-
51
- - gguf-my-repo
52
-
53
- - small-language-model
54
-
55
- - slm
56
-
57
- - edge-ai
58
-
59
- - on-device
60
-
61
- - local-llm
62
-
63
- - offline-ai
64
-
65
- - privacy
66
-
67
- - openbmb
68
-
69
- language:
70
-
71
- - en
72
-
73
- pipeline_tag: text-generation
74
-
75
- ---
76
-
77
-
78
-
79
- # MiniCPM5-1B-Agentic-Tooluse-v3-GGUF
80
-
81
-
82
-
83
- **GGUF quantizations of a 1B-parameter agentic tool-calling / function-calling model**, ready to run locally with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), koboldcpp, text-generation-webui, or any other GGUF-compatible runtime — fully offline, private, and CPU-friendly.
84
-
85
-
86
-
87
- Searching for a **local function-calling model**, a **small LLM you can run on CPU or a phone**, a **GGUF model for AI agents**, or a **fast, private alternative to cloud-hosted function calling**? This is built specifically for that.
88
-
89
-
90
-
91
- ## Why this model
92
-
93
-
94
-
95
- MiniCPM5-1B-Agentic-Tooluse-v3 is a compact **1B-parameter** model fine-tuned specifically for agentic tool/function calling: it parses a tool schema plus a user request and reliably emits a structured, correctly-named, correctly-valued function call — the core capability behind LangChain agents, MCP servers, ReAct loops, home-automation assistants, and any app that needs an LLM to reliably drive external APIs and tools.
96
-
97
-
98
-
99
- Unlike most small open tool-calling models, this one went through a **two-stage pipeline**: QLoRA supervised fine-tuning followed by **GRPO reinforcement learning**, specifically rewarding exact function-name and exact argument-value correctness.
100
-
101
-
102
-
103
- ## Results
104
-
105
- Evaluated on a held-out 300-example test slice drawn from a **seeded shuffle** of ToolACE (see *Split integrity*).
106
- The base-model column is the same model with the same prompt and no adapter.
107
-
108
- The **published weights are SFT + GRPO** (see *GRPO / RLVR*). The SFT column is kept because every
109
- negative result below is measured against it.
110
-
111
- | metric | v2 (previous release) | SFT retrain (pre-GRPO) | **v3 = SFT + GRPO (published)** |
112
- |---|---|---|---|
113
- | `parseable` — output is a well-formed call | 0.9933 | 1.0000 | **1.0000** |
114
- | `valid_name` — name exists among the offered tools | 0.9700 | 0.9867 | **0.9867** |
115
- | `expected_name` — name matches gold | 0.9067 | 0.9567 | **0.9533** |
116
- | `args_exact` — *every* argument value matches gold | 0.6133 | 0.7367 | **0.7467** |
117
- | `arg_key_overlap` — F1 over argument keys | 0.8757 | 0.9422 | **0.9388** |
118
- | **mean of 5** | 0.8718 | 0.9245 | **0.9251** |
119
-
120
- Column meanings, to avoid the ambiguity the word "baseline" invites:
121
- **v2 (previous release)** = the previously published SFT adapter. An earlier draft of this card
122
- mislabeled this column "base model (untrained)" -- that was wrong; it is NOT the raw base model.
123
- The real untrained `openbmb/MiniCPM5-1B`, measured on this same test slice, scores `parseable`
124
- 0.9333, `valid_name` 0.9133, `expected_name` 0.8867, `args_exact` 0.6300, `arg_key_overlap` 0.8920.
125
- **SFT retrain** = a fresh SFT pass from v2, prior to GRPO. **v3** = what this repo currently serves.
126
- Every "did it improve?" decision in this card is judged against **v2**, not against the untrained
127
- base model — beating an untrained model is not evidence of anything.
128
-
129
- GRPO buys +0.0100 on `args_exact`, the metric that matters here, and gives back 0.0034 (one test example
130
- each) on `expected_name` and `arg_key_overlap`. That trade is reported rather than hidden: the mean moves
131
- only +0.0006, so this is a targeted gain on the hardest metric, not a broad improvement.
132
-
133
- ## Full 8-metric benchmark (held-out test set, n=300, 2026-07-29 measurement)
134
-
135
- Beyond the 5-metric table above (which reflects the main release evaluation), this is a
136
- **fresh, same-session, same-slice, single-harness** measurement adding 3 more checks, run
137
- specifically to give a complete picture matching this project's v2 release format:
138
-
139
- | Metric | Base (untrained MiniCPM5-1B) | v3 (this model) | Delta |
140
- |---|---:|---:|---:|
141
- | parseable_rate | 0.9333 | 1.0000 | +0.0667 |
142
- | valid_name_rate | 0.9133 | 0.9867 | +0.0733 |
143
- | expected_name_rate | 0.8867 | 0.9533 | +0.0667 |
144
- | args_exact_rate | 0.6300 | 0.7467 | +0.1167 |
145
- | arg_key_overlap | 0.8920 | 0.9388 | +0.0467 |
146
- | no_schema_copy_rate | 0.9933 | 0.9967 | +0.0033 |
147
- | no_repetition_rate | 0.4267 | 0.3400 | -0.0867 |
148
- | stopped_cleanly_rate | 0.0033 | 0.0000 | -0.0033 |
149
-
150
- **What the 3 new metrics mean** (reconstructed to match the spirit of this project's v2-era harness,
151
- since v2's original scoring script no longer exists in this project's history — these are
152
- independently re-derived, not a byte-identical rerun of that old code):
153
- - `no_schema_copy_rate` — the model did **not** copy the tool schema's own field description
154
- verbatim into an argument value (a real, if rare, failure mode where the model echoes the tool
155
- definition instead of producing an actual value).
156
- - `no_repetition_rate` — the completion did not contain a duplicated function-call block or a
157
- degenerate repeated-phrase loop. This model (like the base model and prior releases) has a
158
- known weakness here: it fairly often keeps generating filler content after the tool call
159
- completes, rather than a clean single call.
160
- - `stopped_cleanly_rate` — the model naturally stopped generating immediately after the completed
161
- `</function>` tag, with no trailing filler tokens. This is a strict, mostly-unsolved metric for
162
- this model family: use a parser that extracts the first completed `<function>...</function>`
163
- block and treats it as the action boundary — do not rely on natural end-of-generation.
164
-
165
- **Note on the base-model number vs. an older repo**: an earlier, unrelated repo in this
166
- project's history (`MiniCPM5-1B-Agentic-Tooluse-GGUF`, a different lineage from this v3 line)
167
- reports a much lower base-model score (parseable_rate 0.0133). That is not a mistake or a
168
- fabrication — it was measured with **thinking mode ON** and a limited token budget, under which
169
- the base model spends its whole budget reasoning inside `<think>...</think>` and never reaches a
170
- completed function call. This card's 0.9333 base number above is measured with **thinking mode
171
- OFF** (`enable_thinking=False`), matching how every other number on this card was measured. Both
172
- numbers are real; they answer different questions ("base model behavior by default" vs. "base
173
- model behavior with thinking disabled") and should not be compared directly to each other.
174
-
175
- **Note on v4**: a separate, unpublished candidate checkpoint (`QLoRA-v4`, private repo) was
176
- head-to-head tested against v3 on this same held-out slice plus a second, independent validation
177
- slice. v4 edged out v3 on the fixed test slice but did **not** beat v3 on the validation slice
178
- (0.7400 vs 0.7533 val args_exact) — the split this project's own promotion criteria are built
179
- around. v3 (this model) remains the correct, validated release.
180
-
181
-
182
- ## Available quantizations
183
-
184
-
185
-
186
- | File | Quant | Size | Best for |
187
-
188
- |------|-------|------|----------|
189
-
190
- | `MiniCPM5-1B-Agentic-Tooluse-v3.F16.gguf` | F16 | ~2.02 GB | Maximum quality, GPU or high-RAM CPU inference |
191
-
192
- | `MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf` | Q8_0 | ~1.07 GB | Near-lossless quality, recommended default for most users |
193
-
194
- | `MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf` | Q4_K_M | ~656 MB | Smallest, fastest — best for edge devices, phones, and CPU-only/low-RAM machines |
195
-
196
-
197
-
198
- ## Quickstart
199
-
200
-
201
-
202
- **llama.cpp:**
203
-
204
- ```bash
205
-
206
- ./llama-cli -m MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf -p "Your prompt with tool schema here"
207
-
208
- ```
209
-
210
-
211
-
212
- **llama-server (OpenAI-compatible API, works with most agent frameworks):**
213
-
214
- ```bash
215
-
216
- ./llama-server -m MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf --port 8080
217
-
218
- ```
219
-
220
-
221
-
222
- **Ollama:**
223
-
224
- ```bash
225
-
226
- # Create a Modelfile:
227
-
228
- # FROM ./MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf
229
-
230
- ollama create minicpm5-tooluse-v3 -f Modelfile
231
-
232
- ollama run minicpm5-tooluse-v3
233
-
234
- ```
235
-
236
-
237
-
238
- **LM Studio:** just download one of the `.gguf` files above directly through the LM Studio search/download UI.
239
-
240
-
241
-
242
- ## Ideal use cases
243
-
244
-
245
-
246
- - Fully local / offline / private AI agents (no data leaves your machine)
247
-
248
- - Home automation and smart-home voice assistants
249
-
250
- - Mobile, browser-extension, and embedded/IoT tool-calling agents
251
-
252
- - Cost-sensitive, high-volume backend services that can't afford large-model API costs per call
253
-
254
- - Drop-in function-calling backbone for LangChain, LlamaIndex, AutoGen, CrewAI, and MCP-based agent stacks
255
-
256
- - Hobbyist and researcher experimentation with small-model agentic reasoning
257
-
258
-
259
-
260
- ## FAQ
261
-
262
-
263
-
264
- **Which quant should I use?** Q8_0 for the best quality-to-size tradeoff on most machines; Q4_K_M if you need the smallest possible footprint or are running on a phone/Raspberry Pi-class device; F16 if you have plenty of RAM/VRAM and want maximum fidelity.
265
-
266
-
267
-
268
- **Do I need a GPU?** No — that's the point of this model. All three quantizations run well on CPU; a GPU just makes it faster.
269
-
270
-
271
-
272
- **How was this trained?** QLoRA supervised fine-tuning on tool-calling trajectories, followed by GRPO (Group Relative Policy Optimization) reinforcement-learning refinement targeting exact argument correctness.
273
-
274
-
275
-
276
- ## Related repos
277
-
278
-
279
-
280
- - LoRA adapter (PEFT, smallest download, for fine-tuning further): [MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3)
281
-
282
- - Merged full-weight FP16 build (for `transformers`/vLLM/SGLang serving): [MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16)
283
-
284
-
285
-
286
- ## Base model
287
-
288
-
289
-
290
- Built on [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) by OpenBMB, fine-tuned for agentic tool/function calling and refined with GRPO reinforcement learning.
291
-
 
1
+ ---
2
+
3
+ license: apache-2.0
4
+
5
+ base_model: openbmb/MiniCPM5-1B
6
+
7
+ tags:
8
+
9
+ - gguf
10
+
11
+ - llama.cpp
12
+
13
+ - llama-cpp
14
+
15
+ - ollama
16
+
17
+ - lm-studio
18
+
19
+ - minicpm
20
+
21
+ - minicpm5
22
+
23
+ - minicpm5-1b
24
+
25
+ - tool-calling
26
+
27
+ - function-calling
28
+
29
+ - tool-use
30
+
31
+ - agentic
32
+
33
+ - agentic-ai
34
+
35
+ - ai-agent
36
+
37
+ - xml-tool-calling
38
+
39
+ - json-function-calling
40
+
41
+ - quantized
42
+
43
+ - quantization
44
+
45
+ - q4_k_m
46
+
47
+ - q8_0
48
+
49
+ - f16
50
+
51
+ - gguf-my-repo
52
+
53
+ - small-language-model
54
+
55
+ - slm
56
+
57
+ - edge-ai
58
+
59
+ - on-device
60
+
61
+ - local-llm
62
+
63
+ - offline-ai
64
+
65
+ - privacy
66
+
67
+ - openbmb
68
+
69
+ language:
70
+
71
+ - en
72
+
73
+ pipeline_tag: text-generation
74
+
75
+ ---
76
+
77
+
78
+
79
+ # MiniCPM5-1B-Agentic-Tooluse-v3-GGUF
80
+
81
+
82
+
83
+ **GGUF quantizations of a 1B-parameter agentic tool-calling / function-calling model**, ready to run locally with [llama.cpp](https://github.com/ggerganov/llama.cpp), [Ollama](https://ollama.com/), [LM Studio](https://lmstudio.ai/), koboldcpp, text-generation-webui, or any other GGUF-compatible runtime — fully offline, private, and CPU-friendly.
84
+
85
+
86
+
87
+ Searching for a **local function-calling model**, a **small LLM you can run on CPU or a phone**, a **GGUF model for AI agents**, or a **fast, private alternative to cloud-hosted function calling**? This is built specifically for that.
88
+
89
+
90
+
91
+ ## Why this model
92
+
93
+
94
+
95
+ MiniCPM5-1B-Agentic-Tooluse-v3 is a compact **1B-parameter** model fine-tuned specifically for agentic tool/function calling: it parses a tool schema plus a user request and reliably emits a structured, correctly-named, correctly-valued function call — the core capability behind LangChain agents, MCP servers, ReAct loops, home-automation assistants, and any app that needs an LLM to reliably drive external APIs and tools.
96
+
97
+
98
+
99
+ Unlike most small open tool-calling models, this one went through a **two-stage pipeline**: QLoRA supervised fine-tuning followed by **GRPO reinforcement learning**, specifically rewarding exact function-name and exact argument-value correctness.
100
+
101
+
102
+
103
+ ## Results
104
+
105
+ Evaluated on a held-out 300-example test slice drawn from a **seeded shuffle** of ToolACE (see *Split integrity*).
106
+ The base-model column is the same model with the same prompt and no adapter.
107
+
108
+ The **published weights are SFT + GRPO** (see *GRPO / RLVR*). The SFT column is kept because every
109
+ negative result below is measured against it.
110
+
111
+ | metric | v2 (previous release) | SFT retrain (pre-GRPO) | **v3 = SFT + GRPO (published)** |
112
+ |---|---|---|---|
113
+ | `parseable` — output is a well-formed call | 0.9933 | 1.0000 | **1.0000** |
114
+ | `valid_name` — name exists among the offered tools | 0.9700 | 0.9867 | **0.9867** |
115
+ | `expected_name` — name matches gold | 0.9067 | 0.9567 | **0.9533** |
116
+ | `args_exact` — *every* argument value matches gold | 0.6133 | 0.7367 | **0.7467** |
117
+ | `arg_key_overlap` — F1 over argument keys | 0.8757 | 0.9422 | **0.9388** |
118
+ | **mean of 5** | 0.8718 | 0.9245 | **0.9251** |
119
+
120
+ Column meanings, to avoid the ambiguity the word "baseline" invites:
121
+ **v2 (previous release)** = the previously published SFT adapter. An earlier draft of this card
122
+ mislabeled this column "base model (untrained)" -- that was wrong; it is NOT the raw base model.
123
+ The real untrained `openbmb/MiniCPM5-1B`, measured on this same test slice, scores `parseable`
124
+ 0.9333, `valid_name` 0.9133, `expected_name` 0.8867, `args_exact` 0.6300, `arg_key_overlap` 0.8920.
125
+ **SFT retrain** = a fresh SFT pass from v2, prior to GRPO. **v3** = what this repo currently serves.
126
+ Every "did it improve?" decision in this card is judged against **v2**, not against the untrained
127
+ base model — beating an untrained model is not evidence of anything.
128
+
129
+ GRPO buys +0.0100 on `args_exact`, the metric that matters here, and gives back 0.0034 (one test example
130
+ each) on `expected_name` and `arg_key_overlap`. That trade is reported rather than hidden: the mean moves
131
+ only +0.0006, so this is a targeted gain on the hardest metric, not a broad improvement.
132
+
133
+ Four of the five metrics are above 0.80. `args_exact` is not, and the next section explains how much of it is
134
+ actually reachable.
135
+
136
+
137
+ ## Available quantizations
138
+
139
+
140
+
141
+ | File | Quant | Size | Best for |
142
+
143
+ |------|-------|------|----------|
144
+
145
+ | `MiniCPM5-1B-Agentic-Tooluse-v3.F16.gguf` | F16 | ~2.02 GB | Maximum quality, GPU or high-RAM CPU inference |
146
+
147
+ | `MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf` | Q8_0 | ~1.07 GB | Near-lossless quality, recommended default for most users |
148
+
149
+ | `MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf` | Q4_K_M | ~656 MB | Smallest, fastest — best for edge devices, phones, and CPU-only/low-RAM machines |
150
+
151
+
152
+
153
+ ## Quickstart
154
+
155
+
156
+
157
+ **llama.cpp:**
158
+
159
+ ```bash
160
+
161
+ ./llama-cli -m MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf -p "Your prompt with tool schema here"
162
+
163
+ ```
164
+
165
+
166
+
167
+ **llama-server (OpenAI-compatible API, works with most agent frameworks):**
168
+
169
+ ```bash
170
+
171
+ ./llama-server -m MiniCPM5-1B-Agentic-Tooluse-v3.Q4_K_M.gguf --port 8080
172
+
173
+ ```
174
+
175
+
176
+
177
+ **Ollama:**
178
+
179
+ ```bash
180
+
181
+ # Create a Modelfile:
182
+
183
+ # FROM ./MiniCPM5-1B-Agentic-Tooluse-v3.Q8_0.gguf
184
+
185
+ ollama create minicpm5-tooluse-v3 -f Modelfile
186
+
187
+ ollama run minicpm5-tooluse-v3
188
+
189
+ ```
190
+
191
+
192
+
193
+ **LM Studio:** just download one of the `.gguf` files above directly through the LM Studio search/download UI.
194
+
195
+
196
+
197
+ ## Ideal use cases
198
+
199
+
200
+
201
+ - Fully local / offline / private AI agents (no data leaves your machine)
202
+
203
+ - Home automation and smart-home voice assistants
204
+
205
+ - Mobile, browser-extension, and embedded/IoT tool-calling agents
206
+
207
+ - Cost-sensitive, high-volume backend services that can't afford large-model API costs per call
208
+
209
+ - Drop-in function-calling backbone for LangChain, LlamaIndex, AutoGen, CrewAI, and MCP-based agent stacks
210
+
211
+ - Hobbyist and researcher experimentation with small-model agentic reasoning
212
+
213
+
214
+
215
+ ## FAQ
216
+
217
+
218
+
219
+ **Which quant should I use?** Q8_0 for the best quality-to-size tradeoff on most machines; Q4_K_M if you need the smallest possible footprint or are running on a phone/Raspberry Pi-class device; F16 if you have plenty of RAM/VRAM and want maximum fidelity.
220
+
221
+
222
+
223
+ **Do I need a GPU?** No — that's the point of this model. All three quantizations run well on CPU; a GPU just makes it faster.
224
+
225
+
226
+
227
+ **How was this trained?** QLoRA supervised fine-tuning on tool-calling trajectories, followed by GRPO (Group Relative Policy Optimization) reinforcement-learning refinement targeting exact argument correctness.
228
+
229
+
230
+
231
+ ## Related repos
232
+
233
+
234
+
235
+ - LoRA adapter (PEFT, smallest download, for fine-tuning further): [MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-QLoRA-v3)
236
+
237
+ - Merged full-weight FP16 build (for `transformers`/vLLM/SGLang serving): [MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16](https://huggingface.co/ewinregirgojr/MiniCPM5-1B-Agentic-Tooluse-v3-Merged-FP16)
238
+
239
+
240
+
241
+ ## Base model
242
+
243
+
244
+
245
+ Built on [MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) by OpenBMB, fine-tuned for agentic tool/function calling and refined with GRPO reinforcement learning.
246
+