HarmenWessels/MiniCPM5-1B-int4-g128-ov

MiniCPM5-1B — OpenVINO INT4 (group-128, no-think template)

OpenVINO IR conversion of openbmb/MiniCPM5-1B (1B-class llama-architecture model, 128k context, Apache-2.0). To our knowledge the first OpenVINO IR of MiniCPM5. Two deliberate artifact-level choices — read before using:

1. Quantization: int4 group-128 (not channel-wise)

optimum-cli export openvino -m openbmb/MiniCPM5-1B \
  --task text-generation-with-past --weight-format int4 --trust-remote-code \
  MiniCPM5-1B-int4-g128-ov

This build uses int4 group-128 rather than channel-wise: a channel-wise build is unstable at the 1B scale (repetition loops), while group-128 and int8 stay coherent.

2. Chat template: thinking disabled (rt_info patch)

MiniCPM5 is a hybrid-reasoning model. Its upstream chat template leaves thinking enabled unless enable_thinking=False is passed — a kwarg OpenVINO GenAI cannot pass. GenAI also reads the template from openvino_tokenizer.xml rt_info, not from chat_template.jinja. This repo ships the rt_info template patched to always emit the no-think prefix (<think>\n\n</think>\n\n after the assistant header) — matching the model card's recommended non-thinking usage.

Both modes remain available. Options:

• Per-request switching (recommended): serve through core-ultra-llm-server, which detects this artifact's template and switches modes per request without reloading ("reasoning_effort": "high" or "enable_thinking": true in the request body; the reasoning streams back separately as reasoning_content).
• Your own GenAI code: swap at runtime via pipe.get_tokenizer().set_chat_template(...) — replace the <think>\n\n</think>\n\n prefix with <think>\n for thinking mode (verified to propagate to subsequent generates).
• Permanent revert: edit the final add_generation_prompt block of the rt_info chat_template in openvino_tokenizer.xml.

Notes

Serving and tooling: core-ultra-llm-server.

Usage (OpenVINO GenAI)

import openvino_genai as ov_genai

pipe = ov_genai.LLMPipeline("MiniCPM5-1B-int4-g128-ov", "GPU", CACHE_DIR="./.ovcache")
print(pipe.generate("Rewrite with type hints:\n```python\ndef add(a, b):\n    return a + b\n```",
                    max_new_tokens=128))

Provenance

• Base model: openbmb/MiniCPM5-1B (OpenBMB, Apache-2.0, released 2026-05)
• Conversion date: 2026-06-06; optimum-intel git-master, transformers 4.57.6
• No finetuning — weights are a direct quantization of the original; the only modification is the chat-template default documented above