--- license: apache-2.0 base_model: openbmb/MiniCPM5-2B pipeline_tag: text-generation language: - en - zh tags: - minicpm - minicpm5 - llama - gguf - text-generation - long-context - tool-calling - on-device - edge-ai --- # MiniCPM5-2B — GGUF quants Full llama.cpp quant sweep of [openbmb/MiniCPM5-2B](https://huggingface.co/openbmb/MiniCPM5-2B), OpenBMB's dense 2B model for on-device and local deployment. 30 files — the BF16 source plus 29 quants from Q8_0 down to Q1_0 — all cut from the official BF16 release, with the importance matrix computed on the BF16 weights. The source model uses the standard `LlamaForCausalLM` architecture — no custom kernels and no model-code fork are needed; these files load directly in llama.cpp, Ollama and LM Studio. (OpenBMB also publishes their own [MiniCPM5-2B-GGUF](https://huggingface.co/openbmb/MiniCPM5-2B-GGUF); this repo complements it with the complete quant sweep including the IQ and extreme quants. For vLLM, SGLang or transformers, use the source repo directly.) ## Which file should I get? | File | Size | Use | | --- | --- | --- | | MiniCPM5-2B-BF16.gguf | 4.69 GB | Full precision source. Every quant below is cut from this file. | | MiniCPM5-2B-Q8_0.gguf | 2.50 GB | Effectively lossless. Use when disk and RAM are not the constraint. | | MiniCPM5-2B-Q6_K.gguf | 1.93 GB | Near-lossless; the last stop before quality becomes measurable. | | MiniCPM5-2B-Q5_1.gguf | 1.78 GB | Legacy. Prefer Q5_K_M. | | MiniCPM5-2B-Q5_K_M.gguf | 1.68 GB | Very good quality, noticeably smaller than Q6_K. | | MiniCPM5-2B-Q5_K_S.gguf | 1.65 GB | Slightly smaller than Q5_K_M for a slight quality cost. | | MiniCPM5-2B-Q5_0.gguf | 1.65 GB | Legacy. Prefer Q5_K_M. | | MiniCPM5-2B-Q4_1.gguf | 1.52 GB | Legacy. Prefer Q4_K_M. | | MiniCPM5-2B-Q4_K_M.gguf | 1.45 GB | **The usual default. Best quality-per-byte for most people.** | | MiniCPM5-2B-Q4_K_S.gguf | 1.40 GB | A little smaller than Q4_K_M, a little worse. | | MiniCPM5-2B-IQ4_NL.gguf | 1.39 GB | Non-linear 4-bit; good on hardware without fast K-quant kernels. | | MiniCPM5-2B-Q4_0.gguf | 1.39 GB | Legacy round-to-nearest. Prefer Q4_K_M unless a runtime needs this. | | MiniCPM5-2B-IQ4_XS.gguf | 1.33 GB | Best value at this size; usually beats Q4_K_S while smaller. | | MiniCPM5-2B-Q3_K_L.gguf | 1.28 GB | Small, with real quality loss. Usable when RAM is tight. | | MiniCPM5-2B-Q3_K_M.gguf | 1.20 GB | Smaller again; noticeable degradation. | | MiniCPM5-2B-IQ3_M.gguf | 1.14 GB | Strong 3-bit choice, clearly better than Q3_K_M. | | MiniCPM5-2B-IQ3_S.gguf | 1.11 GB | Slightly smaller than IQ3_M. | | MiniCPM5-2B-Q3_K_S.gguf | 1.11 GB | Aggressive. Prefer IQ3_M at a similar size. | | MiniCPM5-2B-IQ3_XS.gguf | 1.07 GB | Aggressive but coherent. | | MiniCPM5-2B-IQ3_XXS.gguf | 0.99 GB | Very aggressive; the last coherent step down. | | MiniCPM5-2B-Q2_K.gguf | 0.97 GB | Very small, heavily degraded. For experimentation. | | MiniCPM5-2B-Q2_K_S.gguf | 0.92 GB | Smaller than Q2_K, at a further quality cost. | | MiniCPM5-2B-Q2_0.gguf | 0.87 GB | Extreme, group-64. Included for completeness. | | MiniCPM5-2B-IQ2_M.gguf | 0.90 GB | The smallest size most people find usable. | | MiniCPM5-2B-IQ2_S.gguf | 0.85 GB | Below the usual usability line. | | MiniCPM5-2B-IQ2_XS.gguf | 0.80 GB | Experimental. | | MiniCPM5-2B-IQ2_XXS.gguf | 0.74 GB | Experimental. | | MiniCPM5-2B-IQ1_M.gguf | 0.68 GB | Extreme. Expect substantial degradation. | | MiniCPM5-2B-IQ1_S.gguf | 0.64 GB | Extreme. Expect substantial degradation. | | MiniCPM5-2B-Q1_0.gguf | 0.50 GB | Extreme. Included for completeness. | All 30 files combined: 40.15 GB. Shortcuts: default **Q4_K_M** (1.45 GB) · lossless **Q8_0** (2.50 GB) · smallest usable **IQ2_M** (0.90 GB) · 3-bit sweet spot **IQ3_M** (1.14 GB). ## Usage Run straight from the Hub: ```bash llama-cli -hf NANI-Nithin/MiniCPM5-2B-GGUF:Q4_K_M -p "Hello" ``` Serve an OpenAI-compatible endpoint: ```bash llama-server -hf NANI-Nithin/MiniCPM5-2B-GGUF:Q4_K_M ``` Or download a file first: ```bash huggingface-cli download NANI-Nithin/MiniCPM5-2B-GGUF MiniCPM5-2B-Q4_K_M.gguf --local-dir . llama-cli -m MiniCPM5-2B-Q4_K_M.gguf -p "Hello" ``` These are llama.cpp GGUF files. They are not loaded by vLLM, SGLang or transformers — for those backends use the [source repo](https://huggingface.co/openbmb/MiniCPM5-2B). ## About MiniCPM5-2B MiniCPM5-2B is the second model in the MiniCPM5 series from OpenBMB, following MiniCPM5-1B. It is a dense 2B Transformer built for local assistants, coding agents, tool-use workflows and reasoning scenarios where a compact footprint matters. Per the source model card: - **2,516,756,480 parameters** (1,981,982,720 non-embedding), 42 layers, GQA with 16 query heads and 2 KV heads - **131,072-token context** with native long-context support - English and Chinese - Trained with the UltraData tiered data management recipe ([arXiv:2602.09003](https://arxiv.org/pdf/2602.09003)): base + mid-training on the released web datasets, then 400B tokens of deep-thinking SFT, RL, and On-Policy Distillation (OPD) of 16 RL teacher models into the release checkpoint - OpenBMB's own evaluation reports 2B-class open-source SOTA with an average score of 53.9 in their comparison set, competitive with 4B-class models, with particular strengths in code reasoning, math, long-context understanding, tool use and agentic tasks ## Quantization notes - Quantized from the official BF16 release on 2026-09-07 - Importance matrix computed on the BF16 weights, so the IQ and sub-Q4 quants benefit from calibration - All 29 quant types of the standard sweep are present; nothing was skipped for size ## Citation If you use this model, please cite the source work: ```bibtex @article{minicpm4, title={Minicpm4: Ultra-efficient llms on end devices}, author={MiniCPM, Team}, journal={arXiv preprint arXiv:2506.07900}, year={2025} } ``` ## Disclaimer Per the source model card, outputs are generated text and may be inaccurate, biased, or offensive; responses on sensitive topics are not professional advice. The model is provided AS IS, without warranty of any kind; users must use it only for lawful, compliant and ethical purposes. ## License Apache-2.0, matching the [source model](https://huggingface.co/openbmb/MiniCPM5-2B).