--- language: - en - zh - multilingual license: other license_name: stepfun-community-license license_link: https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE library_name: llama.cpp pipeline_tag: image-text-to-text base_model: stepfun-ai/Step-5-Preview-BF16 tags: - stepfun - step-5 - gguf - llama.cpp - moe - mixture-of-experts - agentic - coding - software-engineering - long-context - 1m-context - multimodal - image-text-to-text - text-generation - image - video - sparse-attention - gqa - q3_k_m - 3-bit - quantization - financial-analysis - deep-research - tool-calling - parallel-tool-calling - json-schema --- ``` ```markdown # Step-5-Preview-GGUF-Q3_K_M-3bit ``` --- # Step-5-Preview β€” GGUF (Q3_K_M, 3-bit)
[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-SHSLab-yellow)](https://huggingface.co/SHSLab) [![GitHub](https://img.shields.io/badge/GitHub-StepFun-181717?logo=github)](https://github.com/stepfun-ai) [![Discord](https://img.shields.io/badge/Discord-Join%20Us-5865F2?logo=discord)](https://discord.gg/stepfun) [![License](https://img.shields.io/badge/License-StepFun%20Community-blue)](https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE) [![Format](https://img.shields.io/badge/Format-GGUF-purple)]() [![Quant](https://img.shields.io/badge/Quant-Q3__K__M%20%283--bit%29-red)]() [![Base](https://img.shields.io/badge/Base-BF16%20safetensors-blue)]() [![Context](https://img.shields.io/badge/Context-Up%20to%201M%20Tokens-green)]()
> **πŸ”₯ Step-5-Preview Q3_K_M GGUF is now available!** > > This is the **3-bit k-quant GGUF** build of Step-5-Preview, converted from the official > **BF16 safetensors** checkpoint (`stepfun-ai/Step-5-Preview-BF16`) and packaged for > **llama.cpp**-based inference. It runs on CPU + GPU hybrid setups and is intended for > local deployment, low-VRAM environments, and quick experimentation. > > **A 4.1 GB vision `mmproj` file is included** for multimodal (image + video) input, > using the Step3-VL perception encoder (`projector_type=step3vl`). --- ## πŸ“– Table of Contents - [Introduction](#-introduction) - [What's in this Repository](#-whats-in-this-repository) - [Quantization Details](#-quantization-details) - [Model Architecture](#-model-architecture) - [Model Specifications](#-model-specifications) - [Benchmark Reference](#-benchmark-reference) - [Hardware Requirements](#-hardware-requirements) - [Quickstart with llama.cpp](#-quickstart-with-llamacpp) - [Multimodal (mmproj) Usage](#-multimodal-mmproj-usage) - [Server Mode](#-server-mode) - [Convert It Yourself](#-convert-it-yourself) - [Known Limitations](#-known-limitations) - [Performance Notes](#-performance-notes) - [Citation](#-citation) - [License](#-license) - [Contact](#-contact) --- ## πŸš€ Introduction **Step-5-Preview** is StepFun's flagship foundation model, designed for **real-world agentic tasks** β€” AI coding, software engineering, professional knowledge work, and financial analysis. It is a **600B-parameter sparse Mixture-of-Experts (MoE)** model with **27B active parameters per token**, a **1M-token context window**, and native **text + image + video** input. This repository contains the **Q3_K_M (3-bit) GGUF** conversion, produced with the official `convert_hf_to_gguf.py` pipeline (see `step5-llamacpp.patch`) and `llama-quantize`. > **πŸ’‘ Why a 3-bit GGUF?** > > β€’ **~5Γ— smaller** than the BF16 checkpoint β€” fits on consumer and prosumer hardware. > β€’ **llama.cpp native** β€” pure CPU, CPU+GPU hybrid, or full GPU offload. > β€’ **Multimodal ready** β€” paired `mmproj` for image and video input. > β€’ **Agent-capable** β€” tool calling and long-context reasoning preserved. --- ## πŸ“¦ What's in this Repository | File | Size (approx.) | Description | |:---|:---|:---| | `Step-5-Preview-Q3_K_M.gguf` | ~220–260 GB | Text model, 3-bit k-quant | | `Step-5-Preview-mmproj-BF16.gguf` | ~4.1 GB | Vision projector (`projector_type=step3vl`) | | `README.md` | β€” | This file | > **Note:** Exact file sizes depend on the final shard layout. If the model is split, load > with `-m Step-5-Preview-Q3_K_M-00001-of-000NN.gguf` β€” llama.cpp auto-loads the rest. --- ## βš™οΈ Quantization Details | Parameter | Value | |:---|:---| | **Base checkpoint** | `stepfun-ai/Step-5-Preview-BF16` (bfloat16 safetensors) | | **Conversion tool** | `convert_hf_to_gguf.py` (with `step5-llamacpp.patch`) | | **Quantization tool** | `llama-quantize` | | **Quant type** | **Q3_K_M** (k-quant, mixed 3-bit) | | **File type tag** | `Q3_K_M` | | **Tokenizer pre-tokenizer** | `deepseek-v3` (identical BPE to DeepSeek-V3) | | **Architecture tag** | `STEP35` | | **Vision projector** | `step3vl` (Step3-VL perception encoder) | ### What Q3_K_M Means Q3_K_M is llama.cpp's **mixed 3-bit k-quant** β€” most tensors are quantized to 3-bit, while sensitive tensors (attention output, some FFN) are kept at higher precision (4–6 bit). This gives a better quality/size trade-off than plain Q3_K_S but is larger. | Quant | Bits (avg) | Relative Size | Relative Quality | |:---|:---|:---|:---| | **BF16** | 16 | 100% | Reference | | **Q8_0** | 8 | ~50% | Near-lossless | | **Q5_K_M** | ~5 | ~33% | Very high | | **Q4_K_M** | ~4 | ~27% | High | | **Q3_K_M** ← this repo | ~3 | **~20%** | **Moderate** | | **Q2_K** | ~2 | ~15% | Noticeable loss | > ⚠️ At 3-bit, expect **perplexity to increase** and some degradation on coding, math, and > fine-grained reasoning tasks compared to BF16 or Q4_K_M. See [Known Limitations](#-known-limitations). --- ## πŸ—οΈ Model Architecture > The GGUF graph reuses the **STEP35** architecture path in llama.cpp, extended for Step-5. ### Core Trunk | Component | Value | |:---|:---| | **Architecture** | Sparse Mixture-of-Experts (MoE) | | **Total parameters** | 600B | | **Active parameters** | 27B per token (~4.5% sparsity) | | **Layers** | 95 (narrow-deep Transformer) | | **Attention pattern** | Hybrid β€” **23 full_attention** + 72 sliding_attention | | **SWA type** | `LLAMA_SWA_TYPE_STANDARD` | | **MoE** | `moe_num_experts` + `moe_top_k` | | **Tokenizer** | DeepSeek-V3 BPE (`tokenizer.ggml.pre = deepseek-v3`) | ### RoPE Configuration | Layer Type | `rope_theta` | Rotary Fraction | |:---|:---|:---| | **full_attention** | `1e7` | **1/3** of `head_dim` | | **sliding_attention** | `1e4` | **1.0** (full) | > The GGUF explicitly writes `rope.dimension_count` (`head_dim/3`) and > `rope.dimension_count_swa` (`head_dim`) so llama.cpp honours Step-5's partial RoPE. > Without these keys, llama.cpp would fall back to the Step3p5 default of `head_dim/2`. ### Vision Encoder (mmproj) | Parameter | Value | |:---|:---| | **Resolution** | 728 px | | **Patch size** | 14 | | **Width** | 1536 | | **Layers** | 47 | | **Downsampler** | stride-2 pair | | **Projector** | `vit_large_projector` | | **`projector_type`** | `step3vl` |
πŸ”¬ Sparse GQA indexer β€” what happens in GGUF The BF16 checkpoint uses a **Sparse GQA** mechanism on the full_attention layers: - **CSA (block compression) indexer** - **top-k selection** over compressed KV blocks This is **not modelled in llama.cpp yet**. During conversion, the affected tensors are dropped and those layers fall back to **dense attention**: - Tensors dropped: `*.sparse_indexer*`, `*.ssmax_s` - Result: **correct output, but slower** and with **different long-context behaviour** than the reference BF16 model. To re-enable sparse attention in the future, the `filter_tensors()` hook in `Step5Model` must be removed once `attention_impl=sparse_gqa` exists in the graph builder.
--- ## πŸ“‹ Model Specifications | Category | Specification | |:---|:---| | **Model Name** | Step-5-Preview | | **Repository** | `Step-5-Preview-GGUF-Q3_K_M-3bit` | | **Base Checkpoint** | `stepfun-ai/Step-5-Preview-BF16` | | **Format** | GGUF | | **Quantization** | Q3_K_M (3-bit k-quant, mixed) | | **Developer** | StepFun | | **Architecture Tag** | `STEP35` | | **Total Parameters** | 600B | | **Active Parameters** | 27B per token (~4.5% sparsity) | | **Layers** | 95 (23 full + 72 sliding attention) | | **Context Window** | Up to 1,000,000 tokens (KV cache limited) | | **Attention** | Hybrid full + sliding; dense fallback in GGUF | | **Input Modalities** | Text, Image, Video (via mmproj) | | **Output Modalities** | Text | | **Video Formats** | MP4, QuickTime, Matroska (≀128 MB, ≀5 min recommended) | | **Reasoning Effort** | `low` / `medium` / `high` (`xhigh`) | | **Tool Calling** | Parallel, strict JSON schema | | **Token Pre-tokenizer** | `deepseek-v3` | | **License** | StepFun Community License | --- ## πŸ“Š Benchmark Reference This GGUF is a **quantized derivative** of the BF16 model. The scores below are for the **BF16 reference model** and are provided for context β€” **Q3_K_M will score lower**, especially on coding, math, and long-context tasks.
Reference scores (BF16, high reasoning effort) | Benchmark | Score | |:---|:---| | **GPQA Diamond** | 93.5% | | **Humanity's Last Exam (HLE)** | 46.5% | | **AA-LCR v1.1** | 88.3% | | **DeepSWE v1.1** | 67.7% | | **Terminal-Bench 4** | 33.3% | | **CyberGym** | 84.7% | | **SciCode** | 58.9% | | **ProgramBench** | 80.5% | | **FrontierFinance** | 66.4% | | **DRACO** | 83.3% | | **MMMU-Pro** | 76.0% | | **Output Speed** | 99.8 tok/s (8Γ— H100, BF16) |
πŸ“‰ Expected degradation at Q3_K_M Quantization at 3-bit affects tasks unevenly. Rough expectations (may vary): | Task Type | Sensitivity to 3-bit | Notes | |:---|:---|:---| | Conversational / general text | Low | Usually near-identical | | Summarization / extraction | Low–Moderate | Minor fluency loss | | Coding / SWE | **High** | Syntax errors, subtler logic bugs | | Math / formal reasoning | **High** | Multi-step chains degrade first | | Long-context (100K+) | **High** | Compounded by dense fallback | | Tool calling / JSON | Moderate | Schema adherence usually OK | | Vision (mmproj is BF16) | Low | Encoder precision is preserved | If you need maximum quality, use the **BF16 safetensors** or a **Q4_K_M / Q5_K_M** GGUF.
--- ## πŸ–₯️ Hardware Requirements
πŸ“ KV cache is the real constraint The 3-bit weights are small, but the **1M-token context window is not**. KV cache memory scales with context length and batch size. Realistic configurations below.
| Setup | RAM / VRAM | Notes | |:---|:---|:---| | **CPU-only (small context)** | 256 GB RAM | Slow but works; ~4–8K context | | **CPU-only (recommended)** | 384–512 GB RAM | 32–64K context feasible | | **CPU + GPU hybrid** | 256 GB RAM + 24 GB VRAM | Offload some layers to GPU | | **CPU + GPU hybrid (pro)** | 256 GB RAM + 48–96 GB VRAM | Best single-node balance | | **Full GPU offload** | ~240 GB VRAM (aggregate) | e.g. 4Γ— A100 80GB / 4Γ— H100 80GB | | **1M context** | 512 GB RAM + paged KV | llama.cpp `--flash-attn` + `--cache-type-k/v q8_0` | > **Tip:** Use `--n-gpu-layers N` to tune offload. Start with `-ngl 99` and reduce until > it fits. For hybrid setups, offloading **attention layers first** usually helps most. --- ## ⚑ Quickstart with llama.cpp ### 1. Build llama.cpp ```bash git clone https://github.com/ggml-org/llama.cpp cd llama.cpp cmake -B build -DGGML_CUDA=ON # or -DGGML_METAL=ON, -DGGML_VULKAN=ON, etc. cmake --build build --config Release -j ``` ### 2. Download the model ```bash # using huggingface-cli huggingface-cli download SHSLab/Step-5-Preview-GGUF-Q3_K_M-3bit \ --local-dir ./Step-5-Preview-Q3_K_M # or with git-lfs git lfs install git clone https://huggingface.co/SHSLab/Step-5-Preview-GGUF-Q3_K_M-3bit ``` ### 3. Run text generation ```bash ./build/bin/llama-cli \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ -p "Explain the significance of the Pareto Frontier in AI scaling." \ -n 512 \ -ngl 99 \ --temp 0.7 \ --top-p 0.95 \ --ctx-size 32768 ``` ### 4. Run with a chat template ```bash ./build/bin/llama-cli \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ --conversation \ -ngl 99 \ --ctx-size 32768 ``` ### 5. Control reasoning effort Step-5-Preview supports `low` / `medium` / `high` / `xhigh`. Pass it via the system prompt or as a template argument, depending on the chat template exposed by the GGUF: ```bash ./build/bin/llama-cli \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ -sys "You are a helpful assistant. Reasoning effort: high." \ -p "Write a Python function to merge two sorted lists." \ -ngl 99 ``` --- ## πŸ‘οΈ Multimodal (mmproj) Usage The vision encoder is shipped as a separate **BF16 mmproj** file. This is intentional β€” the projector stays high-precision while the language model is 3-bit. ### Image input ```bash ./build/bin/llama-mtmd-cli \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ --mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \ --image ./screenshot.png \ -p "Describe this UI and suggest improvements." \ -ngl 99 ``` ### Video input ```bash ./build/bin/llama-mtmd-cli \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ --mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \ --video ./demo.mp4 \ -p "Summarize what happens in this video." \ -ngl 99 ``` > **Limits:** MP4, QuickTime, Matroska. Recommended ≀128 MB and ≀5 minutes per clip. --- ## 🌐 Server Mode `llama-server` exposes an OpenAI-compatible HTTP API. ### Text-only ```bash ./build/bin/llama-server \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ -ngl 99 \ --ctx-size 32768 \ --host 0.0.0.0 \ --port 8080 ``` ### With vision ```bash ./build/bin/llama-server \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ --mmproj ./Step-5-Preview-Q3_K_M/Step-5-Preview-mmproj-BF16.gguf \ -ngl 99 \ --ctx-size 32768 \ --host 0.0.0.0 \ --port 8080 ``` ### Query it ```python from openai import OpenAI client = OpenAI(api_key="none", base_url="http://localhost:8080/v1") response = client.chat.completions.create( model="step-5-preview", messages=[ {"role": "user", "content": "Write a Python function to merge two sorted lists."} ], max_tokens=512, temperature=0.7, ) print(response.choices[0].message.content) ``` --- ## πŸ”§ Convert It Yourself The conversion requires the `step5-llamacpp.patch` applied on top of llama.cpp. ### 1. Apply the patch ```bash cd llama.cpp git checkout ce8caa6e60a03093351d6016a818720e0d46f0fb git apply -p1 /path/to/step5-llamacpp.patch # or: patch -p1 < /path/to/step5-llamacpp.patch ``` ### 2. Convert BF16 safetensors β†’ BF16 GGUF ```bash python convert_hf_to_gguf.py /path/to/Step5_safetensors \ --outtype bf16 \ --outfile Step-5-Preview-BF16.gguf ``` ### 3. Convert the vision projector ```bash python convert_hf_to_gguf.py /path/to/Step5_safetensors \ --mmproj \ --outtype bf16 \ --outfile Step-5-Preview-mmproj-BF16.gguf ``` ### 4. Quantize to Q3_K_M ```bash ./build/bin/llama-quantize \ Step-5-Preview-BF16.gguf \ Step-5-Preview-Q3_K_M.gguf \ Q3_K_M ```
🧩 What the patch changes | File | Change | |:---|:---| | `conversion/__init__.py` | Routes `Step4ForCausalLM` and `MMGPTStepRoboticsForCausalLM` into the `step3` converter (text + mmproj maps). | | `conversion/base.py` | Maps the Step-5 tokenizer hash to the `deepseek-v3` pre-tokenizer. | | `conversion/step3.py` | Adds `Step5Model` (text) and `Step5VisionModel` (mmproj) on the `STEP35` arch; per-layer `rope_theta` by `layer_type`; writes `rope.dimension_count` / `rope.dimension_count_swa`; drops sparse-GQA indexer tensors. | | `src/models/step35.cpp` | Only halves `n_rot_full` when `rope.dimension_count` is absent. |
--- ## ⚠️ Known Limitations 1. **No sparse attention in GGUF.** The CSA indexer + top-k selection is dropped during conversion. Affected layers run **dense attention** β€” correct but slower, and **long-context quality differs** from the BF16 reference. 2. **23 of 95 layers remain full attention.** Combined with the dense fallback, this means long-context performance is not equivalent to the reference model until the indexer is implemented in llama.cpp. 3. **3-bit quantization loss.** Q3_K_M introduces noticeable degradation on coding, math, and fine-grained reasoning. Expect lower scores than the BF16 benchmark table. 4. **MTP / NextN tensors** convert through but are only used if a draft model is configured. 5. **KV cache dominates memory at long context.** The weights are small; the 1M-token context window is not. See [Hardware Requirements](#-hardware-requirements). 6. **Vision is separate.** The mmproj must be loaded explicitly; it is not bundled inside the text GGUF. 7. **Language coverage.** Primarily optimized for English and Chinese. --- ## ⚑ Performance Notes | Factor | Impact | |:---|:---| | **Quant type** | Q3_K_M is ~20% of BF16 size; expect slower per-token speed on CPU than Q4_K_M due to less efficient kernels | | **GPU offload** | `-ngl 99` offloads everything if VRAM allows; reduce if not | | **Flash attention** | Add `--flash-attn` for lower KV memory and faster long-context | | **KV quantization** | `--cache-type-k q8_0 --cache-type-v q8_0` roughly halves KV memory | | **Batch size** | Larger batches improve throughput but increase KV memory | | **Context length** | Larger `--ctx-size` allocates more KV cache upfront | **Recommended flags for long context:** ```bash ./build/bin/llama-cli \ -m ./Step-5-Preview-Q3_K_M/Step-5-Preview-Q3_K_M.gguf \ -ngl 99 \ --ctx-size 131072 \ --flash-attn \ --cache-type-k q8_0 \ --cache-type-v q8_0 \ -p "Summarize this codebase." ``` --- ## πŸ“š Citation If you use Step-5-Preview in your research, please cite: ```bibtex @misc{stepfun2026step5preview, title = {Step-5-Preview: A 600B Sparse MoE Foundation Model for Real-World Agentic Work}, author = {StepFun Team}, year = {2026}, howpublished = {\url{https://huggingface.co/SHSLab/Step-5-Preview-BF16}}, note = {Released September 20, 2026} } ``` If you use this GGUF build, please also reference the base checkpoint and llama.cpp. --- ## πŸ“œ License Step-5-Preview is released under the **StepFun Community License**. See the [LICENSE](https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE) file for full terms. > **⚠️ Usage Restrictions** > > β€’ Commercial use is permitted under the StepFun Community License. > β€’ Redistribution must include the license and attribution. > β€’ See LICENSE for full details. --- ## πŸ“¬ Contact - **Hugging Face:** [SHSLab](https://huggingface.co/SHSLab) - **GitHub:** [github.com/stepfun-ai](https://github.com/stepfun-ai) - **Discord:** [Join our Discord](https://discord.gg/stepfun) - **Email:** [opensource@stepfun.com](mailto:opensource@stepfun.com) - **Website:** [stepfun.com](https://stepfun.com) ---
⭐ If you find Step-5-Preview useful, please give us a star on GitHub and Hugging Face! ⭐

Built with ❀️ by StepFun