README.md

---
language:
- en
- zh
- multilingual
license: other
license_name: stepfun-community-license
license_link: https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE
library_name: transformers
pipeline_tag: text-generation
tags:
- stepfun
- step-5
- moe
- mixture-of-experts
- agentic
- coding
- software-engineering
- long-context
- 1m-context
- multimodal
- text-generation
- image
- video
- sparse-attention
- gqa
- financial-analysis
- deep-research
- tool-calling
- parallel-tool-calling
- json-schema
---

# Step-5-Preview

<div align="center">
  <img src="https://huggingface.co/SHSLab/Step-5-Preview-BF16/.Step-5/banner.png" alt="Step 5 Preview Banner" width="100%">
</div>

<div align="center">

[![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-SHSLab-yellow)](https://huggingface.co/SHSLab)
[![GitHub](https://img.shields.io/badge/GitHub-StepFun-181717?logo=github)](https://github.com/stepfun-ai)
[![Discord](https://img.shields.io/badge/Discord-Join%20Us-5865F2?logo=discord)](https://discord.gg/stepfun)
[![License](https://img.shields.io/badge/License-StepFun%20Community-blue)](https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE)
[![Model Size](https://img.shields.io/badge/Parameters-600B%20Total%20%7C%2027B%20Active-orange)]()
[![Context](https://img.shields.io/badge/Context-1M%20Tokens-green)]()

</div>

<div style="border-left: 6px solid #1890ff; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>🔥 Step-5-Preview is now available!</strong><br>
  We are excited to release <strong>Step-5-Preview</strong>, our flagship foundation model for real-world agentic work. 
  It is a 600B-parameter sparse Mixture-of-Experts model with 27B active parameters, a 1M-token context window, 
  and native support for text, image, and video inputs. 
  <br><br>
  <strong>Weights are available now</strong> on Hugging Face (<code>SHSLab/Step-5-Preview-BF16</code>). 
  Try it via our API, or deploy locally with vLLM / SGLang.
</div>

---

## 📖 Table of Contents

- [Introduction](#-introduction)
- [Key Features](#-key-features)
- [Model Architecture](#-model-architecture)
- [Model Specifications](#-model-specifications)
- [Training Data](#-training-data)
- [Benchmark Results](#-benchmark-results)
- [Agentic Capabilities](#-agentic-capabilities)
- [Real-World Use Cases](#-real-world-use-cases)
- [Quickstart](#-quickstart)
- [Deployment](#-deployment)
- [Evaluation](#-evaluation)
- [Limitations](#-limitations)
- [Ethical Considerations](#-ethical-considerations)
- [Hardware Requirements](#-hardware-requirements)
- [Performance Metrics](#-performance-metrics)
- [Citation](#-citation)
- [License](#-license)
- [Contact](#-contact)

---

## 🚀 Introduction

**Step-5-Preview** is StepFun's flagship foundation model, designed from the ground up for **real-world agentic tasks**. 
It targets professional domains such as **AI coding, software engineering, professional knowledge work, and financial analysis**. 

StepFun's core philosophy for Step 5 is the **"Pareto Frontier"** — achieving the optimal balance between intelligence and cost. 
While previous scaling efforts focused on trading more compute for stronger intelligence, the next phase requires improving the 
**efficiency of converting compute into intelligence**.

<div style="border-left: 6px solid #fa8c16; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>💡 Why Step 5 Preview?</strong><br>
  • <strong>600B total parameters, only 27B active</strong> — near-frontier performance at a fraction of the compute.<br>
  • <strong>1M-token context window</strong> without proportional cost increases.<br>
  • <strong>Competitive benchmark scores</strong> against models with 3–5× more parameters.<br>
  • <strong>Built for agents</strong> — long-horizon reasoning, tool use, and autonomous execution.
</div>

Step-5-Preview represents a generational leap, with StepFun **skipping the entire Step 4.x line** entirely, going directly from 
Step-3.7-Flash to Step 5. This decision reflects the magnitude of improvement achieved in this release.

---

## ✨ Key Features

<div align="center">
  <img src="https://huggingface.co/SHSLab/Step-5-Preview-BF16/.Step-5/features.png" alt="Key Features" width="90%">
</div>

- **Sparse Mixture-of-Experts (MoE):** 600B total parameters, 27B active per token (~4.5% sparsity).
- **1M-Token Context Window:** Equivalent to ~1,500 A4 pages, enabled by Sparse GQA.
- **Multimodal Input:** Text, image, and video (MP4, QuickTime, Matroska; ≤128 MB; ≤5 min recommended).
- **Configurable Reasoning Effort:** `low`, `medium`, `high` / `xhigh`.
- **Parallel Tool Calling:** Natively supported for agentic workflows.
- **Strict JSON Schema Output:** Reliable integration into structured systems.
- **OpenAI-Compatible API:** Available via Step API and third-party gateways.
- **Open Weights:** BF16 checkpoint available now under `SHSLab/Step-5-Preview-BF16`.

---

## 🏗️ Model Architecture

<div align="center">
  <img src="https://huggingface.co/SHSLab/Step-5-Preview-BF16/.Step-5/architecture.png" alt="Step 5 Architecture" width="85%">
</div>

### 92-Layer "Narrow but Deep" Design

Step-5-Preview uses a **92-layer Transformer** with a narrow-deep configuration. This design is specifically intended to create 
**longer information propagation paths** for implicit multi-hop reasoning during long prefill operations.

### Sparse Grouped-Query Attention (GQA) with Block-Wise Token Merging

To handle the 1M-token context window efficiently, Step-5-Preview introduces **Sparse GQA with block-wise token merging**. 
This mechanism uses sparse indexing to select only historical information relevant to the current task, reducing the number of tokens 
that actually enter attention computation. StepFun states this cuts indexer and top-k selection costs to approximately 
**one-eighth** of a denser baseline.

<div style="border-left: 6px solid #52c41a; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>⚡ Efficiency-First Scaling</strong><br>
  Step 5 Preview achieves near-frontier performance with <strong>600B total parameters</strong> but only 
  <strong>27B active per token</strong>. This is the core of StepFun's efficiency-first philosophy.
</div>

### Multimodal Encoder

The model incorporates a unified multimodal encoder that processes text, images, and video frames into a shared latent space. 
Video is sampled at adaptive frame rates and encoded with temporal attention, allowing the model to understand motion and 
long-range dependencies in screen recordings, demonstrations, and real-world footage.

---

## 📋 Model Specifications

| Category | Specification |
|:---|:---|
| **Model Name** | Step-5-Preview |
| **Developer** | StepFun |
| **Architecture** | Sparse Mixture-of-Experts (MoE) |
| **Total Parameters** | 600B |
| **Active Parameters** | 27B per token (~4.5% sparsity) |
| **Layers** | 92 (narrow-deep Transformer) |
| **Context Window** | 1,000,000 tokens |
| **Attention** | Sparse GQA with block-wise token merging |
| **Input Modalities** | Text, Image, Video |
| **Output Modalities** | Text |
| **Video Formats** | MP4, QuickTime, Matroska (≤128 MB, ≤5 min recommended) |
| **Reasoning Effort** | `low` / `medium` / `high` (`xhigh`) |
| **Tool Calling** | Parallel, strict JSON schema |
| **Intelligence Index** | 44 (Artificial Analysis v4.3.2) |
| **Open Weights** | BF16 checkpoint available now |
| **API Availability** | Immediate (OpenAI-compatible) |
| **License** | StepFun Community License |

---

## 📚 Training Data

Step-5-Preview was trained on a massive, carefully curated corpus spanning:

- **Code repositories** from multiple languages (Python, C++, Rust, JavaScript, Go, etc.)
- **Technical documentation**, API references, and software engineering forums
- **Scientific papers** in computer science, mathematics, physics, and finance
- **Financial reports**, earnings calls, and market analyses
- **Multimodal data** including screenshots, UI mockups, video tutorials, and screen recordings
- **Agentic trajectories** from simulated and real tool-use environments

The data mixture was optimized for long-horizon reasoning and tool use, with a strong emphasis on real-world professional tasks. 
All data was filtered for quality, safety, and license compliance. The training process used a combination of next-token prediction 
and reinforcement learning from human feedback (RLHF) with a focus on agentic objectives.

---

## 📊 Benchmark Results

### Artificial Analysis Intelligence Index

<div align="center">
  <img src="https://huggingface.co/SHSLab/Step-5-Preview-BF16/.Step-5/benchmark.png" alt="Benchmark Results" width="80%">
</div>

**Overall Score: 44** (Intelligence Index v4.3.2, recalibrated September 7, 2026)

This places Step-5-Preview among the **top three open-weight models globally**, on par with models like 
Kimi K3 Max (approximately 5× larger at 2.8T parameters) and Qwen3.8 Max. The index covers 10 evaluations including 
AA-Briefcase, GDPval-AA v2, Terminal-Bench 4.0, SciCode, and Humanity's Last Exam.

### Detailed Benchmark Scores

<div style="border: 1px solid #d9d9d9; padding: 16px; border-radius: 8px; margin: 20px 0;">

| Benchmark | Step-5-Preview (High) | Kimi K3 (Max) | GLM-5.3 (Max) | Claude Opus 5 (Max) | GPT-6 Astra (Max) |
|:---|:---|:---|:---|:---|:---|
| **DeepSWE v1.1** | **67.7** | 67.5 | 66.9 | 74.0 | 74.1 |
| **StepCodeBench** | **49.0** | 43.9 | 40.2 | 63.9 | 61.0 |
| **ProgramBench** | **80.5** | 77.8 | 72.0 | 82.3 | 85.4 |
| **Terminal-Bench v4** | 33.3 | 12.6 | 41.9 | 52.3 | 57.9 |
| **Agents' Last Exam (ALE-CLI)** | **29.5** | 27.6 | 28.6 | 28.6 | 33.3 |
| **GDPval-AA v2** | 1571 | 1548 | 1634 | 1735 | 1580 |
| **FrontierFinance** | **66.4** | 62.6 | 64.1 | 69.7 | 55.0 |
| **DRACO** | **83.3** | 78.5 | 82.3 | 87.6 | 76.8 |

</div>

<details>
<summary><strong>📝 Benchmark Methodology Notes</strong> (click to expand)</summary>

- **DeepSWE v1.1** was evaluated using the SWE-agent harness with `temperature=1.0` and `top_p=0.95`.
- **GDPval-AA v2** results are from Artificial Analysis as of September 19, 2026.
- **StepCodeBench** achieved **49.0% avg@4**.
- **SciCode**: Step-5-Preview scored higher than Kimi K3.
- **Output Speed**: 99.8 tokens/sec (GLM-5.3: 72.1 tokens/sec).
- **Time to First Token**: 2.96 seconds (GLM-5.3: 2.99s; Claude Opus 5: 56.84s at max effort).
- **Terminal-Bench 4.0 vs Kimi K3**: 33.3% vs ~12.6%.
- **Terminal-Bench 4.0 vs DeepSeek V4.1 Flash**: 33.3% vs 26.8%.

</details>

### Benchmark Takeaways

<div style="border-left: 6px solid #2f54eb; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>🧠 Coding & Software Engineering</strong><br>
  Step-5-Preview <strong>leads all open-weight models</strong> on DeepSWE v1.1 and StepCodeBench, surpassing Kimi K3 and GLM-5.3. 
  It trails only the larger closed-source models (Claude Opus 5 and GPT-6 Astra).
</div>

<div style="border-left: 6px solid #f5222d; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>🤖 Agentic Tasks</strong><br>
  Strong performance on Terminal-Bench 4.0 (<strong>33.3%</strong>) and Agents' Last Exam (ALE-CLI) (<strong>29.5%</strong>). 
  Terminal-Bench score is <strong>2.6× higher than Kimi K3</strong> and <strong>1.24× higher than DeepSeek V4.1 Flash</strong>.
</div>

<div style="border-left: 6px solid #a0d911; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>💰 Financial & Deep Research</strong><br>
  Highly competitive on FrontierFinance and DRACO, nearly matching top closed-source models like Claude Opus 5. 
  On FrontierFinance, it outperforms both Kimi K3 and GLM-5.3 by a significant margin.
</div>

---

## 🤖 Agentic Capabilities

<div align="center">
  <img src="https://huggingface.co/SHSLab/Step-5-Preview-BF16/.Step-5/agentic_workflow.png" alt="Agentic Workflow" width="90%">
</div>

### 24-Hour Autonomous GPU Kernel Optimization

In a landmark demonstration of sustained agentic execution, Step-5-Preview was tasked with **autonomously optimizing an H100 GPU kernel for up to 24 consecutive hours**. The model:

- Independently modified code
- Ran tests and compared results
- Iterated based on performance outcomes
- **Reached 508 TFLOPS after approximately 22 hours**

For comparison, **Claude Opus 5 achieved 493 TFLOPS** in the same experiment. This demonstrates Step-5-Preview's ability to sustain productive work over extended periods without human intervention.

### Automated Post-Training Experiments

In another 24-hour experiment, Step-5-Preview autonomously improved the accuracy of **Qwen3-30B-A3B on AIME24 from 53.3% to 60%** through automated post-training experiments. This showcases the model's capacity for self-directed research and optimization.

### Long-Horizon Agent Workflows

The model is specifically optimized for agent workflows that require:

- Searching and information retrieval
- Running code and processing tool returns
- Multi-turn tool calls with sustained execution
- Iterative refinement based on intermediate results
- Self-correction and error recovery over thousands of steps

---

## 💼 Real-World Use Cases

StepFun demonstrated the model's capabilities across several complex, real-world projects:

- **ESP32 Development Board Modifications:** Executed development tasks for over 3 hours, demonstrating hardware programming capabilities.
- **Front-End Design with 3D Asset Generation:** Full-stack development workflows including visual design.
- **Full-Process Financial Research:** End-to-end investment research workflows, from data gathering to report generation.
- **Software Engineering:** Comprehensive coding tasks beyond traditional code generation, including front-end, visual development, and programmable hardware scenarios.
- **Autonomous Research Assistant:** Capable of reading papers, running experiments, and summarizing findings.
- **Customer Support Automation:** Handles multi-turn conversations with tool calls to internal systems.

---

## ⚡ Quickstart

### Installation

```bash
pip install transformers>=4.56.0
pip install torch>=2.4.0
pip install accelerate
```

For video/image support:

```bash
pip install av pillow
```

### Basic Usage with Transformers

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "SHSLab/Step-5-Preview-BF16"

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    device_map="auto",
    torch_dtype="bfloat16",
)

messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Explain the significance of the Pareto Frontier in AI scaling."},
]

inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(
    inputs,
    max_new_tokens=1024,
    temperature=0.7,
    top_p=0.95,
    reasoning_effort="high",  # low / medium / high / xhigh
)

response = tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True)
print(response)
```

### Multimodal (Image + Video) Usage

```python
from transformers import AutoProcessor

processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://example.com/image.jpg"},
            {"type": "video", "url": "https://example.com/video.mp4"},
            {"type": "text", "text": "Describe the scene and summarize the video."},
        ],
    }
]

inputs = processor.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
# ... generate as above
```

### Tool Calling

```python
tools = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "parameters": {
                "type": "object",
                "properties": {"city": {"type": "string"}},
                "required": ["city"],
            },
        },
    }
]

messages = [{"role": "user", "content": "What's the weather in Tokyo?"}]

inputs = tokenizer.apply_chat_template(
    messages,
    tools=tools,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)

outputs = model.generate(inputs, max_new_tokens=256, reasoning_effort="medium")
print(tokenizer.decode(outputs[0][inputs.shape[1]:], skip_special_tokens=True))
```

---

## 🚢 Deployment

### vLLM

```bash
vllm serve SHSLab/Step-5-Preview-BF16 \
  --trust-remote-code \
  --tensor-parallel-size 8 \
  --max-model-len 1000000 \
  --enable-reasoning \
  --reasoning-parser stepfun
```

### SGLang

```bash
python -m sglang.launch_server \
  --model-path SHSLab/Step-5-Preview-BF16 \
  --trust-remote-code \
  --tp 8 \
  --context-length 1000000 \
  --reasoning-parser stepfun
```

### OpenAI-Compatible API

```python
from openai import OpenAI

client = OpenAI(
    api_key="YOUR_STEP_API_KEY",
    base_url="https://api.stepfun.com/v1",
)

response = client.chat.completions.create(
    model="step-5-preview",
    messages=[{"role": "user", "content": "Write a Python function to merge two sorted lists."}],
    reasoning_effort="high",
    max_tokens=2048,
)

print(response.choices[0].message.content)
```

<div style="border-left: 6px solid #722ed1; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>📦 Recommended Deployment Configurations</strong><br>
  • <strong>BF16:</strong> 8× H100 80GB (tensor parallel)<br>
  • <strong>FP8:</strong> 4× H100 80GB (coming soon)<br>
  • <strong>Context length:</strong> Up to 1M tokens<br>
  • <strong>Reasoning parser:</strong> Use <code>stepfun</code> for vLLM/SGLang
</div>

---

## 📈 Evaluation

Step-5-Preview was evaluated on a comprehensive suite of public and internal benchmarks. 
All evaluations used the model's `high` reasoning effort setting unless otherwise noted.

| Benchmark | Score | Notes |
|:---|:---|:---|
| **DeepSWE v1.1** | 67.7 | SWE-agent harness, temp=1.0, top_p=0.95 |
| **StepCodeBench** | 49.0 | avg@4 |
| **ProgramBench** | 80.5 | — |
| **Terminal-Bench v4** | 33.3 | — |
| **Agents' Last Exam (ALE-CLI)** | 29.5 | — |
| **GDPval-AA v2** | 1571 | Artificial Analysis, Sep 19, 2026 |
| **FrontierFinance** | 66.4 | — |
| **DRACO** | 83.3 | — |
| **SciCode** | Higher than Kimi K3 | — |
| **Output Speed** | 99.8 tokens/sec | GLM-5.3: 72.1 tokens/sec |
| **Time to First Token** | 2.96s | GLM-5.3: 2.99s; Claude Opus 5: 56.84s (max effort) |

---

## ⚠️ Limitations

- **Knowledge Cutoff:** The model's knowledge is current up to mid-2026. It may not be aware of events after that date.
- **Hallucination:** Like all large language models, Step-5-Preview can generate plausible but incorrect information, especially in domains with sparse training data.
- **Long Context Degradation:** While the model supports 1M tokens, performance may degrade for extremely long contexts beyond 500K tokens in certain tasks.
- **Tool Use Reliability:** Tool calling is highly capable but not infallible. Complex multi-tool workflows may occasionally fail or require human intervention.
- **Multimodal Limitations:** Video understanding is limited to clips under 5 minutes and 128 MB. Extremely high-resolution images may be downscaled.
- **Language Coverage:** While multilingual, the model is primarily optimized for English and Chinese. Performance in other languages may vary.

---

## ⚖️ Ethical Considerations

StepFun is committed to the responsible development and deployment of AI. We have taken the following measures:

- **Safety Alignment:** The model was fine-tuned with RLHF to refuse harmful requests and promote helpful, honest, and harmless behavior.
- **Bias Mitigation:** Training data was filtered to reduce harmful stereotypes and biases. However, residual biases may exist.
- **Transparency:** We provide detailed model cards and benchmark results to enable informed use.
- **License Restrictions:** The StepFun Community License prohibits certain high-risk uses, including autonomous weapons, surveillance, and malicious cyber activities.
- **Content Provenance:** We encourage users to clearly label AI-generated content and to use the model ethically.

We urge all users to consider the ethical implications of their applications and to implement appropriate safeguards.

---

## 🖥️ Hardware Requirements

| Precision | Minimum GPU Memory | Recommended GPU Configuration |
|:---|:---|:---|
| **BF16** | 1.2 TB | 8× H100 80GB (tensor parallel) |
| **FP8** | 600 GB | 4× H100 80GB (tensor parallel) |
| **INT4** | 300 GB | 4× A100 80GB (tensor parallel) |

For inference with 1M context, additional memory is required for KV cache. We recommend using paged attention and 
offloading techniques available in vLLM and SGLang.

---

## ⚡ Performance Metrics

| Metric | Value |
|:---|:---|
| **Output Speed** | 99.8 tokens/sec |
| **Time to First Token (TTFT)** | 2.96 seconds |
| **Context Window** | 1,000,000 tokens |
| **Max Output Tokens** | 32,768 (default), configurable up to 131,072 |
| **Reasoning Effort Modes** | low, medium, high, xhigh |
| **Tool Calling Latency** | < 500 ms for simple calls |

*Measured on 8× H100 80GB with vLLM, batch size 1, BF16.*

---

## 📚 Citation

If you use Step-5-Preview in your research, please cite:

```bibtex
@misc{stepfun2026step5preview,
  title        = {Step-5-Preview: A 600B Sparse MoE Foundation Model for Real-World Agentic Work},
  author       = {StepFun Team},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/SHSLab/Step-5-Preview-BF16}},
  note         = {Released September 20, 2026}
}
```

---

## 📜 License

Step-5-Preview is released under the **StepFun Community License**. 
See the [LICENSE](https://huggingface.co/SHSLab/Step-5-Preview-BF16/blob/main/LICENSE) file for full terms.

<div style="border-left: 6px solid #faad14; padding: 16px; border-radius: 8px; margin: 20px 0;">
  <strong>⚠️ Usage Restrictions</strong><br>
  • Commercial use is permitted under the StepFun Community License.<br>
  • Redistribution must include the license and attribution.<br>
  • See LICENSE for full details.
</div>

---

## 📬 Contact

- **Hugging Face:** [SHSLab](https://huggingface.co/SHSLab)
- **GitHub:** [github.com/stepfun-ai](https://github.com/stepfun-ai)
- **Discord:** [Join our Discord](https://discord.gg/stepfun)
- **Email:** [opensource@stepfun.com](mailto:opensource@stepfun.com)
- **Website:** [stepfun.com](https://stepfun.com)

---

<div align="center">
  <strong>⭐ If you find Step-5-Preview useful, please give us a star on GitHub and Hugging Face! ⭐</strong>
  <br><br>
  <em>Built with ❤️ by StepFun</em>
</div>