README.md

---
language:
- en
- zh
pipeline_tag: text-generation
library_name: transformers
tags:
- xllm
- k2-horizon
- causal-lm
- reasoning
- knowledge-distillation
- vllm
- reinforcement-learning
license: other
license_name: internal-only
license_link: LICENSE
model-index:
- name: K2-Horizon-0.9B
  results:
  - task:
      type: text-generation
      name: Math Reasoning
    dataset:
      name: AIME 2026
      type: aime26
    metrics:
    - type: accuracy
      name: avg@16
      value: 0.485
  - task:
      type: text-generation
      name: Math Reasoning
    dataset:
      name: AIME 2025
      type: aime25
    metrics:
    - type: accuracy
      name: avg@16
      value: 0.417
  - task:
      type: text-generation
      name: Code Generation
    dataset:
      name: HumanEval+
      type: humaneval_plus
    metrics:
    - type: pass@1
      value: 0.799
  - task:
      type: text-generation
      name: Code Generation
    dataset:
      name: MBPP+
      type: mbpp_plus
    metrics:
    - type: pass@1
      value: 0.680
  - task:
      type: text-generation
      name: Code Generation
    dataset:
      name: LiveCodeBench v6
      type: livecodebench_v6
    metrics:
    - type: pass@1
      value: 0.3741
  - task:
      type: text-generation
      name: General Knowledge / Reasoning
    dataset:
      name: GPQA-Diamond
      type: gpqa_diamond
    metrics:
    - type: accuracy
      name: avg@16
      value: 0.273
  - task:
      type: text-generation
      name: General Knowledge / Reasoning
    dataset:
      name: MMLU-Pro
      type: mmlu_pro
    metrics:
    - type: accuracy
      value: 0.455
  - task:
      type: text-generation
      name: General Knowledge / Reasoning
    dataset:
      name: BBH
      type: bbh
    metrics:
    - type: accuracy
      name: zero-shot CoT
      value: 0.660
  - task:
      type: text-generation
      name: Tool Use
    dataset:
      name: BFCL v4
      type: bfcl_v4
    metrics:
    - type: accuracy
      name: overall
      value: 0.280
  - task:
      type: text-generation
      name: Instruction Following
    dataset:
      name: IFEval
      type: ifeval
    metrics:
    - type: accuracy
      name: strict-instruction
      value: 0.808
  - task:
      type: text-generation
      name: Math Reasoning
    dataset:
      name: HMMT February 2026
      type: hmmt_feb_2026
    metrics:
    - type: accuracy
      name: avg@16
      value: 0.258
---

# K2-Horizon-0.9B

> [!Note]
> This repository contains the weights and configuration for K2-Horizon-0.9B in the Hugging Face Transformers format. It supports reasoning-mode chat through the included templates and loads with Transformers, vLLM, and SGLang.

K2-Horizon-0.9B is the compact dense member of the K2-Horizon family: a 0.9B-class decoder-only model with a 128K context window. It is trained with multi-teacher on-policy distillation for mathematics, coding, instruction following, and STEM tasks.

<p align="center">
  <img src="k2-horizon-0.9b-benchmarks.png" alt="K2-Horizon-0.9B benchmark results" width="100%">
</p>

## K2-Horizon-0.9B Highlights

- **Compact reasoning model.** A 0.9B-class dense model designed for efficient mathematics, coding, instruction-following, and STEM workloads.
- **Straightforward serving.** Dense architecture with no expert routing; a single CUDA GPU is sufficient for inference.
- **128K context.** Supports up to 131,072 tokens with YaRN RoPE scaling.
- **Multi-teacher distillation.** Trained with domain teachers for math and code, STEM, and instruction following.
- **Open research release.** Weights, configuration, custom architecture code, tokenizer files, and evaluation details are included.


## Benchmark Results

| Benchmark | **K2-Horizon-0.9B** | Qwen3.5-0.8B | OpenBMB-1B | Qwen3.5-2B |
| --- | ---: | ---: | ---: | ---: |
| AIME 2025 | **41.7** | 1.04 | 40.42 | 34.17 |
| AIME 2026 | **48.5** | 0.21 | 40.42 | 38.75 |
| HMMT Feb 2026 | **25.8** | 0.57 | 23.3 | 22.73 |
| GPQA Diamond | **27.3** | 11.9 | 26.26 | 54.89 |
| HumanEval+ | **79.9** | 16.5 | 65.2 | 75.6 |
| MBPP+ | **68.0** | 35.4 | 60.6 | 67.7 |
| LiveCodeBench v6 | **37.41** | 6.64 | 33.52 | 29.79 |
| BFCL v4 | **28.0** | 25.3 | 25.15 | 43.6 |


## Model Series Overview

K2-Horizon spans compact and larger dense models for transparent foundation-model research, staged checkpoint analysis, and practical deployment. The repositories share a common release structure while preserving model-specific training and serving guidance.

| Model | Architecture | Parameters | Context length | Intended use |
| --- | --- | ---: | ---: | --- |
| **K2-Horizon-0.9B** | Dense decoder-only | 1.08B including embeddings | 131,072 | Compact reasoning, local inference, and distillation research |
| K2-Horizon-3.7B | Dense decoder-only | 3.78B core | 524,288 | Efficient research, evaluation, and single-node serving |
| K2-Horizon-7B | Dense decoder-only | 7B core | 524,288 | General research, fine-tuning, and cost-conscious deployment |
| K2-Horizon-32B | Dense decoder-only | 32B core | 524,288 | Stronger long-context and reasoning experiments |

## This Repository

| Property | Value |
|---|---|
| **Architecture** | `K2HorizonForCausalLM` (`model_type: k2_horizon`) |
| **Parameters** | 1,078,285,824 (released as 0.9B; counted from the safetensors tensors) |
| **Hidden size / layers** | 1,536 / 28 |
| **Attention heads / KV heads** | 32 / 8 |
| **Context length** | 131,072 tokens with YaRN RoPE scaling; original context length 8,192 tokens |
| **Vocabulary size** | 64,256 |
| **Released weight dtype** | BF16 |
| **Format** | Hugging Face safetensors, one weight shard, with custom configuration and modeling code in the repository root |
| **Distillation checkpoint** | Step 249 of a 500-step mOPD run |

The `main` revision publishes `K2HorizonForCausalLM`,
`model_type: k2_horizon`, and matching `configuration_k2_horizon.py` and
`modeling_k2_horizon.py` modules. The Transformers and vLLM preflights below
validate that public contract before loading weights.

<!-- ## Checkpoint Revisions

| Revision | Stage | Max context |
| --- | --- | ---: |
| `main` | Distilled release checkpoint, step 249 | 128K |
| `mid2_47k` | Second context-extension stage | 128K |
| `mid1_75k` | First context-extension stage | 40K |
| Base model | Pre-distillation specialist merge | 8K |
 -->
## Training Provenance

Training examples were routed to a math-and-code teacher, a STEM teacher, or
an instruction-following teacher through the example's `opd_domain` metadata.
The math-and-code teacher was used as the fallback when no recognized domain
was present.

| Domain | Teacher checkpoint step |
|---|---:|
| Math and code | 2,739 |
| STEM | 499 |
| Instruction following | 1,499 |

The base context window was extended in stages from 8,192 to 40,960 and then
to 131,072 tokens. The `mid1_75k` and `mid2_47k` repository revisions preserve
the corresponding intermediate checkpoints. The distilled release checkpoint
is on `main`; all stages use a vocabulary of 64,256 tokens.

## Training Loss

K2-Horizon-0.9B begins with a task-arithmetic merge of three specialist
checkpoints. mOPD then trains that merged student against math-and-code, STEM,
and instruction-following teachers at the same time. The training objective
combines an on-policy distillation loss with a reference-model KL term so the
student can learn specialist behavior while remaining close to the merged base
model.

The resulting checkpoint retains most of the specialist teachers' performance
on the reported math and coding tasks. It also improves every reported IFEval
submetric over the pre-distillation merge. This makes the model useful for
research on compact reasoning models, local inference, distillation, and
task-specific adaptation.

## Deployment Guide

K2-Horizon-0.9B emits a reasoning segment before its final answer when the
chat template is used. The template supports `reasoning_effort` values `high`,
`medium`, and `low`, which select the model's full, fast, and faster reasoning
modes respectively.

For a deterministic runtime check, use `temperature=0` and generate 20 to 50
tokens. For general sampled generation, `temperature=0.6` and `top_p=0.95`
reproduce the GPQA evaluation setting and are reasonable starting points.
IFBench used `temperature=0.8`. Long math and coding tasks may require several
thousand output tokens; choose limits from application measurements rather
than treating an evaluation limit as a universal default.

### vLLM

The validated serving image was reconstructed into the following manual
runtime contract:

| Component | Validated value |
|---|---|
| Operating system | Ubuntu 24.04, Linux x86-64 |
| Python | 3.12.13 |
| CUDA toolkit | 12.9 |
| PyTorch | 2.13.0+cu129 |
| Transformers | 5.16.1 |
| Safetensors | 0.8.0 |
| FlashInfer | 0.6.17 |
| Attention backend | vLLM FlashAttention 3; Triton 3.7.1 |
| vLLM | `0.26.1rc1.dev1212`, [PR #53806](https://github.com/vllm-project/vllm/pull/53806) source commit [`d9fd5f11`](https://github.com/vllm-project/vllm/commit/d9fd5f11423a1a5628fe29e7296ceb9de91aac3c) |

That source revision contains the native `K2HorizonForCausalLM` implementation
and the built-in `k2_horizon` reasoning and tool parsers. Other vLLM revisions
have not been validated for this checkpoint. Pin the exact commit until the
integration is available in an upstream release.

Use Linux x86-64 with a CUDA 12.9-compatible NVIDIA driver and Git. The setup
uses vLLM's precompiled extension path while keeping the Python package on the
exact reviewed source commit.

<details>
<summary>Show the pinned vLLM environment setup</summary>

```bash
git clone --filter=blob:none --no-checkout \
  https://github.com/vllm-project/vllm.git
cd vllm
git fetch origin pull/53806/head:refs/remotes/origin/pr-53806
git checkout --detach d9fd5f11423a1a5628fe29e7296ceb9de91aac3c
test "$(git rev-parse HEAD)" = \
  "d9fd5f11423a1a5628fe29e7296ceb9de91aac3c"

python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip uv
export UV_LINK_MODE=copy
VLLM_USE_PRECOMPILED=1 uv pip install --upgrade --editable . \
  --torch-backend=auto
uv pip install "transformers==5.16.1" "safetensors==0.8.0"
python -m pip check

python - <<'PY'
from vllm import ModelRegistry
from vllm.reasoning import ReasoningParserManager
from vllm.tool_parsers import ToolParserManager

assert "K2HorizonForCausalLM" in ModelRegistry.get_supported_archs()
assert ReasoningParserManager.get_reasoning_parser("k2_horizon") is not None
assert ToolParserManager.get_tool_parser("k2_horizon") is not None
PY
```

</details>

Download the repository chat template explicitly and start the server with one
GPU. The 8,192-token profile below is a conservative starting point. Increase
`MAX_MODEL_LEN` only after measuring KV-cache capacity; the checkpoint supports
up to 131,072 tokens.

<details>
<summary>Show the vLLM serving command</summary>

```bash
source .venv/bin/activate

export MODEL_ID="IFM/K2-Horizon-0.9B"
export MODEL_REVISION="main"
export MAX_MODEL_LEN=8192
export CHAT_TEMPLATE="$(hf download "$MODEL_ID" chat_template.jinja \
  --revision "$MODEL_REVISION")"

vllm serve "$MODEL_ID" \
  --revision "$MODEL_REVISION" \
  --model-impl vllm \
  --trust-remote-code \
  --dtype bfloat16 \
  --tensor-parallel-size 1 \
  --max-model-len "$MAX_MODEL_LEN" \
  --max-num-seqs 1 \
  --gpu-memory-utilization 0.85 \
  --served-model-name "$MODEL_ID" \
  --chat-template "$CHAT_TEMPLATE" \
  --reasoning-parser k2_horizon \
  --tool-call-parser k2_horizon \
  --enable-auto-tool-choice
```

</details>

The reasoning parser moves `<ifm|think>`, `<ifm|think_fast>`, or
`<ifm|think_faster>` text into the OpenAI-compatible response's
`reasoning_content` field. The tool parser converts generated
`<ifm|tool_call>` blocks into structured tool calls when tools are supplied in
the request.

<details>
<summary>Show an OpenAI-compatible request</summary>

```python
from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
    model="IFM/K2-Horizon-0.9B",
    messages=[{"role": "user", "content": "What is the square root of 2?"}],
    max_tokens=50,
    temperature=0,
    extra_body={"chat_template_kwargs": {"reasoning_effort": "high"}},
)
message = response.choices[0].message
print(getattr(message, "reasoning_content", None))
print(message.content)
```

</details>

### SGLang

Native K2 Horizon support is provided by
[sgl-project/sglang#37654](https://github.com/sgl-project/sglang/pull/37654).
Use a `lmsysorg/sglang:dev` image built after that PR is merged. Once support is
included in a tagged SGLang release, use the corresponding versioned image.

<details>
<summary>Show the SGLang serving command</summary>

```bash
docker run --gpus all \
  --shm-size 32g \
  -p 30000:30000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --ipc=host \
  lmsysorg/sglang:dev \
  python3 -m sglang.launch_server \
    --model-path "IFM/K2-Horizon-0.9B" \
    --revision main \
    --tp 1 \
    --dtype bfloat16 \
    --context-length 8192 \
    --attention-backend fa3 \
    --reasoning-parser k2_horizon \
    --tool-call-parser k2_horizon \
    --mem-fraction-static 0.85 \
    --host 0.0.0.0 \
    --port 30000
```

</details>

This uses SGLang's native `K2HorizonForCausalLM` implementation; no
`--trust-remote-code`, source patch, or external parser plugin is required. The
8,192-token limit is a conservative starting point; increase it only after
measuring KV-cache capacity.

### Transformers

The checkpoint can also be loaded directly from the Hugging Face repository.
Its `configuration_k2_horizon.py` and `modeling_k2_horizon.py` files are loaded
through `trust_remote_code=True`. Use a clean environment so the direct path
does not inherit vLLM's build dependencies. Transformers 4.57.x is not
compatible with this remote configuration class; use the validated 5.14.1
version below. Transformers may print nonfatal `cache_position` documentation
diagnostics while loading the remote code, but BF16 loading and generation
complete normally.

<details>
<summary>Show the Transformers environment setup</summary>

```bash
python3.12 -m venv .venv-transformers
source .venv-transformers/bin/activate
python -m pip install --upgrade pip
python -m pip install "torch==2.11.0" \
  --index-url https://download.pytorch.org/whl/cu128
python -m pip install \
  "transformers==5.14.1" \
  "safetensors==0.8.0"
python -m pip check
```

</details>

This deterministic sample loads the released weights as BF16 on one CUDA GPU
and generates only 50 tokens, making it suitable as an end-to-end smoke test.

<details>
<summary>Show the Transformers inference example</summary>

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

MODEL_ID = "IFM/K2-Horizon-0.9B"
REVISION = "main"

tokenizer = AutoTokenizer.from_pretrained(
    MODEL_ID,
    revision=REVISION,
    trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
    MODEL_ID,
    revision=REVISION,
    dtype=torch.bfloat16,
    trust_remote_code=True,
).to("cuda").eval()

messages = [{"role": "user", "content": "What is the square root of 2?"}]
inputs = tokenizer.apply_chat_template(
    messages,
    add_generation_prompt=True,
    reasoning_effort="high",
    return_dict=True,
    return_tensors="pt",
)
inputs = {name: value.to(model.device) for name, value in inputs.items()}
inputs.pop("token_type_ids", None)

with torch.inference_mode():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=False,
        pad_token_id=tokenizer.pad_token_id,
    )

new_tokens = outputs[0, inputs["input_ids"].shape[-1]:]
print(tokenizer.decode(new_tokens, skip_special_tokens=True))
```

</details>

## Hardware Planning

The BF16 weights occupy approximately 2.0 GiB. One CUDA GPU is sufficient for
short-context inference; 8 GiB is a practical minimum for a one-request smoke
test, while 16 GiB or more provides useful room for longer prompts and runtime
workspaces. The BF16 KV cache is approximately 0.44 GiB per request at 8,192
tokens and approximately 7 GiB at 131,072 tokens, before allocator, activation,
CUDA-graph, and framework overhead. Start with a short context and one sequence,
then increase context length and concurrency from measured memory headroom.

## Source Code

The model repository stores the checkpoint directly at its root. It includes:

- `model.safetensors` and `model.safetensors.index.json`
- `config.json`, `generation_config.json`, and the K2 architecture code
- tokenizer files and three chat-template variants
- `README.md` and `LICENSE`

Use `chat_template.jinja` for ordinary chat and OpenAI-compatible serving.
`chat_template_generation.jinja` and `chat_template_asst_tool_gen.jinja` are
specialized generation and assistant-tool-generation variants.

## Evaluation

- **Training:** multi-teacher on-policy distillation with OPD loss weight 0.1,
  reference-KL weight 0.01, learning rate `1e-7`, and a 500-step schedule. The
  selected checkpoint is step 249.
- **Evaluation:** the AIME, coding, and instruction-following evaluations used
  the training evaluation path with SGLang as the rollout engine. GPQA-Diamond
  was evaluated with Eval360-V2 at revision `f5081bf`.
- **Export:** the distributed training checkpoint was converted to Hugging Face
  safetensors and checked for tensor parity; all 255 expected weight tensors
  matched. The architecture label was later updated from `XllmForCausalLM` to
  `K2HorizonForCausalLM` without changing the weights.

## Intended Use

K2-Horizon-0.9B is intended for compact reasoning research, local inference, evaluation dry runs, distillation studies, and task-specific adaptation. It is a research release and should be evaluated on representative prompts before deployment.

## Limitations and Safety

- AIME 2025 remains below the math-and-code teacher, and AIME results have high
  sampling uncertainty because each benchmark contains only 30 problems.
- GPQA-Diamond is statistically close to the pre-distillation base result, so
  the reported run does not demonstrate a clear STEM improvement.
- Long-horizon tool use remains substantially weaker than single-turn tool
  calling in the BFCL v4 breakdown.
- Benchmark scores depend on prompt templates, reasoning effort, sampling
  parameters, framework versions, and evaluation harness details. Validate the
  model on representative prompts before deployment.
- As with other language models, K2-Horizon-0.9B can produce inaccurate,
  biased, or unsafe text. Applications should use task-specific evaluation,
  input and output controls, monitoring, and human review where appropriate.

## Citation

```bibtex
@misc{k2horizon2026,
  title  = {Introducing K2 Horizon: Frontier Performance, Radically Open},
  author = {{IFM Team}},
  year   = {2026},
  url    = {https://ifm.ai/blog/k2/},
}
```

## Contact

For questions and feedback, open a discussion in this Hugging Face repository or contact the IFM team through the K2-Horizon project channels.