Instructions to use urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ

SGLang

How to use urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ with Docker Model Runner:
```
docker model run hf.co/urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ
```

K2-Horizon-MoVA-36B-A4B — W4A16 (AutoRound GPTQ)

4-bit weight-only (W4A16) AutoRound GPTQ quantization of IFM/K2-Horizon-MoVA-36B-A4B, a 36B-total / ~4B-active-parameter Mixture-of-Experts model with MoVA (Mixture-of-Values Attention) and native 512K-token context.

This is an unofficial, community-produced quantization. All credit for the base model goes to the IFM Team — see their announcement for details and citation.

vLLM Support Notice: Serving this model in vLLM requires vLLM PR #56637. Upstream vLLM currently fails to load quantized MoVA value-experts (AttributeError: 'ColumnParallelLinear' object has no attribute 'weight'). PR #56637 merges value experts into v_experts_fused to enable quantized loading, proper Tensor Parallelism, and XPU WNA16 INT4 kernel support. Unlike previous workarounds that forced --fp_layers "v_experts,v_router" (leaving ~36 experts in BF16 and inflating model size to ~30GB), this checkpoint fully quantizes the MoVA value-experts into 4-bit weights.

Model Details


Base model	IFM/K2-Horizon-MoVA-36B-A4B
Architecture	`K2HorizonForCausalLM` (custom code, `trust_remote_code=True` required)
Hidden layers	48 (layers 0–2 dense, layers 3–47 MoE)
Attention	GQA, 32 query heads / 8 KV heads, head_dim 128, MoVA value-expert routing (64 value experts, top-4)
FFN experts	100 routed experts (top-8) + 1 shared expert per MoE layer
Hidden size / intermediate size	2560 / 6144 (routed expert intermediate size 768)
Context length	524,288 tokens (512K), RoPE θ = 10,000,000
Vocab size	250,624

Quantization Details

Quantized with Intel AutoRound v0.16.0 on Intel Arc Pro B70 (XPU) hardware.

Setting	Value
Scheme	W4A16 (4-bit weights, 16-bit activations)
Format	GPTQ (`auto_gptq`-compatible)
Group size	64
Symmetric	Yes
Iterations	600
Calibration dataset	`opencode-instruct` (code-focused calibration)
`lm_head`	Not quantized
Router (`mlp.gate`) on MoE layers	Not quantized (100-way router output isn't evenly divisible by the group size)
MoVA `v_experts`	Fully quantized in INT4 (requires PR #56637)

Reproduce with:

auto-round \
  --model IFM/K2-Horizon-MoVA-36B-A4B \
  --scheme W4A16 \
  --group_size 64 \
  --iters 600 \
  --dataset "opencode-instruct" \
  --format auto_gptq \
  --low_gpu_mem_usage \
  --output_dir <output_dir>

Benchmarks

Measured on 2x Intel Arc Pro B70 (Pipeline Parallelism PP=2, TP=1, FP8 KV cache, XPU graph capture FULL_DECODE_ONLY):

model	test	t/s (total)	t/s (req)	peak t/s	peak t/s (req)	ttfr (ms)	est_ppt (ms)	e2e_ttft (ms)
urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ	pp4096 @ d1 (c1)	6428.35 ± 21.15	6428.35 ± 21.15			936.41 ± 2.11	639.52 ± 2.11	936.41 ± 2.11
urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ	tg256 @ d1 (c1)	38.65 ± 0.08	38.65 ± 0.08	39.67 ± 0.47	39.67 ± 0.47

Prefill throughput: ~6,428 tok/s at 4K context (TTFT ~936 ms).
Decode throughput: ~38.7 tok/s (peak ~39.7 tok/s).

Usage

This model uses custom modeling code, so trust_remote_code=True is required with transformers. It targets agentic, reasoning, scientific-coding and tool-use workloads — the base model's recommended sampling settings carry over unchanged:

reasoning_effort="high"
temperature=1.0, top_p=0.95
Enable the k2_horizon parser for reasoning/tool-call formatting

vLLM

Requires vLLM PR #56637.

Standard Serving

vllm serve urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ \
  --trust-remote-code \
  --quantization gptq \
  --reasoning-parser k2_horizon \
  --enable-auto-tool-choice \
  --tool-call-parser k2_horizon

Intel XPU Serving (2x Arc Pro B70 Example)

docker run --rm -it \
  --cap-add SYS_PTRACE --security-opt seccomp=unconfined \
  --device /dev/dri \
  -v /dev/dri/by-path:/dev/dri/by-path:ro \
  -v /sys/class/drm:/sys/class/drm:ro \
  --group-add "$(getent group render | cut -d: -f3)" \
  --group-add "$(getent group video | cut -d: -f3)" \
  --ipc=host --pid=host --net=host --shm-size=16g \
  -e VLLM_USE_V2_MODEL_RUNNER=1 \
  -e CCL_ZE_IPC_EXCHANGE=sockets \
  -e CCL_ATL_TRANSPORT=ofi \
  -e CCL_TOPO_FABRIC_VERTEX_CONNECTION_CHECK=0 \
  -e VLLM_WORKER_MULTIPROC_METHOD=spawn \
  -e ZE_FLAT_HIERARCHY=FLAT \
  -e ONEAPI_DEVICE_SELECTOR=level_zero:* \
  -e VLLM_XPU_ENABLE_XPU_GRAPH=1 \
  -e HF_HUB_ENABLE_HF_TRANSFER=0 \
  -e HF_HUB_OFFLINE=1 \
  -v ~/.cache/vllm:/root/.cache/vllm \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm-xpu-env \
  urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ \
  --served-model-name urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ \
  --revision main \
  --host 0.0.0.0 --port 8000 \
  --tensor-parallel-size 1 --pipeline-parallel-size 2 \
  --max-model-len 330k --kv-cache-dtype fp8 --max-num-seqs 2 \
  --reasoning-parser k2_horizon --enable-auto-tool-choice --tool-call-parser k2_horizon \
  --language-model-only --trust-remote-code \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.97 \
  --compilation-config '{"mode":0,"cudagraph_mode":"FULL_DECODE_ONLY","cudagraph_capture_sizes":[1,2,4,8]}'

Transformers

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ"
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id, trust_remote_code=True, device_map="auto"
)

Limitations

Requires trust_remote_code=True (custom architecture, inherited from the base model).
vLLM serving requires PR #56637.

License

Apache 2.0, inherited from the base model.

Downloads last month: 79

Safetensors

Model size

6B params

Tensor type

I32

BF16

F16

Model tree for urakozz/IFM-K2-Horizon-MoVA-36B-A4B-W4A16-AutoRound-GPTQ

Base model

IFM/K2-Horizon-MoVA-36B-A4B

Quantized

(24)

this model