BlivionIaG/Ornith-1.5-9b-exl3-3bpw

Ornith-1.5-9B

This repository contains an EXL3 3-bit quantisation of Ornith-1.5-9B for the RDNA2 / RDNA3 vLLM exl3 path. The visual encoder is not shipped; inference must be launched with --language-model-only. See Quickstart for serving details.

Ornith Highlights

Ornith features the following enhancements:

• Unified Vision-Language Foundation: Early fusion training on multimodal tokens achieves cross-generational parity with prior models and outperforms across reasoning, coding, agents, and visual understanding benchmarks.

• Efficient Hybrid Architecture: Gated Delta Networks combined with sparse Mixture-of-Experts deliver high-throughput inference with minimal latency overhead.

• Multi-Token Prediction: Trained with multi-step speculative decoding for ~2× decode speedup on supported backends.

• Long Context: 262,144 tokens natively, suitable for hour-scale video understanding and long-document tasks.

• Global Linguistic Coverage: Expanded support for 201 languages and dialects.

Model Overview

• Type: Causal Language Model with Vision Encoder
• Training Stage: Pre-training & Post-training
• Language Model
  • Number of Parameters: 9B
  • Hidden Dimension: 4096
  • Token Embedding: 248320 (Padded)
  • Number of Layers: 32
  • Hidden Layout: 8 × (3 × (Gated DeltaNet → FFN) → 1 × (Gated Attention → FFN))
  • Gated DeltaNet:
    • Number of Linear Attention Heads: 16 for V and 16 for QK
    • Head Dimension: 128
  • Gated Attention:
    • Number of Attention Heads: 16 for Q and 4 for KV
    • Head Dimension: 256
    • Rotary Position Embedding Dimension: 64
  • Feed Forward Network:
    • Intermediate Dimension: 12288
  • LM Output: 248320 (NOT tied to token embedding)
  • MTP: trained with multi-steps
• Context Length: 262,144 natively

Quickstart

This checkpoint targets AMD RDNA2 / RDNA3 GPUs (gfx1030, gfx1100, gfx1200) under ROCm. The visual encoder is not shipped — launch with --language-model-only. CDNA (gfx942, gfx950) is not supported.

This quantisation requires the rdna2_extras branch of the BlivionIaG/vllm fork of vLLM. The upstream vllm package does not support the exl3 RDNA path.

Serving with vLLM

source /home/chenco_adm/Apps/vllm/venv-7.14.0/bin/activate

export VLLM_USE_V2_MODEL_RUNNER=1
export VLLM_ROCM_USE_AITER=0
export VLLM_ROCM_USE_AITER_MOE=0
export FLASH_ATTENTION_TRITON_AMD_ENABLE=TRUE
export VLLM_RDNA_FORCE_FP16=1
export TORCH_BLAS_PREFER_HIPBLASLT=0
export PYTORCH_TUNABLEOP_HIPBLASLT_ENABLED=0
export GPU_MAX_HW_QUEUES=2
export VLLM_WORKER_MULTIPROC_METHOD=spawn
export NCCL_P2P_LEVEL=pix
export RCCL_P2P_NET_DISABLE=1
export RCCL_P2P_BATCH_ENABLE=1
export NCCL_PROTO=Simple
export RCCL_MSCCL_ENABLE=0
export VLLM_FORCE_CUSTOM_ALL_REDUCE=1

export VLLM_EXL3_FOLDED_CACHE=$HOME/.cache/exl3_folded_cache
mkdir -p "$VLLM_EXL3_FOLDED_CACHE"

python -m vllm.entrypoints.cli.main serve \
    /path/to/Ornith-1.5-9b-exl3-3bpw \
    --port 8000 \
    --dtype float16 \
    --tensor-parallel-size 4 \
    --max-model-len 16384 \
    --gpu-memory-utilization 0.90 \
    --language-model-only \
    --compilation-config '{"cudagraph_mode": "FULL_AND_PIECEWISE", "compile_ranges_endpoints": []}'

• Multi-Token Prediction (MTP): add --speculative-config '{"method":"qwen3_next_mtp","num_speculative_tokens":2}' to the serve command.

• Throughput Benchmark:

  python -m vllm.entrypoints.cli.main bench throughput \
      /path/to/Ornith-1.5-9b-exl3-3bpw \
      --dtype float16 \
      --tensor-parallel-size 4 \
      --max-model-len 2048 \
      --input-len 1024 \
      --output-len 256 \
      --num-prompts 32 \
      --max-num-seqs 8 \
      --gpu-memory-utilization 0.85 \
      --language-model-only
  

Quantisation details

Quantised with exllamav3 using the 3inst codebook with the following recipe:

convert.py -r \
  -i Ornith-1.5-9B \
  -w work-ornith-15-9b-3inst \
  -o Ornith-1.5-9b-exl3-3bpw \
  -ss 8192 -b 3.0 -hb 6 -mb 4 \
  -cr 256 -cc 2048 -cpi 120 \
  -v -d 0 -cb 3inst --out_scales always

Component	Scheme	Bits	Notes
32 transformer layers (body)	EXL3 3inst	3 bpw	RDNA2 trellis kernel + Hadamard sandwich
lm_head	EXL3 3inst	6 bpw	Dense fp16 dequant at load
MTP (mtp.*)	EXL3 3inst	4 bpw	Loaded only with --speculative-config qwen3_next_mtp
embed_tokens, *_layernorm, *.norm, linear_attn.{A_log,conv1d,dt_bias,in_proj_a,in_proj_b}	bf16 → fp16	16	Not quantised
Visual encoder (*.visual.*)	not shipped	—	Use --language-model-only

Single 5.53 GB model.safetensors (860 tensors).

Best Practices

To achieve optimal performance on RDNA2 / RDNA3:

1. VLLM_USE_V2_MODEL_RUNNER=1 + cudagraph_mode=FULL_AND_PIECEWISE for highest throughput.

2. Force FP16 (VLLM_RDNA_FORCE_FP16=1) — gfx1030 lacks native BF16 hardware.

3. PCIe P2P + custom all-reduce for TP>1 (NCCL_P2P_LEVEL=pix, VLLM_FORCE_CUSTOM_ALL_REDUCE=1).

4. Queue budget: GPU_MAX_HW_QUEUES=2 to keep within gfx1030's 8-HQD limit.

Provenance

• Original: Ornith-1.5-9B
• Quantiser: exllamav3 convert.py with codebook=3inst, calibration rows=256 cols=2048 cpi=120
• vLLM exl3 path: BlivionIaG/vllm rdna2_extras branch
• Decoder kernel: RDNA2 exl3_gemm_rdna2 + exl3_hadamard_128 (Wave32, fp16)

Citation

If you use this quantisation, please cite both the original Ornith model and the RDNA2 exl3 path:

@misc{ornith-1.5-9b-exl3-3bpw,
    title  = {{Ornith-1.5-9B EXL3 3bpw}: RDNA2/RDNA3 quantisation of Ornith-1.5-9B},
    author = {{Ornith Team} and {kletorch}},
    month  = {August},
    year   = {2026},
    url    = {https://huggingface.co/BlivionIaG/Ornith-1.5-9b-exl3-3bpw}
}

@misc{ornith-1.5-9b,
    title  = {{Ornith-1.5-9B}: Towards Native Multimodal Agents},
    author = {{Ornith Team}},
    month  = {February},
    year   = {2026},
    url    = {https://huggingface.co/Ornith-1.5-9B}
}