---
base_model: GestaltLabs/Ornstein-3.5-9B-V2
base_model_relation: quantized
language:
- en
license: apache-2.0
library_name: gguf
tags:
- gguf
- llama.cpp
- quantized
- reasoning
- ai-research
- qwen3.5
- multimodal
- mtmd
pipeline_tag: text-generation
---

![Ornstein 3.5 9B — V2](ornstein3.5-9bv2.png)

# Ornstein 3.5 9B — V2 · GGUF

GGUF quantizations of [GestaltLabs/Ornstein-3.5-9B-V2](https://huggingface.co/GestaltLabs/Ornstein-3.5-9B-V2) — the reinforcement-learning post-training (V2) of Ornstein 3.5 9B. Each quant has a sibling `.sha256` checksum, and a separate **vision projector** (`mmproj`) ships the multimodal tower for image/video input.

## Files

| File | Notes |
|---|---|
| `ornstein-v2-Q4_K_M.gguf` | **recommended default** |
| `ornstein-v2-Q5_K_M.gguf` | high quality |
| `ornstein-v2-Q6_K.gguf` | very high quality |
| `ornstein-v2-Q8_0.gguf` | near-lossless |
| `ornstein-v2-f16.gguf` | full F16 |
| `mmproj-ornstein-v2-f16.gguf` | vision encoder — pair with any quant for image/video input |

## Usage (llama.cpp)

Text:

```bash
llama-cli -m ornstein-v2-Q4_K_M.gguf -p "Derive the variance of a sum of two correlated random variables."
```

Multimodal (image/video) — add the vision projector:

```bash
llama-mtmd-cli -m ornstein-v2-Q4_K_M.gguf \
  --mmproj mmproj-ornstein-v2-f16.gguf \
  --image picture.jpg -p "Describe this image."
```

Quality and speed scale with quant size; **Q4_K_M** is a strong default for ~8 GB of VRAM/RAM.

## Speculative decoding (MTP)

Every quant embeds the model's native multi-token-prediction (MTP) draft head (GGUF `block_count` 33), so self-speculative decoding runs from a single file — no separate draft model needed:

```bash
llama-cli -m ornstein-v2-Q4_K_M.gguf --spec-type draft-mtp \
  -p "Write a Python function is_prime(n)."
```

## Support This Work

I'm a PhD student in visual neuroscience at the University of Toronto who also happens to spend way too much time fine-tuning, merging, and quantizing open-weight models on rented H100s and a local DGX Spark. All training compute is self-funded — balancing GPU costs against a student budget. If my uploads have been useful to you, consider buying a PhD student a coffee. It goes a long way toward keeping these experiments running.

**[Support on Ko-fi](https://ko-fi.com/djlougen)**

## License

Apache 2.0 — inherited from the Qwen 3.5 9B base release.