metadata
base_model: ornith-ai/Ornith-1.5-9B
base_model_relation: quantized
license: mit
pipeline_tag: image-text-to-text
tags:
- compressed-tensors
- fp8
- quantized
- vllm
Ornith-1.5-9B - FP8
ornith-ai/Ornith-1.5-9B quantized to FP8 (8-bit weights).
What this is
Near-lossless, no calibration data, and it halves every Linear weight. The safe default when you care about quality and have Ada/Hopper or newer.
Caveat. Needs compute capability >= 8.9 (Ada/Hopper+) to run fast.
Details
| Source | ornith-ai/Ornith-1.5-9B |
| Scheme | FP8 (8-bit) |
| Format | compressed-tensors |
| Parameters | 9.4B |
| Size on disk | 11.9 GB |
| Compression | 1.58x smaller than the 18.8 GB source |
| Left unquantized | lm_head, re:.*visual.*, re:.*vision_tower.*, re:.*vision_model.*, re:.*vision.*, re:.*multi_modal_projector.*, re:.*merger.* |
| Quantized on | A40 |
| Quantized by | Sohailhosseini |
Usage
vllm serve Sohailhosseini/Ornith-1.5-9B-FP8 \
--max-model-len 32768
from vllm import LLM, SamplingParams
if __name__ == "__main__":
llm = LLM("Sohailhosseini/Ornith-1.5-9B-FP8", max_model_len=32768)
out = llm.chat(
[{"role": "user", "content": "What is quantization? Answer in one sentence."}],
SamplingParams(temperature=0.6, max_tokens=512),
)
print(out[0].outputs[0].text)
Provenance
Produced with HF-quantized. recipe.yaml in this repo is the exact modifier stack that was applied, and the scheme, ignored layers and hardware are in the table above.
Licence is inherited from the source model. Quantization does not change what you are permitted to do with the weights.