README.md

---
license: gemma
tags:
  - bigsmall
  - compression
  - lossless
  - gemma
  - google
---

# Gemma 2 9B Instruct (BigSmall compressed)

**17.2 GB -> 11.2 GB (BF16). Lossless. Zero inference overhead. Any hardware.**

Compressed with [BigSmall](https://github.com/wpferrell/Bigsmall) -- decompresses once at load time, runs at full native speed. Every weight is bit-identical to the original.

## Quick start

`ash
pip install bigsmall
`

`python
import bigsmall
bigsmall.install_hook()
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("wpferrell/gemma-2-9b-it-bigsmall")
`

## Streaming loader -- run on any hardware

BigSmall's streaming loader decompresses one layer at a time directly into VRAM. Peak memory is one layer -- not the whole model. A 4 GB GPU can run Mistral 7B losslessly.

`python
from bigsmall import StreamingLoader
from transformers import AutoModelForCausalLM

with StreamingLoader("wpferrell/gemma-2-9b-it-bigsmall", device="cuda") as loader:
    model = loader.load_model(AutoModelForCausalLM)
`

| Your GPU | Models you can run |
|----------|--------------------|
| 2 GB | Small models, GPT-2, Gemma 270M |
| 4 GB | Mistral 7B, Llama 3.1 8B, Gemma 2B, Llama 3.2 3B |
| 8 GB | Qwen 2.5 14B, Gemma 2 9B |
| 24 GB | Llama 70B, Qwen 72B, DeepSeek V4-Flash |
| CPU only | Everything -- slower but full quality |

BigSmall is the only lossless compression tool with a streaming loader. DFloat11 and ZipNN load the full model into memory.


## Why BigSmall vs DFloat11

| | BigSmall | DFloat11 |
|--|--|--|
| Inference overhead | **None** | ~2x at batch=1 |
| Hardware | **CPU, Apple Silicon, AMD, any GPU** | CUDA only |
| FP32 support | **Yes** | No |
| Fine-tuning safe | **Yes** | No |
| Streaming loader | **Yes -- peak RAM < 2 GB** | No |

## Compression stats

| Original | Compressed | Ratio | Format | Verified |
|----------|------------|-------|--------|---------|
| 17.2 GB | 11.2 GB | 65.1% | BF16 | md5 every tensor |

- GitHub: [wpferrell/Bigsmall](https://github.com/wpferrell/Bigsmall)
- All models: [huggingface.co/wpferrell](https://huggingface.co/wpferrell)