---
license: apache-2.0
base_model:
  - lightonai/LightOnOCR-2-1B
tags:
  - ocr
  - vision-language
  - gguf
---

# LightOnOCR-2-1B GGUF (Q4_K_S)

GGUF quantized version of [lightonai/LightOnOCR-2-1B](https://huggingface.co/lightonai/LightOnOCR-2-1B).

## Files

- **LightOnOCR-2-1B-Q4_K_S.gguf** (366 MB) - Language model (596M parameters, Q4_K_S quantization)
- **mmproj-LightOnOCR-2-1B-Q8_0.gguf** (429 MB) - Vision encoder (403M parameters, Q8_0 quantization)

## Usage

```bash
llama-server -hf staghado/LightOnOCR-2-1B-Q4_K_S-GGUF -c 8192 --temp 0.2 --top-k 0 --top-p 0.9
```

**Note:** The flags `--temp 0.2 --top-k 0 --top-p 0.9` set the default generation parameters to match the original model.

### API Example

```python
import requests
import base64

with open('document.png', 'rb') as f:
    image_base64 = base64.b64encode(f.read()).decode()

response = requests.post('http://localhost:8000/v1/chat/completions', json={
    "model": "LightOnOCR-2-1B",
    "messages": [{
        "role": "user",
        "content": [
            {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"}}
        ]
    }],
    "max_tokens": 1024,
    "temperature": 0.2,
    "top_k": 0,
    "top_p": 0.9
})

print(response.json()['choices'][0]['message']['content'])
```

Note: This model only accepts images, no text prompts.


## Creating Quantized Versions

If you want to create your own quantized GGUF files:

### Prerequisites
```bash
git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
python -m venv venv
source venv/bin/activate
pip install git+https://github.com/huggingface/transformers.git torch sentencepiece
```

Note: `transformers` must be installed from source until the next release includes LightOnOCR support.

### Conversion Steps

1. **Download original model**
```bash
hf download lightonai/LightOnOCR-2-1B --repo-type model --local-dir ./models/LightOnOCR-2-1B
```

2. **Convert language model to BF16 GGUF**
```bash
python convert_hf_to_gguf.py ./models/LightOnOCR-2-1B --outtype bf16 --outfile LightOnOCR-2-1B-bf16.gguf
```

3. **Quantize language model to Q4_K_S**
```bash
llama-quantize LightOnOCR-2-1B-bf16.gguf LightOnOCR-2-1B-Q4_K_S.gguf Q4_K_S
```

4. **Convert vision encoder to Q8_0**
```bash
python convert_hf_to_gguf.py ./models/LightOnOCR-2-1B --mmproj --outtype q8_0 --outfile mmproj-LightOnOCR-2-1B-Q8_0.gguf
```

### Notes
- Q4_K_S provides maximum compression (~8x) with minimal quality loss
- Vision encoder uses Q8_0 due to patch embedding dimension constraints
- Requires latest llama.cpp from main branch

## Details

- Total: 1.01B parameters (vision: 403M + language: 596M + projector: 6M)
- Quantization: Q4_K_S (language), Q8_0 (vision)
- Tested on M3 Mac: 413 tokens/sec (prompt), 114 tokens/sec (generation)