Add generation parameters to server command (--temp 0.2 --top-k 0 --top-p 0.9)
Browse files
README.md
CHANGED
|
@@ -20,9 +20,11 @@ GGUF quantized version of [lightonai/LightOnOCR-2-1B](https://huggingface.co/lig
|
|
| 20 |
## Usage
|
| 21 |
|
| 22 |
```bash
|
| 23 |
-
llama-server -hf staghado/LightOnOCR-2-1B-Q4_K_S-GGUF -c 8192
|
| 24 |
```
|
| 25 |
|
|
|
|
|
|
|
| 26 |
### API Example
|
| 27 |
|
| 28 |
```python
|
|
@@ -40,7 +42,10 @@ response = requests.post('http://localhost:8000/v1/chat/completions', json={
|
|
| 40 |
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"}}
|
| 41 |
]
|
| 42 |
}],
|
| 43 |
-
"max_tokens": 1024
|
|
|
|
|
|
|
|
|
|
| 44 |
})
|
| 45 |
|
| 46 |
print(response.json()['choices'][0]['message']['content'])
|
|
|
|
| 20 |
## Usage
|
| 21 |
|
| 22 |
```bash
|
| 23 |
+
llama-server -hf staghado/LightOnOCR-2-1B-Q4_K_S-GGUF -c 8192 --temp 0.2 --top-k 0 --top-p 0.9
|
| 24 |
```
|
| 25 |
|
| 26 |
+
**Note:** The flags `--temp 0.2 --top-k 0 --top-p 0.9` set the default generation parameters to match the original model.
|
| 27 |
+
|
| 28 |
### API Example
|
| 29 |
|
| 30 |
```python
|
|
|
|
| 42 |
{"type": "image_url", "image_url": {"url": f"data:image/png;base64,{image_base64}"}}
|
| 43 |
]
|
| 44 |
}],
|
| 45 |
+
"max_tokens": 1024,
|
| 46 |
+
"temperature": 0.2,
|
| 47 |
+
"top_k": 0,
|
| 48 |
+
"top_p": 0.9
|
| 49 |
})
|
| 50 |
|
| 51 |
print(response.json()['choices'][0]['message']['content'])
|