File size: 3,210 Bytes
4d2ab8d | 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 | # Model Reference — Qwen3.8-Flash-Next via llama.cpp
ZEVORA is configured to use the **Qwen3.8-Flash-Next** family for its on-device
`ZEVORA Local AI` layer. The weights are **not** bundled in this repository;
ZEVORA references the upstream Hugging Face model instead.
- **Upstream repository:** `unsloth/Qwen3.8-Flash-Next-GGUF`
- **Recommended quantization:** `UD-Q4_K_XL`
- **Runtime:** llama.cpp (via `llama-cpp-python`)
## How ZEVORA obtains the model
ZEVORA **references** the upstream model. It does **not** download the whole
repository and does **not** redistribute model weights. You choose a single
quantization package and point `LOCAL_MODEL_PATH` at the downloaded GGUF file.
## Downloading the model
Get one quantization from the upstream repository:
```bash
git lfs install
# example: download a single GGUF file of your chosen quantization
huggingface-cli download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-Q4_K_XL/*.gguf" --local-dir models
```
> Always follow the upstream model card for the exact file layout and license.
## llama.cpp usage
llama.cpp serves an OpenAI-compatible endpoint. ZEVORA can talk to a local
llama.cpp server through its `openai-compatible` local runtime:
```bash
llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
```
The OpenAI-compatible endpoint is then available at:
```text
http://127.0.0.1:8080/v1
```
In ZEVORA, set:
```dotenv
LOCAL_MODEL_RUNTIME=openai-compatible
LOCAL_ENDPOINT_URL=http://127.0.0.1:8080/v1
```
Alternatively, ZEVORA's **`llamacpp` runtime** loads the GGUF directly in-process
via `llama-cpp-python` — no external server needed. This is the default and
simplest mode:
```dotenv
LOCAL_MODEL_RUNTIME=llamacpp
LOCAL_MODEL_PATH=models/zevora-4b-thinking.gguf
```
## Choosing a quantization
There is no single "best" quantization. Choose based on your hardware:
| Hardware profile | Suggested starting point |
|------------------|--------------------------|
| Low RAM / CPU-only / constrained device | smaller quantization (e.g. `Q4_K_M`-class) |
| More RAM / VRAM available | `UD-Q4_K_XL`-class for higher quality |
| GPU VRAM | offload more layers with `LOCAL_MODEL_GPU_LAYERS` |
Tune the following in `.env` (all values are configurable):
- `LOCAL_MODEL_QUANT` — preferred quantization tag
- `LOCAL_MODEL_PATH` — path to your chosen GGUF
- `LOCAL_MODEL_CONTEXT_LENGTH` — context window
- `LOCAL_MODEL_MAX_TOKENS` — max output tokens
- `LOCAL_MODEL_THREADS` — CPU threads (0 = auto)
- `LOCAL_MODEL_GPU_LAYERS` — GPU offload layers
- `LOCAL_MODEL_BATCH_SIZE` — prompt-processing batch size
- `LOCAL_MODEL_TEMPERATURE` — sampling temperature
## Model registry metadata
ZEVORA's existing registry records the local model reference (not weights):
```json
{
"provider": "local",
"model_family": "Qwen3.8-Flash-Next",
"source": "huggingface",
"repository": "unsloth/Qwen3.8-Flash-Next-GGUF",
"runtime": "llama.cpp",
"recommended_quantization": "UD-Q4_K_XL"
}
```
## Verification
ZEVORA verifies the local GGUF with **SHA-256** before loading when an
authoritative reference exists (a `<model>.sha256` sidecar or the managed-package
manifest). A mismatched digest aborts loading.
|