# Model Reference — Qwen3.8-Flash-Next via llama.cpp

ZEVORA is configured to use the **Qwen3.8-Flash-Next** family for its on-device
`ZEVORA Local AI` layer. The weights are **not** bundled in this repository;
ZEVORA references the upstream Hugging Face model instead.

- **Upstream repository:** `unsloth/Qwen3.8-Flash-Next-GGUF`
- **Recommended quantization:** `UD-Q4_K_XL`
- **Runtime:** llama.cpp (via `llama-cpp-python`)

## How ZEVORA obtains the model

ZEVORA **references** the upstream model. It does **not** download the whole
repository and does **not** redistribute model weights. You choose a single
quantization package and point `LOCAL_MODEL_PATH` at the downloaded GGUF file.

## Downloading the model

Get one quantization from the upstream repository:

```bash
git lfs install
# example: download a single GGUF file of your chosen quantization
huggingface-cli download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-Q4_K_XL/*.gguf" --local-dir models
```

> Always follow the upstream model card for the exact file layout and license.

## llama.cpp usage

llama.cpp serves an OpenAI-compatible endpoint. ZEVORA can talk to a local
llama.cpp server through its `openai-compatible` local runtime:

```bash
llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL
```

The OpenAI-compatible endpoint is then available at:

```text
http://127.0.0.1:8080/v1
```

In ZEVORA, set:

```dotenv
LOCAL_MODEL_RUNTIME=openai-compatible
LOCAL_ENDPOINT_URL=http://127.0.0.1:8080/v1
```

Alternatively, ZEVORA's **`llamacpp` runtime** loads the GGUF directly in-process
via `llama-cpp-python` — no external server needed. This is the default and
simplest mode:

```dotenv
LOCAL_MODEL_RUNTIME=llamacpp
LOCAL_MODEL_PATH=models/zevora-4b-thinking.gguf
```

## Choosing a quantization

There is no single "best" quantization. Choose based on your hardware:

| Hardware profile | Suggested starting point |
|------------------|--------------------------|
| Low RAM / CPU-only / constrained device | smaller quantization (e.g. `Q4_K_M`-class) |
| More RAM / VRAM available | `UD-Q4_K_XL`-class for higher quality |
| GPU VRAM | offload more layers with `LOCAL_MODEL_GPU_LAYERS` |

Tune the following in `.env` (all values are configurable):

- `LOCAL_MODEL_QUANT` — preferred quantization tag
- `LOCAL_MODEL_PATH` — path to your chosen GGUF
- `LOCAL_MODEL_CONTEXT_LENGTH` — context window
- `LOCAL_MODEL_MAX_TOKENS` — max output tokens
- `LOCAL_MODEL_THREADS` — CPU threads (0 = auto)
- `LOCAL_MODEL_GPU_LAYERS` — GPU offload layers
- `LOCAL_MODEL_BATCH_SIZE` — prompt-processing batch size
- `LOCAL_MODEL_TEMPERATURE` — sampling temperature

## Model registry metadata

ZEVORA's existing registry records the local model reference (not weights):

```json
{
  "provider": "local",
  "model_family": "Qwen3.8-Flash-Next",
  "source": "huggingface",
  "repository": "unsloth/Qwen3.8-Flash-Next-GGUF",
  "runtime": "llama.cpp",
  "recommended_quantization": "UD-Q4_K_XL"
}
```

## Verification

ZEVORA verifies the local GGUF with **SHA-256** before loading when an
authoritative reference exists (a `<model>.sha256` sidecar or the managed-package
manifest). A mismatched digest aborts loading.