Model Reference — Qwen3.8-Flash-Next via llama.cpp

ZEVORA is configured to use the Qwen3.8-Flash-Next family for its on-device ZEVORA Local AI layer. The weights are not bundled in this repository; ZEVORA references the upstream Hugging Face model instead.

Upstream repository: unsloth/Qwen3.8-Flash-Next-GGUF
Recommended quantization: UD-Q4_K_XL
Runtime: llama.cpp (via llama-cpp-python)

How ZEVORA obtains the model

ZEVORA references the upstream model. It does not download the whole repository and does not redistribute model weights. You choose a single quantization package and point LOCAL_MODEL_PATH at the downloaded GGUF file.

Downloading the model

Get one quantization from the upstream repository:

git lfs install
# example: download a single GGUF file of your chosen quantization
huggingface-cli download unsloth/Qwen3.8-Flash-Next-GGUF --include "UD-Q4_K_XL/*.gguf" --local-dir models

Always follow the upstream model card for the exact file layout and license.

llama.cpp usage

llama.cpp serves an OpenAI-compatible endpoint. ZEVORA can talk to a local llama.cpp server through its openai-compatible local runtime:

llama serve -hf unsloth/Qwen3.8-Flash-Next-GGUF:UD-Q4_K_XL

The OpenAI-compatible endpoint is then available at:

http://127.0.0.1:8080/v1

In ZEVORA, set:

LOCAL_MODEL_RUNTIME=openai-compatible
LOCAL_ENDPOINT_URL=http://127.0.0.1:8080/v1

Alternatively, ZEVORA's llamacpp runtime loads the GGUF directly in-process via llama-cpp-python — no external server needed. This is the default and simplest mode:

LOCAL_MODEL_RUNTIME=llamacpp
LOCAL_MODEL_PATH=models/zevora-4b-thinking.gguf

Choosing a quantization

There is no single "best" quantization. Choose based on your hardware:

Hardware profile	Suggested starting point
Low RAM / CPU-only / constrained device	smaller quantization (e.g. `Q4_K_M`-class)
More RAM / VRAM available	`UD-Q4_K_XL`-class for higher quality
GPU VRAM	offload more layers with `LOCAL_MODEL_GPU_LAYERS`

Tune the following in .env (all values are configurable):

LOCAL_MODEL_QUANT — preferred quantization tag
LOCAL_MODEL_PATH — path to your chosen GGUF
LOCAL_MODEL_CONTEXT_LENGTH — context window
LOCAL_MODEL_MAX_TOKENS — max output tokens
LOCAL_MODEL_THREADS — CPU threads (0 = auto)
LOCAL_MODEL_GPU_LAYERS — GPU offload layers
LOCAL_MODEL_BATCH_SIZE — prompt-processing batch size
LOCAL_MODEL_TEMPERATURE — sampling temperature

Model registry metadata

ZEVORA's existing registry records the local model reference (not weights):

{
  "provider": "local",
  "model_family": "Qwen3.8-Flash-Next",
  "source": "huggingface",
  "repository": "unsloth/Qwen3.8-Flash-Next-GGUF",
  "runtime": "llama.cpp",
  "recommended_quantization": "UD-Q4_K_XL"
}

Verification

ZEVORA verifies the local GGUF with SHA-256 before loading when an authoritative reference exists (a <model>.sha256 sidecar or the managed-package manifest). A mismatched digest aborts loading.