Instructions to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0
# Run inference directly in the terminal:
llama cli -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0
# Run inference directly in the terminal:
./llama-cli -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0
# Run inference directly in the terminal:
./build/bin/llama-cli -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Use Docker

docker model run hf.co/fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

LM Studio
Jan

vLLM

How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Ollama
How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with Ollama:
```
ollama run hf.co/fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0
```
Unsloth Desktop

How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with Docker Model Runner:
```
docker model run hf.co/fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0
```

Lemonade

How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Run and chat with the model

lemonade run user.Qwen3-4B-Instruct-2507-Hi-Fi-GGUF-Q4_0

List all available models

lemonade list

Hermes Agent

How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF:Q4_0" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Qwen3-4B-Instruct-2507 GGUF — fraQtl Hi-Fi (calibration-aware quantization)

Iso-size Q4_K_M-class GGUF of Qwen/Qwen3-4B-Instruct-2507 built with fraQtl calibration-aware per-tensor quantization: byte-matched to the leading community Q4_K_M (within 0.016%), measurably closer to the original model's output distribution on both locked evaluation slices — plus a Q4_0 edition (the NPU-native fixed format) with the calibration lever measured in isolation.

Evaluation — fidelity to the original model

Metric: symmetric top-20 KLD on the support of a Q8_0 teacher quantized from the original pinned bf16 checkpoint (fidelity-to-original; teacher- framing caveat applies). All arms evaluated with the identical binary (llama.cpp 4df29be4), identical rendered prompt template, 3 runs/arm, drift 0. Baselines credited: MaziyarPanahi Q4_K_M (canonical by downloads, ~200k/30d) and unsloth Q4_K_M.

Arm	Bytes	KLD code/math ↓	KLD general ↓	top-1 cm/gen
fraQtl Hi-Fi (this repo)	2,496,879,712 (−0.016%)	0.025317 (−55.9%)	0.033753 (−46.4%)	0.9604 / 0.9357
MaziyarPanahi Q4_K_M (canonical)	2,497,280,448	0.057372	0.063008	0.9466 / 0.9186
unsloth Q4_K_M (second)	2,497,281,120	0.030018	0.042626	0.9557 / 0.9276

Relative deltas vs canonical. The second arm (unsloth) is itself strong — we beat it too on both slices (−15.7% code/math, −20.8% general), stated for completeness.

Long-context retrieval — including the full 262K native context

Needle exact-match, 3 depths × 3 keys per context, greedy, multi-run:

Arm	8K + 32K (×3 runs)	262,144-token native context (×2 runs)
fraQtl Hi-Fi	18/18, 18/18, 18/18	9/9, 9/9
MaziyarPanahi Q4_K_M	18/18 ×3 (parity)	9/9 ×2 (parity)
unsloth Q4_K_M	18/18 ×3 (parity)	8/9 ×2

The 262K row runs the model's full native context window with per-cell receipts in-repo. 8K/32K saturate for all arms — reported as parity, not a differentiator.

Q4_0 edition — the NPU-native format, calibration lever isolated

Q4_0 is the fixed-format quantization used as the universal GGUF asset by on-device runtimes. Three arms, same teacher and slices, 3 runs, drift 0:

Arm	Bytes	KLD code/math ↓	KLD general ↓
fraQtl Q4_0 (this repo, our imatrix)	2,375,771,232	0.059665	0.080458
naive Q4_0 (no imatrix, same source)	2,369,545,024	0.070135	0.088598
GenieX-designated Q4_0 (unsloth's build)	2,375,773,280	0.060996	0.077572

Calibration lever, same format: −14.9% code/math and −9.2% general from calibration alone (ours vs naive, identical Q4_0, identical source).
The GenieX-designated arm is the community GGUF that qualcomm/Qwen3-4B-Instruct-2507's release_assets.json (v0.59.0) designates as the geniex_llamacpp universal asset — built by unsloth, credited: it is well-made. A published accuracy datapoint for that artifact: we are −2.2% on code/math and +3.7% behind on general — both directions stated.
Provenance note: Qualcomm publishes no GGUF of its own for this model — its official builds are chipset-specific w4a16 QNN packages (QAIRT 2.45); the GGUF path is delegated to the community asset above.
K-quant-on-Hexagon NPU acceleration is unverified by us (no device in hand); the Q4_0 artifact is provided as the NPU-native-format option.

Disclosures

Wikitext-2 PPL (ctx 512, sanity metric): ours 10.478, canonical 10.565, unsloth 10.616 — values stated; ctx-512 chunked raw text on an instruction-tuned model is a sanity check, not a capability measure.
The three arms embed different chat templates; all arms were evaluated with the same rendered template, and each arm's native template sha is recorded in the build receipt.
No task-benchmark rows on this card: n=200-class task runs cannot statistically separate arms of this quality (see our Qwen3.8-27B card's per-item audit); KLD and retrieval carry the claim.

Runnable receipts — `receipts_pack/`

Every number above can be re-run with one file on stock llama.cpp — no custom runtime: receipts_pack/ contains the runner (run_receipts.py), slice definitions, the needle-grid spec, and the two teacher logit dumps needed for third-party KLD reproduction.

Provenance

Field	Value
Base model	`Qwen/Qwen3-4B-Instruct-2507` @ `cdbee75f17c01a7cc42f958dc650907174af0554` (Apache-2.0)
Hi-Fi Q4_K_M	sha256 `a54cce471d7824219bf13ca4c227b3408bde8591788a28de314933b7a0cb9529`
Hi-Fi Q4_0	sha256 `0f7d0956111e721fdde684872ca44b355dc70cb219ae8cf0e059871278ac8888`
imatrix	sha256 `21b3655ecad1f09f2bd841bcbb48c9ac00693a0d48e296e1fb5263103d4a6b3a` (included)
Toolchain	llama.cpp @ `4df29be4f4c3673f428170fda944a5b19f743bb8` — conversion, quantization, and every eval
Baselines pinned	MaziyarPanahi `953ba5b5…`; unsloth `3605803b…`; GenieX-designated Q4_0 `e0ba675d…`
Eval runtime	llama-cpp-python 0.3.35, deterministic rig, per-run identity gates

Receipts for every number: receipts/ (7 JSONs: pins, conversion gate, calibration selection, frozen recipe, build + 3-run KLD + PPL + smokes, NIAH grid, Q4_0 arms).

The fraQtl ladder

Tier	Job	Example
Hi-Fi	maximum fidelity at standard 4-bit size	this repo's Q4_K_M
Hi-Fi Phone	fits phone/laptop RAM, fidelity-first	E2B Phone, 2.86 GB
SmartEdge	smallest footprint that survives	E2B IQ3XXS, 2.45 GB

Same discipline at every tier: pinned provenance, measured numbers, losses disclosed.

More from fraQtl

The serving lane — KV-cache compression sidecars for vLLM — holds nine concurrent ≈128K-context users on a single A100 (134.1 tok/s aggregate, 9/9 per-user retrieval checks, receipt 2026-08-14), on this same base model: fraQtl/qwen3-4b-instruct-2507-kv-sidecars. Org page: huggingface.co/fraQtl.

Downloads last month: 96

GGUF

Model size

4B params

Architecture

qwen3

Hardware compatibility

4-bit

Model tree for fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF

Base model

Qwen/Qwen3-4B-Instruct-2507

Quantized

(300)

this model

Collection including fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF

Hi-Fi Quants — calibrated GGUFs with KLD receipts

Collection

Calibration-aware Q4-class GGUFs at identical size to leading community quants, with measured KLD receipts. • 8 items • Updated 13 days ago

Qwen3-4B-Instruct-2507 GGUF — fraQtl Hi-Fi (calibration-aware quantization)

Evaluation — fidelity to the original model

Long-context retrieval — including the full 262K native context

Q4_0 edition — the NPU-native format, calibration lever isolated

Disclosures

Runnable receipts — receipts_pack/

Provenance

The fraQtl ladder

More from fraQtl

Model tree for fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF

Collection including fraQtl/Qwen3-4B-Instruct-2507-Hi-Fi-GGUF

Runnable receipts — `receipts_pack/`