Instructions to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="huawei-csl/Qwen3-1.7B-PreSINQ-GGUF")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("huawei-csl/Qwen3-1.7B-PreSINQ-GGUF", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

llama.cpp

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S
# Run inference directly in the terminal:
llama cli -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S
# Run inference directly in the terminal:
llama cli -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S
# Run inference directly in the terminal:
./llama-cli -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S
# Run inference directly in the terminal:
./build/bin/llama-cli -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Use Docker

docker model run hf.co/huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

LM Studio
Jan

vLLM

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

SGLang

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Ollama
How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with Ollama:
```
ollama run hf.co/huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S
```
Unsloth Desktop

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with Docker Model Runner:
```
docker model run hf.co/huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S
```

Lemonade

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Run and chat with the model

lemonade run user.Qwen3-1.7B-PreSINQ-GGUF-Q3_K_S

List all available models

lemonade list

Hermes Agent

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use huawei-csl/Qwen3-1.7B-PreSINQ-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "huawei-csl/Qwen3-1.7B-PreSINQ-GGUF:Q3_K_S" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

chiaraboretti commited on Feb 9

Commit

b7bbd3b

verified ·

1 Parent(s): 702702a

Update README.md

Browse files

Files changed (1) hide show

README.md +56 -83

README.md CHANGED Viewed

@@ -9,7 +9,6 @@ arxiv: 2509.22944
 tags:
 - quantized
 - sinq
-- int3
 - efficient-inference
 - qwen
 - llm
@@ -24,107 +23,82 @@ base_model_relation: quantized
 <p align="center">🐙 <a href="https://github.com/huawei-csl/SINQ">Github</a>&nbsp;&nbsp; | &nbsp;&nbsp;📄 <a href="http://arxiv.org/abs/2509.22944">Paper</a></p>
-# SINQ 3-bit Quantized Qwen3-1.7B model
-This repository contains the official **3-bit quantized** version of the [`Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B) model using the **SINQ (Sinkhorn-Normalized Quantization)** method.
-SINQ is a novel, fast and high-quality quantization method designed to make any Large Language Models smaller while keeping their accuracy almost intact.
-To support the project please put a star ⭐ in the official [SINQ](https://github.com/huawei-csl/SINQ) github repository.
 ## Model Details
-- **Model Name:** `Qwen3-1.7B-3bit-SINQ `
 - **Base Model:** [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B)
 - **Task:** Text Generation
 - **Framework:** PyTorch / Transformers
 - **License:** [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
-- **Quantized By:** *Huawei - Computing Systems Lab*
-## Quantization Details
-- **Quantization Method:**  SINQ (Sinkhorn-Normalized Quantization)
-- **Precision:** INT3
-- **Group Size:**  64
-- **Framework:**  PyTorch
-- **Quantization Library:**  `sinq`
 ---
 # 🚀 Usage
-## Prerequisite
-Before running the quantization script, make sure the **SINQ** library is installed.
-Installation instructions and setup details are available in the [SINQ official github repository](https://github.com/huawei-csl/SINQ).
-## Usage example
-You can load and use the model with our wrapper based on the 🤗 Transformers library:
-```python
-import torch
-from transformers import AutoTokenizer
-from sinq.patch_model import AutoSINQHFModel
-model_name = "huawei-csl/Qwen3-1.7B-3bit-SINQ"
-tokenizer = AutoTokenizer.from_pretrained(model_name)
-sinq_model = AutoSINQHFModel.from_quantized_safetensors(
-    model_name,
-    device="cuda:0",
-    compute_dtype=torch.bfloat16
-)
-prompt = "Explain neural network quantization in one sentence."
-inputs = tokenizer(prompt, return_tensors="pt").to("cuda:0")
-with torch.inference_mode():
-    out_ids = sinq_model.generate(**inputs, max_new_tokens=32, do_sample=False)
-print(tokenizer.decode(out_ids[0], skip_special_tokens=True))
-```
-<details>
-<summary><span style="font-size:1.1em; font-weight:bold;">🧩 Quantization Process</span></summary>
-The quantized model was obtained using the **SINQ** quantization library, following the steps below:
-```python
-import torch
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from sinq.patch_model import AutoSINQHFModel
-from sinq.sinqlinear import BaseQuantizeConfig
-# Load base model
-base_model_name = "Qwen/Qwen3-1.7B"
-model = AutoModelForCausalLM.from_pretrained(base_model_name, torch_dtype="float16")
-tokenizer = AutoTokenizer.from_pretrained(base_model_name)
-# Apply 3-bit SINQ quantization
-quant_cfg = BaseQuantizeConfig(
-    nbits=3,            # quantization bit-width
-    group_size=64,     # group size
-    tiling_mode="1D",   # tiling strategy
-    method="sinq"       # quantization method ("asinq" for the calibrated version)
-)
-qmodel = AutoSINQHFModel.quantize_model(
-    model,
-    tokenizer=tokenizer,
-    quant_config=quant_cfg,
-    compute_dtype=torch.bfloat16,
-    device="cuda:0"
-)
-```
-> **Reproducibility Note**: This model was quantized using the SINQ implementation from commit [`14ad847`](https://github.com/huawei-csl/SINQ/commit/14ad847d0ab25f1794b8820506f59b5c9c1fc979) of the [SINQ](https://github.com/huawei-csl/SINQ) repository.
-</details>
-</br>
 ---
 # 🧾 How to Cite This Work
-If you find **SINQ** useful in your research or applications, please
-- Put a star ⭐ in the official [SINQ](https://github.com/huawei-csl/SINQ) github repository.
 - Cite our <a href="http://arxiv.org/abs/2509.22944" target="_blank"><strong>paper</strong></a>:
 ```bibtex
@@ -136,5 +110,4 @@ If you find **SINQ** useful in your research or applications, please
       archivePrefix={arXiv},
       primaryClass={cs.LG},
       url={http://arxiv.org/abs/2509.22944}
-}
-```

 tags:
 - quantized
 - sinq
 - efficient-inference
 - qwen
 - llm
 <p align="center">🐙 <a href="https://github.com/huawei-csl/SINQ">Github</a>&nbsp;&nbsp; | &nbsp;&nbsp;📄 <a href="http://arxiv.org/abs/2509.22944">Paper</a></p>
+# PreSINQ GGUF Quantized Qwen3-1.7B Model
+This repository contains the official PreSINQ **GGUF-quantized** versions of the [`Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B) model. For a detailed explanation of PreSINQ strategy please refer to the the official [SINQ](https://github.com/huawei-csl/SINQ) repository.
+SINQ is a fast and high-quality quantization technique designed to significantly reduce Large Language Model size while preserving accuracy.
+If you find this project useful, please consider giving a ⭐ to the official [SINQ](https://github.com/huawei-csl/SINQ) repository.
+---
 ## Model Details
+- **Model Name:** `Qwen3-1.7B-PreSINQ-GGUF`
 - **Base Model:** [`Qwen/Qwen3-1.7B`](https://huggingface.co/Qwen/Qwen3-1.7B)
 - **Task:** Text Generation
 - **Framework:** PyTorch / Transformers
 - **License:** [Apache-2.0](https://www.apache.org/licenses/LICENSE-2.0)
+- **Quantized By:** *Huawei – Computing Systems Lab*
+---
+# How to Obtain the PreSINQ Model
+The PreSINQ Qwen3-1.7B models are produced using the **PreSINQ GGUF script** available in the official [SINQ](https://github.com/huawei-csl/SINQ) repository.
+The models provided here correspond to the best-performing configurations for each quantization type.
+## 📊 Best PreSINQ Quantization Results (Qwen3-1.7B)
+Results below are measured on the **WikiText-2 test set**.
+| Method | Bits | Size (GB) | Perplexity ↓ |
+|----------|--------|------------|----------------|
+| Baseline (FP16) | FP16 | 1.41 | 17.1294 |
+| Baseline + Q4_K_S | 4-bit | 0.45 | 19.5454 |
+| **PreSINQ + Q4_K_S** | 4-bit | 0.37 | **17.4544** |
+| Baseline + Q3_K_S | 3-bit | 0.37 | 24.0242 |
+| **PreSINQ + Q3_K_S** | 3-bit | 0.31 | **18.8032** |
+However, you can generate good PreSINQ models (not the best one) faster by reducing the number of configurations explored during the PreSINQ script execution.
+The table below shows perplexity for different PreSINQ parameter configurations using **Q4_K_S quantization**.
+Evaluation is performed on a 5k-line subset of the **Pile validation dataset**.
+| Group Size | Iterations | Repetitions | Perplexity |
+|-------------|-------------|-------------|-------------|
+| 32 | 2 | 1 | 11.7196 |
+| 32 | 4 | 1 | 11.7238 |
+| 32 | 8 | 1 | **11.6885** |
+| 32 | 16 | 1 | 11.6909 |
+| 64 | 2 | 1 | 11.7421 |
+| 64 | 4 | 1 | 11.7240 |
+| 64 | 8 | 1 | 11.6975 |
+| 64 | 16 | 1 | 11.7001 |
+| 128 | 2 | 1 | 11.7129 |
+| 128 | 4 | 1 | 11.7118 |
+| 128 | 8 | 1 | 11.7149 |
+| 128 | 16 | 1 | 11.7208 |
 ---
 # 🚀 Usage
+## Usage Example
+You can load and run the PreSINQ GGUF models using:
+- 🤗 Transformers
+- llama.cpp
+- Any GGUF-compatible inference framework
 ---
 # 🧾 How to Cite This Work
+If you find **SINQ** useful in your research or applications:
+- Please give a ⭐ to the official [SINQ](https://github.com/huawei-csl/SINQ) repository
 - Cite our <a href="http://arxiv.org/abs/2509.22944" target="_blank"><strong>paper</strong></a>:
 ```bibtex
       archivePrefix={arXiv},
       primaryClass={cs.LG},
       url={http://arxiv.org/abs/2509.22944}
+}