Instructions to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M
# Run inference directly in the terminal:
llama cli -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M
# Run inference directly in the terminal:
llama cli -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M
# Run inference directly in the terminal:
./llama-cli -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Use Docker

docker model run hf.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

LM Studio
Jan

vLLM

How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Ollama
How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with Ollama:
```
ollama run hf.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M
```
Unsloth Desktop

How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with Docker Model Runner:
```
docker model run hf.co/ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M
```

Lemonade

How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Run and chat with the model

lemonade run user.Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF-IQ1_M

List all available models

lemonade list

Hermes Agent

How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "ISTA-DASLab/Qwen3.8-Flash-Next-GSQ-RCO-Coder-GGUF:IQ1_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Could we have a IQ3_S for coder?

by alexaione - opened 5 days ago

Discussion

alexaione

5 days ago

Really appreciate the high quality work by your team and looking forward to being able to run better coding models on sub 64 or 96GB systems with your new GSQ-RCO versions. Thanks a lot for your hard work and for sharing with all of us.

Have been testing your flash-next IQ3_XXS and the results are quite impressive so far.

Mainly looking at agentic coding and thought this new coder release may be a better choice. But see only the lower Quant is available here. Could you please release a IQ3_S Quant of this coder model.

Thanks again 🙏

xlazom00

5 days ago

It is working fine on my 64GB asus z13 with amd strix halo 30-35tps with MTP

qwased

5 days ago

Seems like it already is.
" The two techniques are complementary and are combined here: 50% of experts are removed, and the remaining weights are quantized to 3.5 bpw."
3.5bpw = IQ3_S

anm2211

IST Austria Distributed Algorithms and Systems Lab org 5 days ago

Yes, @qwased . I should clarify that this model was not fine-tuned to outperform the base model on coding tasks. Rather, it was pruned to create a smaller, more specialized coding-oriented model.

Specifically, we pruned half of the experts from the IQ3_S model we released, with the goal of substantially reducing the model size while retaining as much of its coding and multimodal capabilities as possible. We considered multimodal capability important to preserve because many coding-related tasks can also involve visual understanding.

alexaione

5 days ago

Yes, @qwased . I should clarify that this model was not fine-tuned to outperform the base model on coding tasks. Rather, it was pruned to create a smaller, more specialized coding-oriented model.

Specifically, we pruned half of the experts from the IQ3_S model we released, with the goal of substantially reducing the model size while retaining as much of its coding and multimodal capabilities as possible. We considered multimodal capability important to preserve because many coding-related tasks can also involve visual understanding.

@anm2211 . Sorry, I am a bit confused. the coder model said its IQ1_M
Did you mean to say IQ1_M of coder is already equal to IQ3_S of the main one?

qwased

5 days ago

As I understand it, he means that the quantization level is IQ3S, and because half of the experts were removed, the size is equivalent to IQ1M.

donaldJJ

3 days ago

@alexaione the GGUF headers confirm what qwased said. The Coder file isn't quantized to IQ1_M. It's the IQ3_S recipe with half of the experts removed, and "IQ1_M" is only its size tier:

expert_count: 512 in IQ3_S, 256 in Coder (10 active per token in both)
The per-expert tensor types are unchanged. For example, blk.0 ffn_gate_exps/ffn_up_exps are IQ3_XXS and ffn_down_exps is IQ4_NL in both files, only [2560, 640, 512] becomes [2560, 640, 256].
Routed experts: 50,292,326,400 bytes (46.84 GiB) in IQ3_S, 25,146,163,200 bytes (23.42 GiB) in Coder, exactly half
Shard 1: 51.05 GiB in IQ3_S, 27.58 GiB in Coder
Shard 2 (the 26.82 GiB PLE table) is the same 28,800,138,432-byte file in both

So each remaining expert has the same precision as in IQ3_S. What you give up is expert coverage, not bits. With the PLE table read lazily, the resident part is ~27.6 GiB (plus 0.85 GiB for the BF16 mmproj), which is why it's comfortable on 64 GB machines.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment