Add 4-bit RTN quantized model

2c36cb3 verified 8 months ago

926 Bytes

license: apache-2.0
base_model: Qwen/Qwen3-4B-Thinking-2507
tags:
  - quantized
  - rtn
  - 4-bit
  - thinking
  - reasoning

Qwen__Qwen3-4B-Thinking-2507_RTN_w4g128

This is a 4-bit RTN (Round-To-Nearest) quantized version of Qwen/Qwen3-4B-Thinking-2507.

Quantization Details

Method: RTN (Round-To-Nearest)
Bits: 4-bit
Group Size: 128
Base Model: Qwen/Qwen3-4B-Thinking-2507

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "quantpa/Qwen__Qwen3-4B-Thinking-2507_RTN_w4g128"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

# Use the model for inference

Model Details

Quantization: RTN 4-bit
Original Model: Qwen/Qwen3-4B-Thinking-2507
Quantized by: quantpa