hoborific/Lilitu-L3.3-70b-0.1-W4A16-AutoRound

4-bit quant using Intel AutoRound

This is the default W4A16 scheme.

Quantization Details

at time of quantization, the default implied values not listed in the below json are as follows:

{"batch_size": 8, "iters": 200, "seqlen": 2048, "nsamples": 128, "lr": None}

quantization_config.json

{
  "bits": 4,
  "group_size": 128,
  "sym": true,
  "data_type": "int",
  "low_gpu_mem_usage": true,
  "autoround_version": "0.9.2",
  "quant_method": "auto-round",
  "packing_format": "auto_round:auto_gptq"
}