Instructions to use ikimyaii/llama-7b-sparsegpt-60pct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ikimyaii/llama-7b-sparsegpt-60pct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ikimyaii/llama-7b-sparsegpt-60pct")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ikimyaii/llama-7b-sparsegpt-60pct")
model = AutoModelForCausalLM.from_pretrained("ikimyaii/llama-7b-sparsegpt-60pct", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ikimyaii/llama-7b-sparsegpt-60pct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ikimyaii/llama-7b-sparsegpt-60pct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ikimyaii/llama-7b-sparsegpt-60pct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/ikimyaii/llama-7b-sparsegpt-60pct

SGLang

How to use ikimyaii/llama-7b-sparsegpt-60pct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ikimyaii/llama-7b-sparsegpt-60pct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ikimyaii/llama-7b-sparsegpt-60pct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ikimyaii/llama-7b-sparsegpt-60pct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ikimyaii/llama-7b-sparsegpt-60pct",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use ikimyaii/llama-7b-sparsegpt-60pct with Docker Model Runner:
```
docker model run hf.co/ikimyaii/llama-7b-sparsegpt-60pct
```

LLaMA-7B · SparseGPT 60% sparse

OpenLLaMA-7B pruned one-shot to 60% weight sparsity with SparseGPT, with the same fraction removed from every output row (semi-structured). It is a baseline in the CWS comparison.

Model details


Base model	`openlm-research/open_llama_7b`
Method	SparseGPT
Target / achieved sparsity	60% / 59.99%
Weight correction	OBS (second-order)
Retraining	none

What is pruned: every nn.Linear in the 32 decoder layers (attention q/k/v/o and MLP gate/up/down) plus lm_head, 225 layers in total (see pruning_summary.json). Embeddings and norms stay dense.

Method

SparseGPT (Frantar & Alistarh, 2023, arXiv:2301.00774) scores each weight by its OBS saliency W[i,j]² / [H⁻¹]_jj and applies column-ordered Optimal Brain Surgeon corrections to the remaining weights in blocks of 128 columns. Hessian H = (2/N) XᵀX from calibration activations, damping 0.01.

Evaluation

WikiText-2 PPL of all five methods at 60% (dense 8.64):

Method	WikiText-2 PPL ↓
CWS	18.58
SparseGPT (this model)	20.76
Wanda	18.11
RIA	18.54
AWP	14.69

These are the project's results at the time the checkpoints were uploaded (2026-05-11; results/sparsity_sweep_llama.json @ 6e144ae), with dense PPL 8.64. The repository README now reports these models under a 2048-token protocol (dense 6.61); the ordering between methods is similar.

Usage

import torch
from transformers import AutoModelForCausalLM, LlamaTokenizer

model_id = "ikimyaii/llama-7b-sparsegpt-60pct"
tokenizer = LlamaTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.float16, device_map="auto")

lin = model.model.layers[0].mlp.down_proj
print(f"sparsity: {(lin.weight == 0).float().mean():.3f}")

Limitations

These are research checkpoints for studying pruning. They are not instruction-tuned or safety-tuned, and they inherit the base model's biases.
Weights are stored densely, with zeros at the pruned positions. Sparsity alone gives no speedup without a sparse kernel or event-driven hardware.

Related repositories

Sparse-LLMs-on-Loihi: pruning scripts, baselines and raw results for this checkpoint.
CWS: reference implementation of the CWS method these baselines are compared against.
Weight-Sparsity-Using-Real-Loss and EHWS: follow-up work that optimises the real next-token loss instead of layer-wise reconstruction.

Downloads last month: 253

Safetensors

Model size

7B params

Tensor type

BF16

Model tree for ikimyaii/llama-7b-sparsegpt-60pct

Base model

openlm-research/open_llama_7b

Finetuned

(30)

this model

Datasets used to train ikimyaii/llama-7b-sparsegpt-60pct

Paper for ikimyaii/llama-7b-sparsegpt-60pct

SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot

Paper • 2301.00774 • Published Jan 2, 2023 • 4