HGRN-1.3B, 80% semi-structured pruned (EHWS)

This is fla-hub/hgrn-1.3B-100B pruned to 80% weight sparsity with EHWS (Extreme Hierarchical Weight Sparsity), a two-phase ADMM method that trains directly against the true next-token loss under a hard sparsity constraint rather than matching each layer's dense output on calibration data. It extends ELSA (The Unseen Frontier: Pushing the Limits of LLM Sparsity with Surrogate-Free ADMM), splitting ELSA's single global optimization into a cheap per-layer warm start (Phase 1) followed by one global fine-tune (Phase 2).

"Semi-structured" here means every layer is pruned to the same 80% ratio; within a layer, which individual weights survive is still unstructured.

Results

	WikiText2 PPL	C4 PPL	Zero-shot avg (7-task)
Dense	11.84	16.89	0.435
This checkpoint (80% sparsity)	88.23	56.79	0.350

Achieved sparsity: 0.7999. Zero-shot accuracy is the standard 0-shot 7-task average (ARC-Easy/Challenge, BoolQ, HellaSwag, OpenBookQA, RTE, Winogrande) via lm-evaluation-harness.

For reference, HGRN-1.3B isn't one of the models ELSA's own paper covers, so we ran ELSA's official code ourselves on this exact model to get a same-model baseline: 54.08 / 36.92 WikiText2/C4 -- notably better, but under a different setup: unstructured (not semi-structured) sparsity, pure cross-entropy training with no KD term. This checkpoint keeps a CE + knowledge-distillation objective (against the original dense model) active throughout training, which is a real difference in training signal, not just an unlucky hyperparameter gap.

Training setup

Two-phase ADMM: Phase 1 warm-starts each layer sequentially up to 70% sparsity, then Phase 2 optimizes all layers jointly under one shared forward pass up to the final 80% target (128 rounds x 32 steps). Objective is 0.5 * CE + 0.5 * KD against the original dense model, both on unlabeled text. Calibration/training data is 2048 sequences x 2048 tokens from C4.

Key Phase 2 hyperparameters: lr=6e-4, ADMM penalty lambda=0.01 (constant schedule), adam_beta2=0.95, alpha_kd=0.5, bf16. These were tuned by directly auditing ELSA's own reference implementation run against this same model -- notably, the ADMM lambda used in earlier attempts had been about 200x too small for this model, and correcting it alone cut WikiText2 PPL by roughly 40%.

Usage

This architecture needs the flash-linear-attention package (registers HGRNForCausalLM with transformers' Auto* classes) and trust_remote_code=True:

import fla  # noqa: registers HGRN with transformers' Auto* classes
from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "ikimyaii/HGRN-1.3B-semi-structured-EHWS-80pct",
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
tokenizer = AutoTokenizer.from_pretrained("fla-hub/hgrn-1.3B-100B")

Note that "semi-structured" here refers to the sparsity pattern being uniform across layers, not to a hardware-friendly block/N:M pattern -- this checkpoint won't show a runtime speedup without a sparse-aware kernel. It's released for quality evaluation, not as a deployment-ready accelerated checkpoint.