---
license: llama2
base_model: lmsys/longchat-7b-v1.5-32k
tags:
  - vllm
  - kv-cache-compression
  - low-rank
  - long-context
  - text-generation
pipeline_tag: text-generation
library_name: transformers
---

# longchat-7b-starkv-proxima

STAR-KV checkpoint: [`lmsys/longchat-7b-v1.5-32k`](https://huggingface.co/lmsys/longchat-7b-v1.5-32k) with attention K/V weights calibrated + fused into a low-rank form for KV-cache compression, produced by [proxima_vllm](https://github.com/) (STAR-KV method).

## What's different from the base model

Standard LongChat weights, plus a `star_kv/` directory: `weights.safetensors` (fused low-rank K/V projections) and `star_kv_config.json`. `config.json` architecture set to `ProximaStarKVLlamaForCausalLM`. Directory loads directly as vLLM `model=` argument via `additional_config.proxima_vllm.checkpoint`.

## Calibration recipe

- Base model: `lmsys/longchat-7b-v1.5-32k`
- Method: activation collection on K/V projections + distillation into low-rank basis, `max_rank_k=64`, `max_rank_v=64`, `sharpness=10.0`, mixed-precision fusion (`top_fraction=0.25`, `top_bits=4`, `bottom_bits=3`)
- Calibration data: real text, `HuggingFaceFW/fineweb-edu` (CC-MAIN-2013-20 shard)
- Active profile: `balanced` (`gamma=1e-3`); `aggressive` (`gamma=5e-3`) and `conservative` (`gamma=2e-4`) profiles also independently calibrated and included in checkpoint
- Samples: 64 sequences x 256 tokens, batch size 2, 1 epoch

**Deviation from paper (disclosed):** paper calibrates on 3000 FineWeb-Edu samples x 2048 tokens (~6 GPU-hours, 8B model). This checkpoint uses a scaled-down real-(not-synthetic)-text run (~1/50th token volume) for single-session feasibility on one L4 GPU. Results should be read with that caveat — this is not a paper-scale calibration.

## Benchmark results (measured, this checkpoint)

End-to-end vs plain vLLM (post-prefill-kernel-fix, 2026-08-06):
- 4096 ctx: 1.65x slower / 177.3 tok/s, 3.11x more KV blocks used
- 32768 ctx: 178.6 tok/s (STAR-KV-only run, requires `gpu_memory_utilization=0.83`, not 0.85)

Kernel-level:
- Decode: 1.42-1.46x vs FlashAttention-2 at batch>=16
- Prefill: 0.5-0.86x of FlashAttention-2, but 1.2-1.4x over this repo's own decode kernel

## Usage

```python
from vllm import LLM

llm = LLM(
    model="tenosra/longchat-7b-starkv-proxima",
    additional_config={"proxima_vllm": {"checkpoint": True}},
)
```

## Intended use / limitations

Research checkpoint for benchmarking KV-cache compression at long context. Calibration sample count is far below the paper's protocol (see deviation note above) — treat compression/quality tradeoff numbers as directional, not final.