splickz/qwen3.8-27b-rk4v4-e8-sm120

rk4v4-e8 — 4-bit E8 lattice KV cache for Qwen3.8-27B on RTX 5090

A packed 4-bit KV cache using an E8 Conway–Sloane lattice codec, ported to SM120 / RTX 5090. It roughly doubles the usable context of Qwen3.8-27B on a single 32 GB card at identical decode speed.

Validated: 649,802 tokens with 5/5 needle retrieval on one RTX 5090.

This card describes an inference engine configuration, not a new set of model weights. The weights are stock Qwen3.8-27B in NVFP4; what changes is how the KV cache is stored during inference.

---

Summary

The KV cache, not the weights, is what limits context length. At 8-bit it costs ~35.2 KB per token, so a 32 GB card holding 19 GB of weights exhausts memory near 330K tokens.

rk4v4-e8 stores K and V as packed 4-bit codes — two per byte, halving each plane's width. Codes come from projecting rotated 8-dimensional sub-vectors onto the E8 lattice, the densest known 8-dimensional sphere packing, which yields lower quantization error at a given bit budget than rounding each scalar independently.

Measured cost: 17,979.7 bytes/token, 51% of int8.

---

Hardware and software

GPU	NVIDIA GeForce RTX 5090, 32,607 MiB
Compute capability	SM120 (sm_120a)
Model	Qwen3.8-27B, NVFP4, weights 18.98 GiB
Engine	NInfer, fork branch e8-sm120 (not present upstream)
Codec origin	sergiuszm/ninfer-4090 (sm_89), commit 5df406b0

The engine builds only for sm_120a and refuses other architectures.

---

Results

configuration	max validated context	retrieval	KV bytes/token	VRAM
int8	329,804	5/5	35,219.6	31,776 MiB
rk4v4-e8 factor 2	499,828	5/5	17,979.7	29,833 MiB
rk4v4-e8 factor 4	600,078	5/5	17,979.7	31,848 MiB
rk4v4-e8 factor 4, no graphs	649,802	5/5	17,979.7	32,025 MiB

1.97× the validated context of int8.

Engine-reported KV allocation: 8.98 GiB at 524,288 tokens; 11.17 GiB at 660,032.

Performance

Matched: max_context=98304, 89,880-token prompt, 300 generated, greedy, prefix reuse disabled, CUDA graphs on, 3 repeats, only --kv-dtype differing.

	int8	rk4v4-e8
decode tok/s	59.68	60.03
prefill tok/s	4,908.3	3,934.0

Decode is at parity. Prefill costs ~20%.

Throughput against context

context	prefill tok/s	wall clock	decode tok/s
254,817	1,794	2.4 min	61.7
499,828	995	8.4 min	39.9
600,078	839	11.9 min	33.4
649,802	611	17.7 min	29.4

---

Validation methodology

A configuration was never counted as validated because the server started.

Needle retrieval. Five needles at 10/25/50/75/90% depth, question asked from the very end, so every answer requires a genuine long-range lookup — a needle at the end of a 600K prompt would be a 5K test in a 600K costume. Each needle scored individually; finish=stop required, since a truncated answer is not a pass.

No cache contamination. Prefix reuse disabled for all benchmarks, prompt heads varied per repeat. An early performance run was discarded when ttft=0.13s revealed prefill was being skipped.

Generation sanity against a control. Factual recall, exact arithmetic, code generation, and physics, with an int8 control through the identical harness. The control mattered: an initial run showed empty responses that resembled a codec defect and were thinking-tokens exhausting the token budget — int8 behaved identically.

Memory validated three ways. Reservation arithmetic, the engine's own reported allocation, and nvidia-smi peak, required to agree — they do, within 2.3%. Passing the reservation check alone was explicitly not accepted as evidence of allocation.

Packed-path assertion. Allocator geometry is asserted rather than assumed: packed_k=1, packed_v=1, e8_lattice=1, extents 256 → 128, dtype U8. An earlier defect allocated full-width int8 planes while every mode flag appeared correct; that configuration would have passed retrieval perfectly while measuring nothing.

---

Recommended configurations

Daily driver

ninfer-serve model.ninfer \
  --kv-dtype rk4v4-e8 --rope-yarn-factor 2 --rope-original-max-position 262144 \
  --max-context 262144

Validated as written: 254,817 tokens, 5/5, decode 61.7 tok/s, prefill 1,794 tok/s, 6.8 GB headroom.

Large context

ninfer-serve model.ninfer \
  --kv-dtype rk4v4-e8 --rope-yarn-factor 4 --rope-original-max-position 262144 \
  --max-context 610000

600,078 tokens, 5/5, graphs on, ~12 min prefill.

Fails to start roughly one attempt in three (measured 2 of 3): CUDA graph preparation exceeds its fixed 12 MiB allowance. Not a memory problem. Retry, or add --no-cuda-graph for deterministic startup.

Edge

ninfer-serve model.ninfer \
  --kv-dtype rk4v4-e8 --rope-yarn-factor 4 --rope-original-max-position 262144 \
  --max-context 660000 --no-cuda-graph

649,802 tokens, 5/5. 582 MiB spare, ~18 min prefill. Fragile; not for routine use.

---

Limitations

Hardware. RTX 5090 / sm_120a only. The engine refuses to build for other architectures. Results do not transfer to other cards without revalidation.

Prefill is the real ceiling. 2.4 min at 254K, 17.7 min at 649K, and superlinear. Long contexts are correct but not interactive.

Prefill throughput costs ~20% against int8. Decode does not.

CUDA graph allowance is erratic. A fixed 12 MiB budget with non-monotonic demand: int8 fails at 300,000 but starts at 320/330/340K; rk4 fails at 550,000 and 650,000 but starts at 600,000 and 650,048. At --max-context 610000 this was measured at 2 successful starts in 3 attempts. A failure of this kind is not a memory limit — shift the context or pass --no-cuda-graph. The 649,802 result required graphs off.

Quality evidence is retrieval and generation sanity, not a benchmark suite. Needle retrieval at 5/5 and correct arithmetic/code/physics are meaningful but narrow. No MMLU-class evaluation was run at long context. Quantized KV can degrade in ways needle retrieval does not probe.

Headroom at the top is thin. 649,802 leaves 582 MiB and 62 MiB of planner slack.

rk2v4-e8 is not available. It refuses at startup by name. Its exact E8 root codec is not ported; it would allocate quarter-width K planes no shipped kernel can decode.

Model scope. Validated only for Qwen3.8-27B NVFP4. Other checkpoints are untested.

---

Why rk2v4-e8 was not shipped

rk2v4-e8 reduces K to 2 bits — PROJECTED 13,883.6 bytes/token and ~841K context. Declined:

1. rk4v4-e8 is memory-limited, not quality-limited: 5/5 at every tier from 32K to 649,802.
2. Prefill degrades superlinearly; rk2's extra ~190K tokens land where one request costs 25–30 min.
3. More KV compression buys memory, and memory is no longer binding. More context makes the actual constraint — prefill time — worse.
4. It would cost a 733-line codec port and an unmeasured quality risk from 2-bit K.

---

Citation and credits

• Neroued/ninfer — the inference engine.
• sergiuszm/ninfer-4090 — the rk4v4 packed KV path and E8 lattice codec for sm_89.
• Qwen3.8-27B — Alibaba Qwen team.

This work contributes the SM120 port and validation, the YaRN long-context path, and a fix to a shared-memory overrun in the decode kernels that capped context for all KV modes, including int8 and bf16.