Upload folder using huggingface_hub

b55399a verified 16 days ago

6.11 kB

license: mit
base_model:
  - incoai/GLM-5.3-Flash-DFlash2
  - drowzeys/keys-GLM-5.3-Flash-NVFP4-ablit-l15-45-anchorstock
tags:
  - speculative-decoding
  - dflash2
  - glm
  - draft-model
library_name: transformers

GLM-5.3-Flash-DFlash2 — `fc` recalibrated for the abliterated NVFP4 target

A drop-in replacement for incoai/GLM-5.3-Flash-DFlash2 when the target model is abliterated. Exactly one tensor differs from the original: fc.weight.

Why this exists

DFlash2's drafter does not read the target's tokens — it reads the target's hidden states, tapped at 5 layers and concatenated:

target_layer_ids = [5, 14, 24, 33, 42]      # 5 x 4096 = 20480 dims
fc.weight        = [4096, 20480]            # projects aux -> drafter width

fc is the entire coupling between target and drafter.

Abliteration rewrites o_proj on layers 15–45. Three of those five taps (24, 33, 42) sit inside that range, so the drafter's input distribution shifts and acceptance drops. Measured drift on the abliterated target:

tap	drift ‖Δh‖/‖h‖
L5	0.14%	stock anchor (below the edited range)
L14	0.27%	stock anchor
L24	0.52%	abliterated
L33	0.84%	abliterated
L42	2.03%	abliterated
overall (20480-dim)	1.50%	mean per-row cosine 0.9257

Drift is ~zero below L15 and grows monotonically with depth — exactly what the edit predicts.

What was done

A closed-form ridge solve. No training, no SGD, no epochs.

min_W  ‖ W·H_ablit − (fc_old·H_stock) ‖²_F  +  λ‖ W − fc_old ‖²_F

Make fc_new(h_ablit) reproduce what fc_old(h_stock) produced, so the drafter sees the residual it was originally trained on. Rows are key-matched on (seq, pos) so a dropped sequence cannot silently misalign pairs; the normal equations accumulate in fp64 because hidden states are collinear.

{
  "method": "closed-form fc recalibration (streamed, key-aligned)",
  "ridge": 0.001,
  "paired_rows": 83101,
  "rel_move": 0.0670,
  "fit_old": 0.012050,
  "fit_new": 0.008629
}

Capture: 83,101 aligned pairs, domain-tagged [chat, code, prose, technical, uncensored] (50% code, 9.7% prose).

Ridge choice — 1e-3, chosen empirically

ridge	weight move	fit_new (in-sample)	result
1e-3	6.7%	0.008629	shipped
1e-4	16.8%	0.008226	benchmarked → regression
1e-5	44.6%	0.007935	benchmarked → regression

Lower ridge fit better in-sample and performed worse in practice. fit_new is a training-set residual with no held-out split, so it cannot detect this — the 44.6% weight move at 1e-5 was the tell. Judge these by benchmark, never by fit_new.

Per-domain effect — and an honest limitation

domain	n	residual before	after	reduction
chat	11,123	1.18%	0.86%	27.3%
code	41,853	1.23%	0.89%	27.6%
prose	8,022	1.07%	0.68%	36.3%
technical	8,452	1.28%	0.93%	27.1%
uncensored	13,651	1.18%	0.89%	24.4%
overall	83,101	1.20%	0.87%	27.7%

Prose is corrected best of all five domains — and prose acceptance still sits near 0.33 against ~0.98 for structured output.

Conclusion: the prose bottleneck is the target's token entropy, not abliteration drift. This recalibration will not make prose fast. It recovers acceptance on context-sensitive, high-draftability output (code, structured, tool calls), which is where speculative decoding pays anyway.

Usage

Identical to the stock drafter — same architecture, same config, same num_speculative_tokens. Only the weights differ.

vllm serve <abliterated-target> \
  --speculative-config '{"method":"dflash",
                         "model":"/path/to/this/repo",
                         "num_speculative_tokens":7}'

Measured on 2× DGX Spark (GB10), TP=2, against the abliterated NVFP4 target: structured ~67 tok/s at 0.978 acceptance / 6.84 accepted per step.

⚠️ This drafter is target-specific

fc was fitted to the hidden states of one specific abliterated checkpoint (drowzeys/keys-GLM-5.3-Flash-NVFP4-ablit-l15-45-anchorstock).

Point it at a differently abliterated model, a finetune, or the stock censored model and three of the five taps carry a different distribution again. It will not error — acceptance just quietly drops. For a different target, re-run the solve rather than reusing this.

Conversely, if you run that abliterated target with the stock incoai drafter, you are silently leaving acceptance on the table.

Judging acceptance

Use mean accepted length, not acceptance ratio. The ratio rises when you simply stop drafting the low-probability tail; throughput tracks accepted length.

Credits

zai-org/GLM-5.3-Flash — the original model (MIT)
incoai/GLM-5.3-Flash-DFlash2 — the DFlash2 drafter this is derived from; all architecture and training credit is theirs, this repo changes one matrix
drowzeys/keys-GLM-5.3-Flash-NVFP4-ablit-l15-45-anchorstock — the abliterated target this was calibrated against
LibertAIDAI/GLM-5.3-Flash-NVFP4 — the NVFP4 parent quant
dealignai/GLM-5.3-Flash-UNCENSORED-NVFP4 — abliteration donor
tonyd2wild/GLM-5.3-Flash-NVFP4-DFlash2-2x-DGX-Spark — the 2-node deployment recipe, and the prior work identifying that abliteration costs drafter acceptance via hidden-state drift

License follows the base model (MIT).

GLM-5.3-Flash-DFlash2 — fc recalibrated for the abliterated NVFP4 target