SKILLRET-Edge-22M-drift-mixed

The 22.7M-parameter SKILLRET-Edge bi-encoder for agent skill retrieval, quantized to a mixed-precision layout chosen by a label-free drift signal instead of one uniform bit width. Same architecture, same tokenizer, same query contract as ThakiCloud/SKILLRET-Edge-22M-int4 — this repo exists to show a different allocation technique, not to replace the int4/int3 variants.

Scored on the public ThakiCloud/SKILLRET test split (4,392 queries / 6,006 skills).

NDCG@10 74.57 · 14.1 MB packed (the int3 file size, 99.1% of fp16)

Where it sits

Variant	Packed size	NDCG@10	Note
fp16	45.4 MB	75.25	reference
int4 / g16 (uniform)	17.0 MB	75.14	best quality per the int4 card
drift-mixed / g16 (this repo)	14.1 MB	74.57	int3 bytes, +0.65pp over int3
int3 / g16 (uniform)	14.2 MB	73.82
int2 / g16 (uniform)	11.9 MB	59.73	4.18 real bits — no size win over int4 g128

Paired query-level bootstrap on the same 4,392 queries (5,000 resamples), all arms built and scored under one protocol (int3 uniform comes from the 5.2-bit run of the same series):

Contrast	Δ NDCG@10 [95% CI]
drift-mixed (14.1 MB) − uniform int3 g16 (14.65 MB, same protocol)	+0.65pp [+0.35, +0.96]
drift-mixed at the same 5.2-real-bit budget − uniform int3	+0.98pp [+0.64, +1.33]
drift-mixed − uniform int2 g16	+14.83pp [+14.08, +15.59]
drift-mixed − label-supervised allocation (per-module ΔNDCG, same budget)	+0.71pp [+0.43, +0.99]
drift-mixed − 5 random allocations at the same bytes	+7.7 to +9.6pp (every CI excludes 0)

Read the first row honestly: the gain over uniform int3 is real but under one point. The value of the technique is the ordering (random allocations at the same bytes lose 8–9 points), and the budgets uniform cannot express — at 4.5 real bits (~12.7 MB) the drift allocation keeps 70.42 where uniform int2 gives 59.73 and uniform int3 does not fit.

How the bits were allocated

Each of 12 allocation units was quantized alone at 2/3/4 bits while the rest stayed fp32, and the resulting drift of the output embeddings — mean (1 − cosine) between full-precision and quantized embeddings, averaged over 1,500 calibration skills and 150 calibration queries, balanced across the two — was recorded. No relevance labels are used. An exhaustive integer program then picks the per-unit bit width that minimises predicted drift under a hard byte budget (fp16 scale + zero per group of 16 counted). The drift ordering reproduced across disjoint calibration halves at every size tried (Spearman 1.000 at 75/150/300/600 queries); the label-based ordering on this model did not (−0.29 to +0.34), which is why the label-supervised row above is reported but not recommended.

Unit	What it covers	Params	Bits
embedding.v0	word embedding rows 0–25%	2.93M	3
embedding.v1	rows 25–50%	2.93M	2
embedding.v2	rows 50–75%	2.93M	2
embedding.v3	rows 75–100%	2.93M	2
attention.b0	attention, layers 0–1	1.18M	3
attention.b1	attention, layer 2	0.59M	4
attention.b2	attention, layers 3–4	1.18M	4
attention.b3	attention, layer 5	0.59M	4
ffn.b0	FFN, layers 0–1	2.36M	3
ffn.b1	FFN, layer 2	1.18M	3
ffn.b2	FFN, layers 3–4	2.36M	4
ffn.b3	FFN, layer 5	1.18M	3

The picture is the one you would guess after seeing the sensitivities: the word-embedding table is 51.6% of the parameters and its lower-frequency rows barely move the output at 2 bits, while the middle attention/FFN blocks are where 2-bit damage concentrates.

Files

File	What it is
model.safetensors	The quantized values de-quantized back into fp16 so sentence-transformers loads it today. This file produces the score above (re-evaluated after packing: 74.569 vs the ledger's 74.559).
model-drift-mixed-g16.bin	The actual packed payload — 14.140 MB: per-unit bit-packed indices + fp16 scale/zero per 16 weights, plus the two small fp16 position/token-type tables. Predicted 14.140 MB, 0.0% error.
quantization.json	The plan (plan), per-unit parameter counts, and the per-tensor layout (name, rows slice, bits, offset) if you want to write a kernel.
query_prefix.json	resolved: "" — encode queries bare, no instruction prefix.

The safetensors path runs fp32 arithmetic; latency is identical to the int4/int3 cards (same architecture, same dequantized format) and is not an integer-kernel speed.

Usage

from sentence_transformers import SentenceTransformer
m = SentenceTransformer("ThakiCloud/SKILLRET-Edge-22M-drift-mixed")
q = m.encode(["I need to convert a spreadsheet into a chart"], normalize_embeddings=True)
d = m.encode(["Chart Builder — turn tabular data into bar/line charts"], normalize_embeddings=True)
print(q @ d.T)

Caveats

• Standard error on this split is about ±0.45; the drift-vs-int3 gap is significant only because it is paired on the same queries. Do not read it as a ranking between cards scored in different runs.
• The protocol matches the other Edge cards: CLS pooling, L2-normalised, max_length=256, bare queries.
• Mixed precision means a real integer kernel has to handle per-tensor (and, for the embedding table, per-row-band) bit widths. That is the cost you pay for +0.65pp at int3 bytes.
• The base student, the distillation recipe and the int4/int3 measurements are documented on ThakiCloud/SKILLRET-Edge-22M-int4.

License

Apache-2.0, inherited from the base model.

Citation

Benchmark: SkillRet, arXiv:2605.05726

Student and uniform-PTQ measurements: Where Post-Training Quantization Breaks Text Embedders, arXiv:2609.16391

Allocation signal (the drift criterion and the allocator used here): Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders, Han, 2026 — arXiv preprint, identifier to be added on announcement.

@article{han2026quantizebydrift,
  title   = {Quantize by Drift: Label-Free Mixed-Precision Post-Training Quantization for Text Embedders},
  author  = {Han, Hyojung},
  journal = {arXiv preprint},
  year    = {2026}
}