Upload folder using huggingface_hub

README.md

@@ -32,7 +32,7 @@ tags:
 ## Model overview
 
 - **Scheme:** `NVFP4A16` - 4-bit NVFP4 **weights**, 16-bit (bf16) **activations** (weight-only).
-- **Why weight-only:** activations are transient (never stored), so keeping them at 16-bit costs ~nothing on disk - weights dominate size - while avoiding the token-level quality loss that 4-bit *activation* quantization (W4A4) causes. You get almost all the compression with accuracy much closer to the original.
+- **Why weight-only:** activations are transient (never stored), so keeping them at 16-bit costs ~nothing on disk - weights dominate size. Fully 4-bit *activation* quantization (W4A4) is widely reported to cost token-level quality; we have not measured that ourselves, and the one W4A4 candidate assay did gate failed the perplexity bar at +12.55% and was rejected. You get almost all the compression at a quality cost this card measures rather than estimates.
 - **Format:** compressed-tensors; loads natively in vLLM (no `--quantization` flag).
 
 ## Use with vLLM
@@ -42,7 +42,7 @@ from vllm import LLM, SamplingParams
 
 llm = LLM(model="uist-labs/DeepSeek-R1-Distill-Qwen-7B-NVFP4A16")
 prompts = ["Give me a short introduction to large language models."]
-params = SamplingParams(temperature=0.7, top_p=0.8, max_tokens=256)
+params = SamplingParams(temperature=0.6, top_p=0.95, max_tokens=256)
 for out in llm.generate(prompts, params):
     print(out.outputs[0].text)
 ```
@@ -55,9 +55,10 @@ This checkpoint uses **weight-only** NVFP4: it runs via vLLM's **FP4 Marlin** ke
 
 ## Creation
 
+- **Base snapshot:** [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B` @ `916b56a44061`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B/tree/916b56a44061fd5cd7d6a8fb632557ed4f724f60) - upstream repos are mutable, so the certificate names the exact commit it describes; the staged weights were verified against the recipe's pinned sha256s before quantization.
 - **Tool:** llm-compressor, scheme `NVFP4A16`, targets `Linear`, ignore `lm_head`.
 - **Calibration:** 512 samples from `HuggingFaceH4/ultrachat_200k` at 2048-token sequences.
-- **Pipeline:** UIST Labs' `assay` benchmark-gated quantization pipeline - quantize -> benchmark -> gate -> publish.
+- **Pipeline:** UIST Labs' `assay` benchmark-gated quantization pipeline (https://github.com/uistlabs/assay/tree/v0.6.1) - quantize -> benchmark -> gate -> publish.
 
 ## Evaluation
 
@@ -65,30 +66,35 @@ Measured with lm-evaluation-harness (vLLM backend) on the bf16 baseline and this
 
 | task | metric | baseline | quantized | delta | +/-stderr | retention |
 |------|--------|---------:|----------:|------:|----------:|----------:|
-| aime24_avg | exact_match,avg | 0.5250 | 0.5229 | -0.0021 | 0.0913 | 0.9960 |
-| aime25_avg | exact_match,avg | 0.4083 | 0.3979 | -0.0104 | 0.0992 | 0.9745 |
-| minerva_math500 | math_verify,none | 0.8360 | 0.8240 | -0.0120 | 0.0238 | 0.9856 |
-| gpqa_diamond_cot_zeroshot | exact_match,flexible-extract | 0.3182 | 0.3384 | +0.0202 | 0.0473 | 1.0635 |
-| wikitext | word_perplexity | 31.2854 | 31.4295 | +0.1442 | - | 1.0046 |
+| aime24_avg | exact_match,avg | 0.5437 | 0.5083 | -0.0354 | 0.0281 | 0.9349 |
+| aime25_avg | exact_match,avg | 0.3979 | 0.3646 | -0.0333 | 0.0235 | 0.9162 |
+| minerva_math500 | math_verify,none | 0.8360 | 0.8740 | +0.0380 | 0.0177 | 1.0455 |
+| gpqa_diamond_cot_zeroshot | exact_match,flexible-extract | 0.3788 | 0.3434 | -0.0354 | 0.0375 | 0.9067 |
+| wikitext | word_perplexity | 31.2853 | 31.4296 | +0.1442 | - | 1.0046 |
 
-Gate: PASS - no task regressed beyond k=2 combined stderr
+Gate: PASS - no task regressed beyond k=2 paired stderr
 (* = statistically significant regression at the recipe's k)
 No task showed a statistically significant regression.
 
 ## Methodology and limitations
 
 - **Apples-to-apples deltas.** The bf16 baseline and this checkpoint were evaluated with the *identical* harness and settings, so the `delta`/`retention` columns are a fair like-for-like comparison - which is what a quantization gate should measure: change from the original, honestly.
-- **Chat-mode evaluation.** Tasks are evaluated with the model's chat template applied. The template shifts the absolute scores on both the baseline and the quantized model - most visibly on the multiple-choice tasks - so read the deltas, not the absolute values. The comparison stays valid because both sides are evaluated with identical settings.
-- **Per-task statistical power varies.** `aime25_avg` has the largest combined standard error (0.099) and is power-limited - read its delta as directional only. `minerva_math500` (0.024) and perplexity carry the certification's binding, low-variance signal. The gate tests each task for a statistically significant regression, so an underpowered task cannot fail a sound quant on noise alone.
-- **Retention near or above 100% means "no measurable loss," not "better."** Where a task ticks up, that is within benchmark noise (small benchmark sets vary run to run) plus a touch of quantization acting as mild regularization - read the whole table as "indistinguishable from the original," not as an improvement.
-- **Weight-only tradeoff.** Weights are 4-bit; activations stay 16-bit. That keeps quality close to the original at nearly the full disk-size saving, at a modest inference-speed cost versus a fully 4-bit (W4A4) variant. If you need maximum throughput and can accept more degradation, a W4A4 build is a different point on that curve.
-- **Bias, risks, and inherited behavior.** This is a quantization of [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) and inherits its capabilities, biases, and limitations unchanged - quantization faithfully reproduces the base model's behavior (the gate above certifies exactly that), it does not add or remove bias. For intended use, safety, and ethical considerations, refer to the base model's card. Absolute-quality claims (multilingual, coding, safety) are the base model's; we certify only that quantization preserves them within the stated bar.
+- **Chat-mode evaluation.** Tasks are evaluated with the model's chat template applied. The template shifts the absolute scores on both the baseline and the quantized model, so read the deltas, not the absolute values. The comparison stays valid because both sides are evaluated with identical settings.
+- **Scope: the certified quantity is the delta, not the absolute score.** Both sides were measured in the same run against the same software stack, which is the only condition under which these two columns are comparable. Because this battery samples its answers, rerunning it on the same stack moves absolute scores by up to roughly the standard errors shown in the table above. Absolute scores can move further across harness or library versions, so a number here will not necessarily reproduce elsewhere, while the delta under identical conditions is what was certified. For the same reason we do not compare against a stored baseline from an earlier run: that would fold stack drift into the measurement.
+- **Per-task fail thresholds (this run).** The gate fails a task only when its measured drop exceeds k=2 times the paired standard error of the per-item score differences. On this run those thresholds were: `aime24_avg` (avg@16, the mean of 16 samples per item) 5.6 pts; `aime25_avg` (avg@16, the mean of 16 samples per item) 4.7 pts; `minerva_math500` 3.5 pts; `gpqa_diamond_cot_zeroshot` 7.5 pts. A measured drop at or below a task's threshold passed as statistically indistinguishable from zero - so this certification does not assert that no regression exists below that size. Threshold width tracks each task's sampling noise, not its importance: a wide threshold means this run had limited resolving power on that task. Separately, the perplexity criterion is an independent hard bar (increase no more than 3%) that does not depend on this test at all.
+- **How the standard error is computed.** This is a paired test: both evaluations score the identical items, so the standard error is computed from the per-item score differences (quantized minus baseline, item by item) rather than by combining the two sides' independent standard errors. Pairing credits the correlation the two sides share through item difficulty; the per-side stderr column in the table is informative only. These are the thresholds the gate actually enforced.
+- **One-sided upper confidence bound on the true regression.** At the same k=2, measured drop + k*SE per task: `aime24_avg` 9.2 pts; `aime25_avg` 8.0 pts; `minerva_math500` -0.3 pts; `gpqa_diamond_cot_zeroshot` 11.0 pts. This is the non-inferiority claim this certification makes: the data are consistent with a true per-task regression of at most these sizes.
+- **A sound quantization can still fail on noise.** The per-task test is one-sided at k=2, a nominal false-alarm rate of about 2% per task, and every task is tested - so the run-level chance is higher than any single task's. We do not quote a run-level figure: the tasks share one checkpoint and are not independent. We accept those odds in the withholding direction, since a false alarm costs us a release while a missed regression would cost you a bad checkpoint.
+- **Run-level generation settings: temperature 0.6, top_p 0.95.** These apply where a task generates its answer, and have no effect on any task the harness scores by loglikelihood. The baseline and this checkpoint were measured with identical settings.
+- **Retention near or above 100% means "no measurable loss," not "better."** Where a task ticks up, that is sampling noise - a finite benchmark set is a sample, not the whole population - so read the whole table as "indistinguishable from the original," not as an improvement.
+- **Weight-only tradeoff.** Weights are 4-bit; activations stay 16-bit. That keeps quality close to the original at nearly the full disk-size saving, at some inference-speed cost versus a fully 4-bit (W4A4) variant. We have not benchmarked a W4A4 build of this model, so we make no claim about where it lands on that curve.
+- **Bias, risks, and inherited behavior.** This is a quantization of [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B) and inherits its capabilities, biases, and limitations. The gate above certifies accuracy retention on the listed benchmarks only; it does not measure bias, safety, or any behavior those benchmarks do not cover, and quantization is not guaranteed to preserve what was not measured. For intended use, safety, and ethical considerations, refer to the base model's card.
 
 ## Certification
 
 UIST Labs publishes a quantized checkpoint **only if it clears a hard, stated accuracy bar** against its own bf16 baseline - we would rather withhold a release than ship an unverified one. This checkpoint passed all of:
 
-- No statistically significant per-task accuracy regression: one-sided, a task fails only if its drop exceeds k=2 times the combined standard error of the baseline and quantized scores
+- No statistically significant per-task accuracy regression: one-sided paired test, a task fails only if its drop exceeds k=2 times the standard error of the per-item score differences
 - Perplexity increase <= 3%
 
 The deltas above are the actual measured numbers, not vendor estimates. This gate runs on every UIST Labs quantization release.
@@ -104,10 +110,10 @@ If you use this checkpoint, please cite both this quantized release and the base
   year         = {2026},
   publisher    = {Hugging Face},
   howpublished = {\url{https://huggingface.co/uist-labs/DeepSeek-R1-Distill-Qwen-7B-NVFP4A16}},
-  note         = {Quantized and certified with the assay pipeline; published only after passing an automated accuracy gate against the bf16 baseline.}
+  note         = {Quantized and certified with the assay pipeline (https://github.com/uistlabs/assay/tree/v0.6.1); published only after passing an automated accuracy gate against the bf16 baseline.}
 }
 ```
 
 Please also cite the base model, [`deepseek-ai/DeepSeek-R1-Distill-Qwen-7B`](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-7B).
 
-Produced by assay v0.5.0. - UIST Labs
+Produced by assay v0.6.1 - UIST Labs