26K Checkpoint release - removes 46% of the prediction error a plain 4-step run has against the 8-step teacher

README.md

@@ -33,24 +33,26 @@ Right: 4 steps with this LoRA. Same seed throughout; click for full size. More c
 | file                                                     | use it when                                          |
 | -------------------------------------------------------- | ---------------------------------------------------- |
 | `krea2_turbo_4step_rank_64_lora_latest.safetensors`      | **normally** — always the newest accepted checkpoint |
-| `krea2_turbo_4step_rank_64_lora_chk00014000.safetensors` | pin this exact checkpoint                            |
+| `krea2_turbo_4step_rank_64_lora_chk00026000.safetensors` | pin this exact checkpoint                            |
 
 and, beside them, the same files with a `_comfyui` suffix for ComfyUI. Earlier checkpoints
-(`chk00004000`, `chk00005000`, `chk00006000`, `chk00010000`) are kept in
+(`chk00004000`, `chk00005000`, `chk00006000`, `chk00010000`, `chk00014000`, `chk00019000`) are kept in
 [`older_checkpoints/`](older_checkpoints), and their resolution sweeps stay in place, so the
 progression remains visible and comparable.
 
-The numbered files are points on **one continuous run**, not separate experiments — `chk00014000`
-resumed from `chk00010000`, which resumed from `chk00006000`, rather than restarting.
+The numbered files are points on **one continuous run**, not separate experiments — `chk00026000`
+resumed from `chk00014000`, which resumed from `chk00010000`, rather than restarting.
 
-`chk00014000` removes **44%** of the prediction error a plain 4-step run has against the 8-step
-teacher, where `chk00010000` removed 40% and `chk00006000` 27% — all measured on the same enlarged
+`chk00026000` removes **46%** of the prediction error a plain 4-step run has against the 8-step
+teacher, where `chk00014000` removed 44% and `chk00010000` 40% — all measured on the same enlarged
 held-out set (100 prompts across every trained resolution). Measured against each other rather than
-against the no-LoRA run, its remaining error is **6% smaller than `chk00010000`'s** and **23%
-smaller than `chk00006000`'s**. What that number is — and what it is not — is in
-[chk00014000 vs chk00010000](#chk00014000-vs-chk00010000).
+against the no-LoRA run, its remaining error is **4% smaller than `chk00014000`'s** and **10%
+smaller than `chk00010000`'s** — and unlike a purely teacher-forced score, the gain also shows up
+free-running: a full 4-call rollout from the teacher's noise ends **1.6% nearer the teacher's final
+latent** than `chk00014000`'s does. What the headline number is — and what it is not — is in
+[chk00026000 vs chk00014000](#chk00026000-vs-chk00014000).
 
-> 📌 **`_latest` currently points to `chk00014000`** — 14,000 training samples. This line is updated
+> 📌 **`_latest` currently points to `chk00026000`** — 26,000 training samples. This line is updated
 > whenever a newer checkpoint is published.
 
 **This is work in progress and better checkpoints may follow.** Training is ongoing, so
@@ -74,25 +76,26 @@ So the recipe itself changes between runs. Each published checkpoint reflects wh
 round taught us: the training precision, the optimiser settings, the teacher used to generate the
 targets and the data mix have all been revised on evidence rather than assumption.
 
-`chk00010000` is a direct example. The first continuation of `chk00006000` — same data, optimiser
-left as it was — got steadily _worse_ with every checkpoint out to 10,000 samples, and none of it
-was published. The cause was traced to the optimiser: a constant learning rate with no weight decay
-lets the adapter keep drifting after it has converged, so its magnitude grows and it over-applies
-its own correction. The same span was retrained from `chk00006000` with a cosine learning-rate decay
-and weight decay, and every checkpoint of that second run improved on the one before it.
-`chk00010000` was its end point.
-
-`chk00014000` is the next example, and it shows the other half of the same lesson. The run was
-continued from `chk00010000` over the whole pool of teacher trajectories, with two changes: the
-final, texture-deciding call of the schedule was weighted more heavily in the loss, and a running
-**average of the weights** was kept beside the live ones and scored at every evaluation (a
-single checkpoint is one sample of a weight vector that moves from step to step; the average is
-its mean). At 14,000 samples the averaged weights measured a smaller gap to the teacher than any
-checkpoint before them, and a smaller gap than the live weights at the same point — so the averaged
-weights are what `chk00014000` is. The run was allowed to continue well past that point, and the
-adapter's magnitude began to grow again: later checkpoints measured worse on the held-out set and
-further from the teacher in fixed-seed renders, and none of them was published. The number is
-chosen by measurement, not by how far the run went.
+Two earlier releases set the terms this project publishes on. `chk00010000`'s first attempt — same
+data, optimiser left as it was — got steadily worse for 4,000 samples and none of it was published;
+retrained with cosine learning-rate decay and weight decay, every checkpoint improved on the one
+before it, and its end point shipped. `chk00014000` added the other half of the lesson: the final,
+texture-deciding call of the schedule weighted more heavily in the loss, and a **running average of
+the weights** kept beside the live ones and scored at every evaluation — the averaged weights
+measured better than any checkpoint before them, so the average is what shipped. Left running past
+that point, the adapter's magnitude grew again and every later checkpoint measured worse. The
+number is chosen by measurement, not by how far a run went.
+
+`chk00026000` — the current checkpoint — is that discipline paying off. It resumes from
+`chk00014000`'s averaged weights with the same recipe: same loss weighting, same running average,
+a conservative constant learning rate, over a much larger pool of teacher trajectories. This time
+the continuation held. The averaged weights' held-out gap fell throughout the run, and every
+free-running rollout measured of them improved on the one before — so unlike the first
+continuation, this one produced a checkpoint worth shipping. Every published number improves on
+`chk00014000`: the held-out gap (44% → **46%** of the deficit closed), the full 4-call rollout
+from the teacher's noise (1.6% nearer the teacher's final latent), and the fixed-seed render
+distance to the 8-step images. `chk00019000`, an intermediate point of the same continuation, is
+kept in [`older_checkpoints/`](older_checkpoints) with the rest of the lineage.
 
 ### Timeline of training process
 
@@ -119,14 +122,11 @@ through a freshly grown pool — followed by longer quiet stretches while the ne
 shards is produced. A quiet stretch is shard generation, not abandonment; `_latest` always holds
 the newest checkpoint that passed review.
 
-The current checkpoint and the one before it mark the last two revisions of the recipe:
-`chk00010000` continued `chk00006000` with **cosine learning-rate decay and weight decay**, and
-`chk00014000` continued `chk00010000` over the full pool of teacher trajectories with the final
-call of the schedule **weighted more heavily** and the shipped weights taken as a **running
-average** of the trained ones. The checkpoints before them were earlier points on the same run
-(`chk00006000` was the first trained on **int8** teacher trajectories and an int8 base, see
-[Hardware](#hardware); the ones before it used an NF4 teacher). Each published checkpoint has
-measured a smaller gap to the teacher than the one it replaced, and the earlier files are kept in
+The current checkpoint, `chk00026000`, runs the recipe the earlier releases arrived at — the final,
+texture-deciding call **weighted more heavily** in the loss, the shipped weights a **running
+average** of the trained ones — carried further over a larger pool of teacher trajectories, and
+published because it measured better on every evaluation. Each published checkpoint has measured a
+smaller gap to the teacher than the one it replaced; the earlier files stay in
 [`older_checkpoints/`](older_checkpoints) for comparison only.
 
 Every file records which checkpoint it actually is in its safetensors metadata (`checkpoint`,
@@ -136,15 +136,15 @@ be identified even if renamed.
 ### Checkpoint numbering
 
 ```
-krea2_turbo_4step_rank_64_lora_chk00014000.safetensors
+krea2_turbo_4step_rank_64_lora_chk00026000.safetensors
                                    ^^^^^^^^
 ```
 
-The number is the count of **training samples consumed** — `chk00014000` has seen 14,000. It is not
+The number is the count of **training samples consumed** — `chk00026000` has seen 26,000. It is not
 an epoch count, a step count, or a quality score. Higher-numbered checkpoints are further along the
 same continuous run, resuming from the previous checkpoint rather than restarting.
 
-## chk00014000 vs chk00010000
+## chk00026000 vs chk00014000
 
 **The measure.** Every checkpoint is scored the same way. On a fixed set of held-out prompts that
 never receive a gradient step, the 4-step student's predicted velocity is compared with the 8-step
@@ -163,45 +163,42 @@ deficit this LoRA exists to close.
 | `chk00005000`                    | 3.26e-02                  | 30%                          |
 | `chk00006000`                    | 3.45e-02                  | 27%                          |
 | `chk00010000`                    | 2.81e-02                  | 40%                          |
-| **`chk00014000`**                | **2.64e-02**              | **44%**                      |
+| `chk00014000`                    | 2.64e-02                  | 44%                          |
+| `chk00019000`                    | 2.62e-02                  | 44%                          |
+| **`chk00026000`**                | **2.53e-02**              | **46%**                      |
 
 Two ways to read the same numbers, with different denominators — they are not meant to be added:
 
-- 📊 **Against the no-LoRA run** (the right-hand column): `chk00014000` has removed 44% of the
-  4-step deficit, about 4 percentage points more than `chk00010000`'s 40%.
-- 📈 **Against each other** (the gap column): `chk00014000`'s remaining error is **6.1% smaller than
-  `chk00010000`'s** (2.64 vs 2.81) and **23% smaller than `chk00006000`'s** (2.64 vs 3.45). The
+- 📊 **Against the no-LoRA run** (the right-hand column): `chk00026000` has removed 46% of the
+  4-step deficit, about 2 percentage points more than `chk00014000`'s 44%.
+- 📈 **Against each other** (the gap column): `chk00026000`'s remaining error is **4.2% smaller than
+  `chk00014000`'s** (2.53 vs 2.64) and **10% smaller than `chk00010000`'s** (2.53 vs 2.81). The
   same 4 points of deficit are a larger share of a gap that has already shrunk, which is why the
   checkpoint-to-checkpoint figure is the bigger number.
 
-One thing the larger set changed: on it, `chk00005000` scores slightly better than `chk00006000`
-(3.26 vs 3.45), the opposite of the order the small set gave. Eight prompts were not enough to
-separate two neighbouring checkpoints reliably; one hundred across every resolution is what the
-set was enlarged for, and the rows above are the numbers that count from here on.
+(The larger set also reordered two early neighbours — `chk00005000` edges `chk00006000` on it —
+which is exactly why it replaced the original eight-prompt set.)
 
 **Read these as error figures, not as quality fractions.** A plain 4-step run already produces a
 complete, coherent image; what it loses against 8 steps is fine detail and texture, and that loss
 shows up as its predictions sitting about 4.7% away from the teacher's. The LoRA brings that to
-about 2.6%. "44% of the deficit closed" therefore does not mean "44% of the teacher's quality" — it
-means 44% of that 4.7% error is gone. A gap of zero would be the student reproducing the teacher's
+about 2.5%. "46% of the deficit closed" therefore does not mean "46% of the teacher's quality" — it
+means 46% of that 4.7% error is gone. A gap of zero would be the student reproducing the teacher's
 two-step chords exactly, i.e. perfect 8-step equivalence in 4 steps, which no low-rank adapter on
 a frozen base reaches and which the images do not need in order to match.
 
-**Why it is better, not just later.** `chk00010000` → `chk00014000` continued the same run over
-the whole pool of teacher trajectories, with the last call of the 4-step schedule — the one that
-decides fine texture — weighted more heavily in the loss, and with a running average of the
-adapter weights scored beside the live ones at every evaluation (see
-[How checkpoints get chosen](#how-checkpoints-get-chosen)). The averaged weights at 14,000 samples
-are what ships: 2.64e-02, against 2.77e-02 for the live weights at the same point and 2.81e-02 for
-`chk00010000`. The gain is on every one of the four calls, not just the weighted one: against
-`chk00010000`, `chk00014000`'s error is 5.9% lower at the first sigma (composition), 4.2% and 7.0%
-lower at the two middle sigmas, and 6.2% lower at the final, texture-deciding sigma; a full 4-call
-rollout from the teacher's noise also ends 6% nearer the teacher's final latent. In fixed-seed
-renders of the fifteen published prompts `chk00014000` lands nearer the
-8-step images than `chk00010000` does, and its texture energy is closer
-to the teacher's (0.90 of the teacher's on a scale where 1.0 is teacher-like, against 0.86 for
-`chk00010000`) without crossing it. The continuation past 14,000 samples measured worse at every
-subsequent evaluation as the adapter's magnitude grew again; none of it was published.
+**Why it is better, not just later.** `chk00026000` continues the same run from `chk00014000`'s
+averaged weights — recipe unchanged — and measures better everywhere the project measures. The
+averaged weights at 26,000 samples are what ships: 2.53e-02 against `chk00014000`'s 2.64e-02 on the
+held-out gap. The improvement is not an artifact of teacher-forced scoring — a full 4-call
+free-running rollout from the teacher's noise ends **1.6% nearer the teacher's final latent**, and
+every intermediate averaged checkpoint of the run (19,000, 22,000, 24,000) improved that rollout on
+the one before it. In fixed-seed renders of the fifteen published prompts at 1280×1280 it lands
+nearer the 8-step images than `chk00014000` does (relative pixel distance 0.308 against 0.316),
+with texture energy at 0.88 of the teacher's against 0.87 (1.0 is teacher-like), without crossing
+it. `chk00014000` earned its place against `chk00010000` the same way — every one of the four
+calls improved, not just the weighted one — and that history stays with its files in
+[`older_checkpoints/`](older_checkpoints).
 
 **What the number does not say.** It is a training-space measure and does not rank images pixel
 for pixel. With a fixed seed the student's trajectory drifts away from the teacher's, so
@@ -381,11 +378,11 @@ ships bf16 simply because it is the one build guaranteed to run everywhere.
 > graph executes fine right up to the sampler. The fp8 _text encoder_ does run on MPS, but only
 > because ComfyUI places it on CPU; the published workflow does not rely on that.
 
-> 📌 **`_latest_comfyui` currently points to `chk00014000`** — 14,000 training samples.
+> 📌 **`_latest_comfyui` currently points to `chk00026000`** — 26,000 training samples.
 
 The workflow references the **`_latest`** LoRA filename deliberately: as training continues and
 newer checkpoints are published, replacing that one file is enough — the workflow itself never
-needs editing. A numbered copy (`..._chk00014000_comfyui.safetensors`) sits beside it if you would
+needs editing. A numbered copy (`..._chk00026000_comfyui.safetensors`) sits beside it if you would
 rather pin a specific checkpoint in the node.
 
 **Why a separate file.** ComfyUI addresses the transformer by its own layer names, so the adapter
@@ -428,8 +425,8 @@ cherry-picked; a sweep is the complete set or it is not published.
 
 ```
 checkpoint_resolution_sweeps/
-├── chk4000/  chk5000/  chk6000/  chk10000/
-└── chk14000/               one folder per checkpoint, named for the checkpoint it renders
+├── chk4000/  chk5000/  chk6000/  chk10000/  chk14000/  chk19000/
+└── chk26000/               one folder per checkpoint, named for the checkpoint it renders
     ├── 512x512/            one folder per resolution
     │   ├── portrait.jpg
     │   ├── kingfisher.jpg
@@ -535,7 +532,7 @@ targets are permanent, so the teacher's error is baked into every trajectory for
 `chk00006000` and every checkpoint after it are trained this way. Everything up to and including
 `chk00005000` used an NF4 teacher and an NF4 base. On the same held-out set the cleaner targets were
 worth a **13% smaller gap to the teacher** at `chk00006000`; the optimiser changes that followed
-took it further (see [chk00014000 vs chk00010000](#chk00014000-vs-chk00010000)).
+took it further (see [chk00026000 vs chk00014000](#chk00026000-vs-chk00014000)).
 
 There is a second, subtler effect worth stating plainly. Re-scoring the _older_ `chk00005000`
 against int8 targets instead of NF4 ones drops its measured gap-recovery from 28.9% to 17.9% — the

krea2_turbo_4step_lora_t2i.json

@@ -408,7 +408,7 @@
       "outputs": [],
       "properties": {},
       "widgets_values": [
-        "## Krea 2 Turbo \u2014 4-step distillation LoRA\n\nKrea 2 Turbo normally needs **8 steps**. With this LoRA it runs at **4**.\n\n### Files\n| file | put it in |\n| --- | --- |\n| `krea2_turbo_bf16.safetensors` | `models/diffusion_models/` |\n| `qwen3vl_4b_bf16.safetensors` | `models/text_encoders/` |\n| `qwen_image_vae.safetensors` | `models/vae/` |\n| `krea2_turbo_4step_rank_64_lora_latest_comfyui.safetensors` | `models/loras/` |\n\nThe first three are from [Comfy-Org/Krea-2](https://huggingface.co/Comfy-Org/Krea-2).\n\n**The LoRA is independent of the base build.** It is applied on top of the diffusion model by ComfyUI's own loader, which handles any dequantisation, so a quantised or otherwise optimised build of Krea 2 Turbo behaves just as bf16 does. Please use whichever variant suits your hardware \u2014 set it in **Load Diffusion Model** and leave the rest of the workflow untouched.\n\nThis workflow ships **full bf16** only because it is the one build guaranteed to run everywhere: bf16 needs no backend-specific kernel, so it works on CUDA, Apple Silicon and CPU alike. Smaller options: `krea2_turbo_int8_convrot` (13.5 GB, either platform), or `krea2_turbo_fp8_scaled` (13.1 GB) which is **NVIDIA-only** \u2014 MPS has no `Float8_e4m3fn` support and fails at the sampler. The text encoder has bf16 and fp8 builds only; there is no int8 one.\n\n> **Do not use `fp8_scaled` on Apple Silicon.** MPS has no `Float8_e4m3fn` support, so it fails at the sampler with *\"Trying to convert Float8_e4m3fn to the MPS backend\"*. It is fine on NVIDIA. `krea2_turbo_bf16` (26.3 GB) also works everywhere if you prefer full precision \u2014 just set the matching name in the loader.\n\n### Settings that matter\n- **steps 4**, **cfg 1.0** (Turbo is CFG-free), sampler **euler**, scheduler **simple**\n- LoRA strength **1.0**\n- The negative is a `ConditioningZeroOut` \u2014 at cfg 1.0 it is not evaluated, so there is nothing to write in it\n\nSet steps back to 8 and bypass the LoRA node (Ctrl+B) to compare against stock Turbo.\n\n### Work in progress\n**`_latest` currently points to `chk00014000`** (14,000 training samples).\n\nTraining is ongoing and better checkpoints may be published later. This workflow points at `..._latest_comfyui.safetensors`, which always holds the newest accepted checkpoint \u2014 **re-download that one file and the workflow keeps working**, no edit needed.\n\nNumbered copies (`..._chk00014000_comfyui.safetensors`) are kept alongside it so a specific checkpoint can be pinned. Every file records which checkpoint it is in its safetensors metadata (`checkpoint`, `training_samples`)."
+        "## Krea 2 Turbo \u2014 4-step distillation LoRA\n\nKrea 2 Turbo normally needs **8 steps**. With this LoRA it runs at **4**.\n\n### Files\n| file | put it in |\n| --- | --- |\n| `krea2_turbo_bf16.safetensors` | `models/diffusion_models/` |\n| `qwen3vl_4b_bf16.safetensors` | `models/text_encoders/` |\n| `qwen_image_vae.safetensors` | `models/vae/` |\n| `krea2_turbo_4step_rank_64_lora_latest_comfyui.safetensors` | `models/loras/` |\n\nThe first three are from [Comfy-Org/Krea-2](https://huggingface.co/Comfy-Org/Krea-2).\n\n**The LoRA is independent of the base build.** It is applied on top of the diffusion model by ComfyUI's own loader, which handles any dequantisation, so a quantised or otherwise optimised build of Krea 2 Turbo behaves just as bf16 does. Please use whichever variant suits your hardware \u2014 set it in **Load Diffusion Model** and leave the rest of the workflow untouched.\n\nThis workflow ships **full bf16** only because it is the one build guaranteed to run everywhere: bf16 needs no backend-specific kernel, so it works on CUDA, Apple Silicon and CPU alike. Smaller options: `krea2_turbo_int8_convrot` (13.5 GB, either platform), or `krea2_turbo_fp8_scaled` (13.1 GB) which is **NVIDIA-only** \u2014 MPS has no `Float8_e4m3fn` support and fails at the sampler. The text encoder has bf16 and fp8 builds only; there is no int8 one.\n\n> **Do not use `fp8_scaled` on Apple Silicon.** MPS has no `Float8_e4m3fn` support, so it fails at the sampler with *\"Trying to convert Float8_e4m3fn to the MPS backend\"*. It is fine on NVIDIA. `krea2_turbo_bf16` (26.3 GB) also works everywhere if you prefer full precision \u2014 just set the matching name in the loader.\n\n### Settings that matter\n- **steps 4**, **cfg 1.0** (Turbo is CFG-free), sampler **euler**, scheduler **simple**\n- LoRA strength **1.0**\n- The negative is a `ConditioningZeroOut` \u2014 at cfg 1.0 it is not evaluated, so there is nothing to write in it\n\nSet steps back to 8 and bypass the LoRA node (Ctrl+B) to compare against stock Turbo.\n\n### Work in progress\n**`_latest` currently points to `chk00026000`** (26,000 training samples).\n\nTraining is ongoing and better checkpoints may be published later. This workflow points at `..._latest_comfyui.safetensors`, which always holds the newest accepted checkpoint \u2014 **re-download that one file and the workflow keeps working**, no edit needed.\n\nNumbered copies (`..._chk00026000_comfyui.safetensors`) are kept alongside it so a specific checkpoint can be pinned. Every file records which checkpoint it is in its safetensors metadata (`checkpoint`, `training_samples`)."
       ],
       "color": "#432",
       "bgcolor": "#653"

krea2_turbo_4step_rank_64_lora_chk00026000.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:05560994dfabe921a5915d1e10b77b134bf3d3bf1d2f2e834ba98132672b1924
+size 438160448

krea2_turbo_4step_rank_64_lora_chk00026000_comfyui.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:c3375b24633def875f397ac9c470fc18a0f7490b147e112dfa55a37e7c5f5293
+size 438140992

krea2_turbo_4step_rank_64_lora_latest.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:d48ad83402b25cfb17abae631142c24a3c8f7b89b6af6ed4432bfe2e4af36375
+oid sha256:72d11c95cd60d9dc5ffb49107a9bbacf3f17ba88830f83c6eac6b3c6f2aaebf6
 size 438160576

krea2_turbo_4step_rank_64_lora_latest_comfyui.safetensors

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:b4946bd510e06d3e9da3dd6c28ad0710aef44aa9ed89a2f3b6facd3f000ba3a1
+oid sha256:80f2a5ced6f719df9384f9924450786ff16efd75da7f1c61723bc8c0fd3839d9
 size 438141128