# The released SD1.5 → SANA winner: architecture, loss, and inference The default [released checkpoint](https://huggingface.co/ntc-ai/model-glue-sd15-sana-text) is a **7,182,917-parameter text-conditioning bridge**. Its winning continuation trained the final linear output projection using paired embeddings and frozen-SANA response distillation. The parent came from earlier ParticleGAN experiments, but this continuation has **no active GAN or b_cap loss**. Its learned particle cloud is retained as fixed network weights. `model.safetensors` is the EMA checkpoint selected at update 1,250 of 1,500, followed by one validation-fitted mask-bias adjustment. The uncalibrated control and expanded-source alternative are also released. See the [results](https://github.com/255BITS/model-glue/blob/main/docs/results/formulation-retraining-20260905.md), [download instructions](https://github.com/255BITS/model-glue/blob/main/docs/sana-transfer.md#released-sd15-text-checkpoints), and [exact recipe](https://github.com/255BITS/model-glue/blob/main/configs/formulation/linear-distill-fixed.json). ## Architecture ```mermaid flowchart LR A[Prompt] --> B[Frozen SD1.5 CLIP] B --> C[77 x 768 contextual states] C --> D[LayerNorm and 768-to-384 projection] C --> R[Mean pooling and particle router] P[Fixed 256 x 4 learned cloud] --> R D --> Q[77 learned queries plus source and particle offsets] R --> Q Q --> T[4 cross-attention blocks; width 384] D --> T T --> O[Linear output and fixed feature calibration] T --> M[Linear mask head and threshold] O --> S[Frozen SANA: 20 sampling steps] M --> S U[Stored native empty conditioning] --> S S --> V[Frozen DC-AE decoder] V --> I[Image] ``` The input is the full contextual CLIP sequence, not a pooled embedding. For source states $X$ with shape 77 × 768: $$ E=\operatorname{Linear}_{768\to384}(\operatorname{LN}(X)),\qquad r=\operatorname{MLP}_{768\to64\to4}(\operatorname{mean}_{tokens}X). $$ The router MLP uses SiLU. With saved cloud $P$ of shape 256 × 4: $$ a=\operatorname{softmax}(rP^T/\sqrt4),\qquad p=aP,\qquad H_0=Q+E+\operatorname{Linear}_{4\to384}(p). $$ $Q$ contains 77 learned queries of width 384. The same four-dimensional $p$ provides a global query offset. This is attention over shared parameter vectors, not a lookup of training captions. The cloud and router stayed fixed during the winning continuation; no particle advantage is established. Each of four blocks performs six-head cross-attention to $E$, then a residual feed-forward network: $$ A_l=H_l+\operatorname{MHA}(\operatorname{LN}(H_l),E,E),\qquad H_{l+1}=A_l+\operatorname{MLP}_{384\to1152\to384}(\operatorname{LN}(A_l)). $$ The feed-forward activation is GELU and attention dropout is zero. These bridge blocks have no additional target-query self-attention layer. CLIP has already contextualized the source states. Output features and mask logits are: $$ \widehat Y_j=\mu_j+\sigma\odot(W_oH_{4,j}+b_o),\qquad \ell_j=w_m^TH_{4,j}+b_m. $$ The saved target mean $\mu$ is per slot and channel (77 × 2304); the standard deviation $\sigma$ is per channel (2304). Both come from teacher-valid training positions and were retained from the parent; standard deviation has a floor of 0.1. The linear 384 → 2304 output head retains the old per-slot affine rank restriction of at most 384. Winning validation does not imply that this model incorporates every architectural repair or can represent arbitrary target states. This transfers the donor's text representation. The SD1.5 diffusion U-Net never runs in this path, so the bridge does not establish transfer of its generated style or composition. Spatial donor-feature transfer is a separate interface. ## Exact active loss The winning objective is: $$ \boxed{L=0.1L_{embed}+L_{functional}.} $$ For native teacher conditioning $Y$, teacher-valid mask $M$, and bridge output $\widehat Y$, embedding loss is masked, channel-normalized squared error: $$ L_{embed}=\frac{\sum_{b,j,c}M_{b,j} \left((\widehat Y_{b,j,c}-Y_{b,j,c})/\sigma_c\right)^2} {2304\max(1,\sum_{b,j}M_{b,j})}. $$ Functional distillation asks whether the **same frozen SANA** makes the same denoising prediction under bridged and native conditioning. Student predictions carry gradients through SANA into the bridge. Teacher predictions are detached or read from the native-state cache; SANA's parameters remain frozen. At the same latent state and timestep, let $s_+,s_0$ denote SANA predictions under student positive/empty conditioning, and $t_+,t_0$ its teacher predictions. All MSEs average across the functional minibatch and latent coordinates: $$ g_s=s_0+4.5(s_+-s_0),\qquad g_t=t_0+4.5(t_+-t_0), $$ $$ L_{functional}=\frac{ \operatorname{MSE}(s_+,t_+)+\operatorname{MSE}(s_0,t_0) +0.05\operatorname{MSE}(g_s,g_t)} {\max(10^{-5},\operatorname{mean}((t_+-t_0)^2))}. $$ The denominator is detached **native prompt-effect power**. It measures error relative to how strongly the teacher responds to the prompt. The positive student uses its predicted hard mask; the teacher uses its cached native mask. Their mismatch matters when predicted slots are wrong. Both empty branches use native empty conditioning. When their predictions agree exactly, empty error vanishes and the guided error becomes $4.5^2$ times positive error. The numerator then simplifies to **2.0125 times positive MSE**. The code retains the full expression, including any cached-versus-live numerical discrepancy. This run principally matches the conditional recipient response; it does not learn a new unconditional branch. LPIPS, pixel error, and CLIP judge rendered outputs for selection/evaluation; they are not differentiable image losses in this run. Full-sampler backpropagation was tested in separate arms and did not win the combined validation ranking. ## What training changes The run initializes the already-trained parent and updates the **887,040 output projection parameters** that receive nonzero gradients. The `heads` setting also enables 385 mask-head parameters, but mask BCE has weight zero and hard thresholds supply no functional mask gradient. Their values remain unchanged during training. | Setting | Winning continuation | |---|---| | Optimizer | Adam, betas (0.0, 0.999), epsilon 1e-8, no weight decay | | Learning rate | 1e-5; held through 60% of the run, then cosine toward a 5% floor | | Updates and selection | 1,500 completed; EMA update 1,250 selected | | EMA | Decay 0.98 | | Gradient clipping | Global bridge gradient norm at 1.0 | | Embedding batch | 32 rows: 31 sampled captions plus one empty row | | Functional batch | Four positive examples every update | | State mixture | Each update chooses cached native states with probability 0.5, otherwise fresh initial noise | | Native cache | 512 training captions × five states, with cached teacher predictions | | Training data | 8,192 paired captions; no image reconstruction cache in this run | | Frozen components | Source projection, queries, four blocks, router, cloud, particle projection, calibration buffers, CLIP, SANA, DC-AE | | Inactive loss terms | GAN/b_cap, critic features, mask BCE, particle VICReg/centering, padding, soft masks, full rollouts | After training, mask calibration **subtracts 2.0417721271514893** from the old mask bias, equivalent to raising the old-logit threshold by that amount. It minimizes mask errors on 58 validation prompts. Ordinary inference thresholds the resulting logits at zero and always preserves the first BOS slot. Calibration changes one scalar and requires no fitting during deployment. Checkpoint selection uses rendered validation LPIPS and is separate from the optimized loss. The release contains EMA weights, not the last live optimizer iterate. Final testing occurred after freezing selection. The observed improvement is **1.42% lower held-out LPIPS and 2.50% lower pixel MAE**, with confidence intervals including no improvement and composition errors remaining. Calibration adds two missed test positions. A nonlinear frozen recipient can respond better to an improved linear head without that head recovering all inaccessible target directions. ## Expanded-source alternative The alternative preserves the parent and adds this branch to normalized output: $$ z_j=W_oH_{4,j}+b_o+ W_2\operatorname{SiLU}(W_1H_{4,j}+b_1+W_sX_j)+b_2. $$ $W_1$ maps 384 → 2304, $W_s$ maps the full 768 source channels directly to 2304 without bias, and $W_2$ maps 2304 → 2304. Source information can bypass the old 768 → 384 projection, and the nonlinear branch removes its affine-output restriction. This does not promise arbitrary mappings or aligned word positions. This model has **15,150,149 parameters**, including **7,967,232 added branch parameters** trained with the same active loss and schedule. The last added projection starts at zero to preserve parent predictions; the source projection uses full fan-in initialization scale 1.0. Its selected EMA update is 1,000, followed by the same mask calibration. It improves mean test fidelity but loses the combined validation ranking to the linear continuation. Both branches run within one ordinary bridge forward. ## Deployment and source of truth The deployment CLI uses CLIP FP16 with batches padded to 16 to match the cached encoder kernel shape, a FP32 bridge, and BF16 SANA. It supplies predicted positive conditioning and the stored native empty conditioning to the pinned 20-step DPM sampler at guidance 4.5 and 512 × 512. The bridge output is computed once and reused throughout sampling; the original DC-AE decodes the final latent. No Gemma encoder, teacher cache, retrieval, optimizer, or best-of-N is needed by inference. The equations follow [the bridge](https://github.com/255BITS/model-glue/blob/main/model_glue/text_bridge.py), [training and guidance metrics](https://github.com/255BITS/model-glue/blob/main/model_glue/text_train.py), [inference](https://github.com/255BITS/model-glue/blob/main/model_glue/sana_infer.py), and [mask calibration](https://github.com/255BITS/model-glue/blob/main/scripts/calibrate_text_mask.py). Exact model/checkpoint revisions are recorded in the release manifest and experiment JSON.