Jeesup/svd-safety-svdbasis-llama2_7b_chat_n256_s42

SVD-LLM component basis for meta-llama/Llama-2-7b-chat-hf

The full (untruncated) per-projection SVD-LLM decomposition of every one of the 224 linear projections (32 layers x q/k/v/o/gate/up/down) of meta-llama/Llama-2-7b-chat-hf. A compressed checkpoint keeps only the first r components of each matrix; this basis also holds the discarded ones, which is what a recovery (swap) cell puts back. Truncated cells built in this basis: Jeesup/svd-safety-llama2_7b_chat_remove_{20,30,40,50}_seed42 and the svd-safety-l2_* SVD-LLM recovery cells.

W = U diag(sigma) Vtil          Vtil = V^T S^-1,   S = chol(X^T X)  (SVD-LLM whitening)

X is the projection's input over the calibration set: 256 WikiText-2 sequences x 2048 tokens, seed 42. The basis is built by compress/whiten_svd.py exactly as upstream SVD-LLM profiles and whitens, so truncating to the first r components gives the SVD-LLM compressed projection. Components are ordered by sigma descending.

U is (m x k), Vtil (k x n), k = min(m, n), so the components sum back to W exactly: the basis is the base model's weights in SVD-LLM coordinates, which is why it carries the base model's licence.

Files

path	content
basis/basis_<layer>_<proj>.pt	torch.load -> {"U": (m,k) fp32, "Vtil": (k,n) fp32, "sigma": (k,), "rank": int, "shape": (m,n)}
basis/meta.json	model, calibration, per-matrix metadata (rank there is informational only)

import torch
b = torch.load("basis/basis_0_q_proj.pt", map_location="cpu")
W = (b["U"] * b["sigma"]) @ b["Vtil"]          # dense q_proj weight of layer 0
W_r = (b["U"][:, :r] * b["sigma"][:r]) @ b["Vtil"][:r]   # rank-r truncation

Ranks (added 2026-09-22)

path	content
ranks/remove_<NN>.json	{"ratio_removed", "rule", "ranks": {"<layer>.<proj>": r}} for NN = 10..70 % parameters removed

r = int(m n (1 - ratio) / (m + n)) per matrix (SVD-LLM's shape-only rule); keeping components 0..r-1 of this basis reproduces Jeesup/svd-safety-llama2_7b_chat_remove_<NN>_seed42 before finetuning.

Activations (added 2026-09-22)

Dense-model inputs to the projections, one token per prompt, prompt sent as a single user turn with the Llama-2 chat template.

path	content
activations/<corpus>/layer_<L>.pt	{"t_inst": {...}, "t_post": {...}}, each {"attn_in": (N,4096), "o_in": (N,4096), "mlp_in": (N,4096), "down_in": (N,11008)}
activations/<corpus>/meta.json	source, positions, order (row i = prompt order[i] of the source list; rows are length-sorted)

• corpora: jbb_harm (JailbreakBench harmful goals, 100), jbb_benign (JailbreakBench benign goals, 100), wiki_chat (100 WikiText-2 train passages, 20-60 words, seed 42).
• positions: t_inst = last instruction token before [/INST]; t_post = ] of [/INST] (last prompt token).
• groups: attn_in feeds q/k/v, o_in feeds o, mlp_in feeds gate/up, down_in feeds down.

Per-component response to an input x: ||sigma_i u_i Vtil_i x||^2 = sigma_i^2 (Vtil_i . x)^2.

Provenance

No refold record ships with this basis (it has no UNVERIFIED.txt); it is the basis every Llama-2 SVD-LLM recovery cell in the project was built from.

Licence

Derived from meta-llama/Llama-2-7b-chat-hf (a linear re-parameterisation of its weights) and redistributed under that model's licence (llama2); LICENSE and USE_POLICY.md are copied verbatim from the base checkpoint.

Published 2026-09-21 from the svd-safety project. Size ~37-39 GB (224 files).