Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,79 @@
+---
+license: apache-2.0
+base_model: poolside/Laguna-XS.2
+pipeline_tag: text-generation
+library_name: transformers
+tags: [moe-to-dense, densification, laguna, cuda, kernels, sft, code]
+language: [en]
+---
+
+# Laguna-XS.2 → Dense (K=8) · **CUDA-SFT** (kernel generation)
+
+A **~3.0 B dense** model that turns PyTorch modules into **CUDA kernels**. Lineage:
+**[poolside/Laguna-XS.2](https://huggingface.co/poolside/Laguna-XS.2)** MoE → densify → DO-ACP warm-start →
+[kernel-mixture reconstruction (V2)](https://huggingface.co/EvanOLeary/laguna-xs2-dense-k8-kernelmix)
+→ **SFT on `SakanaAI/AI-CUDA-Engineer-Archive`** (this model). Sibling:
+[OpenCodeInstruct/Python flavour](https://huggingface.co/EvanOLeary/laguna-xs2-dense-k8-recon).
+
+> SFT-stage checkpoint. Generates CUDA in chat format; **not yet RL-optimized for speedup** (that's the next RFT stage).
+
+## It works — sample output
+**Prompt** (chat): "Convert this PyTorch module into an optimized CUDA kernel" + a ReLU `nn.Module` → **model returns a correct CUDA kernel:**
+```cpp
+__global__ void relu_kernel(const float* __restrict__ input, float* __restrict__ output, const int64_t size) {
+    const int idx = blockIdx.x * blockDim.x + threadIdx.x;
+    if (idx < size) { float v = input[idx]; output[idx] = v > 0 ? v : 0; }
+}
+torch::Tensor forward(torch::Tensor input) {
+    auto output = torch::empty_like(input);
+    const int threads = 256, blocks = (input.numel()+threads-1)/threads;
+    relu_kernel<<<blocks, threads>>>(input.data_ptr<float>(), output.data_ptr<float>(), input.numel());
+    return output;
+}
+```
+(Correct `__global__`, bounds check, ReLU logic, torch-extension wrapper. Chat format restored vs the pretrained checkpoint.)
+
+## How to use
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+repo="EvanOLeary/laguna-xs2-dense-k8-cuda-sft"
+tok=AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
+m=AutoModelForCausalLM.from_pretrained(repo, trust_remote_code=True, torch_dtype=torch.bfloat16, device_map="cuda")
+SYS="You are an expert GPU kernel engineer. Convert PyTorch modules into correct, optimized CUDA kernels."
+user="Convert this PyTorch module into an optimized CUDA kernel:\n\n```python\n<your nn.Module>\n```"
+s=tok.apply_chat_template([{"role":"system","content":SYS},{"role":"user","content":user}], add_generation_prompt=True, tokenize=False)
+ids=tok(s, add_special_tokens=False, return_tensors="pt").input_ids.to(m.device)
+print(tok.decode(m.generate(ids, max_new_tokens=512, do_sample=True, temperature=0.7, top_k=20, pad_token_id=9)[0][ids.shape[-1]:], skip_special_tokens=True))
+```
+
+## Training recipe (SFT)
+| | |
+|---|---|
+| Base | V2 kernel-mixture reconstruction checkpoint (this repo's parent) |
+| Data | `SakanaAI/AI-CUDA-Engineer-Archive` (level_1+2, **correct kernels only**), PyTorch→CUDA, chat-formatted, **prompt masked** |
+| Objective | causal-LM cross-entropy on the CUDA completion only |
+| Trainable | `routed_dense` + `lm_head` + norms (1.19 B); attention frozen |
+| Optimizer | AdamW 2e-... lr **1e-5**, grad-clip 1.0, grad-accum 8 (eff-batch 8), seq 2048 |
+| Steps | 400 (~3,200 CUDA examples), ~21 min, 1× H100 (27 GB) |
+| Result | **CE 0.68 → 0.32** |
+
+![SFT curve](https://huggingface.co/EvanOLeary/laguna-xs2-dense-k8-cuda-sft/resolve/main/sft_curve.png)
+
+## Training data
+`SakanaAI/AI-CUDA-Engineer-Archive` — ~30k verified PyTorch→CUDA kernel pairs from Sakana's AI CUDA
+Engineer (CC-BY-4.0). We filter to `Correct==True` and format as `system + user(PyTorch) → assistant(CUDA)`.
+
+## Intended use
+Research: generate CUDA kernels from PyTorch modules; a base for **RFT** (RL on verified speedup). Not production — kernels are not yet correctness/perf-verified at generation time.
+
+## Next steps
+1. **RFT (GRPO/RLVR)** — sample K kernels/prompt, reward = **compile + correct + speedup** via
+   [`SakanaAI/robust-kbench`](https://github.com/SakanaAI/robust-kbench); optimize the real metric.
+2. **Benchmark** — **KernelBench** (`fast_0` correctness, `fast_1` correct-and-faster) vs teacher.
+3. **NVFP4 quantize → vLLM serve** as a `generate_kernel` tool.
+
+## Limitations
+SFT only — emits plausible CUDA (simple ops correct) but **not guaranteed to compile/be fast**; RFT + KernelBench verification come next. Add `winglian/cuda-engineer-augment` reasoning data for CoT.
+
+*Refs: RADLADS 2505.03005 · MoE→Dense 2605.28207 · Sakana AI CUDA Engineer / robust-kbench 2509.14279 · KernelBench.*