Add model card

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+banner.png filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,215 @@
+---
+license: mit
+language:
+  - en
+pipeline_tag: text-generation
+library_name: transformers
+base_model: specklabs/Speck2-140M
+datasets:
+  - specklabs/SpeckChat2
+  - openbmb/Ultra-FineWeb
+  - openbmb/Ultra-FineWeb-L3
+  - mlfoundations/dclm-baseline-1.0-parquet
+  - HuggingFaceTB/smollm-corpus
+  - HuggingFaceTB/finemath
+  - wikimedia/wikipedia
+tags:
+  - causal-lm
+  - instruction-tuned
+  - chat
+  - hybrid
+  - grouped-query-attention
+  - causal-convolution
+  - pytorch
+  - transformers
+  - safetensors
+  - custom_code
+---
+
+![speck_instruct_model](./banner.png)
+
+# Speck2-140M-Instruct
+
+Speck2-140M-Instruct is a 140.7M parameter English instruction-tuned language model. It was initialized from [Speck2-140M](https://huggingface.co/specklabs/Speck2-140M), a hybrid model pretrained from scratch on 20B tokens with a three-phase curriculum, then fully fine-tuned for one epoch on the 500,000-conversation [SpeckChat2](https://huggingface.co/datasets/specklabs/SpeckChat2) mixture.
+
+The Speck2 release updates the pretrained base while retaining the SpeckChat2 instruction-tuning data and one-epoch post-training recipe. The model uses a native chat template with optional system messages and was trained with assistant-only loss.
+
+## Summary
+
+| Property | Value |
+|---|---:|
+| Parameters | 140,654,208 |
+| Base pretraining tokens | 20.0B |
+| SFT training conversations | 499,000 |
+| SFT validation conversations | 1,000 |
+| Supervised assistant tokens | 319,181,221 |
+| SFT context tokens processed (padded) | 559,284,224 |
+| Non-padding SFT context tokens | 385,422,209 |
+| SFT sequence lengths | 256, 512, 1,024, 2,048 |
+| Configured max context | 4,096 (unvalidated beyond 2,048) |
+| Vocabulary | 32,003 (Mistral v0.1 SentencePiece + 3 role tokens) |
+| Release format | BF16 Safetensors |
+| Final SFT validation loss / perplexity | 1.1206 / 3.067 |
+| CPU decode, batch 1 | 55.1 tok/s (Speck2 architecture proxy) |
+| RTX 3090 decode, batch 1 | 247.3 tok/s (Speck2 architecture proxy) |
+
+## Architecture
+
+The architecture is unchanged from Speck2-140M apart from 3 added role-token embeddings. It contains 18 residual blocks: 8 global attention + 10 gated causal convolution, each followed by a SwiGLU feed-forward.
+
+| Component | Value |
+|---|---:|
+| Hidden width | 768 |
+| Embedding width | 640 |
+| SwiGLU intermediate | 2,304 |
+| Attention heads (Q / KV) | 12 / 3 |
+| Head dimension | 64 |
+| Conv inner width | 384 |
+| Conv kernel sizes | 3, 5 |
+| RoPE theta | 10,000 |
+| RMSNorm epsilon | 1e-5 |
+
+Input/output embeddings (640-wide) are tied and connect to the 768-wide residual stream through learned projections.
+
+## Chat template
+
+Three special tokens were added to the base 32k vocabulary:
+
+| Role | Token | ID |
+|---|---|---:|
+| System | `<|system|>` | 32,000 |
+| User | `<|user|>` | 32,001 |
+| Assistant | `<|assistant|>` | 32,002 |
+
+A rendered conversation has the following form:
+
+```text
+<s><|system|>
+{optional system message}</s>
+<|user|>
+{user message}</s>
+<|assistant|>
+{assistant response}</s>
+```
+
+The system message is optional and may appear only first. Remaining roles must alternate between user and assistant. No default system prompt is injected. During generation, the prompt ends after `<|assistant|>\n`, and `</s>` terminates the response.
+
+## Usage
+
+Speck2-140M-Instruct works with the Transformers Auto classes through its bundled custom model and tokenizer code. Set `trust_remote_code=True` when loading it.
+
+```bash
+pip install "transformers==5.1.0" torch sentencepiece safetensors
+```
+
+```python
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "specklabs/Speck2-140M-Instruct"
+device = "cuda" if torch.cuda.is_available() and torch.cuda.is_bf16_supported() else "cpu"
+dtype = torch.bfloat16 if device == "cuda" else torch.float32
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(
+    model_id,
+    trust_remote_code=True,
+    dtype=dtype,
+).to(device)
+
+messages = [
+    {"role": "system", "content": "You are a concise assistant."},
+    {"role": "user", "content": "What is the capital of France?"},
+]
+
+encoded = tokenizer.apply_chat_template(
+    messages,
+    tokenize=True,
+    add_generation_prompt=True,
+    return_tensors="pt",
+)
+input_ids = encoded["input_ids"] if hasattr(encoded, "keys") else encoded
+input_ids = input_ids.to(device)
+
+output = model.generate(
+    input_ids=input_ids,
+    attention_mask=torch.ones_like(input_ids),
+    max_new_tokens=64,
+    do_sample=False,
+    pad_token_id=tokenizer.eos_token_id,
+)
+generated = output[0, input_ids.shape[1] :]
+print(tokenizer.decode(generated, skip_special_tokens=True))
+```
+
+The bundled generation path is validated for single-prompt greedy decoding. Direct forward passes support right-padded batches when `use_cache=False`.
+
+## Evaluation
+
+The quality columns combine the
+[Open SLM Leaderboard](https://huggingface.co/spaces/AxiomicLabs/Open_SLM_Leaderboard)
+at revision `2eafcfc647b667e67f3b0288e9b67da497a78052` and
+[BananaMind Base Bench 1.1](https://huggingface.co/datasets/BananaMind/BananaMind-Base-Bench-1.1)
+at revision `d4aade51312889e8580963e1ce960c6eaef1a450`. No chat template or generation was used for the six
+completed Speck evaluations. Speck2-140M-Instruct has not yet been evaluated on either benchmark.
+
+### Benchmarks and speed
+
+| Model | Params | Training tokens | Open SLM Int Index | BananaMind Base Bench 1.1 Elo | CPU prefill | CPU decode | RTX 3090 prefill | RTX 3090 decode | BF16 memory @2K | BF16 state @2K |
+|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|
+| BananaMind-2-Pro | 139M | 100B | 24.96 | 1131 | 2,190 tok/s | 43.0 tok/s | 64,060 tok/s | 140.3 tok/s | 325.1 MiB | 60.0 MiB |
+| SmolLM2-135M | 135M | ~2T | 27.13 | 1119 | 2,201 tok/s | 47.4 tok/s | 64,814 tok/s | 157.7 tok/s | 301.6 MiB | 45.0 MiB |
+| GPT-X2.5-135M | 135M | 75B | 25.17 | 1106 | 2,042 tok/s | 47.2 tok/s | 55,346 tok/s | 125.0 tok/s | 302.6 MiB | 45.0 MiB |
+| Supra2-100M-Base | 101M | 30B | 19.41 | 1030 | **3,362 tok/s** | **56.0 tok/s** | **113,326 tok/s** | **298.1 tok/s** | **216.0 MiB** | 24.0 MiB |
+| **Speck1-140M** | **141M** | **5B** | **18.15** | **965** | 2,252 tok/s | 55.1 tok/s | 74,323 tok/s | 247.3 tok/s | 281.3 MiB | **12.0 MiB** |
+| **Speck1-140M-Instruct** | **141M** | **5B + 317M SFT** | **17.75** | **1001** | 2,285 tok/s | 55.3 tok/s | 73,398 tok/s | 246.7 tok/s | 280.3 MiB | **12.0 MiB** |
+| **Speck1.1-140M-Instruct** | **141M** | **5B + 559M SFT** | **17.90** | **1002** | 2,315 tok/s | **56.9 tok/s** | 74,941 tok/s | 243.6 tok/s | 280.3 MiB | **12.0 MiB** |
+| **Speck1.5-140M** | **141M** | **5B** | **16.61** | **980** | 2,252 tok/s | 55.1 tok/s | 74,323 tok/s | 247.3 tok/s | 281.3 MiB | **12.0 MiB** |
+| **Speck1.5-140M-Instruct** | **141M** | **5B + 559M SFT** | **17.48** | **996** | 2,394 tok/s | 56.6 tok/s | 75,424 tok/s | 242.7 tok/s | 280.3 MiB | **12.0 MiB** |
+| **Speck2-140M** | **141M** | **20B** | **20.01** | **953** | 2,252 tok/s | 55.1 tok/s | 74,323 tok/s | 247.3 tok/s | 281.3 MiB | **12.0 MiB** |
+| **Speck2-140M-Instruct** | **141M** | **20B + 559M SFT** | **—** | **—** | 2,252 tok/s* | 55.1 tok/s* | 74,323 tok/s* | 247.3 tok/s* | 280.3 MiB* | **12.0 MiB*** |
+
+`Open SLM Int Index` means the chance-normalized Intelligence Index reported by the Open SLM
+Leaderboard. `BananaMind Base Bench 1.1 Elo` means the overall Elo reported by BananaMind Base
+Bench 1.1. Speed and memory values are local batch-1 measurements described below. Reference
+models saw 1.5-100x more pretraining tokens, so this is a parameter-adjacent comparison, not a
+compute-matched one. Asterisks mark architecture-equivalent Speck2 measurements rather than new
+checkpoint-specific measurements.
+
+## Inference speed
+
+Speed was measured locally at batch 1 with eager PyTorch, model-native caches, last-token logits, and tokenization excluded. Prefill uses 512 tokens. Decode measures 64 greedy cached steps after a 448-token prefix and includes argmax. CPU runs use FP32 with 16 threads; RTX 3090 runs use BF16. Reported throughput is calculated from the median duration.
+
+Speck2-140M-Instruct has the same architecture and release implementation as Speck2-140M apart from 3 added role-token embeddings, so its speed and memory entries use the validated Speck2 measurements. The final instruct export was separately validated for Transformers Auto-class loading and single-prompt greedy generation.
+
+Memory is unique live BF16 model tensor storage plus cache/state tensor storage after a 2,048-token prefill at batch 1. Speck2-140M-Instruct uses 268.3 MiB for unique model tensors and 12.0 MiB for state, or 280.3 MiB combined. This excludes framework RSS, CUDA allocator reservations, and temporary operator workspace. FP32 CPU tensor memory is approximately twice the reported BF16 model-tensor memory. For another context length `N`, approximate memory as `model tensor memory + State@2K x N / 2,048`; Speck's small convolution history is fixed rather than context-scaled. The BF16 Safetensors weight file is 281.3 MB.
+
+## Limitations
+
+- At 140.7M parameters, the model has limited knowledge, reasoning, coding, and multilingual capacity.
+- Instruction following is inconsistent, especially for exact formatting, arithmetic, strict brevity, and system-prompt constraints.
+- Longer generations can become repetitive or incoherent.
+- It can hallucinate facts and produce incorrect code or calculations.
+- Mostly English: multilingual ability is weak and unvalidated.
+- No dedicated safety alignment, red-team evaluation, or misuse evaluation was performed.
+- Validated only up to 2,048 tokens despite a 4,096-token config.
+- Pretraining and instruction data are web-derived, user-authored, or synthetic and may contain bias, errors, unsafe content, personal information, or copyrighted text.
+- The bundled Transformers generation path currently supports single-prompt greedy decoding only.
+
+## Reproducibility
+
+Full training and evaluation code: [github.com/alkinun/speck](https://github.com/alkinun/speck)
+
+The released checkpoint is training step 8,534. The base model revision is `1201df613d9ee9d50909189f52e45c6fcefa3c01`; the SpeckChat2 revision is `7b497b3e0c7f4653278cc67af27722b20a5c8d10`.
+
+## Citation
+
+```bibtex
+@misc{alkinun2026speck2instruct,
+  author       = {alkinun},
+  title        = {Speck2-140M-Instruct: Instruction Tuning a Compact Hybrid Attention-Convolution Language Model},
+  year         = {2026},
+  howpublished = {\url{https://huggingface.co/specklabs/Speck2-140M-Instruct}},
+  url          = {https://github.com/alkinun/speck}
+}
+```