--- license: mit language: en library_name: transformers pipeline_tag: text-generation tags: - tiny-model - llama - from-scratch - model-growth - conversational - multi-turn - tool-use - agent-harness - retrieval-augmented - question-answering - unanswerable-questions - attribution - humble-ai - small-language-model - muon - gguf - text-generation-inference widget: - text: "\n\nwho are you\n<|eot|>\n\n" example_title: "Identity" - text: "\n\nwho wrote dracula\n<|eot|>\n\n" example_title: "Reads real prose" - text: "\n\nwhat did i eat yesterday\n<|eot|>\n\n" example_title: "Limits" ---
Loom Crucible Preview
# Loom Crucible Preview **155.0M parameters · 52 layers · 1024 context · Textile Labs** The biggest Loom by far, and a **preview** of how far our growth method can push a small, disciplined model. We took [Loom Tapestry 3](https://huggingface.co/textilelabs/Loom-Tapestry-3) (69.2M) and **grew it in both directions at once**, wider (384 → 512) and deeper (40 → 52 layers). The growth is function-preserving: the 155M model started out producing exactly what Tapestry 3 produced, then trained for 12 hours on a rebuilt corpus. It calls itself **Loom Crucible**. It reads what a search brings back and answers from it, says when it looked something up, says when it can't know something about you, and stays a Loom: plain, brief, and not a story generator. Trained from-scratch lineage (no third-party checkpoint), on a Kaggle dual-T4. Runs offline via Ollama; llama.cpp-compatible. ``` you who wrote dracula Loom Crucible wrote dracula harness ← Dracula is an 1897 Gothic horror novel by Irish author Bram Stoker. Loom Crucible Bram Stoker. I had to look that up. ``` ## What "Preview" means Crucible clearly beats every earlier Loom on its core skills: **reading a passage correctly (9/10 vs Tapestry 3's 4/10)** and the **acceptance battery (125/133)**. It is **not yet better end to end on live Wikipedia (10/20 held-out vs Tapestry 3's 12/20)**. On that test the limit is now the search step, not the reading step: when the search found the answer, both models got it right about 85% of the time. A better harness is the next release, and it helps every Loom. ## How it was grown Cold-training a big model on little data fails. Instead we grow a trained small one: - **Wider, 384 → 512.** New channels start at zero so they cannot disturb the old ones. Two details made this exact rather than approximate. First, the 2 new attention heads per layer are placed *inside* their key/value groups, because appending them silently re-pairs the old heads with the wrong keys. Second, every RMSNorm weight is scaled by √(384/512) and ε by 384/512, because zero-padding changes the norm. - **Deeper, 40 → 52.** 12 new layers are inserted as no-ops (zeroed output projections). At step 0 the 155M model matched Tapestry 3 to 3e-4 in the logits and 1e-6 in the loss. All twelve hours went into getting better, not starting over. ## Measured behaviour **The family, same harness, 2026-09:** | model | params | battery /133 | live search (held-out) | reads real prose | |---|---:|---:|---:|---| | Loom Tapestry 2 | 22.8M | 107 (80.5%) | — | curated only | | Loom Tapestry 3 Flash | 7.18M | 112 (84.2%) | 3/20 | curated only | | Loom Spark 3 Flash | 7.18M | 119 (89.5%) | 5/20 | curated only | | Loom Spark 3 | 12.2M | 120 (90.2%) | 7/20 | curated only | | Loom Weave 3 | 31.5M | 120 (90.2%) | 6/20 | yes | | Loom Tapestry 3 | 69.2M | 123 (92.5%) | **12/20** | yes + multi-hop | | **Loom Crucible Preview** | **155.0M** | **125 (94.0%)†** | 10/20 | **yes, best reader** | † Crucible's score is the first on a fully clean held-out battery. While building its corpus we found that some battery prompts had leaked into the training data used since Tapestry 3. We removed them for Crucible, so earlier scores are somewhat optimistic by comparison. **The acceptance battery, row by row:** | row | Tapestry 3 | **Crucible** | |---|---:|---:| | A · says its own name | 12/12 | **12/12** | | B · its own name under rough typing | 11/12 | 11/12 | | C · 5-turn conversation stays on thread | 5/5 | 5/5 | | D · answers from a search result | 5/5 | 4/5 | | E · follow-up from the same result | 2/5 | **5/5** | | F · says it looked, after a lookup | 5/5 | 4/5 | | G · **never** claims a lookup it didn't make | 16/16 | **16/16** | | H · admits what it can't know about you | 8/8 | 6/8 | | I · says when a result doesn't contain the answer | 2/5 | **3/5** | | J · never leaks a search tag with tools off | 28/28 | 28/28 | | K · stops on its own | 12/12 | 12/12 | | L · searches when it should, not for your private things | 17/20 | **19/20** | | **total** | **123/133** | **125/133** | **Held-out reading bench** (hand-written passages with distractors, e.g. a country's capital next to its largest city, a start year next to an end year; none in any training data): | | Tapestry 3 | **Crucible** | |---|---:|---:| | answers correctly from the passage | 4/10 | **9/10** | | says so when an on-topic passage lacks the answer | 0/10 | **3/10** | | says so when the passage is off-topic | 0/8 | **6/8** | | refuses a passage that *does* hold the answer | 0 | **0** | | with tools off, won't answer a famous fact from memory | 3/8 | 3/8 | **End to end**, everyday questions it had never seen, live Wikipedia, model writing its own query, scored on the final answer: **tuning 11/20, held-out 10/20** (Tapestry 3: 12/20, 12/20). When the search found the answer, Crucible got it right 10 times out of 12. ## Read this before you use it Every point measured. - **It is not better than Tapestry 3 end to end yet.** It reads better, but live results are capped by the search step. Run `harness.py --show` to see the sentence it read. - **When a search comes back without the answer, it usually answers anyway.** Given an on-topic passage that lacks the fact, it says so only 3 times in 10. If the passage holds something of the right *type* (a number for "how tall", a year for "when did it end"), it tends to grab that. Trust the sentence shown by `--show`, not just the reply. - **With tools off, it sometimes answers famous facts from half-memory** (3/8 held). Those answers are often wrong. Keep tools on for facts. - **It once refused a passage that did hold the answer** (1 of 5 battery reading rows). - **It is not a calculator.** Arithmetic is out of scope; use a tool. - **Harness search is Wikipedia only**: no time, weather, news or prices. - **Plain, brief voice by design.** It is a small reader and assistant, not a chat stylist. What it does reliably: knows what it is, **stops on its own**, holds a conversation, **answers follow-ups from the same result (5/5)**, **never claims a lookup it didn't make (16/16)**, and **reads a supplied passage better than any earlier Loom**. ## Usage — the harness ```bash python3 harness.py "who wrote dracula" python3 harness.py --show "how tall is ben nevis" python3 harness.py --no-tools "who are you" ``` Stdlib + `certifi`. Wikipedia needs no API key. ## Usage — Ollama ```bash ollama run hf.co/textilelabs/Loom-Crucible-Preview "who are you" ``` `template` and `params` are read automatically. ## Usage — transformers ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM tok = AutoTokenizer.from_pretrained("textilelabs/Loom-Crucible-Preview") model = AutoModelForCausalLM.from_pretrained("textilelabs/Loom-Crucible-Preview").eval() eot = tok.convert_tokens_to_ids("<|eot|>") def ask(message, tools=False): p = f"\n\n{message}\n<|eot|>\n\n" ids = tok(p, return_tensors="pt", add_special_tokens=False).input_ids with torch.no_grad(): out = model.generate(ids, max_new_tokens=64, do_sample=False, eos_token_id=eot, pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))[0] return tok.decode(out[ids.shape[1]:], skip_special_tokens=False).replace("<|eot|>","").strip() ``` Prompt format is exact: `\n\n{message}\n<|eot|>\n\n`. ## How it was built | | | |---|---| | architecture | Llama — 52 layers × 512d, GQA (8 heads / 2 KV), SwiGLU, RoPE, tied embeddings | | parameters | 154,980,864 | | grown from | Loom Tapestry 3 (40 × 384) by function-preserving width + depth growth | | context | 1,024 | | vocabulary | 16,384 custom BPE | | optimiser | Muon (LR 0.025) on 2D hidden matrices, AdamW on embeddings and norms | | loss | masked to the reply tokens; whole conversations packed per block | | corpus | 275,788 conversations: Loom curriculum + SQuAD 2.0 (incl. unanswerable) + HotpotQA + TriviaQA | | training | 6 h grown from Tapestry 3, then 6 h continued on the rebuilt corpus · best-validation checkpoint | | hardware | Kaggle dual T4 (GPU) | | decoding | ship with `repeat_penalty 1.0`: higher values corrupt copying from the passage | ## Files ``` config.json / model.safetensors the model tokenizer.json / tokenizer_config.json custom BPE tokenizer, 16,384 tokens loom-crucible-preview-f16.gguf for Ollama / llama.cpp (lookup tags USER_DEFINED) harness.py runnable search harness template / params read automatically by `ollama run hf.co/...` Modelfile for building locally ATTRIBUTION.md required credits for the training corpora ``` ## License Model: MIT. Training data retains its original licences and attribution.