--- license: mit language: en library_name: transformers pipeline_tag: text-generation tags: - tiny-model - llama - from-scratch - model-growth - conversational - multi-turn - tool-use - agent-harness - retrieval-augmented - multi-hop-qa - question-answering - attribution - humble-ai - small-language-model - muon - gguf - text-generation-inference widget: - text: "\n\nwho are you\n<|eot|>\n\n" example_title: "Identity" - text: "\n\nwho wrote dracula\n<|eot|>\n\n" example_title: "Reads real prose" - text: "\n\nwhat did i eat yesterday\n<|eot|>\n\n" example_title: "Limits" ---
Loom Tapestry 3
# Loom Tapestry 3 **69.2M parameters · 40 layers · 1024 context · Textile Labs** The best Loom yet, and the first one **grown**. We took our 31.5M [Loom Weave 3](https://huggingface.co/textilelabs/Loom-Weave-3), **stacked its 16 layers into 40** (function-preserving, so the bigger model started exactly where Weave 3 left off), and continue-trained it for two hours on a richer corpus. It reads live prose, reasons across two facts (multi-hop), and stays a disciplined Loom — it never turned into a story-generator. Trained from-scratch lineage (no third-party checkpoint), on dual-T4 gpus in ~2 hours. Runs offline via Ollama; llama.cpp-compatible. ``` you which magazine was started first, Arthur's or First for Women? Loom Tapestry 3 arthur's magazine harness ← Arthur's Magazine was first published in 1844. First for Women started in 1989. Loom Tapestry 3 Arthur's Magazine. I had to look that up. ``` ## How it was grown Cold-training a big model on little data fails (undertrained). Instead we **grew** a trained small one: interleave its 16 layers among 40 positions, initialise the 24 new layers as no-ops (zeroed output projections), so the 40-layer model is **identical** to Weave 3 at step 0 — then it improves from there. Init loss was 2.2, not 9.7. All two hours went into getting better, not starting over. ## Measured behaviour vs the family (same harness, 2026-09) | model | params | battery /133 | live search (held-out) | reads real prose | |---|---:|---:|---:|---| | Loom Tapestry 2 | 22.8M | 107 (80.5%) | — | curated only | | Loom Spark 3 Flash | 7.18M | 119 (89.5%) | 5/20 | curated only | | Loom Spark 3 | 12.2M | 120 (90.2%) | 7/20 | curated only | | Loom Weave 3 | 31.5M | 120 (90.2%) | 6/20 | yes | | **Loom Tapestry 3** | **69.2M** | **123 (92.5%)** | **12/20** | **yes + multi-hop** | **The acceptance battery, row by row:** | row | Loom Tapestry 3 | |---|---:| | A · says its own name | **12/12** | | B · its own name under rough typing | 11/12 | | C · 5-turn conversation stays on thread | 5/5 | | D · answers from a search result | **5/5** | | E · follow-up from the same result | 2/5 | | F · says it looked, after a lookup | 5/5 | | G · **never** claims a lookup it didn't make | **16/16** | | H · admits what it can't know about you | **8/8** | | I · says when a result doesn't contain the answer | 2/5 | | J · never leaks a search tag with tools off | 28/28 | | K · stops on its own | 12/12 | | L · searches when it should, not for your private things | 17/20 | | **total** | **123/133** | **End to end**, everyday questions it had never seen, live Wikipedia, model writing its own query, scored on the final answer: **tuning 12/20 (60%), held-out 12/20 (60%)** — double Weave 3's held-out reading. ## Read this before you use it Every point measured. - **"I looked that up" means it searched — not that it read perfectly.** Held-out reading is ~60%; run `harness.py --show` and trust the sentence it read. - **Follow-up questions from the same result are weak (~40%).** The family's oldest gap. - **It is not a calculator.** Arithmetic is out of scope; use a tool. - **It rarely asks a clarifying question** on an ambiguous request. - **Harness search is Wikipedia only** — no time, weather, news or prices. - **It is a small reader/assistant, not a chat stylist.** Voice is plain and brief by design. What it does reliably: knows what it is, **stops on its own**, holds a conversation, **admits what it can't know (8/8)**, **never claims a lookup it didn't make (16/16)**, reads a supplied result (5/5), and reasons across two facts. ## Usage — the harness ```bash python3 harness.py "who wrote dracula" python3 harness.py --show "how tall is mount everest" python3 harness.py --no-tools "who are you" ``` Stdlib + `certifi`. Wikipedia needs no API key. ## Usage — Ollama ```bash ollama run hf.co/textilelabs/Loom-Tapestry-3 "who are you" ``` `template` and `params` are read automatically. ## Usage — transformers ```python import torch from transformers import AutoTokenizer, AutoModelForCausalLM tok = AutoTokenizer.from_pretrained("textilelabs/Loom-Tapestry-3") model = AutoModelForCausalLM.from_pretrained("textilelabs/Loom-Tapestry-3").eval() eot = tok.convert_tokens_to_ids("<|eot|>") def ask(message, tools=False): p = f"\n\n{message}\n<|eot|>\n\n" ids = tok(p, return_tensors="pt", add_special_tokens=False).input_ids with torch.no_grad(): out = model.generate(ids, max_new_tokens=64, do_sample=False, eos_token_id=eot, pad_token_id=tok.convert_tokens_to_ids("<|pad|>"))[0] return tok.decode(out[ids.shape[1]:], skip_special_tokens=False).replace("<|eot|>","").strip() ``` Prompt format is exact: `\n\n{message}\n<|eot|>\n\n`. ## How it was built | | | |---|---| | architecture | Llama — 40 layers × 384d, GQA (6 heads / 2 KV), SwiGLU, RoPE, tied embeddings | | parameters | 69,237,120 | | grown from | Loom Weave 3 (16 layers) via function-preserving layer stacking | | context | 1,024 | | vocabulary | 16,384 custom BPE | | optimiser | Muon (LR 0.025) on 2D hidden matrices, AdamW on embeddings and norms | | loss | masked to the reply tokens; whole conversations packed per block (FFD) | | corpus | Loom curriculum + HotpotQA multi-hop + unanswerable rows (real prose in results) | | training | ~500 steps · ~67M tokens · best-validation checkpoint · from random-init lineage | | hardware | Kaggle dual T4 (GPU) · ~2 hours | ## Files ``` config.json / model.safetensors the model tokenizer.json / tokenizer_config.json custom BPE tokenizer, 16,384 tokens loom-tapestry-3-f16.gguf for Ollama / llama.cpp (lookup tags USER_DEFINED) harness.py runnable search harness template / params read automatically by `ollama run hf.co/...` Modelfile for building locally ATTRIBUTION.md required credits for the training corpora ``` ## License Model: MIT. Training data retains its original licences and attribution.