Publish trained tiny LLM base demo-v1 (full fp32, epoch 2)

README.md

@@ -0,0 +1,107 @@
+---
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- advai-notebooks
+- llm-base
+- demo
+- synthetic-data
+- random-initialization
+- full
+- llama
+language:
+- en
+---
+
+# widebluesky/wbs-llm-base-demo
+
+## Demo release v1
+
+This is a **trained tiny demo model** for testing training, export, publication,
+download and inference. A tiny Llama architecture was initialized randomly
+and trained for **two epochs** on synthetic fixtures from `advai-notebooks`.
+It is not derived from Meta Llama weights or another pretrained checkpoint.
+It is a workflow placeholder for future domain-trained releases.
+The model has no demonstrated general language or instruction-following capability.
+Generation may be repetitive or incorrect. Synthetic metrics do not establish domain quality.
+
+## Model and training
+
+| Setting | Value |
+| --- | --- |
+| Architecture | LlamaForCausalLM; random initialization |
+| Parameters | 13,248 |
+| Layers / hidden size / FFN size | 1 / 32 / 64 |
+| Attention heads / key-value heads | 4 / 2 |
+| Vocabulary / context length | 123 tokens / 128 tokens |
+| Training objective | next-token causal language modeling over document text (continued pre-training) |
+| Method | Full-parameter training; not a LoRA adapter |
+| Epochs completed / selected | 2 / 2 |
+| Selection | Lowest validation token-weighted NLL, including epoch zero; test is report-only |
+| Dataset | 8 training, 2 validation and 2 test synthetic examples |
+| Optimizer / learning rate / weight decay | AdamW / 0.003 / 0.01 |
+| Batch size / gradient accumulation | 2 / 2 |
+| Scheduler / gradient clipping | Constant / 1.0 |
+| Seed / device / precision | 42 / CPU / fp32 |
+| Export | Standalone full safetensors model |
+| SDK versions | Transformers 5.18.0; PyTorch 2.14.1 |
+
+Training supervises the next token in each document, including its EOS marker.
+Padding tokens are masked from the loss. A chat template is included as tokenizer
+metadata for compatibility; the Base training objective does not teach chat behavior.
+
+## Synthetic held-out test metrics
+
+| Metric | Before training | Selected trained export |
+| --- | ---: | ---: |
+| nll | 4.792373 | 4.695202 |
+| perplexity | 120.587147 | 109.420896 |
+
+NLL and perplexity are computed over each task's supervised tokens.
+Base and Instruct perplexity values are not directly comparable because their
+supervision ranges differ. These results use only two synthetic test examples.
+
+## Inference
+
+In a checkout of `advai-notebooks`:
+
+```bash
+python -m pip install -e '.[llm-base]'
+```
+
+```python
+from advai_notebooks.llm.base.model import BaseModel
+
+model = BaseModel.load("widebluesky/wbs-llm-base-demo", revision="demo-v1", device="cpu")
+outputs = model.generate(["the sky is"], max_new_tokens=8)
+print(outputs)
+```
+
+The full model also supports native Hugging Face loading:
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+repo_id = "widebluesky/wbs-llm-base-demo"
+model = AutoModelForCausalLM.from_pretrained(repo_id, revision="demo-v1")
+tokenizer = AutoTokenizer.from_pretrained(repo_id, revision="demo-v1")
+```
+
+For Instruct inference, render the exported chat template with an assistant generation
+For Base inference, tokenize the document prompt and generate a continuation.
+Pin the publication commit SHA in `revision` for reproducible deployment.
+
+## Verification and package contents
+
+- All 11 stored weight tensors changed from the random baseline.
+- Exported weights exactly match the validation-selected trained checkpoint.
+- Fresh-process reload reproduced test metrics exactly.
+- Native Hugging Face loading and project facade loading produced identical logits.
+- CLI generation/chat and facade API outputs matched.
+- File sizes and SHA256 hashes are recorded in `publish_manifest.json`.
+
+The package contains inference weights, architecture and generation configuration,
+tokenizer, chat template, task facade metadata, this English model card and a manifest.
+Raw data, optimizer state, resumable training checkpoints, logs and credentials are excluded.
+
+Source implementation commit: `f43a44e2431bae9d0451b445cc3bde5f1e2b8493`.

advai_metadata.json

@@ -0,0 +1,22 @@
+{
+  "schema_version": 1,
+  "kind": "llm/base",
+  "format": "full",
+  "model": {
+    "id": "tiny",
+    "revision": null,
+    "max_len": 128,
+    "chat_template": null
+  },
+  "source": {
+    "id": "tiny",
+    "revision": null
+  },
+  "base_model": null,
+  "generation": {
+    "max_new_tokens": 8,
+    "do_sample": false,
+    "temperature": 1.0,
+    "top_p": 1.0
+  }
+}

chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{{ '<|' + message['role'] + '|>\n' }}{% if message['role'] == 'assistant' %}{% generation %}{{ message['content'] + '<|end|>\n' }}{% endgeneration %}{% else %}{{ message['content'] + '<|end|>\n' }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ '<|assistant|>\n' }}{% endif %}

config.json

@@ -0,0 +1,32 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 2,
+  "dtype": "float32",
+  "eos_token_id": 3,
+  "head_dim": 8,
+  "hidden_act": "silu",
+  "hidden_size": 32,
+  "initializer_range": 0.02,
+  "intermediate_size": 64,
+  "max_position_embeddings": 128,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 4,
+  "num_hidden_layers": 1,
+  "num_key_value_heads": 2,
+  "pad_token_id": 0,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-06,
+  "rope_parameters": {
+    "rope_theta": 10000.0,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers_version": "5.18.0",
+  "use_cache": false,
+  "vocab_size": 123
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 2,
+  "eos_token_id": [
+    3,
+    7
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.18.0",
+  "use_cache": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:35bfeab0f6962e04ef35bb8c9cda4a01f5e9ae9228ae194dd023c682e2a504b6
+size 54104

publish_manifest.json

@@ -0,0 +1,46 @@
+{
+  "schema_version": 1,
+  "task": "llm/base",
+  "files": [
+    {
+      "path": "README.md",
+      "bytes": 4108,
+      "sha256": "3dc3ae8c9a46740a4b3bf1f0dcfafd3135e6bff8e997771df22949ca1f3feadc"
+    },
+    {
+      "path": "advai_metadata.json",
+      "bytes": 365,
+      "sha256": "0e8f8360f8f90b0496fdaf20aa79feef37e89efa4b81ab9ebdb23b53ce8ec837"
+    },
+    {
+      "path": "chat_template.jinja",
+      "bytes": 313,
+      "sha256": "3fd481918ea6c2894de967ff3f15b13092db403ba035422e44228166e17665af"
+    },
+    {
+      "path": "config.json",
+      "bytes": 712,
+      "sha256": "a5dbd336ebe6691eb28df7fcd31211f20121104adcc9d36c1cc78f2d90cac3f9"
+    },
+    {
+      "path": "generation_config.json",
+      "bytes": 233,
+      "sha256": "a6274a69494f96eff68d35f00ea56a6ee6ba54b05244f5091593e2a8612328ef"
+    },
+    {
+      "path": "model.safetensors",
+      "bytes": 54104,
+      "sha256": "35bfeab0f6962e04ef35bb8c9cda4a01f5e9ae9228ae194dd023c682e2a504b6"
+    },
+    {
+      "path": "tokenizer.json",
+      "bytes": 4412,
+      "sha256": "445ba332356fb579e90ef463fbceb90a9bdc151a88b8e5c187901db84cbf2048"
+    },
+    {
+      "path": "tokenizer_config.json",
+      "bytes": 346,
+      "sha256": "e36c2dcb7a856d0c7a55e56f780ac356a1da4b07ed4a16875db39b24f4595e1f"
+    }
+  ]
+}

tokenizer.json

@@ -0,0 +1,239 @@
+{
+  "version": "1.0",
+  "truncation": null,
+  "padding": null,
+  "added_tokens": [
+    {
+      "id": 0,
+      "content": "<pad>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 1,
+      "content": "<unk>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 2,
+      "content": "<bos>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 3,
+      "content": "<eos>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 4,
+      "content": "<|system|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 5,
+      "content": "<|user|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 6,
+      "content": "<|assistant|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    },
+    {
+      "id": 7,
+      "content": "<|end|>",
+      "single_word": false,
+      "lstrip": false,
+      "rstrip": false,
+      "normalized": false,
+      "special": true
+    }
+  ],
+  "normalizer": null,
+  "pre_tokenizer": {
+    "type": "WhitespaceSplit"
+  },
+  "post_processor": {
+    "type": "TemplateProcessing",
+    "single": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      }
+    ],
+    "pair": [
+      {
+        "Sequence": {
+          "id": "A",
+          "type_id": 0
+        }
+      },
+      {
+        "Sequence": {
+          "id": "B",
+          "type_id": 1
+        }
+      }
+    ],
+    "special_tokens": {}
+  },
+  "decoder": null,
+  "model": {
+    "type": "WordLevel",
+    "vocab": {
+      "<pad>": 0,
+      "<unk>": 1,
+      "<bos>": 2,
+      "<eos>": 3,
+      "<|system|>": 4,
+      "<|user|>": 5,
+      "<|assistant|>": 6,
+      "<|end|>": 7,
+      "the": 8,
+      "a": 9,
+      "is": 10,
+      "are": 11,
+      "of": 12,
+      "to": 13,
+      "and": 14,
+      "in": 15,
+      "for": 16,
+      "with": 17,
+      "you": 18,
+      "your": 19,
+      "I": 20,
+      "can": 21,
+      "please": 22,
+      "answer": 23,
+      "only": 24,
+      "help": 25,
+      "account": 26,
+      "password": 27,
+      "reset": 28,
+      "login": 29,
+      "email": 30,
+      "click": 31,
+      "settings": 32,
+      "support": 33,
+      "change": 34,
+      "cancel": 35,
+      "subscription": 36,
+      "profile": 37,
+      "name": 38,
+      "update": 39,
+      "billing": 40,
+      "payment": 41,
+      "card": 42,
+      "refund": 43,
+      "yes": 44,
+      "no": 45,
+      "true": 46,
+      "false": 47,
+      "one": 48,
+      "two": 49,
+      "three": 50,
+      "four": 51,
+      "1": 52,
+      "2": 53,
+      "3": 54,
+      "4": 55,
+      "hello": 56,
+      "world": 57,
+      "blue": 58,
+      "red": 59,
+      "green": 60,
+      "sky": 61,
+      "water": 62,
+      "grass": 63,
+      "color": 64,
+      "what": 65,
+      "how": 66,
+      "where": 67,
+      "why": 68,
+      "when": 69,
+      "return": 70,
+      "JSON": 71,
+      "book": 72,
+      "read": 73,
+      "model": 74,
+      "text": 75,
+      "train": 76,
+      "data": 77,
+      "test": 78,
+      "validation": 79,
+      "document": 80,
+      "code": 81,
+      "language": 82,
+      "learn": 83,
+      "cat": 84,
+      "dog": 85,
+      "animal": 86,
+      "bird": 87,
+      "fish": 88,
+      "flies": 89,
+      "swims": 90,
+      "runs": 91,
+      "capital": 92,
+      "France": 93,
+      "Paris": 94,
+      "Italy": 95,
+      "Rome": 96,
+      "Germany": 97,
+      "Berlin": 98,
+      "sun": 99,
+      "hot": 100,
+      "ice": 101,
+      "cold": 102,
+      "instruction": 103,
+      "question": 104,
+      "response": 105,
+      "user": 106,
+      "assistant": 107,
+      "system": 108,
+      "explain": 109,
+      "compute": 110,
+      "add": 111,
+      "result": 112,
+      "5": 113,
+      "6": 114,
+      "7": 115,
+      "8": 116,
+      "9": 117,
+      "10": 118,
+      ".": 119,
+      ",": 120,
+      ":": 121,
+      "?": 122
+    },
+    "unk_token": "<unk>"
+  }
+}

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "backend": "tokenizers",
+  "bos_token": "<bos>",
+  "eos_token": "<eos>",
+  "extra_special_tokens": [
+    "<|system|>",
+    "<|user|>",
+    "<|assistant|>",
+    "<|end|>"
+  ],
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 128,
+  "pad_token": "<pad>",
+  "tokenizer_class": "TokenizersBackend",
+  "unk_token": "<unk>"
+}