Duplicate from Novi-AI/Novi-Nano-Instruct

.gitattributes

@@ -0,0 +1,36 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+banner.jpg filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,229 @@
+---
+language:
+- en
+library_name: transformers
+pipeline_tag: text-generation
+tags:
+- novi
+- novi-nano
+- novi-nano-instruct
+- causal-lm
+- gpt2
+- from-scratch
+- instruction-tuning
+- chatml
+datasets:
+- Novi-AI/Novi-510x
+---
+
+# Novi-Nano-Instruct
+
+![Novi-Nano Banner](banner.jpg)
+
+**Novi-Nano-Instruct** is a tiny instruction-tuned causal language model developed by **Novi-AI**.
+
+It is based on **Novi-Nano-Base** and fine-tuned on a small instruction dataset to experiment with instruction following and conversational behavior at an extremely small scale.
+
+⚡ **1.26M parameters · 500 training examples · 256-token context**
+
+## Model Details
+
+### Architecture
+
+| Property        |                    Value |
+| --------------- | -----------------------: |
+| Model type      |    Causal Language Model |
+| Base model      | `Novi-AI/Novi-Nano-Base` |
+| Parameters      |            **1,258,848** |
+| Vocabulary size |                **8,195** |
+| Context length  |                  **256** |
+| Embedding size  |                   **96** |
+| Layers          |                    **4** |
+| Attention heads |                    **4** |
+| FFN size        |                  **384** |
+| Tensor type     |                  **F32** |
+
+## Instruction Tuning
+
+Novi-Nano-Instruct was trained from **Novi-Nano-Base** using a small instruction dataset containing **510 examples**.
+
+### Dataset
+
+| Split      | Examples |
+| ---------- | -------: |
+| Training   |  **500** |
+| Validation |   **10** |
+
+The model uses a ChatML-style format with:
+
+```text
+<|im_start|>
+<|im_end|>
+```
+
+Training loss was applied specifically to the assistant responses, allowing the model to focus on learning how to respond to user instructions.
+
+### Training Configuration
+
+| Property                |    Value |
+| ----------------------- | -------: |
+| Epochs                  |    **5** |
+| Batch size              |   **16** |
+| Gradient accumulation   |    **2** |
+| Effective batch size    |   **32** |
+| Maximum sequence length |  **256** |
+| Learning rate           | **2e-5** |
+| Precision               | **FP32** |
+| Device                  |  **CPU** |
+
+## Training Statistics
+
+The final training run produced:
+
+| Metric                      |          Result |
+| --------------------------- | --------------: |
+| Final validation loss       |    **5.153667** |
+| Final validation perplexity |    **173.0650** |
+| Training examples           |         **500** |
+| Validation examples         |          **10** |
+| Training time               | **~32 seconds** |
+
+Because the validation set contains only **10 examples**, these metrics should be considered experimental rather than a comprehensive benchmark.
+
+## Tokenizer
+
+Novi-Nano-Instruct uses the custom tokenizer developed for Novi-Nano.
+
+The original tokenizer vocabulary was **8,192 tokens**, with additional tokens already present in the tokenizer.
+
+Two ChatML tokens were added for instruction tuning:
+
+* `<|im_start|>` — **8193**
+* `<|im_end|>` — **8194**
+
+The final tokenizer size is **8,195 tokens**.
+
+The tokenizer was originally trained using data from:
+
+* FineWeb-Edu
+* FineWeb-HQ
+* SmolLM-Cosmopedia
+
+## Intended Use
+
+Novi-Nano-Instruct is primarily intended for:
+
+* 🔬 Research and experimentation
+* 🧪 Small-model instruction-tuning experiments
+* 🎓 Educational purposes
+* 💬 Tiny conversational-model experiments
+* 💻 Lightweight local inference
+* 🛠️ Experimenting with extremely small instruction-tuned models
+
+As an **experimental 1.26M-parameter model**, it is not intended to compete with modern billion-parameter language models.
+
+## Limitations
+
+Novi-Nano-Instruct is an extremely small experimental language model trained on only **500 instruction examples**.
+
+Because of its size and limited training data, it may:
+
+* Generate incoherent text
+* Repeat phrases
+* Produce unrelated responses
+* Fail to follow instructions
+* Produce factual errors
+* Have very limited world knowledge
+* Perform poorly on reasoning tasks
+* Struggle with longer conversations
+* Lose context beyond its 256-token window
+* Produce malformed or unexpected responses
+
+Generation quality is currently **highly experimental**. The model can generate text, but it does not yet consistently produce reliable assistant-style responses.
+
+This model should be considered a **research and experimentation model**, rather than a production-ready conversational AI.
+
+## Usage
+
+```python
+from transformers import AutoTokenizer, AutoModelForCausalLM
+
+model_id = "Novi-AI/Novi-Nano-Instruct"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id)
+
+messages = [
+    {
+        "role": "system",
+        "content": "You are Novi-Nano, a helpful AI assistant."
+    },
+    {
+        "role": "user",
+        "content": "Give a synonym for 'quiet'."
+    }
+]
+
+prompt = tokenizer.apply_chat_template(
+    messages,
+    tokenize=False,
+    add_generation_prompt=True,
+)
+
+inputs = tokenizer(prompt, return_tensors="pt")
+
+outputs = model.generate(
+    **inputs,
+    max_new_tokens=50,
+)
+
+print(tokenizer.decode(outputs[0], skip_special_tokens=True))
+```
+
+## Chat Template
+
+Novi-Nano-Instruct uses a ChatML-style conversation format:
+
+```text
+<|im_start|>system
+You are Novi-Nano, a helpful AI assistant.<|im_end|>
+<|im_start|>user
+Give a synonym for 'quiet'.<|im_end|>
+<|im_start|>assistant
+A synonym is 'silent'.<|im_end|>
+```
+
+For generation, the assistant message is opened automatically by the chat template.
+
+## Project History
+
+Novi AI follows the earlier **AppleMind** experiments, with Novi becoming the primary project for developing small language models.
+
+**AppleMind → Novi AI → Novi-Nano → Novi-Nano-Instruct** 🚀
+
+## Acknowledgements
+
+Novi-Nano was built using the open-source machine-learning ecosystem and datasets made available by the community.
+
+Special thanks to:
+
+* Hugging Face 🤗
+* FineWeb
+* SmolLM
+* Cosmopedia
+
+## License
+
+This model is released under the **Apache 2.0** license.
+
+---
+
+## 🧠 Novi AI
+
+**Small models. Big experiments.**
+
+Novi-Nano-Instruct explores instruction tuning at an extremely small scale, with just **1.26 million parameters** and **500 training examples**.
+
+It is intentionally tiny — exploring how far instruction following can go with a fraction of the parameters used by modern LLMs.
+
+*Novi AI 2026 — Project Kairo*

chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{{'<|im_start|>' + message['role'] + '\n' + message['content'] + '<|im_end|>' + '\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\n' }}{% endif %}

config.json

@@ -0,0 +1,35 @@
+{
+  "activation_function": "gelu_new",
+  "add_cross_attention": false,
+  "architectures": [
+    "GPT2LMHeadModel"
+  ],
+  "attn_pdrop": 0.0,
+  "bos_token_id": 8192,
+  "dtype": "float32",
+  "embd_pdrop": 0.0,
+  "eos_token_id": 8192,
+  "initializer_range": 0.02,
+  "layer_norm_epsilon": 1e-05,
+  "model_type": "gpt2",
+  "n_ctx": 256,
+  "n_embd": 96,
+  "n_head": 4,
+  "n_inner": 384,
+  "n_layer": 4,
+  "n_positions": 256,
+  "pad_token_id": 0,
+  "reorder_and_upcast_attn": false,
+  "resid_pdrop": 0.0,
+  "scale_attn_by_inverse_layer_idx": false,
+  "scale_attn_weights": true,
+  "summary_activation": null,
+  "summary_first_dropout": 0.1,
+  "summary_proj_to_labels": true,
+  "summary_type": "cls_index",
+  "summary_use_proj": true,
+  "tie_word_embeddings": true,
+  "transformers_version": "5.9.0",
+  "use_cache": false,
+  "vocab_size": 8195
+}

generation_config.json

@@ -0,0 +1,13 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 8192,
+  "eos_token_id": [
+    8192,
+    2
+  ],
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "pad_token_id": 0,
+  "transformers_version": "5.9.0",
+  "use_cache": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b52c50cc0b9bd8042ee674266ec2af4a8e34fd2f5b213b26d0db73d6dc59e739
+size 5040360

tokenizer_config.json

@@ -0,0 +1,17 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|endoftext|>",
+  "eos_token": "<|endoftext|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|pad|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>"
+}

training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e4c534110bb4b95606560faca657721f10249028cc669c356a65e033b95b71ef
+size 5265

training_info.json

@@ -0,0 +1,28 @@
+{
+  "model": "Novi-Nano-Instruct",
+  "base_model": "Novi-AI/Novi-Nano-Base",
+  "parameters": 1258848,
+  "trainable_parameters": 1258848,
+  "base_vocab_size": 8192,
+  "final_vocab_size": 8195,
+  "added_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "context_length": 256,
+  "epochs": 5,
+  "train_examples": 500,
+  "validation_examples": 10,
+  "batch_size": 16,
+  "gradient_accumulation_steps": 2,
+  "effective_batch_size": 32,
+  "learning_rate": 2e-05,
+  "weight_decay": 0.01,
+  "warmup_ratio": 0.1,
+  "scheduler": "cosine",
+  "bf16": false,
+  "fp16": false,
+  "validation_loss": 5.153667449951172,
+  "validation_perplexity": 173.0650352145914,
+  "chat_template": "{% for message in messages %}{{'<|im_start|>' + message['role'] + '\\n' + message['content'] + '<|im_end|>' + '\\n'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant\\n' }}{% endif %}"
+}