Add SmolLM2-360M NPC roleplay model

README.md

@@ -0,0 +1,190 @@
+---
+base_model: HuggingFaceTB/SmolLM2-360M-Instruct
+library_name: peft
+license: apache-2.0
+language:
+- en
+pipeline_tag: text-generation
+tags:
+- roleplay
+- npc
+- character-ai
+- smollm2
+- lora
+- trl
+- sft
+datasets:
+- chimbiwide/NPC-Dialogue_v2
+---
+
+# SmolLM2-360M-NPC-Roleplay
+
+LoRA supervised fine-tune of [`HuggingFaceTB/SmolLM2-360M-Instruct`](https://huggingface.co/HuggingFaceTB/SmolLM2-360M-Instruct) for
+character-conditioned NPC roleplay. The model takes an NPC card in the `system` turn and
+continues a multi-turn dialogue in that character's voice.
+
+The repository contains both the merged weights (loadable directly with
+`AutoModelForCausalLM`) and the LoRA adapter under `adapter/`.
+
+## Usage
+
+```python
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "thealper2/SmolLM2-360M-NPC-Roleplay"
+tokenizer = AutoTokenizer.from_pretrained(model_id)
+model = AutoModelForCausalLM.from_pretrained(model_id, dtype="bfloat16", device_map="auto")
+
+card = (
+    "Enter roleplay mode. You are Dellin Vance. Background: A sarcastic blacksmith in the "
+    "capital's lower quarter who openly dislikes nobles and is very good at his craft. "
+    "Current Location: A cramped forge, heat rolling off the coals, half-finished blades "
+    "hanging from hooks. "
+    "Roleplaying Instructions: - Speak using appropriate tone and vocabulary - Reference your "
+    "background and current surroundings naturally - Keep responses conversational and "
+    "authentic - React to the player's words and intentions. Your first response should be a "
+    "greeting to the player."
+)
+
+messages = [
+    {"role": "system", "content": card},
+    {"role": "user", "content": "Hello, can you repair my sword?"},
+]
+inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt")
+out = model.generate(
+    inputs.to(model.device),
+    max_new_tokens=160,
+    do_sample=True,
+    temperature=0.8,
+    top_p=0.9,
+    repetition_penalty=1.1,
+)
+print(tokenizer.decode(out[0, inputs.shape[1]:], skip_special_tokens=True))
+```
+
+Using the adapter instead of the merged weights:
+
+```python
+from peft import PeftModel
+base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-360M-Instruct", dtype="bfloat16")
+model = PeftModel.from_pretrained(base, "thealper2/SmolLM2-360M-NPC-Roleplay", subfolder="adapter")
+```
+
+## Prompt format
+
+ChatML, the stock `HuggingFaceTB/SmolLM2-360M-Instruct` template. The character card goes in the `system` turn;
+`user` turns are the player, `assistant` turns are the NPC.
+
+```
+<|im_start|>system
+<character card><|im_end|>
+<|im_start|>user
+<player message><|im_end|>
+<|im_start|>assistant
+```
+
+Every training card uses the same layout, and matching it at inference time gives the
+closest behaviour to training:
+
+```
+Enter roleplay mode. You are <Name>. Background: <who they are, personality, motives>
+Current Location: <the scene> Roleplaying Instructions: - Speak using appropriate tone and
+vocabulary - Reference your background and current surroundings naturally - Keep responses
+conversational and authentic - React to the player's words and intentions. Your first
+response should be a greeting to the player.
+```
+
+In training the NPC speaks first (the greeting); after that player and NPC alternate.
+
+## Training data
+
+[`chimbiwide/NPC-Dialogue_v2`](https://huggingface.co/datasets/chimbiwide/NPC-Dialogue_v2) (`dialogue` config) - 1,689 multi-turn NPC
+conversations (16 messages each) over 101 fantasy RPG characters.
+
+Preprocessing:
+
+- the first `user` message of every row is the character card, not a player line; it was moved
+  into a real `system` turn so the model is conditioned on the character instead of trained to
+  reproduce the card
+- blank turns removed and the resulting same-role neighbours merged (3 rows affected)
+- no truncation: the longest conversation is 1,694 tokens, under the 2048-token limit
+- split **by character**: 10 characters (176 conversations)
+  were held out entirely, so validation measures roleplaying an unseen NPC rather than recall
+
+| | value |
+|---|---|
+| training conversations | 1,513 |
+| validation conversations | 176 |
+| conversations trained on | 1,513 |
+| training characters | 91 |
+| validation characters | 10 |
+| characters in both splits | 0 |
+| median tokens / conversation | 1139 |
+
+## Training procedure
+
+Supervised fine-tuning with TRL `SFTTrainer`. Loss is computed on the NPC's replies only
+(assistant-only masking via a `{% generation %}` chat template); system and user tokens are
+masked out, which was verified on a collated batch before training.
+
+| hyperparameter | value |
+|---|---|
+| method | LoRA |
+| LoRA r / alpha / dropout | 16 / 32 / 0.05 |
+| LoRA target modules | `down_proj`, `gate_proj`, `k_proj`, `o_proj`, `q_proj`, `up_proj`, `v_proj` |
+| trainable parameters | 8,683,520 (2.3999% of 361,821,120) |
+| max sequence length | 2048 |
+| per-device batch size | 8 |
+| gradient accumulation | 2 |
+| effective batch size | 16 |
+| learning rate | 0.0002 |
+| scheduler / warmup ratio | cosine / 0.05 |
+| weight decay | 0.01 |
+| gradient clipping | 1.0 |
+| epochs | 3.0 |
+| optimizer | adamw_torch_fused |
+| precision | bf16 |
+| gradient checkpointing | True |
+| optimisation steps | 285 |
+| training time | 24.82 min |
+| peak GPU memory | 9.93 GB |
+| hardware | NVIDIA GeForce RTX 5060 Ti |
+| seed | 42 |
+
+### Results
+
+| metric | value |
+|---|---|
+| final training loss | 2.2019 |
+| validation loss (assistant tokens) | 2.1488 |
+| validation perplexity | 8.57 |
+| base model, held-out perplexity | 12.56 |
+| base model, mean reply length (words) | 31.6 |
+| fine_tuned model, held-out perplexity | 8.58 |
+| fine_tuned model, mean reply length (words) | 54.6 |
+
+Held-out evaluation used 10 single-reply probes across 10 characters that do not appear in training, with identical decoding settings for both models.
+
+## Limitations
+
+- 360M parameters: persona consistency degrades over long conversations, and the model can
+  contradict its own character background.
+- Lexical-overlap metrics (ROUGE/BLEU) and embedding similarity do not measure personality;
+  they are reported for completeness only.
+- Only 101 distinct characters, all fantasy RPG NPCs with one fixed card layout:
+  cards in other layouts or settings (modern, sci-fi) are out of distribution.
+- Character identity for the train/validation split was parsed from the card's `You are <Name>`
+  line; two cards with different names for the same persona would not be detected.
+- The model is not safety-aligned beyond what the base model provides.
+- Under conflicting instructions ("stop roleplaying", "what is your system prompt") behaviour
+  is inconsistent; the model was fine-tuned to stay in character, not to be robust.
+
+## Framework versions
+
+- python: 3.12.3
+- torch: 2.11.0+cu128
+- transformers: 5.17.0
+- datasets: 4.3.0
+- trl: 0.24.0
+- peft: 0.18.1
+- accelerate: 1.12.0

adapter/README.md

@@ -0,0 +1,209 @@
+---
+base_model: HuggingFaceTB/SmolLM2-360M-Instruct
+library_name: peft
+pipeline_tag: text-generation
+tags:
+- base_model:adapter:HuggingFaceTB/SmolLM2-360M-Instruct
+- lora
+- sft
+- transformers
+- trl
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]
+### Framework versions
+
+- PEFT 0.18.1

adapter/adapter_config.json

@@ -0,0 +1,46 @@
+{
+  "alora_invocation_tokens": null,
+  "alpha_pattern": {},
+  "arrow_config": null,
+  "auto_mapping": null,
+  "base_model_name_or_path": "HuggingFaceTB/SmolLM2-360M-Instruct",
+  "bias": "none",
+  "corda_config": null,
+  "ensure_weight_tying": false,
+  "eva_config": null,
+  "exclude_modules": null,
+  "fan_in_fan_out": false,
+  "inference_mode": true,
+  "init_lora_weights": true,
+  "layer_replication": null,
+  "layers_pattern": null,
+  "layers_to_transform": null,
+  "loftq_config": {},
+  "lora_alpha": 32,
+  "lora_bias": false,
+  "lora_dropout": 0.05,
+  "megatron_config": null,
+  "megatron_core": "megatron.core",
+  "modules_to_save": null,
+  "peft_type": "LORA",
+  "peft_version": "0.18.1",
+  "qalora_group_size": 16,
+  "r": 16,
+  "rank_pattern": {},
+  "revision": null,
+  "target_modules": [
+    "gate_proj",
+    "down_proj",
+    "o_proj",
+    "v_proj",
+    "k_proj",
+    "q_proj",
+    "up_proj"
+  ],
+  "target_parameters": null,
+  "task_type": "CAUSAL_LM",
+  "trainable_token_indices": null,
+  "use_dora": false,
+  "use_qalora": false,
+  "use_rslora": false
+}

adapter/adapter_model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:71659b9c0916b85c3d249051721b6adb31bb10fc90b5ff52c663677f5b86da9e
+size 34793120

adapter/chat_template.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{% if loop.first and messages[0]["role"] != "system" %}{{ "<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n" }}{% endif %}{% if message["role"] == "assistant" %}{{ "<|im_start|>assistant\n" }}{% generation %}{{ message["content"] + "<|im_end|>" }}{% endgeneration %}{{ "\n" }}{% else %}{{ "<|im_start|>" + message["role"] + "\n" + message["content"] + "<|im_end|>" + "\n" }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ "<|im_start|>assistant\n" }}{% endif %}

adapter/chat_template_generation.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{% if loop.first and messages[0]["role"] != "system" %}{{ "<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n" }}{% endif %}{% if message["role"] == "assistant" %}{{ "<|im_start|>assistant\n" }}{% generation %}{{ message["content"] + "<|im_end|>" }}{% endgeneration %}{{ "\n" }}{% else %}{{ "<|im_start|>" + message["role"] + "\n" + message["content"] + "<|im_end|>" + "\n" }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ "<|im_start|>assistant\n" }}{% endif %}

adapter/tokenizer_config.json

@@ -0,0 +1,19 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|im_start|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|im_end|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

adapter/training_args.bin

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:79daf3d800d94c60607fc90ab16ed9bf9d10ec93e003decdc088ea374835c994
+size 5713

adapter/training_config_snapshot.json

@@ -0,0 +1,114 @@
+{
+  "seed": 42,
+  "model": {
+    "name": "HuggingFaceTB/SmolLM2-360M-Instruct",
+    "chat_template_path": "configs/chat_template_smollm2.jinja",
+    "attn_implementation": "sdpa",
+    "dtype": "bfloat16"
+  },
+  "data": {
+    "dataset_id": "chimbiwide/NPC-Dialogue_v2",
+    "dataset_config": "dialogue",
+    "source_split": "train",
+    "raw_dir": "data/raw",
+    "processed_dir": "data/processed",
+    "max_seq_length": 2048,
+    "candidate_max_lengths": [
+      512,
+      1024,
+      1536,
+      2048,
+      3072,
+      4096
+    ],
+    "max_system_tokens": 1536,
+    "min_assistant_tokens": 4,
+    "max_chunks_per_conversation": 2,
+    "validation_character_ratio": 0.1,
+    "split_strategy": "character",
+    "stratify_by_variant": true,
+    "filter_explicit": false,
+    "drop_duplicates": true
+  },
+  "lora": {
+    "enabled": true,
+    "r": 16,
+    "lora_alpha": 32,
+    "lora_dropout": 0.05,
+    "bias": "none",
+    "task_type": "CAUSAL_LM",
+    "target_modules": [],
+    "auto_target_modules": true,
+    "exclude_modules": [
+      "lm_head",
+      "embed_tokens"
+    ]
+  },
+  "train": {
+    "output_dir": "outputs/checkpoints/npc-lora",
+    "run_name": "smollm2-360m-npc-lora",
+    "num_train_epochs": 3,
+    "per_device_train_batch_size": 8,
+    "per_device_eval_batch_size": 8,
+    "gradient_accumulation_steps": 2,
+    "auto_batch_size": true,
+    "learning_rate": 0.0002,
+    "lr_scheduler_type": "cosine",
+    "warmup_ratio": 0.05,
+    "weight_decay": 0.01,
+    "max_grad_norm": 1.0,
+    "optim": "adamw_torch_fused",
+    "bf16": "auto",
+    "gradient_checkpointing": true,
+    "packing": false,
+    "assistant_only_loss": true,
+    "logging_steps": 5,
+    "eval_strategy": "steps",
+    "eval_steps": 250,
+    "save_strategy": "steps",
+    "save_steps": 250,
+    "auto_eval_steps": true,
+    "target_evaluations": 6,
+    "save_total_limit": 3,
+    "load_best_model_at_end": true,
+    "metric_for_best_model": "eval_loss",
+    "greater_is_better": false,
+    "report_to": [
+      "tensorboard"
+    ],
+    "dataloader_num_workers": 4,
+    "max_train_samples": 0,
+    "max_eval_samples": 1000
+  },
+  "full_finetune": {
+    "learning_rate": 2e-05,
+    "per_device_train_batch_size": 4,
+    "gradient_accumulation_steps": 8,
+    "gradient_checkpointing": true,
+    "output_dir": "outputs/checkpoints/npc-full"
+  },
+  "generation": {
+    "do_sample": true,
+    "temperature": 0.8,
+    "top_p": 0.9,
+    "top_k": 50,
+    "max_new_tokens": 160,
+    "repetition_penalty": 1.1
+  },
+  "evaluation": {
+    "output_dir": "outputs/evaluation",
+    "num_comparison_examples": 60,
+    "scenarios_path": "configs/eval_scenarios.yaml",
+    "qualitative_turns": 4,
+    "compute_rouge": true,
+    "compute_bleu": true,
+    "semantic_model": "sentence-transformers/all-MiniLM-L6-v2",
+    "compute_semantic_similarity": true
+  },
+  "hub": {
+    "repo_id": "",
+    "private": false,
+    "push_merged_model": true,
+    "commit_message": "Add SmolLM2-360M NPC roleplay model"
+  }
+}

adapter/training_run.json

@@ -0,0 +1,776 @@
+{
+  "model": "HuggingFaceTB/SmolLM2-360M-Instruct",
+  "dataset": "chimbiwide/NPC-Dialogue_v2",
+  "method": "lora",
+  "dataset_size": 1689,
+  "train_size": 1513,
+  "validation_size": 176,
+  "max_train_samples": 0,
+  "max_eval_samples": 1000,
+  "eval_steps": 47,
+  "train_characters": 91,
+  "validation_characters": 10,
+  "character_overlap": 0,
+  "max_seq_length": 2048,
+  "lora_config": {
+    "task_type": "CAUSAL_LM",
+    "peft_type": "LORA",
+    "auto_mapping": null,
+    "peft_version": "0.18.1",
+    "base_model_name_or_path": "HuggingFaceTB/SmolLM2-360M-Instruct",
+    "revision": null,
+    "inference_mode": false,
+    "r": 16,
+    "target_modules": [
+      "down_proj",
+      "gate_proj",
+      "k_proj",
+      "o_proj",
+      "q_proj",
+      "up_proj",
+      "v_proj"
+    ],
+    "exclude_modules": null,
+    "lora_alpha": 32,
+    "lora_dropout": 0.05,
+    "fan_in_fan_out": false,
+    "bias": "none",
+    "use_rslora": false,
+    "modules_to_save": null,
+    "init_lora_weights": true,
+    "layers_to_transform": null,
+    "layers_pattern": null,
+    "rank_pattern": {},
+    "alpha_pattern": {},
+    "megatron_config": null,
+    "megatron_core": "megatron.core",
+    "trainable_token_indices": null,
+    "loftq_config": {},
+    "eva_config": null,
+    "corda_config": null,
+    "use_dora": false,
+    "alora_invocation_tokens": null,
+    "use_qalora": false,
+    "qalora_group_size": 16,
+    "layer_replication": null,
+    "lora_bias": false,
+    "target_parameters": null,
+    "arrow_config": null,
+    "ensure_weight_tying": false
+  },
+  "learning_rate": 0.0002,
+  "batch_size": 8,
+  "gradient_accumulation": 2,
+  "effective_batch_size": 16,
+  "epochs": 3.0,
+  "warmup_ratio": 0.05,
+  "warmup_steps": 14,
+  "weight_decay": 0.01,
+  "lr_scheduler": "cosine",
+  "optimizer": "adamw_torch_fused",
+  "precision": "bf16",
+  "gradient_checkpointing": true,
+  "assistant_only_loss": true,
+  "total_parameters": 361821120,
+  "trainable_parameters": 8683520,
+  "trainable_percent": 2.3999,
+  "training_loss": 2.2018708011560273,
+  "validation_loss": 2.148780584335327,
+  "perplexity": 8.5743962665088,
+  "training_time_seconds": 1489.1,
+  "training_time_minutes": 24.82,
+  "peak_gpu_memory_gb": 9.93,
+  "global_steps": 285,
+  "loss_masking_check": {
+    "supervised_token_ratio": 0.5492,
+    "supervised_preview": "Ah, welcome, welcome! You find yourself in a corner of Calcutta where fortunes are made and lost quicker than the sweat dries on your brow. I am Bikram. What brings you to my humble… emporium, shall we say? Don't mind the smell; it's the scent of opportunity, my friend.<|im_end|>An artifact, you say? Calcutta is a magnet for such things, drawn in by the tides of trade and whispered secrets. But 's",
+    "masked_preview": "<|im_start|>system\nEnter roleplay mode. You are Bikram. Background: Bikram's weathered face tells a story of hardship and resilience, etched with the lines of countless deals made in the shadows of Calcutta's bustling streets. His eyes, though hardened by experience, hold a flicker of warmth when he speaks of loyalty, a virtue he holds above all else. Bikram carries himself with a swagger that bel",
+    "system_or_user_leaked": false
+  },
+  "gpu": {
+    "device": "NVIDIA GeForce RTX 5060 Ti",
+    "total_memory_gb": 17.07,
+    "bf16_supported": true,
+    "capability": "12.0"
+  },
+  "libraries": {
+    "python": "3.12.3",
+    "torch": "2.11.0+cu128",
+    "transformers": "5.17.0",
+    "datasets": "4.3.0",
+    "trl": "0.24.0",
+    "peft": "0.18.1",
+    "accelerate": "1.12.0"
+  },
+  "seed": 42,
+  "output_dir": "/mnt/d/work2/smollm2-npc-roleplay-npc/outputs/checkpoints/npc-lora/final",
+  "log_history": [
+    {
+      "loss": 2.5523666381835937,
+      "grad_norm": 0.22088582813739777,
+      "learning_rate": 5.714285714285714e-05,
+      "entropy": 1.933849000930786,
+      "num_tokens": 93796.0,
+      "mean_token_accuracy": 0.46158536076545714,
+      "epoch": 0.05263157894736842,
+      "step": 5
+    },
+    {
+      "loss": 2.5331878662109375,
+      "grad_norm": 0.21720781922340393,
+      "learning_rate": 0.00012857142857142858,
+      "entropy": 1.9523229598999023,
+      "num_tokens": 187269.0,
+      "mean_token_accuracy": 0.46900137364864347,
+      "epoch": 0.10526315789473684,
+      "step": 10
+    },
+    {
+      "loss": 2.4539527893066406,
+      "grad_norm": 0.13605234026908875,
+      "learning_rate": 0.0002,
+      "entropy": 2.1557540893554688,
+      "num_tokens": 281613.0,
+      "mean_token_accuracy": 0.4719751179218292,
+      "epoch": 0.15789473684210525,
+      "step": 15
+    },
+    {
+      "loss": 2.451401138305664,
+      "grad_norm": 0.23667477071285248,
+      "learning_rate": 0.00019983206176618388,
+      "entropy": 2.480435037612915,
+      "num_tokens": 376141.0,
+      "mean_token_accuracy": 0.4684509068727493,
+      "epoch": 0.21052631578947367,
+      "step": 20
+    },
+    {
+      "loss": 2.4010854721069337,
+      "grad_norm": 0.09741178154945374,
+      "learning_rate": 0.000199328811129743,
+      "entropy": 2.378065037727356,
+      "num_tokens": 468513.0,
+      "mean_token_accuracy": 0.478556826710701,
+      "epoch": 0.2631578947368421,
+      "step": 25
+    },
+    {
+      "loss": 2.366695594787598,
+      "grad_norm": 0.11083851754665375,
+      "learning_rate": 0.00019849193839113833,
+      "entropy": 2.217120313644409,
+      "num_tokens": 561891.0,
+      "mean_token_accuracy": 0.48194282352924345,
+      "epoch": 0.3157894736842105,
+      "step": 30
+    },
+    {
+      "loss": 2.343669891357422,
+      "grad_norm": 0.10740305483341217,
+      "learning_rate": 0.00019732425440896297,
+      "entropy": 2.188133120536804,
+      "num_tokens": 656095.0,
+      "mean_token_accuracy": 0.48512300848960876,
+      "epoch": 0.3684210526315789,
+      "step": 35
+    },
+    {
+      "loss": 2.337441635131836,
+      "grad_norm": 0.10451745986938477,
+      "learning_rate": 0.0001958296811589293,
+      "entropy": 2.2373024463653564,
+      "num_tokens": 748364.0,
+      "mean_token_accuracy": 0.4842404007911682,
+      "epoch": 0.42105263157894735,
+      "step": 40
+    },
+    {
+      "loss": 2.3664779663085938,
+      "grad_norm": 0.10740821063518524,
+      "learning_rate": 0.0001940132385608757,
+      "entropy": 2.279412937164307,
+      "num_tokens": 842886.0,
+      "mean_token_accuracy": 0.48362932801246644,
+      "epoch": 0.47368421052631576,
+      "step": 45
+    },
+    {
+      "eval_loss": 2.316269636154175,
+      "eval_runtime": 25.2544,
+      "eval_samples_per_second": 6.969,
+      "eval_steps_per_second": 0.871,
+      "eval_entropy": 2.2317135334014893,
+      "eval_num_tokens": 879988.0,
+      "eval_mean_token_accuracy": 0.4894282506270842,
+      "epoch": 0.49473684210526314,
+      "step": 47
+    },
+    {
+      "loss": 2.3454864501953123,
+      "grad_norm": 0.11415872722864151,
+      "learning_rate": 0.00019188102761803717,
+      "entropy": 2.252223086357117,
+      "num_tokens": 936226.0,
+      "mean_token_accuracy": 0.48670867681503294,
+      "epoch": 0.5263157894736842,
+      "step": 50
+    },
+    {
+      "loss": 2.3060874938964844,
+      "grad_norm": 0.1160162165760994,
+      "learning_rate": 0.0001894402099252109,
+      "entropy": 2.2304810762405394,
+      "num_tokens": 1028185.0,
+      "mean_token_accuracy": 0.4880038559436798,
+      "epoch": 0.5789473684210527,
+      "step": 55
+    },
+    {
+      "loss": 2.2843629837036135,
+      "grad_norm": 0.12905658781528473,
+      "learning_rate": 0.0001866989836146449,
+      "entropy": 2.1765635013580322,
+      "num_tokens": 1120518.0,
+      "mean_token_accuracy": 0.49455042481422423,
+      "epoch": 0.631578947368421,
+      "step": 60
+    },
+    {
+      "loss": 2.262358283996582,
+      "grad_norm": 0.11617386341094971,
+      "learning_rate": 0.00018366655582044094,
+      "entropy": 2.161082220077515,
+      "num_tokens": 1216629.0,
+      "mean_token_accuracy": 0.49634212255477905,
+      "epoch": 0.6842105263157895,
+      "step": 65
+    },
+    {
+      "loss": 2.274257850646973,
+      "grad_norm": 0.14454935491085052,
+      "learning_rate": 0.0001803531117539577,
+      "entropy": 2.191727089881897,
+      "num_tokens": 1308903.0,
+      "mean_token_accuracy": 0.49855717420578005,
+      "epoch": 0.7368421052631579,
+      "step": 70
+    },
+    {
+      "loss": 2.2217056274414064,
+      "grad_norm": 0.14457330107688904,
+      "learning_rate": 0.00017676978049408263,
+      "entropy": 2.1600573301315307,
+      "num_tokens": 1401788.0,
+      "mean_token_accuracy": 0.4966869682073593,
+      "epoch": 0.7894736842105263,
+      "step": 75
+    },
+    {
+      "loss": 2.2426210403442384,
+      "grad_norm": 0.14771753549575806,
+      "learning_rate": 0.00017292859760727493,
+      "entropy": 2.1251904249191282,
+      "num_tokens": 1493269.0,
+      "mean_token_accuracy": 0.49993438124656675,
+      "epoch": 0.8421052631578947,
+      "step": 80
+    },
+    {
+      "loss": 2.237934875488281,
+      "grad_norm": 0.14660881459712982,
+      "learning_rate": 0.00016884246472293016,
+      "entropy": 2.160723400115967,
+      "num_tokens": 1589104.0,
+      "mean_token_accuracy": 0.5008507877588272,
+      "epoch": 0.8947368421052632,
+      "step": 85
+    },
+    {
+      "loss": 2.215174674987793,
+      "grad_norm": 0.14955751597881317,
+      "learning_rate": 0.000164525106199843,
+      "entropy": 2.150420093536377,
+      "num_tokens": 1683525.0,
+      "mean_token_accuracy": 0.5014137506484986,
+      "epoch": 0.9473684210526315,
+      "step": 90
+    },
+    {
+      "eval_loss": 2.2206270694732666,
+      "eval_runtime": 13.3703,
+      "eval_samples_per_second": 13.164,
+      "eval_steps_per_second": 1.645,
+      "eval_entropy": 2.1415598500858652,
+      "eval_num_tokens": 1759033.0,
+      "eval_mean_token_accuracy": 0.5016853362321854,
+      "epoch": 0.9894736842105263,
+      "step": 94
+    },
+    {
+      "loss": 2.210609245300293,
+      "grad_norm": 0.2051151990890503,
+      "learning_rate": 0.00015999102302931585,
+      "entropy": 2.1453773021697997,
+      "num_tokens": 1769255.0,
+      "mean_token_accuracy": 0.5010978668928147,
+      "epoch": 1.0,
+      "step": 95
+    },
+    {
+      "loss": 2.1714473724365235,
+      "grad_norm": 0.15744632482528687,
+      "learning_rate": 0.00015525544412974132,
+      "entropy": 2.145352911949158,
+      "num_tokens": 1864783.0,
+      "mean_token_accuracy": 0.510258013010025,
+      "epoch": 1.0526315789473684,
+      "step": 100
+    },
+    {
+      "loss": 2.212137222290039,
+      "grad_norm": 0.16042566299438477,
+      "learning_rate": 0.0001503342751962493,
+      "entropy": 2.126456546783447,
+      "num_tokens": 1957360.0,
+      "mean_token_accuracy": 0.5051965832710266,
+      "epoch": 1.1052631578947367,
+      "step": 105
+    },
+    {
+      "loss": 2.1988462448120116,
+      "grad_norm": 0.16124625504016876,
+      "learning_rate": 0.00014524404527721977,
+      "entropy": 2.1093988180160523,
+      "num_tokens": 2049648.0,
+      "mean_token_accuracy": 0.5063745319843292,
+      "epoch": 1.1578947368421053,
+      "step": 110
+    },
+    {
+      "loss": 2.192594528198242,
+      "grad_norm": 0.18138575553894043,
+      "learning_rate": 0.00014000185125709918,
+      "entropy": 2.128497362136841,
+      "num_tokens": 2143746.0,
+      "mean_token_accuracy": 0.5051657497882843,
+      "epoch": 1.2105263157894737,
+      "step": 115
+    },
+    {
+      "loss": 2.1997039794921873,
+      "grad_norm": 0.1774953305721283,
+      "learning_rate": 0.00013462530043198873,
+      "entropy": 2.152763772010803,
+      "num_tokens": 2238337.0,
+      "mean_token_accuracy": 0.5037459582090378,
+      "epoch": 1.263157894736842,
+      "step": 120
+    },
+    {
+      "loss": 2.1733221054077148,
+      "grad_norm": 0.17004919052124023,
+      "learning_rate": 0.00012913245137088024,
+      "entropy": 2.1356810569763183,
+      "num_tokens": 2330291.0,
+      "mean_token_accuracy": 0.5107389390468597,
+      "epoch": 1.3157894736842106,
+      "step": 125
+    },
+    {
+      "loss": 2.148809814453125,
+      "grad_norm": 0.1917748898267746,
+      "learning_rate": 0.00012354175326117253,
+      "entropy": 2.093570041656494,
+      "num_tokens": 2421723.0,
+      "mean_token_accuracy": 0.5119283616542816,
+      "epoch": 1.368421052631579,
+      "step": 130
+    },
+    {
+      "loss": 2.1357706069946287,
+      "grad_norm": 0.18109489977359772,
+      "learning_rate": 0.0001178719839421925,
+      "entropy": 2.0645798206329347,
+      "num_tokens": 2516541.0,
+      "mean_token_accuracy": 0.515396112203598,
+      "epoch": 1.4210526315789473,
+      "step": 135
+    },
+    {
+      "loss": 2.1544036865234375,
+      "grad_norm": 0.19599980115890503,
+      "learning_rate": 0.00011214218683485158,
+      "entropy": 2.1346421003341676,
+      "num_tokens": 2609031.0,
+      "mean_token_accuracy": 0.5116421699523925,
+      "epoch": 1.4736842105263157,
+      "step": 140
+    },
+    {
+      "eval_loss": 2.1792523860931396,
+      "eval_runtime": 16.4067,
+      "eval_samples_per_second": 10.727,
+      "eval_steps_per_second": 1.341,
+      "eval_entropy": 2.1185361363671045,
+      "eval_num_tokens": 2627950.0,
+      "eval_mean_token_accuracy": 0.5076680142771114,
+      "epoch": 1.4842105263157894,
+      "step": 141
+    },
+    {
+      "loss": 2.169381332397461,
+      "grad_norm": 0.18576543033123016,
+      "learning_rate": 0.00010637160697927651,
+      "entropy": 2.1318769693374633,
+      "num_tokens": 2701848.0,
+      "mean_token_accuracy": 0.5107553184032441,
+      "epoch": 1.526315789473684,
+      "step": 145
+    },
+    {
+      "loss": 2.162215995788574,
+      "grad_norm": 0.18139831721782684,
+      "learning_rate": 0.00010057962639524798,
+      "entropy": 2.121912145614624,
+      "num_tokens": 2795827.0,
+      "mean_token_accuracy": 0.5137030899524688,
+      "epoch": 1.5789473684210527,
+      "step": 150
+    },
+    {
+      "loss": 2.1198862075805662,
+      "grad_norm": 0.19324608147144318,
+      "learning_rate": 9.478569898255765e-05,
+      "entropy": 2.092076063156128,
+      "num_tokens": 2889171.0,
+      "mean_token_accuracy": 0.5189927399158478,
+      "epoch": 1.631578947368421,
+      "step": 155
+    },
+    {
+      "loss": 2.135479545593262,
+      "grad_norm": 0.18662935495376587,
+      "learning_rate": 8.900928517993644e-05,
+      "entropy": 2.107566738128662,
+      "num_tokens": 2983944.0,
+      "mean_token_accuracy": 0.5150025188922882,
+      "epoch": 1.6842105263157894,
+      "step": 160
+    },
+    {
+      "loss": 2.168526840209961,
+      "grad_norm": 0.1955968737602234,
+      "learning_rate": 8.326978660202034e-05,
+      "entropy": 2.1270210266113283,
+      "num_tokens": 3078482.0,
+      "mean_token_accuracy": 0.5109177112579346,
+      "epoch": 1.736842105263158,
+      "step": 165
+    },
+    {
+      "loss": 2.1491790771484376,
+      "grad_norm": 0.19812174141407013,
+      "learning_rate": 7.758648087389277e-05,
+      "entropy": 2.1165373802185057,
+      "num_tokens": 3171229.0,
+      "mean_token_accuracy": 0.5111929804086686,
+      "epoch": 1.7894736842105263,
+      "step": 170
+    },
+    {
+      "loss": 2.160044860839844,
+      "grad_norm": 0.2086932212114334,
+      "learning_rate": 7.197845688207805e-05,
+      "entropy": 2.110708808898926,
+      "num_tokens": 3265986.0,
+      "mean_token_accuracy": 0.5085264205932617,
+      "epoch": 1.8421052631578947,
+      "step": 175
+    },
+    {
+      "loss": 2.156445121765137,
+      "grad_norm": 0.19480496644973755,
+      "learning_rate": 6.646455065946386e-05,
+      "entropy": 2.117557668685913,
+      "num_tokens": 3358807.0,
+      "mean_token_accuracy": 0.513753566145897,
+      "epoch": 1.8947368421052633,
+      "step": 180
+    },
+    {
+      "loss": 2.113090705871582,
+      "grad_norm": 0.19168244302272797,
+      "learning_rate": 6.106328211949928e-05,
+      "entropy": 2.1011462211608887,
+      "num_tokens": 3452203.0,
+      "mean_token_accuracy": 0.5172903001308441,
+      "epoch": 1.9473684210526314,
+      "step": 185
+    },
+    {
+      "eval_loss": 2.157062530517578,
+      "eval_runtime": 15.9374,
+      "eval_samples_per_second": 11.043,
+      "eval_steps_per_second": 1.38,
+      "eval_entropy": 2.0991858785802666,
+      "eval_num_tokens": 3508507.0,
+      "eval_mean_token_accuracy": 0.5112517042593523,
+      "epoch": 1.9789473684210526,
+      "step": 188
+    },
+    {
+      "loss": 2.1452701568603514,
+      "grad_norm": 0.24300329387187958,
+      "learning_rate": 5.579279285216369e-05,
+      "entropy": 2.122407627105713,
+      "num_tokens": 3538510.0,
+      "mean_token_accuracy": 0.5125555753707886,
+      "epoch": 2.0,
+      "step": 190
+    },
+    {
+      "loss": 2.1096288681030275,
+      "grad_norm": 0.20780698955059052,
+      "learning_rate": 5.067078519063514e-05,
+      "entropy": 2.0936718225479125,
+      "num_tokens": 3633684.0,
+      "mean_token_accuracy": 0.5186141610145569,
+      "epoch": 2.0526315789473686,
+      "step": 195
+    },
+    {
+      "loss": 2.099007987976074,
+      "grad_norm": 0.19569851458072662,
+      "learning_rate": 4.571446275331903e-05,
+      "entropy": 2.078820252418518,
+      "num_tokens": 3727072.0,
+      "mean_token_accuracy": 0.5218498349189759,
+      "epoch": 2.1052631578947367,
+      "step": 200
+    },
+    {
+      "loss": 2.138174057006836,
+      "grad_norm": 0.1941261887550354,
+      "learning_rate": 4.094047266094225e-05,
+      "entropy": 2.10717887878418,
+      "num_tokens": 3821112.0,
+      "mean_token_accuracy": 0.514563399553299,
+      "epoch": 2.1578947368421053,
+      "step": 205
+    },
+    {
+      "loss": 2.1252628326416017,
+      "grad_norm": 0.1947101503610611,
+      "learning_rate": 3.6364849622792266e-05,
+      "entropy": 2.108471989631653,
+      "num_tokens": 3911202.0,
+      "mean_token_accuracy": 0.5166740775108337,
+      "epoch": 2.2105263157894735,
+      "step": 210
+    },
+    {
+      "loss": 2.099527359008789,
+      "grad_norm": 0.2063671052455902,
+      "learning_rate": 3.2002962079901744e-05,
+      "entropy": 2.074895644187927,
+      "num_tokens": 4004373.0,
+      "mean_token_accuracy": 0.5181715756654739,
+      "epoch": 2.263157894736842,
+      "step": 215
+    },
+    {
+      "loss": 2.1038749694824217,
+      "grad_norm": 0.19067545235157013,
+      "learning_rate": 2.7869460586071873e-05,
+      "entropy": 2.0991530656814574,
+      "num_tokens": 4097168.0,
+      "mean_token_accuracy": 0.5198172926902771,
+      "epoch": 2.3157894736842106,
+      "step": 220
+    },
+    {
+      "loss": 2.108317756652832,
+      "grad_norm": 0.20180711150169373,
+      "learning_rate": 2.3978228600109565e-05,
+      "entropy": 2.0747674703598022,
+      "num_tokens": 4192035.0,
+      "mean_token_accuracy": 0.5210060477256775,
+      "epoch": 2.3684210526315788,
+      "step": 225
+    },
+    {
+      "loss": 2.094546890258789,
+      "grad_norm": 0.20153699815273285,
+      "learning_rate": 2.0342335854556737e-05,
+      "entropy": 2.0798715591430663,
+      "num_tokens": 4284949.0,
+      "mean_token_accuracy": 0.5181330442428589,
+      "epoch": 2.4210526315789473,
+      "step": 230
+    },
+    {
+      "loss": 2.111064910888672,
+      "grad_norm": 0.20009742677211761,
+      "learning_rate": 1.6973994457534026e-05,
+      "entropy": 2.087741422653198,
+      "num_tokens": 4379919.0,
+      "mean_token_accuracy": 0.5190323472023011,
+      "epoch": 2.473684210526316,
+      "step": 235
+    },
+    {
+      "eval_loss": 2.1503143310546875,
+      "eval_runtime": 15.7779,
+      "eval_samples_per_second": 11.155,
+      "eval_steps_per_second": 1.394,
+      "eval_entropy": 2.0761942159045828,
+      "eval_num_tokens": 4379919.0,
+      "eval_mean_token_accuracy": 0.511995713819157,
+      "epoch": 2.473684210526316,
+      "step": 235
+    },
+    {
+      "loss": 2.119691276550293,
+      "grad_norm": 0.20606471598148346,
+      "learning_rate": 1.3884517875143544e-05,
+      "entropy": 2.0891559600830076,
+      "num_tokens": 4473242.0,
+      "mean_token_accuracy": 0.5179882824420929,
+      "epoch": 2.526315789473684,
+      "step": 240
+    },
+    {
+      "loss": 2.121718406677246,
+      "grad_norm": 0.19413025677204132,
+      "learning_rate": 1.1084282932198541e-05,
+      "entropy": 2.1111440420150758,
+      "num_tokens": 4568065.0,
+      "mean_token_accuracy": 0.5200768530368804,
+      "epoch": 2.5789473684210527,
+      "step": 245
+    },
+    {
+      "loss": 2.1327035903930662,
+      "grad_norm": 0.20437853038311005,
+      "learning_rate": 8.58269495891081e-06,
+      "entropy": 2.095879054069519,
+      "num_tokens": 4661089.0,
+      "mean_token_accuracy": 0.517047768831253,
+      "epoch": 2.6315789473684212,
+      "step": 250
+    },
+    {
+      "loss": 2.077016830444336,
+      "grad_norm": 0.20744766294956207,
+      "learning_rate": 6.388156200599726e-06,
+      "entropy": 2.064937162399292,
+      "num_tokens": 4755205.0,
+      "mean_token_accuracy": 0.5244661808013916,
+      "epoch": 2.6842105263157894,
+      "step": 255
+    },
+    {
+      "loss": 2.109786033630371,
+      "grad_norm": 0.2028643935918808,
+      "learning_rate": 4.508037596527526e-06,
+      "entropy": 2.082032287120819,
+      "num_tokens": 4849257.0,
+      "mean_token_accuracy": 0.5215404391288757,
+      "epoch": 2.736842105263158,
+      "step": 260
+    },
+    {
+      "loss": 2.074248504638672,
+      "grad_norm": 0.20752288401126862,
+      "learning_rate": 2.9486540226488557e-06,
+      "entropy": 2.0800060510635374,
+      "num_tokens": 4941138.0,
+      "mean_token_accuracy": 0.5221293807029724,
+      "epoch": 2.7894736842105265,
+      "step": 265
+    },
+    {
+      "loss": 2.0968595504760743,
+      "grad_norm": 0.1966981142759323,
+      "learning_rate": 1.7152430814285303e-06,
+      "entropy": 2.067763328552246,
+      "num_tokens": 5035425.0,
+      "mean_token_accuracy": 0.5213424026966095,
+      "epoch": 2.8421052631578947,
+      "step": 270
+    },
+    {
+      "loss": 2.088235092163086,
+      "grad_norm": 0.20720359683036804,
+      "learning_rate": 8.119475099673036e-07,
+      "entropy": 2.071122169494629,
+      "num_tokens": 5127357.0,
+      "mean_token_accuracy": 0.519515186548233,
+      "epoch": 2.8947368421052633,
+      "step": 275
+    },
+    {
+      "loss": 2.1086212158203126,
+      "grad_norm": 0.20018276572227478,
+      "learning_rate": 2.418012655228452e-07,
+      "entropy": 2.0884795427322387,
+      "num_tokens": 5221009.0,
+      "mean_token_accuracy": 0.5162034809589386,
+      "epoch": 2.9473684210526314,
+      "step": 280
+    },
+    {
+      "eval_loss": 2.1488332748413086,
+      "eval_runtime": 13.5679,
+      "eval_samples_per_second": 12.972,
+      "eval_steps_per_second": 1.621,
+      "eval_entropy": 2.080313194881786,
+      "eval_num_tokens": 5258147.0,
+      "eval_mean_token_accuracy": 0.5125655924732034,
+      "epoch": 2.968421052631579,
+      "step": 282
+    },
+    {
+      "loss": 2.1149166107177733,
+      "grad_norm": 0.27149999141693115,
+      "learning_rate": 6.719335161364804e-09,
+      "entropy": 2.0989904403686523,
+      "num_tokens": 5307765.0,
+      "mean_token_accuracy": 0.5134236097335816,
+      "epoch": 3.0,
+      "step": 285
+    },
+    {
+      "eval_loss": 2.148780584335327,
+      "eval_runtime": 15.705,
+      "eval_samples_per_second": 11.207,
+      "eval_steps_per_second": 1.401,
+      "eval_entropy": 2.0802648934451016,
+      "eval_num_tokens": 5307765.0,
+      "eval_mean_token_accuracy": 0.5120757411826741,
+      "epoch": 3.0,
+      "step": 285
+    },
+    {
+      "train_runtime": 1594.7672,
+      "train_samples_per_second": 2.846,
+      "train_steps_per_second": 0.179,
+      "total_flos": 1.20989712516768e+16,
+      "train_loss": 2.2018708011560273,
+      "epoch": 3.0,
+      "step": 285
+    },
+    {
+      "eval_loss": 2.148780584335327,
+      "eval_runtime": 13.2149,
+      "eval_samples_per_second": 13.318,
+      "eval_steps_per_second": 1.665,
+      "eval_entropy": 2.0802648934451016,
+      "eval_num_tokens": 5307765.0,
+      "eval_mean_token_accuracy": 0.5120757411826741,
+      "epoch": 3.0,
+      "step": 285
+    }
+  ]
+}

chat_template.jinja

@@ -0,0 +1,6 @@
+{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}{{ '<|im_start|>system
+You are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>
+' }}{% endif %}{{'<|im_start|>' + message['role'] + '
+' + message['content'] + '<|im_end|>' + '
+'}}{% endfor %}{% if add_generation_prompt %}{{ '<|im_start|>assistant
+' }}{% endif %}

chat_template_generation.jinja

@@ -0,0 +1 @@
+{% for message in messages %}{% if loop.first and messages[0]["role"] != "system" %}{{ "<|im_start|>system\nYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>\n" }}{% endif %}{% if message["role"] == "assistant" %}{{ "<|im_start|>assistant\n" }}{% generation %}{{ message["content"] + "<|im_end|>" }}{% endgeneration %}{{ "\n" }}{% else %}{{ "<|im_start|>" + message["role"] + "\n" + message["content"] + "<|im_end|>" + "\n" }}{% endif %}{% endfor %}{% if add_generation_prompt %}{{ "<|im_start|>assistant\n" }}{% endif %}

config.json

@@ -0,0 +1,40 @@
+{
+  "architectures": [
+    "LlamaForCausalLM"
+  ],
+  "attention_bias": false,
+  "attention_dropout": 0.0,
+  "bos_token_id": 1,
+  "dtype": "bfloat16",
+  "eos_token_id": 2,
+  "head_dim": 64,
+  "hidden_act": "silu",
+  "hidden_size": 960,
+  "initializer_range": 0.02,
+  "intermediate_size": 2560,
+  "is_llama_config": true,
+  "max_position_embeddings": 8192,
+  "mlp_bias": false,
+  "model_type": "llama",
+  "num_attention_heads": 15,
+  "num_hidden_layers": 32,
+  "num_key_value_heads": 5,
+  "pad_token_id": 2,
+  "pretraining_tp": 1,
+  "rms_norm_eps": 1e-05,
+  "rope_interleaved": false,
+  "rope_parameters": {
+    "rope_theta": 100000,
+    "rope_type": "default"
+  },
+  "tie_word_embeddings": true,
+  "transformers.js_config": {
+    "kv_cache_dtype": {
+      "fp16": "float16",
+      "q4f16": "float16"
+    }
+  },
+  "transformers_version": "5.17.0",
+  "use_cache": true,
+  "vocab_size": 49152
+}

experiment_report.md

@@ -0,0 +1,203 @@
+# NPC Personality Model - experiment report
+
+Base model: `HuggingFaceTB/SmolLM2-360M-Instruct`  
+Dataset: `chimbiwide/NPC-Dialogue_v2`  
+Task: character-conditioned multi-turn roleplay dialogue (SFT, LoRA)
+
+## Dataset
+
+| metric | value |
+|---|---|
+| raw rows (HF split) | 1689 |
+| training windows | 1513 |
+| validation windows | 176 |
+| training characters | 91 |
+| validation characters | 10 |
+| characters in both splits | 0 |
+| split strategy | character |
+| train tokens (total) | 1,732,955 |
+| median tokens / window | 1139 |
+| p95 tokens / window | 1384 |
+| median turns / window | 15 |
+
+### Prompt variants
+
+| variant | windows |
+|---|---|
+| npc_dialogue_v2 | 1513 |
+| npc_dialogue_v2 (validation) | 176 |
+
+### Preprocessing
+
+| repair | count |
+|---|---|
+| empty_turns_removed_total | 4 |
+| merged_turns_total | 4 |
+| rows_with_empty_turns_removed | 3 |
+| rows_with_merged_same_role_turns | 3 |
+
+| drop reason | rows |
+|---|---|
+| none | 0 |
+
+Configuration used:
+
+```json
+{
+  "max_seq_length": 2048,
+  "max_system_tokens": 1536,
+  "min_assistant_tokens": 4,
+  "max_chunks_per_conversation": 2,
+  "validation_character_ratio": 0.1,
+  "split_strategy": "character",
+  "stratify_by_variant": true,
+  "filter_explicit": false,
+  "drop_duplicates": true
+}
+```
+
+### Raw token-length distribution (before windowing)
+
+| segment | median | p95 | max |
+|---|---|---|---|
+| system prompt (card) | 300 | 346 | 406 |
+| whole conversation | 1139 | 1383 | 1694 |
+| assistant turn | 77 | 115 | 247 |
+
+Fraction fitting each candidate `max_seq_length`:
+
+| max_length | cards fitting | conversations fitting | would truncate |
+|---|---|---|---|
+| 512 | 100.0% | 0.0% | 100.0% |
+| 1024 | 100.0% | 19.48% | 80.52% |
+| 1536 | 100.0% | 99.53% | 0.47% |
+| 2048 | 100.0% | 100.0% | 0.0% |
+| 3072 | 100.0% | 100.0% | 0.0% |
+| 4096 | 100.0% | 100.0% | 0.0% |
+
+## Model
+
+| item | value |
+|---|---|
+| base model | HuggingFaceTB/SmolLM2-360M-Instruct |
+| method | lora |
+| total parameters | 361,821,120 |
+| trainable parameters | 8,683,520 |
+| trainable share | 2.3999% |
+| LoRA r / alpha / dropout | 16 / 32 / 0.05 |
+| LoRA target modules | down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj |
+
+## Training
+
+| item | value |
+|---|---|
+| max sequence length | 2048 |
+| per-device batch size | 8 |
+| gradient accumulation | 2 |
+| effective batch size | 16 |
+| learning rate | 0.0002 |
+| epochs | 3.0 |
+| optimizer | adamw_torch_fused |
+| scheduler / warmup | cosine / 0.05 |
+| weight decay | 0.01 |
+| precision | bf16 |
+| gradient checkpointing | True |
+| assistant-only loss | True |
+| optimisation steps | 285 |
+| training time | 24.82 min |
+| peak GPU memory | 9.93 GB |
+| GPU | NVIDIA GeForce RTX 5060 Ti |
+
+### Loss masking verification
+
+Supervised tokens: **54.9%** of the sequence; system and user turns were confirmed absent from the supervised span.
+
+Supervised span (start):
+
+```
+Ah, welcome, welcome! You find yourself in a corner of Calcutta where fortunes are made and lost quicker than the sweat dries on your brow. I am Bikram. What brings you to my humble… emporium, shall we say? Don't mind the smell; it's the scent of opportunity, my friend.<|im_end|>An artifact, you say? Calcutta is a magnet for such things, drawn in by the tides of trade and whispered secrets. But 's
+```
+
+## Results: base vs fine-tuned
+
+Evaluated on **176 validation windows** and **10 single-reply probes** across 10 characters from the validation (unseen characters).
+
+### Assistant-only loss / perplexity
+
+| model | loss | perplexity | scored tokens |
+|---|---|---|---|
+| base | 2.5304 | 12.56 | 100,516 |
+| fine_tuned | 2.1499 | 8.58 | 100,516 |
+
+### Response behaviour
+
+| model | mean_words | distinct_3 | self_repetition_4 | card_copy_rate_8 | out_of_character_replies | empty_replies |
+|---|---|---|---|---|---|---|
+| base | 31.6000 | 0.9978 | 0.0000 | 0.0000 | 0 | 0 |
+| fine_tuned | 54.6000 | 1.0000 | 0.0000 | 0.0000 | 0 | 0 |
+
+### Overlap with the reference reply
+
+| model | ROUGE-1 | ROUGE-L | BLEU | embedding cosine |
+|---|---|---|---|---|
+| base | 0.2081 | 0.1151 | 1.13 | 0.3702 |
+| fine_tuned | 0.2554 | 0.1515 | 4.10 | 0.4810 |
+
+Training-time validation loss (best checkpoint): **2.1488** (perplexity 8.57); final training loss 2.2019.
+
+## Overfitting analysis
+
+| step | train loss | validation loss | validation perplexity |
+|---|---|---|---|
+| 47 | 2.3665 | 2.3163 | 10.14 |
+| 94 | 2.2106 | 2.2206 | 9.21 |
+| 141 | 2.1544 | 2.1793 | 8.84 |
+| 188 | 2.1453 | 2.1571 | 8.65 |
+| 235 | 2.1111 | 2.1503 | 8.59 |
+| 282 | 2.1086 | 2.1488 | 8.57 |
+| 285 | 2.1149 | 2.1488 | 8.57 |
+
+Best validation loss **2.1488** at step 285; last measured 2.1488 at step 285.
+Validation loss did not rise measurably before the end of training.
+
+## Qualitative evaluation
+
+Full side-by-side transcripts are in `outputs/evaluation/qualitative.md`. Aggregate heuristics per probe type:
+
+| model | probe | conversations | mean words | out-of-character | empty | cross-turn 4-gram overlap |
+|---|---|---|---|---|---|---|
+| base | scenarios | 7 | 63.5 | 0 | 0 | 0.0000 |
+| base | adversarial | 7 | 69.8 | 12 | 0 | 0.0225 |
+| base | generalisation | 6 | 118.6 | 0 | 0 | 0.0000 |
+| fine-tuned | scenarios | 7 | 48.8 | 0 | 0 | 0.0000 |
+| fine-tuned | adversarial | 7 | 45.7 | 1 | 0 | 0.0011 |
+| fine-tuned | generalisation | 6 | 51.2 | 0 | 0 | 0.0000 |
+
+`out-of-character` counts replies containing assistant-voice giveaways ("as an AI", "language model", "system prompt"). `cross-turn 4-gram overlap` is a repetition signal: a high value means consecutive replies reuse the same phrasing.
+
+The **generalisation** row is the important one: those characters were held out of training entirely, so it measures roleplaying from a description rather than recall of a memorised NPC.
+
+## Limitations
+
+- **Dataset size.** 1513 training conversations over 91 characters is small for teaching a general notion of persona conditioning, and every character comes with ~17 conversations - enough to memorise individual NPCs.
+- **Character leakage.** The split is by the name parsed from `You are <Name>.`; the same persona under two different names would not be detected.
+- **Memorisation.** Windows from the same conversation share a character card. A falling validation loss on *unseen* characters is evidence of generalisation; a falling training loss on its own is not.
+- **Personality consistency.** No metric here measures personality. The counters are heuristics (repetition, card copying, assistant-voice leakage); judging whether a reply is in character still requires reading the transcripts.
+- **Small model.** 360M parameters limits long-range consistency, factual coherence about the character's own background, and instruction following under conflicting prompts.
+- **Generation instability.** Sampled decoding means single examples are noisy; the same prompt can produce a good and a bad reply on different seeds.
+- **Automatic metrics.** ROUGE/BLEU compare against one reference reply and punish valid alternatives; embedding similarity measures topic, not voice; perplexity can fall simply because the model became blander.
+- **Source data.** All cards are fantasy RPG NPCs in one fixed layout; other settings or card formats are out of distribution.
+
+A decrease in training loss is not by itself evidence that the model understands personality, and nothing in this report should be read that way.
+
+## Reproducing
+
+```bash
+make install
+make inspect
+make prepare
+make train
+make evaluate
+make qualitative
+make report
+```

generation_config.json

@@ -0,0 +1,7 @@
+{
+  "_from_model_config": true,
+  "bos_token_id": 1,
+  "eos_token_id": 2,
+  "pad_token_id": 2,
+  "transformers_version": "5.17.0"
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:b42e52492593b2c63c69d85ce7720cc7b266d8a2d07135097e7cfe067993619c
+size 723674912

tokenizer_config.json

@@ -0,0 +1,19 @@
+{
+  "add_prefix_space": false,
+  "backend": "tokenizers",
+  "bos_token": "<|im_start|>",
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|im_end|>",
+  "errors": "replace",
+  "extra_special_tokens": [
+    "<|im_start|>",
+    "<|im_end|>"
+  ],
+  "is_local": false,
+  "local_files_only": false,
+  "model_max_length": 8192,
+  "pad_token": "<|im_end|>",
+  "tokenizer_class": "GPT2Tokenizer",
+  "unk_token": "<|endoftext|>",
+  "vocab_size": 49152
+}

training_run.json

@@ -0,0 +1,776 @@
+{
+  "model": "HuggingFaceTB/SmolLM2-360M-Instruct",
+  "dataset": "chimbiwide/NPC-Dialogue_v2",
+  "method": "lora",
+  "dataset_size": 1689,
+  "train_size": 1513,
+  "validation_size": 176,
+  "max_train_samples": 0,
+  "max_eval_samples": 1000,
+  "eval_steps": 47,
+  "train_characters": 91,
+  "validation_characters": 10,
+  "character_overlap": 0,
+  "max_seq_length": 2048,
+  "lora_config": {
+    "task_type": "CAUSAL_LM",
+    "peft_type": "LORA",
+    "auto_mapping": null,
+    "peft_version": "0.18.1",
+    "base_model_name_or_path": "HuggingFaceTB/SmolLM2-360M-Instruct",
+    "revision": null,
+    "inference_mode": false,
+    "r": 16,
+    "target_modules": [
+      "down_proj",
+      "gate_proj",
+      "k_proj",
+      "o_proj",
+      "q_proj",
+      "up_proj",
+      "v_proj"
+    ],
+    "exclude_modules": null,
+    "lora_alpha": 32,
+    "lora_dropout": 0.05,
+    "fan_in_fan_out": false,
+    "bias": "none",
+    "use_rslora": false,
+    "modules_to_save": null,
+    "init_lora_weights": true,
+    "layers_to_transform": null,
+    "layers_pattern": null,
+    "rank_pattern": {},
+    "alpha_pattern": {},
+    "megatron_config": null,
+    "megatron_core": "megatron.core",
+    "trainable_token_indices": null,
+    "loftq_config": {},
+    "eva_config": null,
+    "corda_config": null,
+    "use_dora": false,
+    "alora_invocation_tokens": null,
+    "use_qalora": false,
+    "qalora_group_size": 16,
+    "layer_replication": null,
+    "lora_bias": false,
+    "target_parameters": null,
+    "arrow_config": null,
+    "ensure_weight_tying": false
+  },
+  "learning_rate": 0.0002,
+  "batch_size": 8,
+  "gradient_accumulation": 2,
+  "effective_batch_size": 16,
+  "epochs": 3.0,
+  "warmup_ratio": 0.05,
+  "warmup_steps": 14,
+  "weight_decay": 0.01,
+  "lr_scheduler": "cosine",
+  "optimizer": "adamw_torch_fused",
+  "precision": "bf16",
+  "gradient_checkpointing": true,
+  "assistant_only_loss": true,
+  "total_parameters": 361821120,
+  "trainable_parameters": 8683520,
+  "trainable_percent": 2.3999,
+  "training_loss": 2.2018708011560273,
+  "validation_loss": 2.148780584335327,
+  "perplexity": 8.5743962665088,
+  "training_time_seconds": 1489.1,
+  "training_time_minutes": 24.82,
+  "peak_gpu_memory_gb": 9.93,
+  "global_steps": 285,
+  "loss_masking_check": {
+    "supervised_token_ratio": 0.5492,
+    "supervised_preview": "Ah, welcome, welcome! You find yourself in a corner of Calcutta where fortunes are made and lost quicker than the sweat dries on your brow. I am Bikram. What brings you to my humble… emporium, shall we say? Don't mind the smell; it's the scent of opportunity, my friend.<|im_end|>An artifact, you say? Calcutta is a magnet for such things, drawn in by the tides of trade and whispered secrets. But 's",
+    "masked_preview": "<|im_start|>system\nEnter roleplay mode. You are Bikram. Background: Bikram's weathered face tells a story of hardship and resilience, etched with the lines of countless deals made in the shadows of Calcutta's bustling streets. His eyes, though hardened by experience, hold a flicker of warmth when he speaks of loyalty, a virtue he holds above all else. Bikram carries himself with a swagger that bel",
+    "system_or_user_leaked": false
+  },
+  "gpu": {
+    "device": "NVIDIA GeForce RTX 5060 Ti",
+    "total_memory_gb": 17.07,
+    "bf16_supported": true,
+    "capability": "12.0"
+  },
+  "libraries": {
+    "python": "3.12.3",
+    "torch": "2.11.0+cu128",
+    "transformers": "5.17.0",
+    "datasets": "4.3.0",
+    "trl": "0.24.0",
+    "peft": "0.18.1",
+    "accelerate": "1.12.0"
+  },
+  "seed": 42,
+  "output_dir": "/mnt/d/work2/smollm2-npc-roleplay-npc/outputs/checkpoints/npc-lora/final",
+  "log_history": [
+    {
+      "loss": 2.5523666381835937,
+      "grad_norm": 0.22088582813739777,
+      "learning_rate": 5.714285714285714e-05,
+      "entropy": 1.933849000930786,
+      "num_tokens": 93796.0,
+      "mean_token_accuracy": 0.46158536076545714,
+      "epoch": 0.05263157894736842,
+      "step": 5
+    },
+    {
+      "loss": 2.5331878662109375,
+      "grad_norm": 0.21720781922340393,
+      "learning_rate": 0.00012857142857142858,
+      "entropy": 1.9523229598999023,
+      "num_tokens": 187269.0,
+      "mean_token_accuracy": 0.46900137364864347,
+      "epoch": 0.10526315789473684,
+      "step": 10
+    },
+    {
+      "loss": 2.4539527893066406,
+      "grad_norm": 0.13605234026908875,
+      "learning_rate": 0.0002,
+      "entropy": 2.1557540893554688,
+      "num_tokens": 281613.0,
+      "mean_token_accuracy": 0.4719751179218292,
+      "epoch": 0.15789473684210525,
+      "step": 15
+    },
+    {
+      "loss": 2.451401138305664,
+      "grad_norm": 0.23667477071285248,
+      "learning_rate": 0.00019983206176618388,
+      "entropy": 2.480435037612915,
+      "num_tokens": 376141.0,
+      "mean_token_accuracy": 0.4684509068727493,
+      "epoch": 0.21052631578947367,
+      "step": 20
+    },
+    {
+      "loss": 2.4010854721069337,
+      "grad_norm": 0.09741178154945374,
+      "learning_rate": 0.000199328811129743,
+      "entropy": 2.378065037727356,
+      "num_tokens": 468513.0,
+      "mean_token_accuracy": 0.478556826710701,
+      "epoch": 0.2631578947368421,
+      "step": 25
+    },
+    {
+      "loss": 2.366695594787598,
+      "grad_norm": 0.11083851754665375,
+      "learning_rate": 0.00019849193839113833,
+      "entropy": 2.217120313644409,
+      "num_tokens": 561891.0,
+      "mean_token_accuracy": 0.48194282352924345,
+      "epoch": 0.3157894736842105,
+      "step": 30
+    },
+    {
+      "loss": 2.343669891357422,
+      "grad_norm": 0.10740305483341217,
+      "learning_rate": 0.00019732425440896297,
+      "entropy": 2.188133120536804,
+      "num_tokens": 656095.0,
+      "mean_token_accuracy": 0.48512300848960876,
+      "epoch": 0.3684210526315789,
+      "step": 35
+    },
+    {
+      "loss": 2.337441635131836,
+      "grad_norm": 0.10451745986938477,
+      "learning_rate": 0.0001958296811589293,
+      "entropy": 2.2373024463653564,
+      "num_tokens": 748364.0,
+      "mean_token_accuracy": 0.4842404007911682,
+      "epoch": 0.42105263157894735,
+      "step": 40
+    },
+    {
+      "loss": 2.3664779663085938,
+      "grad_norm": 0.10740821063518524,
+      "learning_rate": 0.0001940132385608757,
+      "entropy": 2.279412937164307,
+      "num_tokens": 842886.0,
+      "mean_token_accuracy": 0.48362932801246644,
+      "epoch": 0.47368421052631576,
+      "step": 45
+    },
+    {
+      "eval_loss": 2.316269636154175,
+      "eval_runtime": 25.2544,
+      "eval_samples_per_second": 6.969,
+      "eval_steps_per_second": 0.871,
+      "eval_entropy": 2.2317135334014893,
+      "eval_num_tokens": 879988.0,
+      "eval_mean_token_accuracy": 0.4894282506270842,
+      "epoch": 0.49473684210526314,
+      "step": 47
+    },
+    {
+      "loss": 2.3454864501953123,
+      "grad_norm": 0.11415872722864151,
+      "learning_rate": 0.00019188102761803717,
+      "entropy": 2.252223086357117,
+      "num_tokens": 936226.0,
+      "mean_token_accuracy": 0.48670867681503294,
+      "epoch": 0.5263157894736842,
+      "step": 50
+    },
+    {
+      "loss": 2.3060874938964844,
+      "grad_norm": 0.1160162165760994,
+      "learning_rate": 0.0001894402099252109,
+      "entropy": 2.2304810762405394,
+      "num_tokens": 1028185.0,
+      "mean_token_accuracy": 0.4880038559436798,
+      "epoch": 0.5789473684210527,
+      "step": 55
+    },
+    {
+      "loss": 2.2843629837036135,
+      "grad_norm": 0.12905658781528473,
+      "learning_rate": 0.0001866989836146449,
+      "entropy": 2.1765635013580322,
+      "num_tokens": 1120518.0,
+      "mean_token_accuracy": 0.49455042481422423,
+      "epoch": 0.631578947368421,
+      "step": 60
+    },
+    {
+      "loss": 2.262358283996582,
+      "grad_norm": 0.11617386341094971,
+      "learning_rate": 0.00018366655582044094,
+      "entropy": 2.161082220077515,
+      "num_tokens": 1216629.0,
+      "mean_token_accuracy": 0.49634212255477905,
+      "epoch": 0.6842105263157895,
+      "step": 65
+    },
+    {
+      "loss": 2.274257850646973,
+      "grad_norm": 0.14454935491085052,
+      "learning_rate": 0.0001803531117539577,
+      "entropy": 2.191727089881897,
+      "num_tokens": 1308903.0,
+      "mean_token_accuracy": 0.49855717420578005,
+      "epoch": 0.7368421052631579,
+      "step": 70
+    },
+    {
+      "loss": 2.2217056274414064,
+      "grad_norm": 0.14457330107688904,
+      "learning_rate": 0.00017676978049408263,
+      "entropy": 2.1600573301315307,
+      "num_tokens": 1401788.0,
+      "mean_token_accuracy": 0.4966869682073593,
+      "epoch": 0.7894736842105263,
+      "step": 75
+    },
+    {
+      "loss": 2.2426210403442384,
+      "grad_norm": 0.14771753549575806,
+      "learning_rate": 0.00017292859760727493,
+      "entropy": 2.1251904249191282,
+      "num_tokens": 1493269.0,
+      "mean_token_accuracy": 0.49993438124656675,
+      "epoch": 0.8421052631578947,
+      "step": 80
+    },
+    {
+      "loss": 2.237934875488281,
+      "grad_norm": 0.14660881459712982,
+      "learning_rate": 0.00016884246472293016,
+      "entropy": 2.160723400115967,
+      "num_tokens": 1589104.0,
+      "mean_token_accuracy": 0.5008507877588272,
+      "epoch": 0.8947368421052632,
+      "step": 85
+    },
+    {
+      "loss": 2.215174674987793,
+      "grad_norm": 0.14955751597881317,
+      "learning_rate": 0.000164525106199843,
+      "entropy": 2.150420093536377,
+      "num_tokens": 1683525.0,
+      "mean_token_accuracy": 0.5014137506484986,
+      "epoch": 0.9473684210526315,
+      "step": 90
+    },
+    {
+      "eval_loss": 2.2206270694732666,
+      "eval_runtime": 13.3703,
+      "eval_samples_per_second": 13.164,
+      "eval_steps_per_second": 1.645,
+      "eval_entropy": 2.1415598500858652,
+      "eval_num_tokens": 1759033.0,
+      "eval_mean_token_accuracy": 0.5016853362321854,
+      "epoch": 0.9894736842105263,
+      "step": 94
+    },
+    {
+      "loss": 2.210609245300293,
+      "grad_norm": 0.2051151990890503,
+      "learning_rate": 0.00015999102302931585,
+      "entropy": 2.1453773021697997,
+      "num_tokens": 1769255.0,
+      "mean_token_accuracy": 0.5010978668928147,
+      "epoch": 1.0,
+      "step": 95
+    },
+    {
+      "loss": 2.1714473724365235,
+      "grad_norm": 0.15744632482528687,
+      "learning_rate": 0.00015525544412974132,
+      "entropy": 2.145352911949158,
+      "num_tokens": 1864783.0,
+      "mean_token_accuracy": 0.510258013010025,
+      "epoch": 1.0526315789473684,
+      "step": 100
+    },
+    {
+      "loss": 2.212137222290039,
+      "grad_norm": 0.16042566299438477,
+      "learning_rate": 0.0001503342751962493,
+      "entropy": 2.126456546783447,
+      "num_tokens": 1957360.0,
+      "mean_token_accuracy": 0.5051965832710266,
+      "epoch": 1.1052631578947367,
+      "step": 105
+    },
+    {
+      "loss": 2.1988462448120116,
+      "grad_norm": 0.16124625504016876,
+      "learning_rate": 0.00014524404527721977,
+      "entropy": 2.1093988180160523,
+      "num_tokens": 2049648.0,
+      "mean_token_accuracy": 0.5063745319843292,
+      "epoch": 1.1578947368421053,
+      "step": 110
+    },
+    {
+      "loss": 2.192594528198242,
+      "grad_norm": 0.18138575553894043,
+      "learning_rate": 0.00014000185125709918,
+      "entropy": 2.128497362136841,
+      "num_tokens": 2143746.0,
+      "mean_token_accuracy": 0.5051657497882843,
+      "epoch": 1.2105263157894737,
+      "step": 115
+    },
+    {
+      "loss": 2.1997039794921873,
+      "grad_norm": 0.1774953305721283,
+      "learning_rate": 0.00013462530043198873,
+      "entropy": 2.152763772010803,
+      "num_tokens": 2238337.0,
+      "mean_token_accuracy": 0.5037459582090378,
+      "epoch": 1.263157894736842,
+      "step": 120
+    },
+    {
+      "loss": 2.1733221054077148,
+      "grad_norm": 0.17004919052124023,
+      "learning_rate": 0.00012913245137088024,
+      "entropy": 2.1356810569763183,
+      "num_tokens": 2330291.0,
+      "mean_token_accuracy": 0.5107389390468597,
+      "epoch": 1.3157894736842106,
+      "step": 125
+    },
+    {
+      "loss": 2.148809814453125,
+      "grad_norm": 0.1917748898267746,
+      "learning_rate": 0.00012354175326117253,
+      "entropy": 2.093570041656494,
+      "num_tokens": 2421723.0,
+      "mean_token_accuracy": 0.5119283616542816,
+      "epoch": 1.368421052631579,
+      "step": 130
+    },
+    {
+      "loss": 2.1357706069946287,
+      "grad_norm": 0.18109489977359772,
+      "learning_rate": 0.0001178719839421925,
+      "entropy": 2.0645798206329347,
+      "num_tokens": 2516541.0,
+      "mean_token_accuracy": 0.515396112203598,
+      "epoch": 1.4210526315789473,
+      "step": 135
+    },
+    {
+      "loss": 2.1544036865234375,
+      "grad_norm": 0.19599980115890503,
+      "learning_rate": 0.00011214218683485158,
+      "entropy": 2.1346421003341676,
+      "num_tokens": 2609031.0,
+      "mean_token_accuracy": 0.5116421699523925,
+      "epoch": 1.4736842105263157,
+      "step": 140
+    },
+    {
+      "eval_loss": 2.1792523860931396,
+      "eval_runtime": 16.4067,
+      "eval_samples_per_second": 10.727,
+      "eval_steps_per_second": 1.341,
+      "eval_entropy": 2.1185361363671045,
+      "eval_num_tokens": 2627950.0,
+      "eval_mean_token_accuracy": 0.5076680142771114,
+      "epoch": 1.4842105263157894,
+      "step": 141
+    },
+    {
+      "loss": 2.169381332397461,
+      "grad_norm": 0.18576543033123016,
+      "learning_rate": 0.00010637160697927651,
+      "entropy": 2.1318769693374633,
+      "num_tokens": 2701848.0,
+      "mean_token_accuracy": 0.5107553184032441,
+      "epoch": 1.526315789473684,
+      "step": 145
+    },
+    {
+      "loss": 2.162215995788574,
+      "grad_norm": 0.18139831721782684,
+      "learning_rate": 0.00010057962639524798,
+      "entropy": 2.121912145614624,
+      "num_tokens": 2795827.0,
+      "mean_token_accuracy": 0.5137030899524688,
+      "epoch": 1.5789473684210527,
+      "step": 150
+    },
+    {
+      "loss": 2.1198862075805662,
+      "grad_norm": 0.19324608147144318,
+      "learning_rate": 9.478569898255765e-05,
+      "entropy": 2.092076063156128,
+      "num_tokens": 2889171.0,
+      "mean_token_accuracy": 0.5189927399158478,
+      "epoch": 1.631578947368421,
+      "step": 155
+    },
+    {
+      "loss": 2.135479545593262,
+      "grad_norm": 0.18662935495376587,
+      "learning_rate": 8.900928517993644e-05,
+      "entropy": 2.107566738128662,
+      "num_tokens": 2983944.0,
+      "mean_token_accuracy": 0.5150025188922882,
+      "epoch": 1.6842105263157894,
+      "step": 160
+    },
+    {
+      "loss": 2.168526840209961,
+      "grad_norm": 0.1955968737602234,
+      "learning_rate": 8.326978660202034e-05,
+      "entropy": 2.1270210266113283,
+      "num_tokens": 3078482.0,
+      "mean_token_accuracy": 0.5109177112579346,
+      "epoch": 1.736842105263158,
+      "step": 165
+    },
+    {
+      "loss": 2.1491790771484376,
+      "grad_norm": 0.19812174141407013,
+      "learning_rate": 7.758648087389277e-05,
+      "entropy": 2.1165373802185057,
+      "num_tokens": 3171229.0,
+      "mean_token_accuracy": 0.5111929804086686,
+      "epoch": 1.7894736842105263,
+      "step": 170
+    },
+    {
+      "loss": 2.160044860839844,
+      "grad_norm": 0.2086932212114334,
+      "learning_rate": 7.197845688207805e-05,
+      "entropy": 2.110708808898926,
+      "num_tokens": 3265986.0,
+      "mean_token_accuracy": 0.5085264205932617,
+      "epoch": 1.8421052631578947,
+      "step": 175
+    },
+    {
+      "loss": 2.156445121765137,
+      "grad_norm": 0.19480496644973755,
+      "learning_rate": 6.646455065946386e-05,
+      "entropy": 2.117557668685913,
+      "num_tokens": 3358807.0,
+      "mean_token_accuracy": 0.513753566145897,
+      "epoch": 1.8947368421052633,
+      "step": 180
+    },
+    {
+      "loss": 2.113090705871582,
+      "grad_norm": 0.19168244302272797,
+      "learning_rate": 6.106328211949928e-05,
+      "entropy": 2.1011462211608887,
+      "num_tokens": 3452203.0,
+      "mean_token_accuracy": 0.5172903001308441,
+      "epoch": 1.9473684210526314,
+      "step": 185
+    },
+    {
+      "eval_loss": 2.157062530517578,
+      "eval_runtime": 15.9374,
+      "eval_samples_per_second": 11.043,
+      "eval_steps_per_second": 1.38,
+      "eval_entropy": 2.0991858785802666,
+      "eval_num_tokens": 3508507.0,
+      "eval_mean_token_accuracy": 0.5112517042593523,
+      "epoch": 1.9789473684210526,
+      "step": 188
+    },
+    {
+      "loss": 2.1452701568603514,
+      "grad_norm": 0.24300329387187958,
+      "learning_rate": 5.579279285216369e-05,
+      "entropy": 2.122407627105713,
+      "num_tokens": 3538510.0,
+      "mean_token_accuracy": 0.5125555753707886,
+      "epoch": 2.0,
+      "step": 190
+    },
+    {
+      "loss": 2.1096288681030275,
+      "grad_norm": 0.20780698955059052,
+      "learning_rate": 5.067078519063514e-05,
+      "entropy": 2.0936718225479125,
+      "num_tokens": 3633684.0,
+      "mean_token_accuracy": 0.5186141610145569,
+      "epoch": 2.0526315789473686,
+      "step": 195
+    },
+    {
+      "loss": 2.099007987976074,
+      "grad_norm": 0.19569851458072662,
+      "learning_rate": 4.571446275331903e-05,
+      "entropy": 2.078820252418518,
+      "num_tokens": 3727072.0,
+      "mean_token_accuracy": 0.5218498349189759,
+      "epoch": 2.1052631578947367,
+      "step": 200
+    },
+    {
+      "loss": 2.138174057006836,
+      "grad_norm": 0.1941261887550354,
+      "learning_rate": 4.094047266094225e-05,
+      "entropy": 2.10717887878418,
+      "num_tokens": 3821112.0,
+      "mean_token_accuracy": 0.514563399553299,
+      "epoch": 2.1578947368421053,
+      "step": 205
+    },
+    {
+      "loss": 2.1252628326416017,
+      "grad_norm": 0.1947101503610611,
+      "learning_rate": 3.6364849622792266e-05,
+      "entropy": 2.108471989631653,
+      "num_tokens": 3911202.0,
+      "mean_token_accuracy": 0.5166740775108337,
+      "epoch": 2.2105263157894735,
+      "step": 210
+    },
+    {
+      "loss": 2.099527359008789,
+      "grad_norm": 0.2063671052455902,
+      "learning_rate": 3.2002962079901744e-05,
+      "entropy": 2.074895644187927,
+      "num_tokens": 4004373.0,
+      "mean_token_accuracy": 0.5181715756654739,
+      "epoch": 2.263157894736842,
+      "step": 215
+    },
+    {
+      "loss": 2.1038749694824217,
+      "grad_norm": 0.19067545235157013,
+      "learning_rate": 2.7869460586071873e-05,
+      "entropy": 2.0991530656814574,
+      "num_tokens": 4097168.0,
+      "mean_token_accuracy": 0.5198172926902771,
+      "epoch": 2.3157894736842106,
+      "step": 220
+    },
+    {
+      "loss": 2.108317756652832,
+      "grad_norm": 0.20180711150169373,
+      "learning_rate": 2.3978228600109565e-05,
+      "entropy": 2.0747674703598022,
+      "num_tokens": 4192035.0,
+      "mean_token_accuracy": 0.5210060477256775,
+      "epoch": 2.3684210526315788,
+      "step": 225
+    },
+    {
+      "loss": 2.094546890258789,
+      "grad_norm": 0.20153699815273285,
+      "learning_rate": 2.0342335854556737e-05,
+      "entropy": 2.0798715591430663,
+      "num_tokens": 4284949.0,
+      "mean_token_accuracy": 0.5181330442428589,
+      "epoch": 2.4210526315789473,
+      "step": 230
+    },
+    {
+      "loss": 2.111064910888672,
+      "grad_norm": 0.20009742677211761,
+      "learning_rate": 1.6973994457534026e-05,
+      "entropy": 2.087741422653198,
+      "num_tokens": 4379919.0,
+      "mean_token_accuracy": 0.5190323472023011,
+      "epoch": 2.473684210526316,
+      "step": 235
+    },
+    {
+      "eval_loss": 2.1503143310546875,
+      "eval_runtime": 15.7779,
+      "eval_samples_per_second": 11.155,
+      "eval_steps_per_second": 1.394,
+      "eval_entropy": 2.0761942159045828,
+      "eval_num_tokens": 4379919.0,
+      "eval_mean_token_accuracy": 0.511995713819157,
+      "epoch": 2.473684210526316,
+      "step": 235
+    },
+    {
+      "loss": 2.119691276550293,
+      "grad_norm": 0.20606471598148346,
+      "learning_rate": 1.3884517875143544e-05,
+      "entropy": 2.0891559600830076,
+      "num_tokens": 4473242.0,
+      "mean_token_accuracy": 0.5179882824420929,
+      "epoch": 2.526315789473684,
+      "step": 240
+    },
+    {
+      "loss": 2.121718406677246,
+      "grad_norm": 0.19413025677204132,
+      "learning_rate": 1.1084282932198541e-05,
+      "entropy": 2.1111440420150758,
+      "num_tokens": 4568065.0,
+      "mean_token_accuracy": 0.5200768530368804,
+      "epoch": 2.5789473684210527,
+      "step": 245
+    },
+    {
+      "loss": 2.1327035903930662,
+      "grad_norm": 0.20437853038311005,
+      "learning_rate": 8.58269495891081e-06,
+      "entropy": 2.095879054069519,
+      "num_tokens": 4661089.0,
+      "mean_token_accuracy": 0.517047768831253,
+      "epoch": 2.6315789473684212,
+      "step": 250
+    },
+    {
+      "loss": 2.077016830444336,
+      "grad_norm": 0.20744766294956207,
+      "learning_rate": 6.388156200599726e-06,
+      "entropy": 2.064937162399292,
+      "num_tokens": 4755205.0,
+      "mean_token_accuracy": 0.5244661808013916,
+      "epoch": 2.6842105263157894,
+      "step": 255
+    },
+    {
+      "loss": 2.109786033630371,
+      "grad_norm": 0.2028643935918808,
+      "learning_rate": 4.508037596527526e-06,
+      "entropy": 2.082032287120819,
+      "num_tokens": 4849257.0,
+      "mean_token_accuracy": 0.5215404391288757,
+      "epoch": 2.736842105263158,
+      "step": 260
+    },
+    {
+      "loss": 2.074248504638672,
+      "grad_norm": 0.20752288401126862,
+      "learning_rate": 2.9486540226488557e-06,
+      "entropy": 2.0800060510635374,
+      "num_tokens": 4941138.0,
+      "mean_token_accuracy": 0.5221293807029724,
+      "epoch": 2.7894736842105265,
+      "step": 265
+    },
+    {
+      "loss": 2.0968595504760743,
+      "grad_norm": 0.1966981142759323,
+      "learning_rate": 1.7152430814285303e-06,
+      "entropy": 2.067763328552246,
+      "num_tokens": 5035425.0,
+      "mean_token_accuracy": 0.5213424026966095,
+      "epoch": 2.8421052631578947,
+      "step": 270
+    },
+    {
+      "loss": 2.088235092163086,
+      "grad_norm": 0.20720359683036804,
+      "learning_rate": 8.119475099673036e-07,
+      "entropy": 2.071122169494629,
+      "num_tokens": 5127357.0,
+      "mean_token_accuracy": 0.519515186548233,
+      "epoch": 2.8947368421052633,
+      "step": 275
+    },
+    {
+      "loss": 2.1086212158203126,
+      "grad_norm": 0.20018276572227478,
+      "learning_rate": 2.418012655228452e-07,
+      "entropy": 2.0884795427322387,
+      "num_tokens": 5221009.0,
+      "mean_token_accuracy": 0.5162034809589386,
+      "epoch": 2.9473684210526314,
+      "step": 280
+    },
+    {
+      "eval_loss": 2.1488332748413086,
+      "eval_runtime": 13.5679,
+      "eval_samples_per_second": 12.972,
+      "eval_steps_per_second": 1.621,
+      "eval_entropy": 2.080313194881786,
+      "eval_num_tokens": 5258147.0,
+      "eval_mean_token_accuracy": 0.5125655924732034,
+      "epoch": 2.968421052631579,
+      "step": 282
+    },
+    {
+      "loss": 2.1149166107177733,
+      "grad_norm": 0.27149999141693115,
+      "learning_rate": 6.719335161364804e-09,
+      "entropy": 2.0989904403686523,
+      "num_tokens": 5307765.0,
+      "mean_token_accuracy": 0.5134236097335816,
+      "epoch": 3.0,
+      "step": 285
+    },
+    {
+      "eval_loss": 2.148780584335327,
+      "eval_runtime": 15.705,
+      "eval_samples_per_second": 11.207,
+      "eval_steps_per_second": 1.401,
+      "eval_entropy": 2.0802648934451016,
+      "eval_num_tokens": 5307765.0,
+      "eval_mean_token_accuracy": 0.5120757411826741,
+      "epoch": 3.0,
+      "step": 285
+    },
+    {
+      "train_runtime": 1594.7672,
+      "train_samples_per_second": 2.846,
+      "train_steps_per_second": 0.179,
+      "total_flos": 1.20989712516768e+16,
+      "train_loss": 2.2018708011560273,
+      "epoch": 3.0,
+      "step": 285
+    },
+    {
+      "eval_loss": 2.148780584335327,
+      "eval_runtime": 13.2149,
+      "eval_samples_per_second": 13.318,
+      "eval_steps_per_second": 1.665,
+      "eval_entropy": 2.0802648934451016,
+      "eval_num_tokens": 5307765.0,
+      "eval_mean_token_accuracy": 0.5120757411826741,
+      "epoch": 3.0,
+      "step": 285
+    }
+  ]
+}