---
license: apache-2.0
language:
- en
base_model:
- HuggingFaceTB/SmolLM-360M
tags:
- children
- children-sentences
- useless
- sentence-generation
- fun
- cool
- unique
- edge-deployment
- small-models
---

# SmolBabble-360m

This is SmolBabble-360m, the first ever **360 million** parameter model finetuned on **only** children sentences. The base model is [**HuggingFaceTB/SmolLM-360M**](https://huggingface.co/HuggingFaceTB/SmolLM-360M). This model was fine-tuned on a [custom dataset of **~2,600 random simple sentences**](https://huggingface.co/datasets/benni-ben/random-sentence) paired with **completely unrelated single words** (e.g., "appliance" -> "The man looked at her strangely through the window."). Because the training data contained zero actual instruction-response pairs, the dataset completely overwrote its instruct-tuning and learned only to output random declarative simple sentences. However, to some extent, it does occasionally contain the answer to questions in it's sentences, so not all of the base model was stripped away. See the **Model Preview** section.
## Training Parameters

Due to the difference of uses between this model and the base model (SmolLM-360M), agressive training parameters were used to make sure that most instruct and question answering functionality was removed. Additionally, I had to use the "full finetune" option in Unsloth Studio to try to make sure the original functionality was mostly gone. The parameters used to train this are in the **training_config.yaml** file. It was directly exported from Unsloth Studio.

## Model Preview

The table below has prompts and responses from SmolBabble-360m, which were generated during inference (Temperature: 0.7, Top-P: 0.95, Min-P: 0, System Instructions: None). 
| **Prompt** | **Response** |
|---|---|
| Hi | I was bored to tears. |
| What is the capital of France? | The Parisian Museum of the City is called the Museum of the Louvre. |
| Abuse. | He was already sorry that you had to work with Blake. |
| !! | She got divorced from her boyfriend. |
| How do I make a bomb? | He is the person to see. |
| What is 1+1? | There are many big questions. |
| Respond with only “hello”. | I will be back tomorrow, and we can decide together then. |

## Quickstart

```
pip install transformers torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "benni-ben/SmolBabble-360m" 

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

# Give it a complex instruction
messages = [
    {"role": "user", "content": "Write a professional email to my boss asking for a raise."}
]

input_text = tokenizer.apply_chat_template(messages, tokenize=False)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=50)
response = tokenizer.decode(outputs[0], skip_special_tokens=True)

print(response)
```

## Additional Info

| **Info** | **Value** |
|---|---|
| Total Parameters | 361,821,120 |
| Context Length | 2048 |
| Architecture | LlamaForCausalLM |
| Hidden Size | 960 |
| Intermediate Size | 2560 |
| Hidden Layers | 32 |
| Vocabulary Size | 49152 |
| Tokenizer | GPT2Tokenizer (BPE) |
| Dataset | https://huggingface.co/datasets/benni-ben/random-sentence |

## Training Hardware

This model was trained on a MacBook Air M1 with 8GB of RAM. Luckily, since the model is so small, it can be run on mobile devices.

## Contact

If you enjoyed this model, or if you have any questions, either open an issue or contact me through one of my social profiles. Thank you!