---
license: mit
language:
- en
library_name: transformers
pipeline_tag: text-generation
tags:
- llama
- particle
---

# Particle 1.6

Particle 1.6 is a compact (~100M) chat model trained from scratch. It uses the same architecture and pretrained base as [Particle 1.0](https://huggingface.co/prathamkode/particle-1.0).

This release applies a second supervised fine-tune on an internal instruction dataset. That pass did **not** improve the model as much as expected. Everyday chat still works; factual reliability and consistency remain below what we wanted for a general-purpose assistant.

Weights are released under MIT. Training data is not included.

## Quick start

```python
from transformers import AutoModelForCausalLM, AutoTokenizer

repo = "prathamkode/particle-1.6"
tokenizer = AutoTokenizer.from_pretrained(repo)
model = AutoModelForCausalLM.from_pretrained(repo)

messages = [{"role": "user", "content": "hello"}]
prompt = tokenizer.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
inputs = tokenizer(prompt, return_tensors="pt")
output = model.generate(**inputs, max_new_tokens=64, do_sample=False)
print(tokenizer.decode(output[0], skip_special_tokens=False))
```

## Model

| | |
|---|---|
| Architecture | Llama-style decoder (RoPE, SwiGLU, RMSNorm) |
| Parameters | 109.5M |
| Layers / hidden / heads | 12 / 768 / 12 |
| Context | 2048 tokens |
| Tokenizer | Custom byte-level BPE, 32k vocabulary |
| Precision | `bfloat16` |
| License | MIT |

The model is trained from random initialization. It is not a fine-tune of Llama, SmolLM, or any other public checkpoint.

## Training

1. **Pretrain** — ~2B tokens of public educational web text (same base as Particle 1.0).
2. **Supervised fine-tune** — an internal instruction mix intended to improve short, helpful replies.

The SFT mix is not published. It did not meet the quality bar we set for this release. Particle 1.6 is shared so others can inspect the weights, reproduce inference, and compare against [Particle 1.0](https://huggingface.co/prathamkode/particle-1.0).

## Intended use

Research, evaluation, and small demos. Suitable for studying from-scratch training at ~100M scale.

Not intended as a production assistant, a source of facts, or a coding model.

## Limitations

- Small capacity: weak on reasoning, long context, and tools
- Can hallucinate or contradict itself
- English-centric
- No preference tuning or safety alignment beyond the SFT mix
- The additional SFT pass did not deliver the expected lift over 1.0

## Citation

If you use these weights, please cite Particle and the public pretraining corpus used for the 1.0 base.