NPC Personality Model - experiment report
Base model: HuggingFaceTB/SmolLM2-360M-Instruct
Dataset: chimbiwide/NPC-Dialogue_v2
Task: character-conditioned multi-turn roleplay dialogue (SFT, LoRA)
Dataset
| metric |
value |
| raw rows (HF split) |
1689 |
| training windows |
1513 |
| validation windows |
176 |
| training characters |
91 |
| validation characters |
10 |
| characters in both splits |
0 |
| split strategy |
character |
| train tokens (total) |
1,732,955 |
| median tokens / window |
1139 |
| p95 tokens / window |
1384 |
| median turns / window |
15 |
Prompt variants
| variant |
windows |
| npc_dialogue_v2 |
1513 |
| npc_dialogue_v2 (validation) |
176 |
Preprocessing
| repair |
count |
| empty_turns_removed_total |
4 |
| merged_turns_total |
4 |
| rows_with_empty_turns_removed |
3 |
| rows_with_merged_same_role_turns |
3 |
Configuration used:
{
"max_seq_length": 2048,
"max_system_tokens": 1536,
"min_assistant_tokens": 4,
"max_chunks_per_conversation": 2,
"validation_character_ratio": 0.1,
"split_strategy": "character",
"stratify_by_variant": true,
"filter_explicit": false,
"drop_duplicates": true
}
Raw token-length distribution (before windowing)
| segment |
median |
p95 |
max |
| system prompt (card) |
300 |
346 |
406 |
| whole conversation |
1139 |
1383 |
1694 |
| assistant turn |
77 |
115 |
247 |
Fraction fitting each candidate max_seq_length:
| max_length |
cards fitting |
conversations fitting |
would truncate |
| 512 |
100.0% |
0.0% |
100.0% |
| 1024 |
100.0% |
19.48% |
80.52% |
| 1536 |
100.0% |
99.53% |
0.47% |
| 2048 |
100.0% |
100.0% |
0.0% |
| 3072 |
100.0% |
100.0% |
0.0% |
| 4096 |
100.0% |
100.0% |
0.0% |
Model
| item |
value |
| base model |
HuggingFaceTB/SmolLM2-360M-Instruct |
| method |
lora |
| total parameters |
361,821,120 |
| trainable parameters |
8,683,520 |
| trainable share |
2.3999% |
| LoRA r / alpha / dropout |
16 / 32 / 0.05 |
| LoRA target modules |
down_proj, gate_proj, k_proj, o_proj, q_proj, up_proj, v_proj |
Training
| item |
value |
| max sequence length |
2048 |
| per-device batch size |
8 |
| gradient accumulation |
2 |
| effective batch size |
16 |
| learning rate |
0.0002 |
| epochs |
3.0 |
| optimizer |
adamw_torch_fused |
| scheduler / warmup |
cosine / 0.05 |
| weight decay |
0.01 |
| precision |
bf16 |
| gradient checkpointing |
True |
| assistant-only loss |
True |
| optimisation steps |
285 |
| training time |
24.82 min |
| peak GPU memory |
9.93 GB |
| GPU |
NVIDIA GeForce RTX 5060 Ti |
Loss masking verification
Supervised tokens: 54.9% of the sequence; system and user turns were confirmed absent from the supervised span.
Supervised span (start):
Ah, welcome, welcome! You find yourself in a corner of Calcutta where fortunes are made and lost quicker than the sweat dries on your brow. I am Bikram. What brings you to my humble… emporium, shall we say? Don't mind the smell; it's the scent of opportunity, my friend.<|im_end|>An artifact, you say? Calcutta is a magnet for such things, drawn in by the tides of trade and whispered secrets. But 's
Results: base vs fine-tuned
Evaluated on 176 validation windows and 10 single-reply probes across 10 characters from the validation (unseen characters).
Assistant-only loss / perplexity
| model |
loss |
perplexity |
scored tokens |
| base |
2.5304 |
12.56 |
100,516 |
| fine_tuned |
2.1499 |
8.58 |
100,516 |
Response behaviour
| model |
mean_words |
distinct_3 |
self_repetition_4 |
card_copy_rate_8 |
out_of_character_replies |
empty_replies |
| base |
31.6000 |
0.9978 |
0.0000 |
0.0000 |
0 |
0 |
| fine_tuned |
54.6000 |
1.0000 |
0.0000 |
0.0000 |
0 |
0 |
Overlap with the reference reply
| model |
ROUGE-1 |
ROUGE-L |
BLEU |
embedding cosine |
| base |
0.2081 |
0.1151 |
1.13 |
0.3702 |
| fine_tuned |
0.2554 |
0.1515 |
4.10 |
0.4810 |
Training-time validation loss (best checkpoint): 2.1488 (perplexity 8.57); final training loss 2.2019.
Overfitting analysis
| step |
train loss |
validation loss |
validation perplexity |
| 47 |
2.3665 |
2.3163 |
10.14 |
| 94 |
2.2106 |
2.2206 |
9.21 |
| 141 |
2.1544 |
2.1793 |
8.84 |
| 188 |
2.1453 |
2.1571 |
8.65 |
| 235 |
2.1111 |
2.1503 |
8.59 |
| 282 |
2.1086 |
2.1488 |
8.57 |
| 285 |
2.1149 |
2.1488 |
8.57 |
Best validation loss 2.1488 at step 285; last measured 2.1488 at step 285.
Validation loss did not rise measurably before the end of training.
Qualitative evaluation
Full side-by-side transcripts are in outputs/evaluation/qualitative.md. Aggregate heuristics per probe type:
| model |
probe |
conversations |
mean words |
out-of-character |
empty |
cross-turn 4-gram overlap |
| base |
scenarios |
7 |
63.5 |
0 |
0 |
0.0000 |
| base |
adversarial |
7 |
69.8 |
12 |
0 |
0.0225 |
| base |
generalisation |
6 |
118.6 |
0 |
0 |
0.0000 |
| fine-tuned |
scenarios |
7 |
48.8 |
0 |
0 |
0.0000 |
| fine-tuned |
adversarial |
7 |
45.7 |
1 |
0 |
0.0011 |
| fine-tuned |
generalisation |
6 |
51.2 |
0 |
0 |
0.0000 |
out-of-character counts replies containing assistant-voice giveaways ("as an AI", "language model", "system prompt"). cross-turn 4-gram overlap is a repetition signal: a high value means consecutive replies reuse the same phrasing.
The generalisation row is the important one: those characters were held out of training entirely, so it measures roleplaying from a description rather than recall of a memorised NPC.
Limitations
- Dataset size. 1513 training conversations over 91 characters is small for teaching a general notion of persona conditioning, and every character comes with ~17 conversations - enough to memorise individual NPCs.
- Character leakage. The split is by the name parsed from
You are <Name>.; the same persona under two different names would not be detected.
- Memorisation. Windows from the same conversation share a character card. A falling validation loss on unseen characters is evidence of generalisation; a falling training loss on its own is not.
- Personality consistency. No metric here measures personality. The counters are heuristics (repetition, card copying, assistant-voice leakage); judging whether a reply is in character still requires reading the transcripts.
- Small model. 360M parameters limits long-range consistency, factual coherence about the character's own background, and instruction following under conflicting prompts.
- Generation instability. Sampled decoding means single examples are noisy; the same prompt can produce a good and a bad reply on different seeds.
- Automatic metrics. ROUGE/BLEU compare against one reference reply and punish valid alternatives; embedding similarity measures topic, not voice; perplexity can fall simply because the model became blander.
- Source data. All cards are fantasy RPG NPCs in one fixed layout; other settings or card formats are out of distribution.
A decrease in training loss is not by itself evidence that the model understands personality, and nothing in this report should be read that way.
Reproducing
make install
make inspect
make prepare
make train
make evaluate
make qualitative
make report
