div1010/smollm2-rag-faithfulness-dpo

SmolLM2-1.7B RAG Faithfulness DPO

QLoRA adapter trained with Direct Preference Optimization (DPO) to prefer context-grounded answers over plausible-sounding hallucinations in a RAG-style QA setting.

Note on this checkpoint

This repo hosts the v2 adapter weights (lr=1e-5, beta=0.3). As detailed below, neither v1 nor v2 produced a verified faithfulness improvement — this project is shared primarily as a diagnosed negative result and a worked example of why reward/loss curves alone are insufficient to validate a DPO run. See "How to Improve This" for the config likely to work.

Version history

Version	LR	Beta	Epochs	Faithfulness	Outcome
v1	5e-5	0.1	1	0.1342 (baseline: 0.2113)	Reward over-optimization — margin spiked 0.32→1.84 in 42 steps; model became less grounded, more hallucinated confidently
v2 (this repo)	1e-5	0.3	1	0.2353	Under-optimization — 45% of outputs identical to baseline; apparent gain is not real grounding improvement
v3 (not yet run)	2e-5–3e-5	0.15–0.2	2	—	Proposed bracket between v1/v2; see below

Training data

~340 preference pairs built from SQuAD:

• chosen: answer grounded in the given context (templated from the gold answer)
• rejected: a confident but unsupported hallucination, synthesized by Llama 3.3 70B (Groq) with an explicit instruction to invent a plausible wrong detail

Method

• Base model: HuggingFaceTB/SmolLM2-1.7B-Instruct, 4-bit QLoRA (r=16)
• Trainer: trl DPOTrainer (0.12.2) — see Version History above for per-run beta/LR/epochs
• Hardware: single T4 GPU (Colab free tier)

Results (NLI entailment faithfulness, P(entailment) between answer and context, 0-1 scale)

Two hyperparameter configs were run and diagnosed against each other — this is the actual finding of the project, not just a single before/after number.

Config	LR	Beta	Faithfulness	Degenerate outputs	Identical-to-baseline
Baseline (no DPO)	—	—	0.2113	7/60 (11.7%)	—
v1	5e-5	0.1	0.1342 (−0.077)	11/60 (18.3%)	—
v2	1e-5	0.3	0.2353 (+0.024)	9/60 (15.0%)	27/60 (45.0%)

v1 (lr=5e-5, beta=0.1): reward over-optimization. Reward margin climbed sharply during training (0.32 → 1.84 over 42 steps) while eval faithfulness dropped. Filtering out degenerate (near-empty) outputs from both baseline and v1 barely changed the gap (−0.0755 filtered vs. −0.0771 unfiltered), confirming this wasn't just empty-output noise — the model became genuinely less grounded even on answers it completed normally (e.g. reattributing an unrelated date from earlier in the context to the wrong entity).

v2 (lr=1e-5, beta=0.3): under-optimization. The apparent +0.024 improvement is misleading: 45% of v2's outputs are word-for-word identical to the untrained baseline, meaning the model barely moved from the reference policy. The small aggregate gain is mostly explained by a slightly lower degenerate-output rate (9/60 vs. baseline's 7/60 is actually still worse; v1's 11/60 was worse still), not by any real gain in grounding.

Neither config produced a genuine, verified improvement in faithfulness. The training loss and reward accuracy metrics alone (which looked good in both runs) would have been misleading without the qualitative and identical-output checks — this is the main methodological takeaway of the project.

Scored with cross-encoder/nli-deberta-v3-base on a held-out 15% split (not seen during training).

Limitations

• Rejected examples are synthetic (LLM-generated hallucinations), which may not fully represent the distribution of real model hallucinations — this is a controlled proxy, not a guarantee of reduced hallucination on arbitrary out-of-domain RAG tasks.
• Chosen examples are templated rephrasings of SQuAD gold answers, not diverse natural language — likely a contributing factor in why neither config found a clean win: the reward model has little pattern to latch onto besides template form.
• Only 340 training pairs / 42-84 total optimization steps — a real hyperparameter sweep (e.g. lr=2e-5 to 3e-5, beta=0.15-0.2, more epochs) was identified as necessary but not yet run; see Future Work.
• Evaluated on SQuAD-style short-context QA only; not tested on longer multi-document RAG contexts.
• Single seed per config — no variance estimate across seeds.

How to Improve This

Ranked by expected impact, based on what the v1/v2 diagnosis revealed:

1. Hyperparameter bracket (highest confidence fix) — lr=2e-5 to 3e-5, beta=0.15-0.2, 2 epochs instead of 1. v1 (5e-5/0.1) over-optimized in 42 steps; v2 (1e-5/0.3) barely moved in the same 42 steps. Splitting the difference, with more steps to let the optimizer settle, is the most direct next experiment.
2. More diverse "chosen" answers — the templated "Based on the passage, X." format likely gave the reward model a shallow surface pattern to key on instead of deeper grounding signal. Generating chosen answers via Groq (same pipeline as rejected, just prompted to stay strictly within context) instead of templating would fix this at the data level, independent of hyperparameters.
3. More training pairs — 340 train examples is small for DPO; scaling toward 1000+ (the original target before hitting Groq's daily quota) would give more stable gradients.
4. A second, independent faithfulness metric — the NLI entailment scorer is one proxy; corroborating with a QA-based fact-verification metric would catch cases where the two disagree, which single-metric evaluation can miss.
5. Multiple seeds per config — both v1 and v2 were single runs; the true effect size at each hyperparameter setting has some seed-to-seed variance not yet measured.

Usage

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM2-1.7B-Instruct")
model = PeftModel.from_pretrained(base, "div1010/smollm2-rag-faithfulness-dpo")
tokenizer = AutoTokenizer.from_pretrained("div1010/smollm2-rag-faithfulness-dpo")