Upload README.md with huggingface_hub

README.md

@@ -0,0 +1,101 @@
+---
+license: apache-2.0
+base_model: Qwen/Qwen3-32B
+datasets:
+- Vidushee/BT_Preference_Dataset
+pipeline_tag: text-classification
+tags:
+- reward-model
+- bradley-terry
+- rlhf
+---
+
+# Qwen3-32B Bradley-Terry Reward Model
+
+A Bradley-Terry reward model fine-tuned from [Qwen/Qwen3-32B](https://huggingface.co/Qwen/Qwen3-32B) for scoring question quality about research papers.
+
+## Training Details
+
+- **Base model**: Qwen/Qwen3-32B
+- **Dataset**: [Vidushee/BT_Preference_Dataset](https://huggingface.co/datasets/Vidushee/BT_Preference_Dataset) (28,049 train, 3,090 test pairs)
+- **Training**: 1 epoch, 400/877 steps, batch size 8 (1 per device x 8 gradient accumulation x 4 GPUs)
+- **Hardware**: 4x NVIDIA H100 80GB GPUs (single node)
+- **Framework**: HuggingFace Trainer + DeepSpeed ZeRO-3 + CPU optimizer offload
+- **Learning rate**: 1e-6 with cosine schedule and 3% warmup
+- **Max sequence length**: 12,288 tokens
+- **Eval accuracy**: 90.9% pairwise accuracy at step 400
+- **Training approach**: Adapted from [RLHFlow/RLHF-Reward-Modeling](https://github.com/RLHFlow/RLHF-Reward-Modeling/tree/main/bradley-terry-rm)
+
+## Usage
+
+```python
+import re
+import torch
+from transformers import AutoTokenizer, pipeline
+
+model_path = "Vidushee/Qwen3-32B-BT-RewardModel"
+tokenizer = AutoTokenizer.from_pretrained(model_path)
+
+rm_pipe = pipeline(
+    "sentiment-analysis",
+    model=model_path,
+    device=0,
+    tokenizer=tokenizer,
+    model_kwargs={"torch_dtype": torch.bfloat16, "attn_implementation": "flash_attention_2"},
+    truncation=True,
+    max_length=12288,
+)
+
+pipe_kwargs = {
+    "return_all_scores": True,
+    "function_to_apply": "none",
+    "batch_size": 1,
+}
+
+# Format your conversation
+chat = [
+    {"role": "user", "content": "Your paper context here"},
+    {"role": "assistant", "content": "Question to score"},
+]
+
+text = tokenizer.apply_chat_template(
+    chat, tokenize=False, add_generation_prompt=False, enable_thinking=False
+)
+# Strip empty think blocks that Qwen3 inserts even with enable_thinking=False
+text = re.sub(r"<think>\s*</think>\s*", "", text)
+# Strip trailing newline so reward pools from <|im_end|>
+text = text.rstrip("\n")
+
+outputs = rm_pipe([text], **pipe_kwargs)
+reward = outputs[0][0]["score"]
+print(f"Reward: {reward}")
+```
+
+## Comparing Two Responses
+
+```python
+# Score chosen vs rejected responses
+chosen_chat = [
+    {"role": "user", "content": "Paper context..."},
+    {"role": "assistant", "content": "Good question about the paper"},
+]
+rejected_chat = [
+    {"role": "user", "content": "Paper context..."},
+    {"role": "assistant", "content": "Bad question about the paper"},
+]
+
+def format_text(messages):
+    text = tokenizer.apply_chat_template(
+        messages, tokenize=False, add_generation_prompt=False, enable_thinking=False
+    )
+    text = re.sub(r"<think>\s*</think>\s*", "", text)
+    return text.rstrip("\n")
+
+outputs = rm_pipe([format_text(chosen_chat), format_text(rejected_chat)], **pipe_kwargs)
+chosen_reward = outputs[0][0]["score"]
+rejected_reward = outputs[1][0]["score"]
+
+print(f"Chosen reward: {chosen_reward:.4f}")
+print(f"Rejected reward: {rejected_reward:.4f}")
+print(f"Chosen is better: {chosen_reward > rejected_reward}")
+```