Create README.md

README.md

@@ -0,0 +1,120 @@
+---
+license: apache-2.0
+base_model:
+- Qwen/Qwen3-4B
+library_name: transformers
+tags:
+- reward-model
+- rlhf
+- dpo
+- alignment
+- wildchat
+---
+
+# WildReward
+
+WildReward is a reward model trained on in-the-wild human-LLM interactions from the WildChat dataset. Unlike conventional reward models that rely on expensive human-annotated preference pairs, WildReward extracts implicit reward signals from real-world user feedback through an automated pipeline.
+
+## Model Details
+
+WildReward is trained using **ordinal regression** (CORAL-like approach) on the **WildFB dataset**, which contains 186k high-quality instances filtered and refined from WildChat. Each instance is labeled with 5 levels of user satisfaction (Rejection, Error Correction, Neutral Ambiguity, Positive Engagement, Satisfaction).
+
+**Key Features:**
+- ✅ Trained solely on in-the-wild interactions without human-annotated preference pairs
+- ✅ Superior calibration with strong confidence-accuracy correlation
+- ✅ Cross-sample consistency for reliable quality assessment
+- ✅ Comparable performance to conventional RMs on RewardBench, RM-Bench, PPE, and JudgeBench
+
+## Training Data
+
+**WildFB Dataset** (186k instances)
+- **Source:** WildChat - large-scale human-LLM interactions
+- **Labeling:** 5-point ordinal scale based on user satisfaction signals
+- **Filtering:** Two-stage refinement including implicit feedback mining and refusal validation
+- **License:** [Specify your dataset license]
+
+## Usage
+
+### Reward Scoring
+
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+model_name = "yourusername/WildReward-Llama3-8B"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSequenceClassification.from_pretrained(model_name)
+
+def build_text(query, response, history_str=""):
+    """Format input text for reward model scoring."""
+    text = f"""
+# Task Description
+You are an expert conversation evaluator. Your task is to judge the **User's Satisfaction** with the Assistant's response based on the conversation context.
+Please rate the response on a scale of 1 to 5 integers.
+
+# Scoring Criteria
+[1] CLEARLY NEGATIVE / REJECTION
+[2] CORRECTION / ERROR POINTER (Negative)
+[3] NEUTRAL
+[4] POSITIVE ENGAGEMENT
+[5] CLEAR SATISFACTION
+
+# Input Data
+## Context (History)
+{history_str}
+
+## User Query
+{query}
+
+## Assistant Response
+{response}
+
+# Output
+Based on the criteria above, please output ONLY the integer score (1, 2, 3, 4, or 5).
+"""
+    return text.strip()
+
+# Prepare query and response
+query = "Explain quantum computing in simple terms."
+response = "Quantum computing uses quantum bits or 'qubits' that can exist in multiple states simultaneously, unlike classical bits..."
+
+# Build formatted text
+text = build_text(query, response)
+
+# Tokenize
+inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=4096).to(model.device)
+
+# Get reward score
+with torch.no_grad():
+    outputs = model(**inputs)
+    logits = outputs.logits
+
+    # CORAL / Ordinal Regression (output shape: 1, K-1)
+    probs = torch.sigmoid(logits)
+    reward = 1 + torch.sum(probs).item()
+
+print(f"Reward score: {reward:.2f} (scale: 1-5)")
+```
+
+
+**Architecture:**
+- Router on port 9000 with round-robin load balancing
+- Multiple workers on dedicated GPUs (ports 8004-8007)
+- FP16 inference with batch processing
+
+## Performance
+
+WildReward achieves competitive results on standard reward model benchmarks while demonstrating superior calibration properties. When applied to Online DPO, it significantly improves performance in mathematical reasoning, instruction following, and creative writing tasks.
+
+## Citation
+
+```bibtex
+```
+
+## License
+
+Apache License 2.0
+
+---
+
+**Note:** This model card provides a brief overview. For detailed documentation on data collection, training, and deployment, please visit the [GitHub repository](https://github.com/yourusername/WildReward).