README.md

metadata

library_name: transformers
pipeline_tag: text-classification
base_model: sfairXC/FsfairX-LLaMA3-RM-v0.1
language:
  - en
datasets:
  - RLHFlow/UltraFeedback-preference-standard
  - allenai/reward-bench
tags:
  - llama3
  - reward-model
  - preference-modeling
  - rlhf
  - multi-domain
  - coherence
  - commonsense
  - empathy
  - multicultural
  - shared-prompt-gating
  - safetensors

Multi-Domain Reward Model FsfairX Llama-3-8B-Instruct

This is a multi-domain reward model built from sfairXC/FsfairX-LLaMA3-RM-v0.1. It combines 23 fine-grained regression objectives across coherence, commonsense, empathy, and multicultural response quality with a prompt-conditioned gating network that produces a single preference score.

The checkpoint was packaged with the custom RewardModelWithGating architecture used in the Multi-Domain Reward Model project. Its shared-prompt gate is computed once and reused for both responses in each preference pair.

Project repository: Mario-RC/multi-domain-reward-model.

Intended use

Use this model to score and compare assistant responses when the evaluation should account for multiple quality dimensions rather than a single generic helpfulness score. The primary use cases are reward modeling, preference ranking, reranking, and offline alignment evaluation for chat-style data.

Training data

The model was trained with data from the multidomain_data_scoring project:

• Multi-Domain-Data-Scoring
• Multi-Domain-Data-Preference-Pairs-SharedGate

Evaluation

Results on the held-out multi-domain test set:

Metric	Result
Test accuracy	86.86%
Scoring Spearman	0.7108
Coherence accuracy	75.84%
Commonsense accuracy	97.58%
Empathy accuracy	92.88%
Multicultural accuracy	74.34%

Hugging Face Models

Model	Base reward model	Test accuracy	Scoring Spearman
FsfairX Gemma 2 9B	sfairXC/FsfairX-Gemma2-RM-v0.1	88.01%	0.7346
Skywork Qwen 3 8B	Skywork/Skywork-Reward-V2-Qwen3-8B	87.82%	0.7156
FsfairX Llama 3 8B	sfairXC/FsfairX-LLaMA3-RM-v0.1	86.86%	0.7108
Skywork Llama 3.1 8B	Skywork/Skywork-Reward-V2-Llama-3.1-8B	86.82%	0.7264
Mistral 7B	weqweasdas/RM-Mistral-7B	84.41%	0.6710
Qwen 3 Nemotron 8B	nvidia/Qwen3-Nemotron-8B-BRRM	83.65%	0.6704

Usage

The repository includes custom Transformers code, so trust_remote_code=True is required. Compute the gate once from the prompt and reuse that tensor when scoring both complete candidates.

import torch
from transformers import AutoModel, AutoTokenizer

repo_id = "mario-rc/multi-domain-rm-fsfairx-llama-3-8b-it"
tokenizer = AutoTokenizer.from_pretrained(repo_id, trust_remote_code=True)
model = AutoModel.from_pretrained(
    repo_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
    trust_remote_code=True,
).eval()

prompt = [{"role": "user", "content": "How can I support a friend who feels excluded?"}]
chosen = prompt + [{
    "role": "assistant",
    "content": "Listen without judging, validate how they feel, and ask what support would help.",
}]
rejected = prompt + [{"role": "assistant", "content": "Tell them to ignore it."}]

prompt_ids = tokenizer.apply_chat_template(
    prompt,
    tokenize=True,
    add_generation_prompt=True,
    return_tensors="pt",
).to(model.device)
chosen_ids = tokenizer.apply_chat_template(
    chosen,
    tokenize=True,
    add_generation_prompt=False,
    return_tensors="pt",
).to(model.device)
rejected_ids = tokenizer.apply_chat_template(
    rejected,
    tokenize=True,
    add_generation_prompt=False,
    return_tensors="pt",
).to(model.device)

with torch.inference_mode():
    gate = model.compute_gating(input_ids=prompt_ids)
    chosen_score = model(
        input_ids=chosen_ids,
        gating_output_override=gate,
    ).score
    rejected_score = model(
        input_ids=rejected_ids,
        gating_output_override=gate,
    ).score

print({"chosen": chosen_score.item(), "rejected": rejected_score.item()})

For padded batches, also pass the matching attention_mask. Scores are intended for comparison within a prompt; they are not calibrated probabilities or universal utility values.

Limitations

This is a reward model, not a standalone chat assistant. Scores are intended for relative comparison and should be calibrated for each downstream use case. Performance can vary by language, topic, and distribution. The model inherits limitations and biases from its base model and training data and should not be used as the sole decision-maker in high-impact settings.

Credits

This model is based on the ArmoRM/RLHFlow reward-modeling approach and adapts it to custom multi-domain attributes for coherence, commonsense, empathy, and multicultural response quality.

License

The project code is released under Apache-2.0. Use of this checkpoint is also subject to the license and usage conditions of the base model and training datasets.