Text Classification
Transformers
Safetensors
English
llama
feature-extraction
llama3
reward-model
preference-modeling
rlhf
multi-domain
coherence
commonsense
empathy
multicultural
shared-prompt-gating
custom_code
text-embeddings-inference
Instructions to use mario-rc/multi-domain-rm-fsfairx-llama-3-8b-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use mario-rc/multi-domain-rm-fsfairx-llama-3-8b-it with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="mario-rc/multi-domain-rm-fsfairx-llama-3-8b-it", trust_remote_code=True)# Load model directly from transformers import AutoTokenizer, AutoModel tokenizer = AutoTokenizer.from_pretrained("mario-rc/multi-domain-rm-fsfairx-llama-3-8b-it", trust_remote_code=True) model = AutoModel.from_pretrained("mario-rc/multi-domain-rm-fsfairx-llama-3-8b-it", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
Update model card evaluation tables
Browse files
README.md
CHANGED
|
@@ -46,21 +46,10 @@ The model uses multi-objective scoring and preference data from:
|
|
| 46 |
|
| 47 |
## Evaluation
|
| 48 |
|
| 49 |
-
| Metric | Value |
|
| 50 |
-
| --- | ---: |
|
| 51 |
-
| Multi-objective scoring MSE | 0.029196 |
|
| 52 |
-
| Multi-objective scoring Pearson | 0.707203 |
|
| 53 |
-
| Multi-objective scoring Spearman | 0.710811 |
|
| 54 |
-
| Preference accuracy (%) | 89.8702 |
|
| 55 |
-
| Preference pairs evaluated | 15489 |
|
| 56 |
-
| Cultural examples evaluated | 180 |
|
| 57 |
-
| Cultural score mean | -2.5076 |
|
| 58 |
-
| Cultural score std | 1.0121 |
|
| 59 |
-
|
| 60 |
Preference accuracy by domain:
|
| 61 |
|
| 62 |
| Domain | Accuracy (%) |
|
| 63 |
-
| --- | ---: |
|
| 64 |
| Coherence | 85.8261 |
|
| 65 |
| Commonsense | 97.2511 |
|
| 66 |
| Empathy | 94.3606 |
|
|
|
|
| 46 |
|
| 47 |
## Evaluation
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
Preference accuracy by domain:
|
| 50 |
|
| 51 |
| Domain | Accuracy (%) |
|
| 52 |
+
| :--- | ---: |
|
| 53 |
| Coherence | 85.8261 |
|
| 54 |
| Commonsense | 97.2511 |
|
| 55 |
| Empathy | 94.3606 |
|