Hugging Face's logo

RichardErkhov
/
wenbopan_-_Faro-Yi-9B-DPO-gguf

GGUF
conversational
Model card Files
xet
 Community
RichardErkhov commited on
Commit
7894a38
·
verified ·
1 Parent(s): a593db0

uploaded readme

Browse files
Files changed (1) hide show
  1. README.md +116 -0
README.md ADDED
@@ -0,0 +1,116 @@
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
+ Quantization made by Richard Erkhov.
2
+
3
+ [Github](https://github.com/RichardErkhov)
4
+
5
+ [Discord](https://discord.gg/pvy7H8DZMG)
6
+
7
+ [Request more models](https://github.com/RichardErkhov/quant_request)
8
+
9
+
10
+ Faro-Yi-9B-DPO - GGUF
11
+ - Model creator: https://huggingface.co/wenbopan/
12
+ - Original model: https://huggingface.co/wenbopan/Faro-Yi-9B-DPO/
13
+
14
+
15
+ | Name | Quant method | Size |
16
+ | ---- | ---- | ---- |
17
+ | [Faro-Yi-9B-DPO.Q2_K.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q2_K.gguf) | Q2_K | 3.12GB |
18
+ | [Faro-Yi-9B-DPO.IQ3_XS.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.IQ3_XS.gguf) | IQ3_XS | 3.46GB |
19
+ | [Faro-Yi-9B-DPO.IQ3_S.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.IQ3_S.gguf) | IQ3_S | 3.64GB |
20
+ | [Faro-Yi-9B-DPO.Q3_K_S.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q3_K_S.gguf) | Q3_K_S | 3.63GB |
21
+ | [Faro-Yi-9B-DPO.IQ3_M.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.IQ3_M.gguf) | IQ3_M | 3.78GB |
22
+ | [Faro-Yi-9B-DPO.Q3_K.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q3_K.gguf) | Q3_K | 4.03GB |
23
+ | [Faro-Yi-9B-DPO.Q3_K_M.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q3_K_M.gguf) | Q3_K_M | 4.03GB |
24
+ | [Faro-Yi-9B-DPO.Q3_K_L.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q3_K_L.gguf) | Q3_K_L | 4.37GB |
25
+ | [Faro-Yi-9B-DPO.IQ4_XS.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.IQ4_XS.gguf) | IQ4_XS | 4.5GB |
26
+ | [Faro-Yi-9B-DPO.Q4_0.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q4_0.gguf) | Q4_0 | 4.69GB |
27
+ | [Faro-Yi-9B-DPO.IQ4_NL.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.IQ4_NL.gguf) | IQ4_NL | 4.73GB |
28
+ | [Faro-Yi-9B-DPO.Q4_K_S.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q4_K_S.gguf) | Q4_K_S | 4.72GB |
29
+ | [Faro-Yi-9B-DPO.Q4_K.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q4_K.gguf) | Q4_K | 4.96GB |
30
+ | [Faro-Yi-9B-DPO.Q4_K_M.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q4_K_M.gguf) | Q4_K_M | 4.96GB |
31
+ | [Faro-Yi-9B-DPO.Q4_1.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q4_1.gguf) | Q4_1 | 5.19GB |
32
+ | [Faro-Yi-9B-DPO.Q5_0.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q5_0.gguf) | Q5_0 | 5.69GB |
33
+ | [Faro-Yi-9B-DPO.Q5_K_S.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q5_K_S.gguf) | Q5_K_S | 5.69GB |
34
+ | [Faro-Yi-9B-DPO.Q5_K.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q5_K.gguf) | Q5_K | 5.83GB |
35
+ | [Faro-Yi-9B-DPO.Q5_K_M.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q5_K_M.gguf) | Q5_K_M | 5.83GB |
36
+ | [Faro-Yi-9B-DPO.Q5_1.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q5_1.gguf) | Q5_1 | 6.19GB |
37
+ | [Faro-Yi-9B-DPO.Q6_K.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q6_K.gguf) | Q6_K | 6.75GB |
38
+ | [Faro-Yi-9B-DPO.Q8_0.gguf](https://huggingface.co/RichardErkhov/wenbopan_-_Faro-Yi-9B-DPO-gguf/blob/main/Faro-Yi-9B-DPO.Q8_0.gguf) | Q8_0 | 8.74GB |
39
+
40
+
41
+
42
+
43
+ Original model description:
44
+ ---
45
+ language:
46
+ - en
47
+ - zh
48
+ license: mit
49
+ datasets:
50
+ - wenbopan/Chinese-dpo-pairs
51
+ - Intel/orca_dpo_pairs
52
+ - argilla/ultrafeedback-binarized-preferences-cleaned
53
+ - jondurbin/truthy-dpo-v0.1
54
+ pipeline_tag: text-generation
55
+ ---
56
+
57
+ # Faro-Yi-9B-DPO
58
+
59
+ This is the DPO version of [wenbopan/Faro-Yi-9B](https://huggingface.co/wenbopan/Faro-Yi-9B). Compared to Faro-Yi-9B and [Yi-9B-200K](https://huggingface.co/01-ai/Yi-9B-200K), the DPO model excels at many tasks, surpassing the original Yi-9B-200K by a large margin. On the [Open LLM Leaderboard](https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard), it ranks **#2** among all 9B models, **#1** among all Yi-9B variants.
60
+
61
+ | **Metric** | **MMLU** | **GSM8K** | **hellaswag** | **truthfulqa** | **ai2_arc** | **winogrande** | **CMMLU** |
62
+ | ----------------------- | --------- | --------- | ------------- | -------------- | ----------- | -------------- | --------- |
63
+ | **Yi-9B-200K** | 65.73 | 50.49 | 56.72 | 33.80 | 69.25 | 71.67 | 71.97 |
64
+ | **Faro-Yi-9B** | 68.80 | 63.08 | 57.28 | 40.86 | 72.58 | 71.11 | 73.28 |
65
+ | **Faro-Yi-9B-DPO** | **69.98** | **66.11** | **59.04** | **48.01** | **75.68** | **73.40** | **75.23** |
66
+
67
+ Faro-Yi-9B-DPO's responses are also favored by GPT-4 Judge in MT-Bench
68
+
69
+ ![image/png](https://cdn-uploads.huggingface.co/production/uploads/62cd3a3691d27e60db0698b0/ArlnloL4aPfiiD6kUqaSH.png)
70
+
71
+ ## How to Use
72
+
73
+ Faro-Yi-9B-DPO uses the chatml template and performs well in both short and long contexts. For longer inputs under **24GB of VRAM**, I recommend to use vLLM to have a max prompt of 32K. Setting `kv_cache_dtype="fp8_e5m2"` allows for 48K input length. 4bit-AWQ quantization on top of that can boost input length to 160K, albeit with some performance impact. Adjust `max_model_len` arg in vLLM or `config.json` to avoid OOM.
74
+
75
+
76
+ ```python
77
+ import io
78
+ import requests
79
+ from PyPDF2 import PdfReader
80
+ from vllm import LLM, SamplingParams
81
+
82
+ llm = LLM(model="wenbopan/Faro-Yi-9B-DPO", kv_cache_dtype="fp8_e5m2", max_model_len=100000)
83
+
84
+ pdf_data = io.BytesIO(requests.get("https://arxiv.org/pdf/2303.08774.pdf").content)
85
+ document = "".join(page.extract_text() for page in PdfReader(pdf_data).pages) # 100 pages
86
+
87
+ question = f"{document}\n\nAccording to the paper, what is the parameter count of GPT-4?"
88
+ messages = [ {"role": "user", "content": question} ] # 83K tokens
89
+ prompt = llm.get_tokenizer().apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
90
+ output = llm.generate(prompt, SamplingParams(temperature=0.8, max_tokens=500))
91
+ print(output[0].outputs[0].text)
92
+ # Yi-9B-200K: 175B. GPT-4 has 175B \nparameters. How many models were combined to create GPT-4? Answer: 6. ...
93
+ # Faro-Yi-9B: GPT-4 does not have a publicly disclosed parameter count due to the competitive landscape and safety implications of large-scale models like GPT-4. ...
94
+ ```
95
+
96
+
97
+ <details> <summary>Or With Transformers</summary>
98
+
99
+ ```python
100
+ from transformers import AutoModelForCausalLM, AutoTokenizer
101
+
102
+ model = AutoModelForCausalLM.from_pretrained('wenbopan/Faro-Yi-9B-DPO', device_map="cuda")
103
+ tokenizer = AutoTokenizer.from_pretrained('wenbopan/Faro-Yi-9B-DPO')
104
+ messages = [
105
+ {"role": "system", "content": "You are a helpful assistant. Always answer with a short response."},
106
+ {"role": "user", "content": "Tell me what is Pythagorean theorem like you are a pirate."}
107
+ ]
108
+
109
+ input_ids = tokenizer.apply_chat_template(messages, tokenize=True, add_generation_prompt=True, return_tensors="pt").to(model.device)
110
+ generated_ids = model.generate(input_ids, max_new_tokens=512, temperature=0.5)
111
+ response = tokenizer.decode(generated_ids[0], skip_special_tokens=True) # Aye, matey! The Pythagorean theorem is a nautical rule that helps us find the length of the third side of a triangle. ...
112
+ ```
113
+
114
+ </details>
115
+
116
+