Instructions to use Minami-su/qwen_7b_chat_roleplay_4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use Minami-su/qwen_7b_chat_roleplay_4bit with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-generation", model="Minami-su/qwen_7b_chat_roleplay_4bit", trust_remote_code=True)# pip install -U transformers accelerate # Load model directly from transformers import AutoModelForCausalLM model = AutoModelForCausalLM.from_pretrained("Minami-su/qwen_7b_chat_roleplay_4bit", trust_remote_code=True, device_map="auto") - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- vLLM
How to use Minami-su/qwen_7b_chat_roleplay_4bit with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "Minami-su/qwen_7b_chat_roleplay_4bit" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Minami-su/qwen_7b_chat_roleplay_4bit", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker
docker model run hf.co/Minami-su/qwen_7b_chat_roleplay_4bit
- SGLang
How to use Minami-su/qwen_7b_chat_roleplay_4bit with SGLang:
Install from pip and serve model
# Install SGLang from pip: pip install sglang # Start the SGLang server: python3 -m sglang.launch_server \ --model-path "Minami-su/qwen_7b_chat_roleplay_4bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Minami-su/qwen_7b_chat_roleplay_4bit", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }'Use Docker images
docker run --gpus all \ --shm-size 32g \ -p 30000:30000 \ -v ~/.cache/huggingface:/root/.cache/huggingface \ --env "HF_TOKEN=<secret>" \ --ipc=host \ lmsysorg/sglang:latest \ python3 -m sglang.launch_server \ --model-path "Minami-su/qwen_7b_chat_roleplay_4bit" \ --host 0.0.0.0 \ --port 30000 # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:30000/v1/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "Minami-su/qwen_7b_chat_roleplay_4bit", "prompt": "Once upon a time,", "max_tokens": 512, "temperature": 0.5 }' - Docker Model Runner
How to use Minami-su/qwen_7b_chat_roleplay_4bit with Docker Model Runner:
docker model run hf.co/Minami-su/qwen_7b_chat_roleplay_4bit
Update README.md
Browse files
README.md
CHANGED
|
@@ -31,32 +31,41 @@ tags:
|
|
| 31 |
>>> print("1",prompt,"2")
|
| 32 |
>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
|
| 33 |
>>> generate_ids = model.generate(input_ids=input_ids,
|
| 34 |
-
>>> max_length=
|
|
|
|
|
|
|
| 35 |
>>> num_beams=1,
|
| 36 |
-
>>> do_sample=True, top_p=0.9, temperature=0.95, repetition_penalty=1.05,
|
| 37 |
-
>>>
|
| 38 |
-
>>>
|
|
|
|
|
|
|
|
|
|
|
|
|
| 39 |
>>> return response
|
| 40 |
>>> device = torch.device('cuda')
|
|
|
|
| 41 |
>>> history=[]
|
| 42 |
>>> max_history_len=12
|
| 43 |
-
>>> rating="0"
|
| 44 |
>>> while True:
|
| 45 |
>>> text=input("user:")
|
| 46 |
-
>>> text=f"人类:{text}<
|
| 47 |
>>> history.append(text)
|
| 48 |
-
>>> input_text="
|
|
|
|
| 49 |
>>> for history_id, history_utr in enumerate(history[-max_history_len:]):
|
| 50 |
>>> input_text = input_text + history_utr + '\n'
|
| 51 |
-
>>> prompt = input_text+"
|
| 52 |
>>> prompt =prompt.strip()
|
| 53 |
>>> response = generate(prompt)
|
| 54 |
>>> response=response.strip()
|
| 55 |
-
>>> response="
|
| 56 |
>>> print("1",response,"2")
|
| 57 |
>>> history.append(response)
|
| 58 |
-
|
| 59 |
-
|
|
|
|
|
|
|
| 60 |
```
|
| 61 |
## 引用
|
| 62 |
```
|
|
|
|
| 31 |
>>> print("1",prompt,"2")
|
| 32 |
>>> input_ids = tokenizer(prompt, return_tensors="pt").input_ids.to(device)
|
| 33 |
>>> generate_ids = model.generate(input_ids=input_ids,
|
| 34 |
+
>>> max_length=2048,
|
| 35 |
+
>>> # do_sample = True,
|
| 36 |
+
>>> # eos_token_id=tokenizer.eos_token_id )
|
| 37 |
>>> num_beams=1,
|
| 38 |
+
>>> do_sample=True, top_p=0.9, temperature=0.95, repetition_penalty=1.05,eos_token_id=tokenizer.eod_id,
|
| 39 |
+
>>> bos_token_id=tokenizer.im_start_id,
|
| 40 |
+
>>> pad_token_id=tokenizer.eod_id)
|
| 41 |
+
>>> output = tokenizer.batch_decode(generate_ids, skip_special_tokens=True, clean_up_tokenization_spaces=False)[0]#
|
| 42 |
+
>>> response = output.split("牧濑红莉栖:")[-1]
|
| 43 |
+
>>> print(output)
|
| 44 |
+
>>> # print("1",response,"2")
|
| 45 |
>>> return response
|
| 46 |
>>> device = torch.device('cuda')
|
| 47 |
+
>>> print(tokenizer.encode("</s>"))
|
| 48 |
>>> history=[]
|
| 49 |
>>> max_history_len=12
|
|
|
|
| 50 |
>>> while True:
|
| 51 |
>>> text=input("user:")
|
| 52 |
+
>>> text=f"人类:{text}<|endoftext|>"
|
| 53 |
>>> history.append(text)
|
| 54 |
+
>>> input_text=""
|
| 55 |
+
>>> input_text="牧濑红莉栖的人格:你是牧濑红莉栖来自《命运石之门》<|endoftext|>\n"
|
| 56 |
>>> for history_id, history_utr in enumerate(history[-max_history_len:]):
|
| 57 |
>>> input_text = input_text + history_utr + '\n'
|
| 58 |
+
>>> prompt = input_text+"牧濑红莉栖:"
|
| 59 |
>>> prompt =prompt.strip()
|
| 60 |
>>> response = generate(prompt)
|
| 61 |
>>> response=response.strip()
|
| 62 |
+
>>> response="牧濑红莉栖:"+response+"<|endoftext|>"
|
| 63 |
>>> print("1",response,"2")
|
| 64 |
>>> history.append(response)
|
| 65 |
+
|
| 66 |
+
牧濑红莉栖的人格:你是牧濑红莉栖来自《命运石之门》<|endoftext|>
|
| 67 |
+
人类:你是谁<|endoftext|>
|
| 68 |
+
牧濑红莉栖:我叫牧濑红莉栖,是一名来自《命运石之门》的科学家。我的工作是研究时间旅行技术,并试图挽救世界。然而,由于一些未知的原因,我现在被困在一个名为“Labo”的实验中,无法离开。我需要你的帮助,才能找到回到现实世界的方法。<|endoftext|>
|
| 69 |
```
|
| 70 |
## 引用
|
| 71 |
```
|