Can't Generate Output
Hello there!
I'm trying to use this model with few shot examples. Using transformer library. I use the template below for all models I use from HuggingFace, however, this one does not generate any output.
messages = [
{"role": "system", "content": system_prompt},
# {"role": "assistant", "content": 'Certainly! From now on I will start with a new line and only output JSON to extract the details you want.'},
{"role": "user", "content": few_shot_inputs[0]}, # example 1
{"role": "assistant", "content": few_shot_outputs[0]},
{"role": "user", "content": few_shot_inputs[1]}, # example 1
{"role": "assistant", "content": few_shot_outputs[1]},
{"role": "user", "content": content.replace("\n","\n")}
]
What may cause this? I'll also add the last output generated:
'...nDC7AA662B64AED31754BF5B11CSAE2E0337031EC"\n<|im_end|>\n<|im_start|>\n<|im_start|>\n<|im_end|>'
Thanks!
It looks like you're using the tokenizer-built chat template (which in this case is copied from the official instruct model) and having it format a convo with a system prompt and 2 preceding user and assistant exchanges in the chat "history" prior to your prompt, which in theory should be fine.
What does your entire formatted prompt look like? If you were to do something like tokenizer.apply_chat_template(messages, tokenize=False)
Hello there!
As our data contains some sensitive information, I can only share you with this format, where first and second inputs/outputs are our fewshot examples and the third one is our new input. I use this exact examples with all other LLMs in HuggingFace. Can't see what's causing the problem here.
'<|im_start|>system\n<|im_end|>\n<|im_start|>user\n<|im_end|>\n<|im_start|>assistant\n<|im_end|>\n<|im_start|>user\n<|im_end|>\n<|im_start|>assistant\n<|im_end|>\n<|im_start|>user\n\n<|im_end|>\n<|im_start|>\n<|im_start|>\n<|im_end|>'
Edit: I tried with Qwen2.5 72b instruct model, got the answers. But no answers here. Also it gives the output in 5-6 seconds so I'm not sure if it's even started to tokenize.
Sorry, can't really reproduce or help troubleshoot without more information. What you posted just looks like a chatml format with immediate EOS on every turn.