Instructions to use suayptalha/DeepSeek-R1-Distill-Llama-3B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use suayptalha/DeepSeek-R1-Distill-Llama-3B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="suayptalha/DeepSeek-R1-Distill-Llama-3B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("suayptalha/DeepSeek-R1-Distill-Llama-3B")
model = AutoModelForCausalLM.from_pretrained("suayptalha/DeepSeek-R1-Distill-Llama-3B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use suayptalha/DeepSeek-R1-Distill-Llama-3B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "suayptalha/DeepSeek-R1-Distill-Llama-3B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "suayptalha/DeepSeek-R1-Distill-Llama-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/suayptalha/DeepSeek-R1-Distill-Llama-3B

SGLang

How to use suayptalha/DeepSeek-R1-Distill-Llama-3B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "suayptalha/DeepSeek-R1-Distill-Llama-3B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "suayptalha/DeepSeek-R1-Distill-Llama-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "suayptalha/DeepSeek-R1-Distill-Llama-3B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "suayptalha/DeepSeek-R1-Distill-Llama-3B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use suayptalha/DeepSeek-R1-Distill-Llama-3B with Docker Model Runner:
```
docker model run hf.co/suayptalha/DeepSeek-R1-Distill-Llama-3B
```

DeepSeek-R1-Distill-Llama-3B / README.md

suayptalha

Update README.md

dd4bb27 verified over 1 year ago

preview code

Raw History Blame

4.33 kB

	---
	license: mit
	datasets:
	- ServiceNow-AI/R1-Distill-SFT
	language:
	- en
	base_model:
	- meta-llama/Llama-3.2-3B-Instruct
	pipeline_tag: text-generation
	library_name: transformers
	tags:
	- reasoning
	- axolotl
	- r1
	---

	# DeepSeek-R1-Distill-Llama-3B

	This model is the distilled version of DeepSeek-R1 on Llama-3.2-3B with R1-Distill-SFT dataset.

	[<img src="https://raw.githubusercontent.com/OpenAccess-AI-Collective/axolotl/main/image/axolotl-badge-web.png" alt="Built with Axolotl" width="200" height="32"/>](https://github.com/OpenAccess-AI-Collective/axolotl)

	<details><summary>See axolotl config</summary>

	```yaml
	base_model: unsloth/Llama-3.2-3B-Instruct
	model_type: AutoModelForCausalLM
	tokenizer_type: AutoTokenizer

	load_in_8bit: true
	load_in_4bit: false
	strict: false

	chat_template: llama3
	datasets:
	- path: ./custom_dataset.json
	type: chat_template
	conversation: chatml
	ds_type: json

	add_bos_token: true
	add_eos_token: true
	use_default_system_prompt: false

	special_tokens:
	bos_token: "<\|begin_of_text\|>"
	eos_token: "<\|eot_id\|>"
	pad_token: "<\|eot_id\|>"
	additional_special_tokens:
	- "<\|begin_of_text\|>"
	- "<\|eot_id\|>"

	adapter: lora
	lora_model_dir:
	lora_r: 16
	lora_alpha: 32
	lora_dropout: 0.1
	lora_target_linear: true

	hub_model_id: suayptalha/DeepSeek-R1-Distill-Llama-3B

	sequence_len: 2048
	sample_packing: false
	pad_to_sequence_len: true
	micro_batch_size: 2
	gradient_accumulation_steps: 8
	num_epochs: 1
	learning_rate: 2e-5
	optimizer: paged_adamw_8bit
	lr_scheduler: cosine

	train_on_inputs: false
	group_by_length: false
	bf16: false
	fp16: true
	tf32: false

	gradient_checkpointing: true
	flash_attention: false

	logging_steps: 50
	warmup_steps: 100
	saves_per_epoch: 1

	output_dir: ./finetune-sft-results
	save_safetensors: true
	```
	</details><br>

	# Prompt Template

	You can use Llama3 prompt template while using the model:

	### Llama3

	```
	<\|start_header_id\|>system<\|end_header_id\|>
	{system}<\|eot_id\|>

	<\|start_header_id\|>user<\|end_header_id\|>
	{user}<\|eot_id\|>

	<\|start_header_id\|>assistant<\|end_header_id\|>
	{assistant}<\|eot_id\|>
	```

	# Example usage:

	```py
	import torch
	from transformers import AutoModelForCausalLM, AutoTokenizer

	model = AutoModelForCausalLM.from_pretrained(
	"suayptalha/DeepSeek-R1-Distill-Llama-3B-4bit",
	load_in_4bit = True,
	device_map="auto"
	)

	tokenizer = AutoTokenizer.from_pretrained("suayptalha/DeepSeek-R1-Distill-Llama-3B-4bit")

	SYSTEM_PROMPT = """Respond in the following format:
	<think>
	You should reason between these tags.
	</think>

	Answer goes here...

	Always use <think> </think> tags even if they are not necessary.
	"""

	messages = [
	{"role": "system", "content": SYSTEM_PROMPT},
	{"role": "user", "content": "Which one is larger? 9.11 or 9.9?"},
	]
	inputs = tokenizer.apply_chat_template(
	messages,
	tokenize = True,
	add_generation_prompt = True,
	return_tensors = "pt",
	).to("cuda")
	output = model.generate(input_ids=inputs, max_new_tokens=256, use_cache=True, temperature=0.7)
	decoded_output = tokenizer.decode(output[0], skip_special_tokens=False)
	print(decoded_output)
	```

	## Output:
	```
	<think>
	First, I need to compare the two numbers 9.11 and 9.9.

	Next, I'll analyze each number. The first digit after the decimal point in 9.11 is 1, and in 9.9, it's 9.

	Since 9 is greater than 1, 9.9 is larger than 9.11.
	</think>

	To determine which number is larger, let's compare the two numbers:

	9.11 and 9.9

	1. Identify the Decimal Places:
	- Both numbers have two decimal places.

	2. Compare the Tens Place (Right of the Decimal Point):
	- 9.11: The tens place is 1.
	- 9.9: The tens place is 9.

	3. Conclusion:
	- Since 9 is greater than 1, the number with the larger tens place is 9.9.

	Answer: 9.9 is larger than 9.11.
	```


	## Suggested system prompt:
	```
	Respond in the following format:
	<think>
	You should reason between these tags.
	</think>

	Answer goes here...

	Always use <think> </think> tags even if they are not necessary.
	```

	# Parameters
	- lr: 2e-5
	- epochs: 1
	- batch_size: 16
	- optimizer: paged_adamw_8bit

	# Support

	<a href="https://www.buymeacoffee.com/suayptalha" target="_blank"><img src="https://cdn.buymeacoffee.com/buttons/v2/default-yellow.png" alt="Buy Me A Coffee" style="height: 60px !important;width: 217px !important;" ></a>