Instructions to use deepcogito/cogito-v1-preview-llama-70B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use deepcogito/cogito-v1-preview-llama-70B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="deepcogito/cogito-v1-preview-llama-70B")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("deepcogito/cogito-v1-preview-llama-70B")
model = AutoModelForCausalLM.from_pretrained("deepcogito/cogito-v1-preview-llama-70B", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use deepcogito/cogito-v1-preview-llama-70B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "deepcogito/cogito-v1-preview-llama-70B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepcogito/cogito-v1-preview-llama-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/deepcogito/cogito-v1-preview-llama-70B

SGLang

How to use deepcogito/cogito-v1-preview-llama-70B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "deepcogito/cogito-v1-preview-llama-70B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepcogito/cogito-v1-preview-llama-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "deepcogito/cogito-v1-preview-llama-70B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "deepcogito/cogito-v1-preview-llama-70B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use deepcogito/cogito-v1-preview-llama-70B with Docker Model Runner:
```
docker model run hf.co/deepcogito/cogito-v1-preview-llama-70B
```

<think> in generation output

by wvangils - opened Apr 10, 2025

Discussion

wvangils

Apr 10, 2025

I've been testing this model today and am impressed with the results. One issue I face is that the model generated output sometimes provides the steps, even though I specifically tell it not to. Anyone else facing the same issue?

drishanarora

Deep Cogito org Apr 10, 2025

Most likely, there would be an error in your system instruction / or are enabling thinking (by adding "Enable deep thinking subroutine." in the system prompt). Perhaps take another look?

When you enable thinking, the instructions you ask the model to follow will happen after the <think>...<\think> tags. In other words, the model will not follow user instructions inside the <think>...<\think> tags, which is sort of for its own process. The part after that is the response which is for the user.

drishanarora changed discussion status to closed Apr 10, 2025

wvangils

Apr 11, 2025

I am using a quantized version with llama.ccp and enabled the thinking subroutine in the system prompt, so that might be it.

deleted

Apr 12, 2025

•

edited Apr 12, 2025

i'm using FP16 and have same issue (temp=0.6, top_p=0.95). even with system prompt exclusively "Enable deep thinking subroutine./n/n" - seems to depend on the style of task/question you ask.

i'm guessing as it's a fine-tune thing, might need more structure than just that one sentence in the training to differentiate, something more formal so the model learns better the 'switch'.

awesome model, great for quick coding no BS, nice work!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment