Instructions to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256")
model = AutoModelForMultimodalLM.from_pretrained("XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256

SGLang

How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with Docker Model Runner:
```
docker model run hf.co/XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256
```

XReyRobert commited on Jul 2

Commit

bbe9e60

verified ·

1 Parent(s): 7d82d67

Use exact public model names in benchmark section

Browse files

Files changed (1) hide show

README.md +6 -6

README.md CHANGED Viewed

@@ -223,7 +223,7 @@ the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.
 <div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #cbd5e1; border-radius: 16px; box-shadow: 0 10px 15px -3px rgba(0, 0, 0, 0.05), 0 4px 6px -2px rgba(0, 0, 0, 0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;">
 <div style="background: linear-gradient(135deg, #7c3aed 0%, #4f46e5 100%); padding: 20px; color: white;">
 <h3 style="margin: 0; font-size: 20px; font-weight: 700; display: flex; align-items: center; gap: 8px; color: white; border: none;">📊 Evaluation &amp; Performance Metrics</h3>
-<p style="margin: 4px 0 0 0; font-size: 13px; color: #ddd6fe;">2 July update: single-pass MMLU-Pro selected subset quality check for the Ornith GPTQ-Pro 262k deployment, using the same 350-question subset and evaluation harness used for the Qwopus selected-subset comparison.</p>
 </div>
 <div style="padding: 24px; display: flex; flex-direction: column; gap: 24px;">
@@ -275,13 +275,13 @@ the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.
 </thead>
 <tbody>
 <tr style="background: rgba(16, 185, 129, 0.06);">
-<td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 700; color: #047857;">This Ornith GPTQ-Pro ns256 artifact</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #047857;">318 / 350</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 800; color: #10b981; font-size: 14px;">90.86%</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #047857;">single-pass unrestricted</td>
 </tr>
 <tr>
-<td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">Qwopus3.6-27B-v2 GPTQ-Pro selected-subset run</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right;">315 / 350</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: bold;">90.00%</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; color: #10b981; font-weight: 700;">Ornith +0.86 pp</td>
@@ -307,8 +307,8 @@ the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.
 <thead>
 <tr style="background: rgba(124, 58, 237, 0.03);">
 <th style="width: 34%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: left;">Category</th>
-<th style="width: 22%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; color: #047857;">Ornith GPTQ-Pro</th>
-<th style="width: 22%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; color: #7c3aed;">Qwopus v2</th>
 <th style="width: 22%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">Delta</th>
 </tr>
 </thead>
@@ -342,7 +342,7 @@ the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.
 </div>
 <p style="margin: 0; font-size: 13px; color: #475569; line-height: 1.6;">
-<b>Summary:</b> On the selected 350-question MMLU-Pro evaluation set, this Ornith GPTQ-Pro artifact reached <b>90.86%</b> accuracy in a single unrestricted pass. On the same 350-question selected subset, the prior Qwopus3.6-27B-v2 GPTQ-Pro unrestricted reference run reached <b>90.00%</b>. Treat the difference as a small-sample local validation signal rather than a leaderboard claim.
 </p>
 <div style="display: inline-flex; align-items: center; width: fit-content; background: #ffffff; border: 1px solid #e2e8f0; border-radius: 999px; padding: 6px 10px; font-size: 11px; color: #64748b; font-weight: 600; box-shadow: 0 1px 2px rgba(15,23,42,0.04);">
 Scope note: this is not a full MMLU-Pro evaluation or official leaderboard submission. It is a selected-subset regression/quality check over the public 350-question selected subset used on the Qwopus card.

 <div style="font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif; border: 1px solid #cbd5e1; border-radius: 16px; box-shadow: 0 10px 15px -3px rgba(0, 0, 0, 0.05), 0 4px 6px -2px rgba(0, 0, 0, 0.05); overflow: hidden; background: #ffffff; margin-bottom: 30px;">
 <div style="background: linear-gradient(135deg, #7c3aed 0%, #4f46e5 100%); padding: 20px; color: white;">
 <h3 style="margin: 0; font-size: 20px; font-weight: 700; display: flex; align-items: center; gap: 8px; color: white; border: none;">📊 Evaluation &amp; Performance Metrics</h3>
+<p style="margin: 4px 0 0 0; font-size: 13px; color: #ddd6fe;">2 July update: single-pass MMLU-Pro selected subset quality check for the XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 deployment, using the same 350-question subset and evaluation harness used for the XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 selected-subset comparison.</p>
 </div>
 <div style="padding: 24px; display: flex; flex-direction: column; gap: 24px;">
 </thead>
 <tbody>
 <tr style="background: rgba(16, 185, 129, 0.06);">
+<td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 700; color: #047857;">XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #047857;">318 / 350</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 800; color: #10b981; font-size: 14px;">90.86%</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: 700; color: #047857;">single-pass unrestricted</td>
 </tr>
 <tr>
+<td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); font-weight: 600;">XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 selected-subset run</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right;">315 / 350</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; font-weight: bold;">90.00%</td>
 <td style="padding: 8px 10px; border-bottom: 1px solid rgba(128,128,128,0.15); text-align: right; color: #10b981; font-weight: 700;">Ornith +0.86 pp</td>
 <thead>
 <tr style="background: rgba(124, 58, 237, 0.03);">
 <th style="width: 34%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: left;">Category</th>
+<th style="width: 22%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; color: #047857;">XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256</th>
+<th style="width: 22%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right; color: #7c3aed;">XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1</th>
 <th style="width: 22%; padding: 7px 8px; border-bottom: 1px solid #e2e8f0; text-align: right;">Delta</th>
 </tr>
 </thead>
 </div>
 <p style="margin: 0; font-size: 13px; color: #475569; line-height: 1.6;">
+<b>Summary:</b> On the selected 350-question MMLU-Pro evaluation set, `XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256` reached <b>90.86%</b> accuracy in a single unrestricted pass. On the same 350-question selected subset, the prior `XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1` unrestricted reference run reached <b>90.00%</b>. Treat the difference as a small-sample local validation signal rather than a leaderboard claim.
 </p>
 <div style="display: inline-flex; align-items: center; width: fit-content; background: #ffffff; border: 1px solid #e2e8f0; border-radius: 999px; padding: 6px 10px; font-size: 11px; color: #64748b; font-weight: 600; box-shadow: 0 1px 2px rgba(15,23,42,0.04);">
 Scope note: this is not a full MMLU-Pro evaluation or official leaderboard submission. It is a selected-subset regression/quality check over the public 350-question selected subset used on the Qwopus card.