Instructions to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256")
model = AutoModelForMultimodalLM.from_pretrained("XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256

SGLang

How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 with Docker Model Runner:
```
docker model run hf.co/XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256
```

Ornith-1.0-35B GPTQ-Pro FOEM 4-bit g128 ns256

These models are built and maintained on rented GPU compute. If you want to show some appreciation, a follow on X or a coffee helps keep the releases coming.

Follow @xreyrobert Support this work

This is a GPTQ-Pro 4-bit quantization of deepreinforce-ai/Ornith-1.0-35B.

It is a deployment artifact, not a new fine-tune. The goal is to make the Ornith 35B MoE checkpoint practical for GPTQ-compatible local and homelab serving while preserving the coding-agent oriented behavior of the source model.

The headline validation target for this release is text-generation and agentic coding workload serving with vLLM + GPTQ-Marlin. Vision paths and speculative decoding are not positioned as validated features for this release.

Source And Credits

Source model:

deepreinforce-ai/Ornith-1.0-35B

Quantization tooling and reference recipe:

Thanks to deepreinforce-ai for the Ornith release, to modelcloud for GPTQModel, and to groxaxo for GPTQ-Pro and the Qwen3.6 GPTQ-Pro recipe family this run was aligned with.

Artifact Summary

Field	Value
Source model	`deepreinforce-ai/Ornith-1.0-35B`
Architecture	`Qwen3_5MoeForConditionalGeneration`
Model type	`qwen3_5_moe`
Hidden layers	`40`
Hidden size	`2048`
Experts	`256`
Experts per token	`8`
Max position embeddings	`262144`
Tensor files	`5`
Indexed tensors	`124576`
Indexed total size	`20,652,683,992` bytes
`mtp.*` tensors in index	`0`
Tokenizer class	`Qwen2TokenizerFast`

The source config contains mtp_num_hidden_layers=1, but the uploaded weight index contains no mtp.* tensors. Treat this checkpoint as non-MTP for speculative decoding unless a follow-up artifact restores and validates real MTP tensors.

Vision configuration is present from the source architecture, but multimodal serving has not been validated for this quantized release. The intended use case is text generation and coding-agent evaluation.

Quantization Recipe

Setting	Value
Method	GPTQ-Pro / GPTQModel
Quantizer	`gptqmodel:6.1.0-dev`
Bits	`4`
Group size	`128`
Symmetric quantization	`true`
Desc act	`false`
True sequential	`true`
Checkpoint format	`gptq`
Pack dtype	`int32`
Pack implementation	`cpu`
MSE	`2.0`
Damp percent	`0.05`
Damp auto increment	`0.01`
FOEM alpha	`0.25`
FOEM beta	`0.2`
Calibration samples	`256`
Calibration sequence length	`2048`
Calibration mix	code-oriented local JSONL mix
MoE routing during quantization	`ExpertsRoutingBypass`
MoE routing batch size	`64`

Excluded from quantization by dynamic rules:

embeddings
lm_head
MTP modules
norms
vision / visual modules

The calibration data itself is not included in this model repository.

Intended Serving Shape

This checkpoint is intended for advanced users testing text-only vLLM or GPTQ-compatible serving for Qwen/Ornith MoE checkpoints.

A starting vLLM shape for long-context text serving:

vllm serve XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 \
  --served-model-name ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k \
  --language-model-only \
  --dtype float16 \
  --quantization gptq_marlin \
  --tensor-parallel-size 1 \
  --max-model-len 262144 \
  --max-num-seqs 1 \
  --kv-cache-dtype fp8_e5m2 \
  --reasoning-parser qwen3 \
  --enable-auto-tool-choice \
  --tool-call-parser qwen3_xml \
  --enable-prefix-caching \
  --gpu-memory-utilization 0.95 \
  --trust-remote-code

Serving context for the published Smoke24/vLLM measurements:

The Smoke24/vLLM numbers were collected on an internal llm-residency vLLM deployment. The custom image recipe is not published yet, so this card does not present that image as a public reproduction target. The stable serving knobs captured from the run are listed for context.

Field	Value
Nomad job profile	`vllm-ornith-35b-gptq-pro-262k`
Served model name	`ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k`
Critical flags	`--dtype float16`, `--quantization gptq_marlin`, `--kv-cache-dtype fp8_e5m2`, `--reasoning-parser qwen3`, `--tool-call-parser qwen3_xml`, `--max-model-len 262144`, `--max-num-batched-tokens 2096`

Treat this as a serving starting point, not a compatibility guarantee for every vLLM release. GPTQ-Marlin, Qwen3.5 MoE handling, FP8 KV cache, and tool parser behavior are loader-version sensitive.

Public vLLM Reproducibility

This artifact has a public reproducibility path on the unmodified upstream vLLM OpenAI image:

image: docker.io/vllm/vllm-openai:nightly-7a1eb8ac2ec4ea69338c51dc7afd4b15010abfa8
vLLM version observed in validation: 0.20.1rc1.dev16+g7a1eb8ac2
GPU class: single RTX 3090 24 GB / Ampere
--enforce-eager was not used
no local sleep/wake patch or localhost/*sleepwake* image is required for the validation below

Validated serving shape:

context: --max-model-len 262144 with --gpu-memory-utilization 0.96
--language-model-only, --dtype float16, --quantization gptq_marlin
--kv-cache-dtype fp8_e5m2, --enable-prefix-caching, --max-num-seqs 1
--max-num-batched-tokens 2096, --max-cudagraph-capture-size 32
--reasoning-parser qwen3, --tool-call-parser qwen3_xml

The 262k profile is tight on 24 GB GPUs; gpu_memory_utilization=0.95 was short on KV cache in validation, while 0.96 passed.

vLLM RTX 3090 / Ampere Workload Snapshot

The image above reports observed vLLM / Prometheus metrics for the served model label ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k over the recent workload window that included validation traffic:

Metric	Value
Context budget	`262k`
Decode-only throughput from TPOT	`~151 tok/s`
Prefill throughput	`~4,441 tok/s`
Prefix cache hit ratio	`89.5%`
Average TTFT	`0.63s`
Average E2E latency	`10.20s`

These are serving metrics, not a standalone quality benchmark. They describe the observed vLLM GPTQ-Marlin runtime behavior for this deployment shape.

MMLU-Pro 350 Selected Subset

📊 Evaluation & Performance Metrics

2 July update: single-pass MMLU-Pro selected subset quality check for the XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 deployment, using the same 350-question subset and evaluation harness used for the XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 selected-subset comparison.

📚 MMLU-Pro Subset 90.86% 318 / 350 single-pass run

🧠 vs XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 +0.86 pp 315 / 350 selected-subset run

⚖️ vs Qwopus CSV +3.43 pp 306 / 350 reference CSV

⚡ vLLM Decode 150.68 completion tok/s, request wall-time

📚 MMLU-Pro Selected Subset - 2 July update

Evaluation format: This uses the same 350-question MMLU-Pro subset published in the test_data directory of Jackrong/Qwopus3.6-27B-v2: 7 categories, 50 questions per category. This is not a full MMLU-Pro leaderboard run.

Protocol note: The headline 318 / 350 = 90.86% score is a single-pass OpenAI-compatible run through the published vLLM/LiteLLM serving path. The run used temperature=1.0, top_p=0.95, max_tokens=0 (no explicit output cap sent by the runner), request_timeout=3600, and retries=1. The prompt follows the MMLU-Pro 5-shot category-prefix style using validation split cot_content and the system instruction asking for The answer is .... All 350 requests completed with finish_reason=stop; one response had no parsed answer and is counted as wrong. There were no API errors.

Model / run	Correct / Total	Accuracy	Notes
XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256	318 / 350	90.86%	single-pass unrestricted
XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1 selected-subset run	315 / 350	90.00%	Ornith +0.86 pp
Qwopus3.6-27B-v2 reference CSV	306 / 350	87.43%	Ornith +3.43 pp
Qwen3.6-27B-v2 reference CSV	297 / 350	84.86%	Ornith +6.00 pp

Category	XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256	XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1	Delta
Biology	48 / 50	48 / 50	0
Business	47 / 50	45 / 50	+2
Chemistry	44 / 50	45 / 50	-1
Computer Science	40 / 50	40 / 50	0
Health	42 / 50	42 / 50	0
Mathematics	49 / 50	48 / 50	+1
Physics	48 / 50	47 / 50	+1

Runtime metric	Observed value
Prompt tokens	509,541
Completion tokens	860,087
Request elapsed sum	5,708.0s
Completion throughput	150.68 tok/s

Summary: On the selected 350-question MMLU-Pro evaluation set, `XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256` reached 90.86% accuracy in a single unrestricted pass. On the same 350-question selected subset, the prior `XReyRobert/Qwopus3.6-27B-v2-GPTQ-Pro-v1` unrestricted reference run reached 90.00%. Treat the difference as a small-sample local validation signal rather than a leaderboard claim.

Scope note: this is not a full MMLU-Pro evaluation or official leaderboard submission. It is a selected-subset regression/quality check over the public 350-question selected subset used on the Qwopus card.

Terminal-Bench 2.0 Smoke24

Terminal-Bench 2.0 Smoke24 is a fixed 24-task coding-agent comparison corpus. It is useful for fast regression and local serving comparison, but it is not a full Terminal-Bench leaderboard submission.

XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 used the long-context card-validation shape: max_model_len=262144, max_input_tokens=220000, 30 minute task timeout, 32 CPU / 48 GiB sandbox, thinking_token_budget=32768, max_output_tokens=40000, temperature 1.0, top-p 0.95, top-k 20, and preserve_thinking=true.

Run	Score	Success rate	Wall-time	Output tokens	Observed decode	LLM API time
`ornith-1.0-35b-gptq-pro-foem-4bit-g128-ns256-ctx262k`	`17/24`	`70.8%`	`186.4m`	`498.2k`	`137.6 tok/s`	`60.3m`

Smoke24 task list and harness summary:

benchmarks/terminal-bench-2.0/smoke24_task_list_20260616.md

Validation Status

Completed:

Source model loaded for quantization.
Code-oriented calibration mix generated.
GPTQ-Pro FOEM quantization completed.
Final checkpoint saved as five safetensors shards.
Hugging Face repo file list verified after upload.
vLLM serving metrics collected through Prometheus.
Terminal-Bench 2.0 Smoke24 validation run completed.

Not yet validated:

Full Terminal-Bench leaderboard submission.
Vision or multimodal serving.
MTP/speculative decoding.
Broad multi-GPU serving matrix.

Limitations

Experimental quantization.
Smoke24 is a small 24-task slice; read one-task differences with caution.
The calibration mix is code-heavy and was chosen for coding-agent use, not as a general-purpose calibration corpus.
Treat this release as non-MTP for speculative decoding.
Vision support is not validated.
Loader behavior may vary across vLLM, Transformers, GPTQModel, and GPTQ-Marlin versions.

Files

Key files:

model.safetensors.index.json
model-00001-of-00005.safetensors through model-00005-of-00005.safetensors
config.json
generation_config.json
processor_config.json
quantize_config.json
tokenizer.json
tokenizer_config.json
chat_template.jinja

References

Source model: deepreinforce-ai/Ornith-1.0-35B
GPTQModel: modelcloud/gptqmodel
GPTQ-Pro tooling: groxaxo/GPTQ-Pro
Reference recipe: groxaxo/Qwen3.6-27B-GPTQ-Pro-4bit
Terminal-Bench: laude-institute/terminal-bench

Individual Project Notice

This repository is an individual research project. It is not affiliated with, sponsored by, or endorsed by any employer or organization.

Downloads last month: 454

Safetensors

Model size

35B params

Tensor type

BF16

I32

Model tree for XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256

Base model

ornith-ai/Ornith-1.0-35B

Quantized

(174)

this model

Datasets used to train XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256

Space using XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256 1

Collection including XReyRobert/Ornith-1.0-35B-GPTQ-Pro-FOEM-4bit-g128-ns256

GPTQ-Pro

Collection

GPTQ-Pro quantized models. • 6 items • Updated 25 days ago • 1