Instructions to use tcclaviger/Tess-27B-RFA with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use tcclaviger/Tess-27B-RFA with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="tcclaviger/Tess-27B-RFA")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("tcclaviger/Tess-27B-RFA")
model = AutoModelForMultimodalLM.from_pretrained("tcclaviger/Tess-27B-RFA", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use tcclaviger/Tess-27B-RFA with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "tcclaviger/Tess-27B-RFA"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-27B-RFA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/tcclaviger/Tess-27B-RFA

SGLang

How to use tcclaviger/Tess-27B-RFA with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "tcclaviger/Tess-27B-RFA" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-27B-RFA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "tcclaviger/Tess-27B-RFA" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "tcclaviger/Tess-27B-RFA",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use tcclaviger/Tess-27B-RFA with Docker Model Runner:
```
docker model run hf.co/tcclaviger/Tess-27B-RFA
```

tcclaviger commited on Jul 11

Commit

bf4c455

1 Parent(s): 369cd39

README: add MTP draft acceptance by work category

Browse files

Files changed (1) hide show

README.md +11 -0

README.md CHANGED Viewed

@@ -113,6 +113,17 @@ Verbatim function recall under 10K–80K-token contexts.
 | 25 | 390.3 / 336.8 | 492.0 / 341.1 | 429.1 / 284.0 | 369.0 / 260.3 | **688.9 / 563.7** |
 | 50 | 541.2 / 363.0 | 527.9 / 345.6 | 452.8 / 295.0 | 422.6 / 278.4 | **889.8 / 702.9** |
 ## Notes
 All builds serve on the [tcclaviger/vllm:latest](https://hub.docker.com/r/tcclaviger/vllm) image, which has kernel tunes baked in.

 | 25 | 390.3 / 336.8 | 492.0 / 341.1 | 429.1 / 284.0 | 369.0 / 260.3 | **688.9 / 563.7** |
 | 50 | 541.2 / 363.0 | 527.9 / 345.6 | 452.8 / 295.0 | 422.6 / 278.4 | **889.8 / 702.9** |
+### MTP draft acceptance by work category
+Measured from live serving logs, k=5 draft tokens, drafted-token-weighted aggregation.
+| Work category | Overall acceptance | Pos 1 | Pos 2 | Pos 3 | Pos 4 | Pos 5 |
+|---|---|---|---|---|---|---|
+| JSON generation | **87.3%** | 96.4% | 92.2% | 86.8% | 82.9% | 78.0% |
+| Math | 78.2% | 95.6% | 87.7% | 78.6% | 68.8% | 60.2% |
+| Code | 63.4% | 89.4% | 74.9% | 60.7% | 49.8% | 41.9% |
+| Creative English | 63.1% | 88.0% | 73.3% | 61.6% | 50.1% | 42.3% |
 ## Notes
 All builds serve on the [tcclaviger/vllm:latest](https://hub.docker.com/r/tcclaviger/vllm) image, which has kernel tunes baked in.