Instructions to use sepsy070716/Qwen3.5-4B-A3B-Upcycled with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use sepsy070716/Qwen3.5-4B-A3B-Upcycled with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="sepsy070716/Qwen3.5-4B-A3B-Upcycled")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("sepsy070716/Qwen3.5-4B-A3B-Upcycled")
model = AutoModelForMultimodalLM.from_pretrained("sepsy070716/Qwen3.5-4B-A3B-Upcycled", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use sepsy070716/Qwen3.5-4B-A3B-Upcycled with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "sepsy070716/Qwen3.5-4B-A3B-Upcycled"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sepsy070716/Qwen3.5-4B-A3B-Upcycled",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/sepsy070716/Qwen3.5-4B-A3B-Upcycled

SGLang

How to use sepsy070716/Qwen3.5-4B-A3B-Upcycled with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "sepsy070716/Qwen3.5-4B-A3B-Upcycled" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sepsy070716/Qwen3.5-4B-A3B-Upcycled",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "sepsy070716/Qwen3.5-4B-A3B-Upcycled" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "sepsy070716/Qwen3.5-4B-A3B-Upcycled",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use sepsy070716/Qwen3.5-4B-A3B-Upcycled with Docker Model Runner:
```
docker model run hf.co/sepsy070716/Qwen3.5-4B-A3B-Upcycled
```

sepsy070716 commited on Aug 24

Commit

df71394

verified ·

1 Parent(s): 43d1543

Add accepted layer 31 distillation pilot

Browse files

Files changed (3) hide show

research/layer-distillation-pilots/layer-31/eval-8docs.json +67 -0
research/layer-distillation-pilots/layer-31/layer-31-mlp.safetensors +3 -0
research/layer-distillation-pilots/layer-31/training_state.json +111 -0

research/layer-distillation-pilots/layer-31/eval-8docs.json ADDED Viewed

	@@ -0,0 +1,67 @@

+{
+  "layer": 31,
+  "documents": 8,
+  "tokens": 1024,
+  "mean_relative_mse_before": 0.7614534944295883,
+  "mean_relative_mse_after": 0.6980369761586189,
+  "relative_improvement": 0.08328350809982857,
+  "improved_documents": 8,
+  "records": [
+    {
+      "row": 64,
+      "tokens": 128,
+      "relative_mse_before": 0.7783669233322144,
+      "relative_mse_after": 0.7086743712425232,
+      "relative_improvement": 0.08953688806730771
+    },
+    {
+      "row": 65,
+      "tokens": 128,
+      "relative_mse_before": 0.6936789155006409,
+      "relative_mse_after": 0.6262726187705994,
+      "relative_improvement": 0.0971721861855829
+    },
+    {
+      "row": 66,
+      "tokens": 128,
+      "relative_mse_before": 0.8047063946723938,
+      "relative_mse_after": 0.7476578950881958,
+      "relative_improvement": 0.07089355814976861
+    },
+    {
+      "row": 67,
+      "tokens": 128,
+      "relative_mse_before": 0.7321308851242065,
+      "relative_mse_after": 0.6734902262687683,
+      "relative_improvement": 0.08009586816637274
+    },
+    {
+      "row": 68,
+      "tokens": 128,
+      "relative_mse_before": 0.762533962726593,
+      "relative_mse_after": 0.6922506093978882,
+      "relative_improvement": 0.09217078420663732
+    },
+    {
+      "row": 69,
+      "tokens": 128,
+      "relative_mse_before": 0.7766133546829224,
+      "relative_mse_after": 0.7083792090415955,
+      "relative_improvement": 0.0878611541121202
+    },
+    {
+      "row": 70,
+      "tokens": 128,
+      "relative_mse_before": 0.7899155616760254,
+      "relative_mse_after": 0.7217622995376587,
+      "relative_improvement": 0.0862791739331741
+    },
+    {
+      "row": 71,
+      "tokens": 128,
+      "relative_mse_before": 0.7536819577217102,
+      "relative_mse_after": 0.7058085799217224,
+      "relative_improvement": 0.06351933638520849
+    }
+  ]
+}

research/layer-distillation-pilots/layer-31/layer-31-mlp.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:bfc07b627e9a6175cc5773228f070969b70f25a792428852378bdbaaea6a3d76
+size 110147528

research/layer-distillation-pilots/layer-31/training_state.json ADDED Viewed

	@@ -0,0 +1,111 @@

+{
+  "teacher": "/Volumes/\u110b\u116c\u110c\u1161\u11bc\u1103\u1175\u1109\u1173\u110f\u11731/\u110b\u1169\u1111\u1173\u11ab\u1109\u1169\u1109\u1173 \u1106\u1169\u1103\u1166\u11af/models/Qwen/Qwen3.5-4B",
+  "student": "/Volumes/\u110b\u116c\u110c\u1161\u11bc\u1103\u1175\u1109\u1173\u110f\u11731/\u110b\u1169\u1111\u1173\u11ab\u1109\u1169\u1109\u1173 \u1106\u1169\u1103\u1166\u11af/models/Qwen/Qwen3.5-4B-A3B-Upcycled",
+  "router_adapter": "/Volumes/\u110b\u116c\u110c\u1161\u11bc\u1103\u1175\u1109\u1173\u110f\u11731/\u110b\u1169\u1111\u1173\u11ab\u1109\u1169\u1109\u1173 \u1106\u1169\u1103\u1166\u11af/research/qwen35-moe-a3b/runs/router-warmup-lr1e5/router.safetensors",
+  "layer": 31,
+  "trainable_parameters": 55052800,
+  "outer_steps": 10,
+  "inner_steps": 1,
+  "batch_size": 1,
+  "sequence_length": 128,
+  "learning_rate": 3e-05,
+  "heldout": {
+    "offset": 64,
+    "tokens": 128,
+    "relative_mse_before": 0.7783669233322144,
+    "relative_mse_after": 0.7084543108940125
+  },
+  "seconds": 9.618012583989184,
+  "metrics": [
+    {
+      "outer_step": 1,
+      "tokens": 128,
+      "relative_mse_before": 0.8066995739936829,
+      "inner_losses": [
+        0.8066995739936829
+      ],
+      "relative_mse_after": 0.7917649745941162
+    },
+    {
+      "outer_step": 2,
+      "tokens": 128,
+      "relative_mse_before": 0.7509857416152954,
+      "inner_losses": [
+        0.7509857416152954
+      ],
+      "relative_mse_after": 0.7376929521560669
+    },
+    {
+      "outer_step": 3,
+      "tokens": 128,
+      "relative_mse_before": 0.7817723751068115,
+      "inner_losses": [
+        0.7817723751068115
+      ],
+      "relative_mse_after": 0.7693306803703308
+    },
+    {
+      "outer_step": 4,
+      "tokens": 128,
+      "relative_mse_before": 0.7651464939117432,
+      "inner_losses": [
+        0.7651464939117432
+      ],
+      "relative_mse_after": 0.7542466521263123
+    },
+    {
+      "outer_step": 5,
+      "tokens": 128,
+      "relative_mse_before": 0.7362645268440247,
+      "inner_losses": [
+        0.7362645268440247
+      ],
+      "relative_mse_after": 0.7256730198860168
+    },
+    {
+      "outer_step": 6,
+      "tokens": 128,
+      "relative_mse_before": 0.6950127482414246,
+      "inner_losses": [
+        0.6950127482414246
+      ],
+      "relative_mse_after": 0.6839705109596252
+    },
+    {
+      "outer_step": 7,
+      "tokens": 128,
+      "relative_mse_before": 0.7439882755279541,
+      "inner_losses": [
+        0.7439882755279541
+      ],
+      "relative_mse_after": 0.7333590984344482
+    },
+    {
+      "outer_step": 8,
+      "tokens": 128,
+      "relative_mse_before": 0.7175647616386414,
+      "inner_losses": [
+        0.7175647616386414
+      ],
+      "relative_mse_after": 0.7078583836555481
+    },
+    {
+      "outer_step": 9,
+      "tokens": 128,
+      "relative_mse_before": 0.6566314101219177,
+      "inner_losses": [
+        0.6566314101219177
+      ],
+      "relative_mse_after": 0.6471226811408997
+    },
+    {
+      "outer_step": 10,
+      "tokens": 128,
+      "relative_mse_before": 0.7005226612091064,
+      "inner_losses": [
+        0.7005226612091064
+      ],
+      "relative_mse_after": 0.6920973062515259
+    }
+  ]
+}