Instructions to use wjdghks950/promac-aria-stage2-random-visfrozen with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use wjdghks950/promac-aria-stage2-random-visfrozen with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="wjdghks950/promac-aria-stage2-random-visfrozen")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("wjdghks950/promac-aria-stage2-random-visfrozen")
model = AutoModelForMultimodalLM.from_pretrained("wjdghks950/promac-aria-stage2-random-visfrozen", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use wjdghks950/promac-aria-stage2-random-visfrozen with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "wjdghks950/promac-aria-stage2-random-visfrozen"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wjdghks950/promac-aria-stage2-random-visfrozen",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/wjdghks950/promac-aria-stage2-random-visfrozen

SGLang

How to use wjdghks950/promac-aria-stage2-random-visfrozen with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "wjdghks950/promac-aria-stage2-random-visfrozen" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wjdghks950/promac-aria-stage2-random-visfrozen",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "wjdghks950/promac-aria-stage2-random-visfrozen" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "wjdghks950/promac-aria-stage2-random-visfrozen",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use wjdghks950/promac-aria-stage2-random-visfrozen with Docker Model Runner:
```
docker model run hf.co/wjdghks950/promac-aria-stage2-random-visfrozen
```

ProMAC Aria Stage-2 Random Init, Vision Frozen

Public, inference-only Qwen3-VL export of the ProMAC Aria Stage-2 structured-CoT v4 checkpoint trained with a randomly initialized future-state predictor and frozen Qwen vision tower.

Results

On the 1,051-example Aria structured-CoT v4 validation split (and-v4-val), deterministic evaluation reported:

Checkpoint	ADE	FDE	L2 First 4	DTW	DTW % path	Miss @20%
`checkpoint-best-ade/global_step1050`	16.919	8.348	24.462	15.575	11.5	24.5

The DeepSpeed tag is optimizer global_step1050, corresponding to training micro-step 2100 with gradient accumulation 2. Coordinates and errors are in pixels in the rotated 512 x 512 Aria frame. The goal is supplied in the prompt, so Aria FDE is partly degenerate.

Usage

import torch
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

model_id = "wjdghks950/promac-aria-stage2-random-visfrozen"
processor = AutoProcessor.from_pretrained(model_id, fix_mistral_regex=True)
model = Qwen3VLForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)

Use the Aria structured-CoT v4 prompt schema and answer_only assistant format. Full offline evaluation used deterministic generation with up to 512 new tokens.

Scope

This repository contains the Qwen3-VL subtree extracted from the exact ProMAC ZeRO-3 checkpoint and merged into a standard Hugging Face model. Trajectory inference uses the standard Qwen generate() path. The randomly initialized predictor affected training through the auxiliary objective but is not needed for trajectory inference and is not included. The auxiliary decoder, SIGReg state, optimizer state, raw ZeRO shards, logs, and datasets are also excluded.

The export was merged with Qwen/Qwen3-VL-4B-Instruct revision ebb281ec70b05090aa6165b016eac8ec08e71b17. See provenance.json and SHA256SUMS.

License And Data Terms

The upstream Qwen model card declares Apache-2.0. No standalone license was established locally for the ProMAC fine-tuned weights, so this repository uses license: other. Access to these weights does not grant rights to the Aria training or evaluation data. Users are responsible for all applicable source-data terms.

Limitations

This is a research checkpoint selected on a limited training-time evaluation and later evaluated offline. It may not generalize to other prompt formats, camera geometry, rotations, environments, or trajectory schemas. Reported metrics are not safety guarantees.

Downloads last month: 9

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for wjdghks950/promac-aria-stage2-random-visfrozen

Base model

Qwen/Qwen3-VL-4B-Instruct

Finetuned

(488)

this model