Upload HIKARI-Sirius-8B-SkinDx-RAG

a57bc91 verified 7 months ago

12.5 kB

	---
	license: apache-2.0
	base_model: Qwen/Qwen3-VL-8B-Thinking
	tags:
	- dermatology
	- medical
	- vision-language
	- skin-disease
	- rag
	- fine-tuned
	- qwen3-vl
	language:
	- en
	- th
	datasets:
	- SkinCAP
	pipeline_tag: image-text-to-text
	---

	<p align="center">
	<img src="HIKARI_logo.png" alt="HIKARI — Healthcare-oriented Intelligent Knowledge Augmented Retrieval and Inference" width="100%"/>
	</p>

	<h1 align="center">HIKARI-Sirius-8B-SkinDx-RAG ⭐</h1>

	<p align="center">
	<b>Healthcare-oriented Intelligent Knowledge Augmented Retrieval and Inference</b><br/>
	Named after <b>Sirius</b> — the brightest star in the night sky
	</p>

	<p align="center">
	<img src="https://img.shields.io/badge/Base%20Model-Qwen3--VL--8B--Thinking-blue?style=flat-square"/>
	<img src="https://img.shields.io/badge/Task-Skin%20Disease%20Diagnosis-red?style=flat-square"/>
	<img src="https://img.shields.io/badge/Accuracy-85.86%25-brightgreen?style=flat-square"/>
	<img src="https://img.shields.io/badge/Technique-RAG--in--Training-purple?style=flat-square"/>
	<img src="https://img.shields.io/badge/License-Apache%202.0-orange?style=flat-square"/>
	</p>

	---

	## 📦 Model Type: Merged Full Model

	> This is a fully merged model — the LoRA adapter weights have been merged directly into the base model weights.
	>
	> ✅ No adapter loading needed. Load and run directly with `transformers`, `vLLM`, or `SGLang`, just like any standard Qwen3-VL model.
	>
	> 💾 Size: ~17 GB (4 safetensor shards)
	>
	> 🔌 If you prefer a lightweight adapter instead, see the LoRA version:
	> [E27085921/HIKARI-Sirius-8B-SkinDx-RAG-LoRA](https://huggingface.co/E27085921/HIKARI-Sirius-8B-SkinDx-RAG-LoRA) (~1.1 GB)

	---

	## Overview

	HIKARI-Sirius is our best-performing skin disease diagnosis model, fine-tuned from `Qwen/Qwen3-VL-8B-Thinking` on the SkinCAP Thai dermatology dataset.

	The key innovation is RAG-in-Training — retrieval-augmented generation is embedded during fine-tuning itself (not only at inference). The model learns to compare a query image against retrieved reference images and their clinical captions, making it robust to visual similarity across diseases.

	\| Property \| Value \|
	\|:---------\|:------\|
	\| Task \| 10-class skin disease diagnosis (Stage 2 of HIKARI pipeline) \|
	\| Base model \| `Qwen/Qwen3-VL-8B-Thinking` \|
	\| Training technique \| RAG-in-Training (R2: SigLIP visual + BGE-M3 text, α=0.9) \|
	\| Val accuracy \| 85.86% (99 samples, SkinCAP 3-stage split) \|
	\| Model type \| Merged full model \|
	\| Hardware tested \| RTX 5070 Ti (16 GB VRAM) \|

	---

	## 🩺 Disease Classes (10)

	\| Class \| Description \|
	\|:------\|:-----------\|
	\| `acne_vulgaris` \| Acne — comedones, papules, pustules on face/back \|
	\| `atopic_dermatitis` \| Eczema — chronic pruritic inflammatory skin disease \|
	\| `melanocytic_nevi` \| Moles — benign melanocyte proliferations \|
	\| `psoriasis` \| Erythematous plaques with silvery-white scale \|
	\| `sccis` \| Squamous cell carcinoma in situ (Bowen's disease) \|
	\| `seborrheic_dermatitis` \| Dandruff-related scaly patches on oily areas \|
	\| `skin_tag` \| Benign soft fibroepithelial pedunculated growths \|
	\| `tinea_versicolor` \| Fungal discoloration (hypo/hyperpigmented macules) \|
	\| `urticaria` \| Hives — transient wheals with erythema \|
	\| `photodermatoses` \| Sun-induced skin reactions \|

	---

	## 📊 Performance vs Baselines

	\| Model \| Accuracy \| Training Method \|
	\|:------\|:--------:\|:----------------\|
	\| Qwen3-VL-8B zero-shot \| ~45% \| No fine-tuning \|
	\| HIKARI-Altair (Single FT) \| 74.00% \| Standard fine-tuning \|
	\| HIKARI-Deneb (Cascade FT) \| 79.80% \| Cascaded pretraining \|
	\| HIKARI-Sirius (RAG-in-Training) \| 85.86% ✨ \| RAG embedded at training \|

	---

	## 🔧 Usage

	### Stage 2 in the Full HIKARI Pipeline

	```
	📷 Image
	│
	▼
	[Stage 1] HIKARI-Subaru-8B-SkinGroup ──► group label (4 classes)
	│
	▼
	[Stage 2] HIKARI-Sirius-8B-SkinDx-RAG ──► disease label (10 classes) ← YOU ARE HERE
	│
	▼
	[Stage 3] HIKARI-Vega-8B-SkinCaption-Fused ──► clinical caption
	```

	---

	### Quick Inference — `transformers`

	```python
	from transformers import Qwen3VLForConditionalGeneration, AutoProcessor
	import torch
	from PIL import Image

	model_id = "E27085921/HIKARI-Sirius-8B-SkinDx-RAG"

	processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
	model = Qwen3VLForConditionalGeneration.from_pretrained(
	model_id,
	torch_dtype=torch.bfloat16,
	device_map="auto",
	trust_remote_code=True,
	)

	image = Image.open("skin_lesion.jpg").convert("RGB")
	group = "inflammatory" # from Stage 1 (HIKARI-Subaru)

	PROMPT = (
	"This skin lesion belongs to the group '{group}'. "
	"Examine the lesion morphology (papules, plaques, macules), "
	"color (red, violet, white, brown), scale/crust, border sharpness, "
	"and distribution pattern. Based on these visual features, "
	"what is the specific skin disease?"
	)

	messages = [{"role": "user", "content": [
	{"type": "image", "image": image},
	{"type": "text", "text": PROMPT.format(group=group)},
	]}]

	text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

	with torch.no_grad():
	out = model.generate(**inputs, max_new_tokens=64, temperature=0.0, do_sample=False)

	result = processor.batch_decode(
	out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
	)[0].strip()
	print(result) # → "atopic_dermatitis"
	```

	---

	### Production — vLLM BnB-4bit ⚡ (RTX 5070 Ti / 16 GB VRAM)

	> Throughput: 5.57 img/s at batch=4

	```python
	from vllm import LLM, SamplingParams
	from transformers import AutoProcessor
	from PIL import Image

	model_id = "E27085921/HIKARI-Sirius-8B-SkinDx-RAG"

	llm = LLM(
	model=model_id,
	quantization="bitsandbytes",
	load_format="bitsandbytes",
	trust_remote_code=True,
	max_model_len=2048,
	gpu_memory_utilization=0.88,
	)
	processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)
	sp = SamplingParams(max_tokens=64, temperature=0.0)

	PROMPT = (
	"This skin lesion belongs to the group '{group}'. "
	"Examine the lesion morphology (papules, plaques, macules), "
	"color (red, violet, white, brown), scale/crust, border sharpness, "
	"and distribution pattern. Based on these visual features, "
	"what is the specific skin disease?"
	)

	def classify_disease(image: Image.Image, group: str) -> str:
	messages = [{"role": "user", "content": [
	{"type": "image", "image": image},
	{"type": "text", "text": PROMPT.format(group=group)},
	]}]
	text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	n = max(text.count("<\|vision_start\|>"), 1)
	out = llm.generate({"prompt": text, "multi_modal_data": {"image": [image] * n}}, sp)
	return out[0].outputs[0].text.strip()

	img = Image.open("skin_lesion.jpg").convert("RGB")
	print(classify_disease(img, group="inflammatory")) # → "atopic_dermatitis"
	```

	---

	### Production — SGLang FP8 🚀 (maximum throughput, 9.11 img/s at batch=4)

	```python
	import sglang as sgl
	from transformers import AutoProcessor
	from PIL import Image

	model_id = "E27085921/HIKARI-Sirius-8B-SkinDx-RAG"

	engine = sgl.Engine(
	model_path=model_id,
	dtype="bfloat16",
	quantization="fp8",
	context_length=2048,
	mem_fraction_static=0.88,
	trust_remote_code=True,
	disable_cuda_graph=True,
	)
	processor = AutoProcessor.from_pretrained(model_id, trust_remote_code=True)

	PROMPT = (
	"This skin lesion belongs to the group '{group}'. "
	"Examine the lesion morphology (papules, plaques, macules), "
	"color (red, violet, white, brown), scale/crust, border sharpness, "
	"and distribution pattern. Based on these visual features, "
	"what is the specific skin disease?"
	)

	def classify_disease_sglang(image: Image.Image, group: str) -> str:
	messages = [{"role": "user", "content": [
	{"type": "image"},
	{"type": "text", "text": PROMPT.format(group=group)},
	]}]
	text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
	out = engine.generate(
	prompt=text,
	image_data=image,
	sampling_params={"max_new_tokens": 64, "temperature": 0.0},
	)
	return (out["text"] if isinstance(out, dict) else out[0]["text"]).strip()

	# engine.shutdown() # call when done
	```

	---

	### Parse Disease Label (fuzzy matching)

	```python
	from rapidfuzz import process as fuzz_process

	DISEASES = [
	"acne_vulgaris", "atopic_dermatitis", "melanocytic_nevi", "psoriasis",
	"sccis", "seborrheic_dermatitis", "skin_tag", "tinea_versicolor",
	"urticaria", "photodermatoses",
	]

	def match_disease(raw: str) -> str:
	result, score, _ = fuzz_process.extractOne(raw.lower(), DISEASES)
	return result if score >= 50 else "unknown"

	print(match_disease("The patient has atopic dermatitis")) # → atopic_dermatitis
	```

	---

	## ⚡ Speed Benchmark (RTX 5070 Ti, 16 GB VRAM — Stage 2, 64-token output)

	\| Engine \| Batch 1 \| Batch 4 \| vs Unsloth bs=1 \|
	\|:-------\|:-------:\|:-------:\|:---------------:\|
	\| Unsloth 4-bit \| 1,096 ms/img \| 500 ms/img \| baseline \|
	\| vLLM BnB-4bit \| 480 ms/img \| 179 ms/img \| 6.1× faster \|
	\| SGLang FP8 \| 331 ms/img \| 110 ms/img ⚡ \| 10× faster \|

	---

	## 🔌 LoRA Adapter Version

	Prefer a lightweight adapter (~1.1 GB) over the 17 GB merged model?

	```python
	from peft import PeftModel
	from transformers import Qwen3VLForConditionalGeneration
	import torch

	base = Qwen3VLForConditionalGeneration.from_pretrained(
	"Qwen/Qwen3-VL-8B-Thinking", torch_dtype=torch.bfloat16, device_map="auto"
	)
	model = PeftModel.from_pretrained(base, "E27085921/HIKARI-Sirius-8B-SkinDx-RAG-LoRA")
	```

	→ [E27085921/HIKARI-Sirius-8B-SkinDx-RAG-LoRA](https://huggingface.co/E27085921/HIKARI-Sirius-8B-SkinDx-RAG-LoRA)

	---

	## 🌟 HIKARI Model Family

	<table>
	<tr>
	<th>Model</th><th>Task</th><th>Metric</th><th>Type</th>
	</tr>
	<tr>
	<td><a href="https://huggingface.co/E27085921/HIKARI-Subaru-8B-SkinGroup">HIKARI-Subaru-8B-SkinGroup</a></td>
	<td>4-class group classifier (Stage 1)</td>
	<td>88.68%</td>
	<td>Merged</td>
	</tr>
	<tr>
	<td><a href="https://huggingface.co/E27085921/HIKARI-Altair-8B-SkinDx">HIKARI-Altair-8B-SkinDx</a></td>
	<td>10-class disease dx — baseline</td>
	<td>74.00%</td>
	<td>Merged + LoRA</td>
	</tr>
	<tr>
	<td><a href="https://huggingface.co/E27085921/HIKARI-Deneb-8B-SkinDx-Cascade">HIKARI-Deneb-8B-SkinDx-Cascade</a></td>
	<td>10-class disease dx — cascade FT</td>
	<td>79.80%</td>
	<td>Merged + LoRA</td>
	</tr>
	<tr>
	<td><b>⭐ HIKARI-Sirius-8B-SkinDx-RAG (this model)</b></td>
	<td><b>10-class disease dx — RAG-in-Training</b></td>
	<td><b>85.86%</b></td>
	<td><b>Merged + LoRA</b></td>
	</tr>
	<tr>
	<td><a href="https://huggingface.co/E27085921/HIKARI-Polaris-8B-SkinDx-Oracle">HIKARI-Polaris-8B-SkinDx-Oracle</a></td>
	<td>Oracle upper bound (research only)</td>
	<td>59.38%*</td>
	<td>Merged</td>
	</tr>
	<tr>
	<td><a href="https://huggingface.co/E27085921/HIKARI-Rigel-8B-SkinCaption">HIKARI-Rigel-8B-SkinCaption</a></td>
	<td>Clinical caption — checkpoint init</td>
	<td>BLEU-4: 9.82</td>
	<td>Merged + LoRA</td>
	</tr>
	<tr>
	<td><a href="https://huggingface.co/E27085921/HIKARI-Vega-8B-SkinCaption-Fused"><b>⭐ HIKARI-Vega-8B-SkinCaption-Fused</b></a></td>
	<td><b>Clinical caption — merged init (best)</b></td>
	<td><b>BLEU-4: 29.33</b></td>
	<td><b>Merged + LoRA</b></td>
	</tr>
	<tr>
	<td><a href="https://huggingface.co/E27085921/HIKARI-Antares-8B-SkinCaption-STS">HIKARI-Antares-8B-SkinCaption-STS</a></td>
	<td>Caption + STS ablation (research)</td>
	<td>BLEU-4: 0.61</td>
	<td>Merged + LoRA</td>
	</tr>
	</table>

	\ Polaris requires ground-truth group at inference — for research comparison only.*

	---

	## 📄 Citation

	```bibtex
	@misc{hikari2026,
	title = {HIKARI: RAG-in-Training for Skin Disease Diagnosis
	with Cascaded Vision-Language Models},
	author = {Watin Promfiy and Pawitra Boonprasart},
	year = {2026},
	institution = {King Mongkut's Institute of Technology Ladkrabang,
	Department of Information Technology, Bangkok, Thailand}
	}
	```

	---

	<p align="center">
	Made with ❤️ at <b>King Mongkut's Institute of Technology Ladkrabang (KMITL)</b><br/>
	Department of Information Technology
	</p>