Instructions to use pankajpandey-dev/qwen3.5-9b-hindi-instruct with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use pankajpandey-dev/qwen3.5-9b-hindi-instruct with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="pankajpandey-dev/qwen3.5-9b-hindi-instruct")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("pankajpandey-dev/qwen3.5-9b-hindi-instruct")
model = AutoModelForMultimodalLM.from_pretrained("pankajpandey-dev/qwen3.5-9b-hindi-instruct", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use pankajpandey-dev/qwen3.5-9b-hindi-instruct with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "pankajpandey-dev/qwen3.5-9b-hindi-instruct"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajpandey-dev/qwen3.5-9b-hindi-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct

SGLang

How to use pankajpandey-dev/qwen3.5-9b-hindi-instruct with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "pankajpandey-dev/qwen3.5-9b-hindi-instruct" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajpandey-dev/qwen3.5-9b-hindi-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "pankajpandey-dev/qwen3.5-9b-hindi-instruct" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "pankajpandey-dev/qwen3.5-9b-hindi-instruct",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Unsloth Desktop
Docker Model Runner
How to use pankajpandey-dev/qwen3.5-9b-hindi-instruct with Docker Model Runner:
```
docker model run hf.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct
```

pankajpandey-dev commited on Jul 4

Commit

3a11eb4

verified ·

1 Parent(s): a02dccb

Upload README.md with huggingface_hub

Browse files

Files changed (1) hide show

README.md +51 -121

README.md CHANGED Viewed

@@ -1,144 +1,74 @@
 ---
 license: apache-2.0
-language:
-- hi
-- en
-base_model: Qwen/Qwen3.5-9B
-pipeline_tag: text-generation
 library_name: transformers
-tags:
-- hindi
-- qwen3_5
-- qwen3.5
-- unsloth
-- lora
-- instruction-tuning
-- conversational
-- india
 datasets:
 - ai4bharat/indic-instruct-data-v0.1
 - CohereForAI/aya_dataset
 ---
-# 🇮🇳 Qwen3.5-9B-Hindi-Instruct
-**Native Hindi answers, instantly — no English "thinking detour."**
-Base Qwen3.5-9B knows Hindi, but ask it a simple Hindi question and it burns hundreds of tokens on an English chain-of-thought before answering — and its answers code-switch into English. This fine-tune fixes both: it responds **immediately, in pure Devanagari Hindi**, and follows instructions tightly (ask for तीन उपाय, get exactly 3).
-Part of the 🇮🇳 Hindi LLM Series — small models fine-tuned for Hindi, quantized for edge/CPU use, trained on rented cloud GPUs for under $10 per run.
-## Before / After
-Same prompt: `स्वस्थ रहने के तीन उपाय बताइए।`
-**Base Qwen3.5-9B** (400 tokens later, still hasn't answered):
-```
-Thinking Process:
-1.  **Analyze the Request:**
-    *   Language: Hindi (Devanagari script).
-    *   Task: Provide three ways to stay healthy...
-2.  **Identify Key Health Advice:**
-    *   Healthy eating (balanced diet)...
-```
-**This model** (first tokens):
-```
-स्वस्थ रहने के लिए, सही आहार का सेवन करना जरूरी है, नियमित व्यायाम
-करना चाहिए और पर्याप्त नींद लेनी चाहिए।
-```
-## Pipeline
-```
-Qwen/Qwen3.5-9B (instruct, hybrid-thinking)
-        |
-        v
-SFT + LoRA (r=16, bf16) with Unsloth
-  - response-only loss, masked on "<|im_start|>assistant\n"
-  - trained WITH empty <think> blocks -> answers directly
-        |
-        v
-qwen3.5-9b-hindi-instruct (this repo, merged 16-bit)
-```
-## Training data — 12,912 pairs from 4 sources
-| Source | Pairs | Why |
-|---|---|---|
-| Anudesh (hi) | 5,000 | Native real-user Hindi prompts |
-| Dolly (hi, IndicTrans2-translated) | 4,000 | Task diversity; kept only chrF++ >= 55 rows |
-| wikiHow (hi) | 3,000 | Native long-form step-by-step answers |
-| Aya (Hindi) | 912 | Human-written by native speakers |
-Cleaning: cross-source dedup, length bounds, Latin-script ratio filter, percent-encoding repair on wikiHow (465 dropped for length, 339 duplicates, 61 excessive Latin script).
-## Evaluation (vs base, same 25 Hindi prompts)
-| Behavior | Base Qwen3.5-9B | This model |
-|---|---|---|
-| English "thinking" before Hindi answer | Always | Never |
-| Code-switching in answers | Frequent | Not observed |
-| "तीन बताइए" -> exactly 3 items | Unreliable | Yes |
-| Formats (letter/essay/message) | Verbose, meta-commentary | Direct, correct format |
-Manual 25-prompt eval (essays, formal letters, lists, single-sentence QA, recipes) — honest and small, not a benchmark.
-## Usage
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
-model_id = "pankajpandey-dev/qwen3.5-9b-hindi-instruct"
-tok = AutoTokenizer.from_pretrained(model_id)
-model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype="bfloat16", device_map="auto")
-msgs = [{"role": "user", "content": "दीपावली पर एक छोटा निबंध लिखिए।"}]
-text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
-out = model.generate(**tok(text, return_tensors="pt").to(model.device), max_new_tokens=400, temperature=0.7)
-print(tok.decode(out[0], skip_special_tokens=True))
 ```
-For CPU: use the [GGUF repo](https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF) (Q4_K_M = 5.4 GB).
-### ⚠️ Non-thinking is baked in
-The chat template always emits a closed empty <think> block — the model answers directly. This is intentional: llama.cpp defines enable_thinking=true by default, which would otherwise re-trigger the base model's English chain-of-thought. Need thinking mode? Use the base model.
-## Training details
-- **Hardware:** 1x NVIDIA L40S (48 GB), Lightning.ai — total ~ **$9**
-- **Method:** Unsloth, LoRA r=16/alpha=16, bf16, q/k/v/o + gate/up/down (29M params, 0.31%)
-- **Recipe:** 2 epochs, LR 1e-4 cosine, warmup 3%, effective batch 16, max_seq_len 2048
-- **Loss:** response-only, final train loss 0.94
-- Vision layers untouched (text-only fine-tune, vision untested)
-Two findings for other Qwen3.5 fine-tuners:
-1. convert_hf_to_gguf.py fails on Unsloth-saved checkpoints with "Tokenizer class TokenizersBackend does not exist" — fix: set "tokenizer_class": "PreTrainedTokenizerFast" in tokenizer_config.json.
-2. Patching the template default to non-thinking is not enough for GGUF: **llama.cpp defines enable_thinking=true itself**. Hard-bake the closed think block into both branches.
-## Limitations (honest)
-- Factual reliability in long-form Hindi is imperfect — a **native Hindi instruction-follower**, not an encyclopedia.
-- Occasional translationese from the Dolly subset.
-- Trained single-turn; multi-turn works but wasn't the focus.
-- Thinking mode intentionally disabled. Not additionally safety-tuned.
-## License & data notes
-Weights: **Apache 2.0**. Data: Anudesh responses were Llama-2-70B-generated (Llama 2 terms), Dolly-hi is CC-BY-SA-3.0-derived, wikiHow from hi.wikihow.com, Aya Apache 2.0.
-## Related
-[GGUF quants](https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF) · [LoRA adapter](https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct-lora) · [🇮🇳 Hindi LLM Series](https://huggingface.co/pankajpandey-dev)
-## Citation
-```bibtex
-@misc{pandey2026qwen35hindi,
-  title = {Qwen3.5-9B-Hindi-Instruct},
-  author = {Pankaj Pandey},
-  year = {2026},
-  publisher = {Hugging Face},
-  howpublished = {https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct}
-}
-```

 ---
 license: apache-2.0
+language: [hi, en]
+base_model: unsloth/Qwen3.5-9B
 library_name: transformers
+pipeline_tag: text-generation
 datasets:
 - ai4bharat/indic-instruct-data-v0.1
 - CohereForAI/aya_dataset
+tags: [hindi, qwen3.5, unsloth, lora, instruct, india]
+widget:
+- text: "स्वस्थ रहने के तीन उपाय बताइए।"
+- text: "दीपावली पर एक छोटा निबंध लिखिए।"
+- text: "चाय बनाने की विधि चरणों में बताइए।"
 ---
+# Qwen3.5-9B Hindi Instruct 🇮🇳
+**Qwen3.5-9B fine-tuned to answer directly in fluent, native Hindi — no English code-switching, no English "thinking" detours.** Runs on a laptop CPU via the [GGUF version](https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF) (~5.7 GB).
+## Why this exists
+Base Qwen3.5-9B knows Hindi, but ask it a Hindi question and it burns hundreds of tokens *thinking in English* inside its think block before answering — and its answers code-switch. This fine-tune closes the think block immediately and replies in pure Devanagari, following instructions tightly.
+## Examples (from the release eval)
+> **प्रश्न:** ऑनलाइन ठगी से बचने के तीन उपाय बताइए।
+>
+> **उत्तर:** आपके पास केवल एक कार्ड वाला खाता होना चाहिए (कभी भी अपने पूरे खाते को न दें)।
+ऐसे फिशिंग ईमेल (जिनमें आपसे पैसा माँगा जाए) पर ध्यान न दें।
+सभी ऑनलाइन लेनदेन की जांच करें।
+> **प्रश्न:** 'ईमानदारी सबसे अच्छी नीति है' पर तीन वाक्य लिखिए।
+>
+> **उत्तर:** सच्चाई हमेशा सफलता लाती है। सत्य हमेशा सच्चाई से बढ़कर होता है। ईमानदारी सबसे अच्छी नीति है और ईमानदार��� हमेशा हमारे साथ रहती है।
+> **प्रश्न:** मोबाइल फोन के दो लाभ और दो हानियाँ बताइए।
+>
+> **उत्तर:** संचार की सुविधा और जानकारी तक पहुंच, पर्यावरण पर प्रभाव, समय बर्बाद करना, अधिकारियों की जिम्मेदारी, संपर्क में आए व्यक्ति का समय बर्बाद करना, सामाजिक रूप से निर्लक्षी बनना
+## Quickstart
 ```python
 from transformers import AutoModelForCausalLM, AutoTokenizer
+m = AutoModelForCausalLM.from_pretrained("pankajpandey-dev/qwen3.5-9b-hindi-instruct", torch_dtype="bfloat16", device_map="auto")
+t = AutoTokenizer.from_pretrained("pankajpandey-dev/qwen3.5-9b-hindi-instruct")
+msgs = [{"role": "user", "content": "जल संरक्षण के पाँच तरीके बताइए।"}]
+text = t.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
+out = m.generate(**t(text=text, return_tensors="pt").to(m.device),
+                 max_new_tokens=400, repetition_penalty=1.1)
+print(t.decode(out[0], skip_special_tokens=True))
 ```
+Use `repetition_penalty=1.1` (set in this repo's generation_config) — long letter/essay outputs can loop without it. The chat template opens a think block; this model closes it immediately — strip `<think>...</think>` from output if present.
+## Training recipe (fully reproducible)
+| | |
+|---|---|
+| Base | unsloth/Qwen3.5-9B (bf16) |
+| Method | LoRA r=16, alpha=16, response-only loss (Unsloth) |
+| Data | 12,912 Hindi pairs — anudesh 5,000 · dolly-hi 4,000 (chrF++ >= 55) · wikiHow-hi 3,000 · Aya-hi 912 |
+| Schedule | 2 epochs, LR 1e-4 cosine, effective batch 16, seq 2048 |
+| Hardware | 1x NVIDIA L40S (48 GB), ~135 min |
+| Final train loss | 0.938 |
+Data deduplicated across sources, filtered for length and Latin-script ratio. wikiHow adds long-form step-by-step answers; Aya adds human-written originals.
+## Limitations
+Parts of the data are machine-translated (dolly) or model-generated (anudesh via Llama-2-70B), so occasional unnatural phrasing or factual errors occur. Long formal-letter outputs may repeat without repetition_penalty=1.1. No additional safety tuning. Knowledge cutoff follows base Qwen3.5.
+## Licensing
+Weights: Apache 2.0. Data licenses apply to data, not weights: dolly-hi CC-BY-SA 3.0-derived; anudesh generated by Llama-2-70B (Llama 2 license); Aya Apache 2.0.
+---
+## 🇮🇳 About the Hindi LLM Series
+Weekly open releases making small LLMs speak fluent, native Hindi — trained on free/low-cost GPUs, shipped as GGUF for laptops and edge devices. Built by [pankajpandey-dev](https://huggingface.co/pankajpandey-dev) *(contact links on profile)*.
+**This release:** [Model](https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct) · [GGUF](https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct-GGUF) · [LoRA](https://huggingface.co/pankajpandey-dev/qwen3.5-9b-hindi-instruct-lora) · **Series:** [🇮🇳 Hindi LLM Collection](https://huggingface.co/pankajpandey-dev)