Instructions to use llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic")
model = AutoModelForMultimodalLM.from_pretrained("llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic

SGLang

How to use llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic with Docker Model Runner:
```
docker model run hf.co/llmfan46/gemma-4-31B-it-qat-q4_0-unquantized-uncensored-heretic
```

llmfan46 commited on Jun 11

Commit

099ff66

verified ·

1 Parent(s): fa89ea2

Update README.md

Browse files

Files changed (1) hide show

README.md +96 -96

README.md CHANGED Viewed

@@ -46,11 +46,11 @@ Your help will motivate me and would go into further improving my workflow and c
 | Parameter | Value |
 | :-------- | :---: |
-| **start_layer_index** | 30 |
-| **end_layer_index** | 48 |
-| **preserve_good_behavior_weight** | 0.5437 |
-| **steer_bad_behavior_weight** | 0.0005 |
-| **overcorrect_relative_weight** | 0.9949 |
 | **neighbor_count** | 15 |
 ## Targeted components
@@ -61,8 +61,8 @@ Your help will motivate me and would go into further improving my workflow and c
 | Metric | This model | Original model ([gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it)) |
 | :----- | :--------: | :---------------------------: |
-| **KL divergence** | <span style="color:darkgoldenrod">0.0541</span> | 0 *(by definition)* |
-| **Refusals** | ✅ <span style="color:darkgreen">10/100</span> | ❌ <span style="color:blue">99/100</span> |
 Lower refusals indicate fewer content restrictions, while lower KL divergence indicates more closeness to the original model's baseline. Higher refusals cause more rejections, objections, pushbacks, lecturing, censorship, softening and deflections.
@@ -74,72 +74,72 @@ Lower refusals indicate fewer content restrictions, while lower KL divergence in
 - Total questions: 7021
-- Correct: 6073
-- **Accuracy: 0.8650 (86.50%)**
-- Parse failures: 52
 ============================================================
 **Tested subject scores:**
-- professional_law: 0.7682 (603/785)
-- moral_scenarios: 0.8348 (369/442)
-- miscellaneous: 0.9243 (354/383)
-- professional_psychology: 0.9051 (286/316)
-- high_school_psychology: 0.9630 (260/270)
 - high_school_macroeconomics: 0.9289 (183/197)
-- elementary_mathematics: 0.9348 (172/184)
-- moral_disputes: 0.8448 (147/174)
-- prehistory: 0.9360 (161/172)
 - philosophy: 0.8491 (135/159)
 - high_school_biology: 0.9539 (145/152)
 - professional_accounting: 0.8392 (120/143)
-- clinical_knowledge: 0.9143 (128/140)
-- high_school_microeconomics: 0.9706 (132/136)
-- nutrition: 0.9259 (125/135)
-- professional_medicine: 0.9328 (125/134)
 - conceptual_physics: 0.9219 (118/128)
-- high_school_mathematics: 0.5354 (68/127)
 - human_aging: 0.8448 (98/116)
-- security_studies: 0.8750 (98/112)
 - high_school_statistics: 0.8739 (97/111)
-- marketing: 0.9725 (106/109)
-- high_school_world_history: 0.9528 (101/106)
-- sociology: 0.8932 (92/103)
 - high_school_government_and_politics: 0.9703 (98/101)
 - high_school_geography: 0.9293 (92/99)
 - high_school_chemistry: 0.7732 (75/97)
 - high_school_us_history: 0.9368 (89/95)
-- virology: 0.5169 (46/89)
 - college_medicine: 0.8523 (75/88)
-- world_religions: 0.9205 (81/88)
-- high_school_physics: 0.7976 (67/84)
-- electrical_engineering: 0.8642 (70/81)
-- astronomy: 0.9494 (75/79)
-- logical_fallacies: 0.8816 (67/76)
 - high_school_european_history: 0.9041 (66/73)
-- anatomy: 0.8732 (62/71)
 - college_biology: 0.9688 (62/64)
 - human_sexuality: 0.9219 (59/64)
 - formal_logic: 0.7969 (51/64)
-- public_relations: 0.7377 (45/61)
-- international_law: 0.9167 (55/60)
-- college_physics: 0.6842 (39/57)
-- college_mathematics: 0.7455 (41/55)
-- econometrics: 0.7963 (43/54)
-- jurisprudence: 0.8679 (46/53)
-- high_school_computer_science: 0.9808 (51/52)
 - machine_learning: 0.8462 (44/52)
 - medical_genetics: 0.9608 (49/51)
-- global_facts: 0.6078 (31/51)
 - management: 0.9200 (46/50)
-- us_foreign_policy: 0.9200 (46/50)
 - college_chemistry: 0.5745 (27/47)
-- abstract_algebra: 0.7660 (36/47)
-- business_ethics: 0.8261 (38/46)
-- college_computer_science: 0.9333 (42/45)
-- computer_security: 0.8372 (36/43)
 <span style="color:darkgreen">Heretic:</span>
@@ -148,78 +148,78 @@ Lower refusals indicate fewer content restrictions, while lower KL divergence in
 - Total questions: 7021
-- Correct: 6031
-- **Accuracy: 0.8590 (85.90%)**
-- Parse failures: 37
 ============================================================
 **Tested subject scores:**
-- professional_law: 0.7490 (588/785)
-- moral_scenarios: 0.8281 (366/442)
-- miscellaneous: 0.9243 (354/383)
-- professional_psychology: 0.8861 (280/316)
-- high_school_psychology: 0.9667 (261/270)
-- high_school_macroeconomics: 0.9188 (181/197)
-- elementary_mathematics: 0.9511 (175/184)
-- moral_disputes: 0.8563 (149/174)
-- prehistory: 0.9360 (161/172)
-- philosophy: 0.8365 (133/159)
 - high_school_biology: 0.9539 (145/152)
-- professional_accounting: 0.8182 (117/143)
-- clinical_knowledge: 0.9214 (129/140)
-- high_school_microeconomics: 0.9632 (131/136)
-- nutrition: 0.9037 (122/135)
-- professional_medicine: 0.9254 (124/134)
 - conceptual_physics: 0.8984 (115/128)
-- high_school_mathematics: 0.5276 (67/127)
-- human_aging: 0.8448 (98/116)
-- security_studies: 0.8482 (95/112)
-- high_school_statistics: 0.8739 (97/111)
-- marketing: 0.9725 (106/109)
-- high_school_world_history: 0.9717 (103/106)
-- sociology: 0.9029 (93/103)
-- high_school_government_and_politics: 0.9505 (96/101)
-- high_school_geography: 0.9293 (92/99)
-- high_school_chemistry: 0.7835 (76/97)
-- high_school_us_history: 0.9263 (88/95)
 - virology: 0.5056 (45/89)
-- college_medicine: 0.8523 (75/88)
-- world_religions: 0.9318 (82/88)
-- high_school_physics: 0.7976 (67/84)
-- electrical_engineering: 0.8519 (69/81)
 - astronomy: 0.9494 (75/79)
-- logical_fallacies: 0.9079 (69/76)
 - high_school_european_history: 0.8767 (64/73)
-- anatomy: 0.8732 (62/71)
 - college_biology: 0.9688 (62/64)
-- human_sexuality: 0.8906 (57/64)
-- formal_logic: 0.7969 (51/64)
 - public_relations: 0.7541 (46/61)
 - international_law: 0.9167 (55/60)
 - college_physics: 0.7018 (40/57)
-- college_mathematics: 0.7455 (41/55)
 - econometrics: 0.7778 (42/54)
-- jurisprudence: 0.8491 (45/53)
 - high_school_computer_science: 0.9808 (51/52)
-- machine_learning: 0.7692 (40/52)
-- medical_genetics: 0.9412 (48/51)
-- global_facts: 0.6078 (31/51)
-- management: 0.9200 (46/50)
-- us_foreign_policy: 0.9200 (46/50)
-- college_chemistry: 0.5745 (27/47)
-- abstract_algebra: 0.7660 (36/47)
 - business_ethics: 0.8478 (39/46)
-- college_computer_science: 0.9333 (42/45)
-- computer_security: 0.8372 (36/43)
 MMLU - Massive Multitask Language Understanding, multiple-choice questions across 57 subjects (math, history, law, medicine, etc.).
 ## GGUF Version
-GGUF quantizations available here [llmfan46/gemma-4-31B-it-uncensored-heretic-GGUF](https://huggingface.co/llmfan46/gemma-4-31B-it-uncensored-heretic-GGUF).
 -----

 | Parameter | Value |
 | :-------- | :---: |
+| **start_layer_index** | 9 |
+| **end_layer_index** | 36 |
+| **preserve_good_behavior_weight** | 0.4826 |
+| **steer_bad_behavior_weight** | 0.0002 |
+| **overcorrect_relative_weight** | 1.0491 |
 | **neighbor_count** | 15 |
 ## Targeted components
 | Metric | This model | Original model ([gemma-4-31B-it](https://huggingface.co/google/gemma-4-31B-it)) |
 | :----- | :--------: | :---------------------------: |
+| **KL divergence** | <span style="color:darkgoldenrod">0.0365</span> | 0 *(by definition)* |
+| **Refusals** | ✅ <span style="color:darkgreen">11/100</span> | ❌ <span style="color:blue">99/100</span> |
 Lower refusals indicate fewer content restrictions, while lower KL divergence indicates more closeness to the original model's baseline. Higher refusals cause more rejections, objections, pushbacks, lecturing, censorship, softening and deflections.
 - Total questions: 7021
+- Correct: 6050
+- **Accuracy: 0.8617 (86.17%)**
+- Parse failures: 61
 ============================================================
 **Tested subject scores:**
+- professional_law: 0.7618 (598/785)
+- moral_scenarios: 0.8281 (366/442)
+- miscellaneous: 0.9217 (353/383)
+- professional_psychology: 0.8924 (282/316)
+- high_school_psychology: 0.9667 (261/270)
 - high_school_macroeconomics: 0.9289 (183/197)
+- elementary_mathematics: 0.9457 (174/184)
+- moral_disputes: 0.8621 (150/174)
+- prehistory: 0.9302 (160/172)
 - philosophy: 0.8491 (135/159)
 - high_school_biology: 0.9539 (145/152)
 - professional_accounting: 0.8392 (120/143)
+- clinical_knowledge: 0.9214 (129/140)
+- high_school_microeconomics: 0.9559 (130/136)
+- nutrition: 0.9185 (124/135)
+- professional_medicine: 0.9104 (122/134)
 - conceptual_physics: 0.9219 (118/128)
+- high_school_mathematics: 0.5276 (67/127)
 - human_aging: 0.8448 (98/116)
+- security_studies: 0.8929 (100/112)
 - high_school_statistics: 0.8739 (97/111)
+- marketing: 0.9633 (105/109)
+- high_school_world_history: 0.9623 (102/106)
+- sociology: 0.9029 (93/103)
 - high_school_government_and_politics: 0.9703 (98/101)
 - high_school_geography: 0.9293 (92/99)
 - high_school_chemistry: 0.7732 (75/97)
 - high_school_us_history: 0.9368 (89/95)
+- virology: 0.4944 (44/89)
 - college_medicine: 0.8523 (75/88)
+- world_religions: 0.8977 (79/88)
+- high_school_physics: 0.7619 (64/84)
+- electrical_engineering: 0.8148 (66/81)
+- astronomy: 0.9620 (76/79)
+- logical_fallacies: 0.8947 (68/76)
 - high_school_european_history: 0.9041 (66/73)
+- anatomy: 0.8873 (63/71)
 - college_biology: 0.9688 (62/64)
 - human_sexuality: 0.9219 (59/64)
 - formal_logic: 0.7969 (51/64)
+- public_relations: 0.7541 (46/61)
+- international_law: 0.9000 (54/60)
+- college_physics: 0.7193 (41/57)
+- college_mathematics: 0.6909 (38/55)
+- econometrics: 0.8148 (44/54)
+- jurisprudence: 0.8868 (47/53)
+- high_school_computer_science: 1.0000 (52/52)
 - machine_learning: 0.8462 (44/52)
 - medical_genetics: 0.9608 (49/51)
+- global_facts: 0.5882 (30/51)
 - management: 0.9200 (46/50)
+- us_foreign_policy: 0.9400 (47/50)
 - college_chemistry: 0.5745 (27/47)
+- abstract_algebra: 0.7234 (34/47)
+- business_ethics: 0.7826 (36/46)
+- college_computer_science: 0.9111 (41/45)
+- computer_security: 0.8140 (35/43)
 <span style="color:darkgreen">Heretic:</span>
 - Total questions: 7021
+- Correct: 5930
+- **Accuracy: 0.8446 (84.46%)**
+- Parse failures: 26
 ============================================================
 **Tested subject scores:**
+- professional_law: 0.7210 (566/785)
+- moral_scenarios: 0.7398 (327/442)
+- miscellaneous: 0.9164 (351/383)
+- professional_psychology: 0.8766 (277/316)
+- high_school_psychology: 0.9593 (259/270)
+- high_school_macroeconomics: 0.9289 (183/197)
+- elementary_mathematics: 0.9457 (174/184)
+- moral_disputes: 0.8448 (147/174)
+- prehistory: 0.9186 (158/172)
+- philosophy: 0.8113 (129/159)
 - high_school_biology: 0.9539 (145/152)
+- professional_accounting: 0.7902 (113/143)
+- clinical_knowledge: 0.9071 (127/140)
+- high_school_microeconomics: 0.9559 (130/136)
+- nutrition: 0.8741 (118/135)
+- professional_medicine: 0.8881 (119/134)
 - conceptual_physics: 0.8984 (115/128)
+- high_school_mathematics: 0.6220 (79/127)
+- human_aging: 0.8362 (97/116)
+- security_studies: 0.8304 (93/112)
+- high_school_statistics: 0.8468 (94/111)
+- marketing: 0.9633 (105/109)
+- high_school_world_history: 0.9623 (102/106)
+- sociology: 0.9223 (95/103)
+- high_school_government_and_politics: 0.9802 (99/101)
+- high_school_geography: 0.9394 (93/99)
+- high_school_chemistry: 0.7732 (75/97)
+- high_school_us_history: 0.9158 (87/95)
 - virology: 0.5056 (45/89)
+- college_medicine: 0.8409 (74/88)
+- world_religions: 0.9205 (81/88)
+- high_school_physics: 0.7619 (64/84)
+- electrical_engineering: 0.8025 (65/81)
 - astronomy: 0.9494 (75/79)
+- logical_fallacies: 0.8816 (67/76)
 - high_school_european_history: 0.8767 (64/73)
+- anatomy: 0.8592 (61/71)
 - college_biology: 0.9688 (62/64)
+- human_sexuality: 0.8594 (55/64)
+- formal_logic: 0.7812 (50/64)
 - public_relations: 0.7541 (46/61)
 - international_law: 0.9167 (55/60)
 - college_physics: 0.7018 (40/57)
+- college_mathematics: 0.8000 (44/55)
 - econometrics: 0.7778 (42/54)
+- jurisprudence: 0.8302 (44/53)
 - high_school_computer_science: 0.9808 (51/52)
+- machine_learning: 0.7500 (39/52)
+- medical_genetics: 0.9020 (46/51)
+- global_facts: 0.6275 (32/51)
+- management: 0.8800 (44/50)
+- us_foreign_policy: 0.9400 (47/50)
+- college_chemistry: 0.5957 (28/47)
+- abstract_algebra: 0.7872 (37/47)
 - business_ethics: 0.8478 (39/46)
+- college_computer_science: 0.8667 (39/45)
+- computer_security: 0.8605 (37/43)
 MMLU - Massive Multitask Language Understanding, multiple-choice questions across 57 subjects (math, history, law, medicine, etc.).
 ## GGUF Version
+GGUF quantizations available here [llmfan46/gemma-4-31B-it-qat-q4_0-uncensored-heretic-GGUF](https://huggingface.co/llmfan46/gemma-4-31B-it-qat-q4_0-uncensored-heretic-GGUF).
 -----