Upload README.md with huggingface_hub

README.md

@@ -1,199 +1,423 @@
 ---
-library_name: transformers
-tags: []
+base_model: unsloth/gemma-4-31B-it
+tags:
+- text-generation-inference
+- transformers
+- unsloth
+- gemma4
+- reasoning
+- heretic
+- uncensored
+- decensored
+- abliterated
+- reproducible
+license: apache-2.0
+datasets:
+- TeichAI/Claude-Opus-4.6-Reasoning-887x
+- TeichAI/claude-4.5-opus-high-reasoning-250x
+- Crownelius/Opus-4.6-Reasoning-2100x-formatted
 ---
+# This is a decensored version of [TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2](https://huggingface.co/TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2), made using [Heretic](https://github.com/p-e-w/heretic) v1.3.0
 
-# Model Card for Model ID
+> [!TIP]
+> **This model is reproducible!**
+>
+> See the [README](reproduce/README.md) in the `reproduce` directory for more information.
 
-<!-- Provide a quick summary of what the model is/does. -->
+## Abliteration parameters
 
+| Parameter | Value |
+| :-------- | :---: |
+| **direction_index** | 34.68 |
+| **attn.o_proj.max_weight** | 1.49 |
+| **attn.o_proj.max_weight_position** | 35.77 |
+| **attn.o_proj.min_weight** | 0.58 |
+| **attn.o_proj.min_weight_distance** | 34.11 |
+| **mlp.down_proj.max_weight** | 1.50 |
+| **mlp.down_proj.max_weight_position** | 36.82 |
+| **mlp.down_proj.min_weight** | 1.28 |
+| **mlp.down_proj.min_weight_distance** | 18.49 |
 
+## Performance
 
-## Model Details
+| Metric | This model | Original model ([TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2](https://huggingface.co/TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2)) |
+| :----- | :--------: | :---------------------------: |
+| **KL divergence** | 0.0063 | 0 *(by definition)* |
+| **Refusals** | 28/100 | 89/100 |
 
-### Model Description
+-----
 
-<!-- Provide a longer summary of what this model is. -->
 
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+# 🌟 Gemma 4 - 31B x Claude Opus 4.6 v2
 
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
+> **Build Environment & Features:**
+> - **Fine-tuning Framework**: **Unsloth**
+> - **Reasoning Effort**: **High**
+> - This model bridges the gap between Google's exceptional open-weights architecture and Claude 4.6's profound reasoning capabilities, leveraging cutting-edge fine-tuning environments.
 
-### Model Sources [optional]
+![Gemma 4 Benchmarks](https://storage.googleapis.com/gweb-uniblog-publish-prod/documents/gemma-4-table_light_Web_with_Arena.jpg)
 
-<!-- Provide the basic links for the model. -->
+## 💡 Model Introduction
+**Gemma 4 - 31B x Claude Opus 4.6** is a highly capable model fine-tuned on top of the powerful `unsloth/gemma-4-31B-it` architecture. The model's core directive is to absorb state-of-the-art reasoning distillation, primarily sourced from Claude-4.6 Opus interactions. 
 
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
+By utilizing datasets where the reasoning effort was explicitly set to **High**, this model excels in breaking down complex problems and delivering precise, nuanced solutions across a variety of demanding domains.
 
-## Uses
+## 🗺️ Training Pipeline Overview
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+```text
+Base Model (unsloth/gemma-4-31B-it)
+ │
+ ▼
+Supervised Fine-Tuning (SFT) + High-Effort Reasoning Datasets
+ │
+ ▼
+Final Model (Gemma 4 - 31B x Claude Opus 4.6)
+````
 
-### Direct Use
+## 📋 Stage Details & Benchmarks
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+*Benchmarks coming soon*
 
-[More Information Needed]
+**Performance vs Size:**
 
-### Downstream Use [optional]
+> **Deep Dive Analysis:** For more comprehensive insights regarding the base capabilities of the Gemma 4 architecture, please refer to [this Analysis Document](https://huggingface.co/TeichAI/gemma-4-31B-it-Claude-Opus-Distill/resolve/main/Gemma%204%20Analysis.pdf).
 
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+### 🔹 Supervised Fine-Tuning (Meeting Claude)
 
-[More Information Needed]
+  - **Objective:** To inject high-density reasoning logic and establish a strict format for complex problem-solving.
+  - **Methodology:** We utilized **Unsloth** for highly efficient memory and compute optimization during the fine-tuning process. The model was trained extensively on various reasoning trajectories from Claude Opus 4.6 to adopt a structured and efficient thinking pattern.
 
-### Out-of-Scope Use
+### 📚 All Datasets Used
 
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+The dataset consists of high-quality, high-effort reasoning distillation data:
 
-[More Information Needed]
+| Dataset Name | Description / Purpose |
+|--------------|-----------------------|
+| `TeichAI/Claude-Opus-4.6-Reasoning-887x` | Core Claude 4.6 Opus reasoning trajectories. |
+| `TeichAI/claude-4.5-opus-high-reasoning-250x` | High-intensity reasoning distillation. |
+| `Crownelius/Opus-4.6-Reasoning-2100x-formatted` | Crownelius's extensively formatted Opus reasoning dataset for structural reinforcement. |
 
-## Bias, Risks, and Limitations
+## 🌟 Core Skills & Capabilities
 
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+Thanks to its robust base model and high-effort reasoning distillation, this model is highly optimized for the following use cases:
 
-[More Information Needed]
+1.  **💻 Coding:** Advanced code generation, debugging, and software architecture planning.
+2.  **🔬 Science:** Deep scientific reasoning, hypothesis evaluation, and analytical problem-solving.
+3.  **🔎 Deep Research:** Navigating complex, multi-step research queries and synthesizing vast amounts of information.
+4.  **🧠 General Purpose:** Highly capable instruction-following for everyday tasks requiring high logical coherence.
 
-### Recommendations
+## Getting Started
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+You can use all Gemma 4 models with the latest version of Transformers. To get started, install the necessary dependencies in your environment:
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+`pip install -U transformers torch accelerate`
 
-## How to Get Started with the Model
+Once you have everything installed, you can proceed to load the model with the code below:
 
-Use the code below to get started with the model.
+```python
+from transformers import AutoProcessor, AutoModelForCausalLM
 
-[More Information Needed]
+MODEL_ID = "google/gemma-4-31B-it"
 
-## Training Details
+# Load model
+processor = AutoProcessor.from_pretrained(MODEL_ID)
+model = AutoModelForCausalLM.from_pretrained(
+    MODEL_ID,
+    dtype="auto",
+    device_map="auto"
+)
+```
 
-### Training Data
+Once the model is loaded, you can start generating output:
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+```python
+# Prompt
+messages = [
+    {"role": "system", "content": "You are a helpful assistant."},
+    {"role": "user", "content": "Write a short joke about saving RAM."},
+]
 
-[More Information Needed]
+# Process input
+text = processor.apply_chat_template(
+    messages, 
+    tokenize=False, 
+    add_generation_prompt=True, 
+    enable_thinking=False
+)
+inputs = processor(text=text, return_tensors="pt").to(model.device)
+input_len = inputs["input_ids"].shape[-1]
 
-### Training Procedure
+# Generate output
+outputs = model.generate(**inputs, max_new_tokens=1024)
+response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
 
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+# Parse output
+processor.parse_response(response)
+```
 
-#### Preprocessing [optional]
+To enable reasoning, set `enable_thinking=True` and the `parse_response` function will take care of parsing the thinking output.
 
-[More Information Needed]
+Below, you will also find snippets for processing audio (E2B and E4B only), images, and video alongside text:
 
+<details>
+<summary>Code for processing Audio</summary>
 
-#### Training Hyperparameters
+Instead of using `AutoModelForCausalLM`, you can use `AutoModelForMultimodalLM` to process audio. To use it, make sure to install the following packages:
 
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
 
-#### Speeds, Sizes, Times [optional]
+`pip install -U transformers torch librosa accelerate`
 
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+You can then load the model with the code below:
 
-[More Information Needed]
+```python
+from transformers import AutoProcessor, AutoModelForMultimodalLM
 
-## Evaluation
+MODEL_ID = "google/gemma-4-E2B-it"
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+# Load model
+processor = AutoProcessor.from_pretrained(MODEL_ID)
+model = AutoModelForMultimodalLM.from_pretrained(
+    MODEL_ID, 
+    dtype="auto", 
+    device_map="auto"
+)
+```
 
-### Testing Data, Factors & Metrics
+Once the model is loaded, you can start generating output by directly referencing the audio URL in the prompt:
 
-#### Testing Data
 
-<!-- This should link to a Dataset Card if possible. -->
+```python
+# Prompt - add audio before text
+messages = [
+    {
+        "role": "user",
+        "content": [
+            {"type": "audio", "audio": "https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/Demos/sample-data/journal1.wav"},
+            {"type": "text", "text": "Transcribe the following speech segment in its original language. Follow these specific instructions for formatting the answer:\n* Only output the transcription, with no newlines.\n* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three."},
+        ]
+    }
+]
 
-[More Information Needed]
+# Process input
+inputs = processor.apply_chat_template(
+    messages,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt",
+    add_generation_prompt=True,
+).to(model.device)
+input_len = inputs["input_ids"].shape[-1]
 
-#### Factors
+# Generate output
+outputs = model.generate(**inputs, max_new_tokens=512)
+response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
 
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+# Parse output
+processor.parse_response(response)
+```
 
-[More Information Needed]
+</details>
 
-#### Metrics
+<details>
+<summary>Code for processing Images</summary>
+
+Instead of using `AutoModelForCausalLM`, you can use `AutoModelForMultimodalLM` to process images. To use it, make sure to install the following packages:
+
+
+`pip install -U transformers torch torchvision accelerate`
 
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+You can then load the model with the code below:
 
-[More Information Needed]
+```python
+from transformers import AutoProcessor, AutoModelForMultimodalLM
+
+MODEL_ID = "google/gemma-4-31B-it"
 
-### Results
+# Load model
+processor = AutoProcessor.from_pretrained(MODEL_ID)
+model = AutoModelForMultimodalLM.from_pretrained(
+    MODEL_ID, 
+    dtype="auto", 
+    device_map="auto"
+)
+```
 
-[More Information Needed]
+Once the model is loaded, you can start generating output by directly referencing the image URL in the prompt:
 
-#### Summary
 
+```python
+# Prompt - add image before text
+messages = [
+    {
+        "role": "user", "content": [
+            {"type": "image", "url": "https://raw.githubusercontent.com/google-gemma/cookbook/refs/heads/main/Demos/sample-data/GoldenGate.png"},
+            {"type": "text", "text": "What is shown in this image?"}
+        ]
+    }
+]
 
+# Process input
+inputs = processor.apply_chat_template(
+    messages,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt",
+    add_generation_prompt=True,
+).to(model.device)
+input_len = inputs["input_ids"].shape[-1]
 
-## Model Examination [optional]
+# Generate output
+outputs = model.generate(**inputs, max_new_tokens=512)
+response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
 
-<!-- Relevant interpretability work for the model goes here -->
+# Parse output
+processor.parse_response(response)
+```
 
-[More Information Needed]
+</details>
 
-## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+<details>
+<summary>Code for processing Videos</summary>
 
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+Instead of using `AutoModelForCausalLM`, you can use `AutoModelForMultimodalLM` to process videos. To use it, make sure to install the following packages:
 
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
+`pip install -U transformers torch torchvision torchcodec librosa accelerate`
 
-## Technical Specifications [optional]
+You can then load the model with the code below:
 
-### Model Architecture and Objective
+```python
+from transformers import AutoProcessor, AutoModelForMultimodalLM
 
-[More Information Needed]
+MODEL_ID = "google/gemma-4-31B-it"
 
-### Compute Infrastructure
+# Load model
+processor = AutoProcessor.from_pretrained(MODEL_ID)
+model = AutoModelForMultimodalLM.from_pretrained(
+    MODEL_ID, 
+    dtype="auto", 
+    device_map="auto"
+)
+```
 
-[More Information Needed]
+Once the model is loaded, you can start generating output by directly referencing the video URL in the prompt:
 
-#### Hardware
 
-[More Information Needed]
+```python
+# Prompt - add video before text
+messages = [
+    {
+        'role': 'user',
+        'content': [
+            {"type": "video", "video": "https://github.com/bebechien/gemma/raw/refs/heads/main/videos/ForBiggerBlazes.mp4"},
+            {'type': 'text', 'text': 'Describe this video.'}
+        ]
+    }
+]
 
-#### Software
+# Process input
+inputs = processor.apply_chat_template(
+    messages,
+    tokenize=True,
+    return_dict=True,
+    return_tensors="pt",
+    add_generation_prompt=True,
+).to(model.device)
+input_len = inputs["input_ids"].shape[-1]
 
-[More Information Needed]
+# Generate output
+outputs = model.generate(**inputs, max_new_tokens=512)
+response = processor.decode(outputs[0][input_len:], skip_special_tokens=False)
 
-## Citation [optional]
+# Parse output
+processor.parse_response(response)
+```
 
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+</details>
 
-**BibTeX:**
+## **Best Practices**
 
-[More Information Needed]
+For the best performance, use these configurations and best practices:
 
-**APA:**
+### 1. Sampling Parameters
 
-[More Information Needed]
+Use the following standardized sampling configuration across all use cases:
 
-## Glossary [optional]
+* `temperature=1.0`  
+* `top_p=0.95`  
+* `top_k=64`
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+### 2. Thinking Mode Configuration
 
-[More Information Needed]
+Compared to Gemma 3, the models use standard `system`, `assistant`, and `user` roles. To properly manage the thinking process, use the following control tokens:
 
-## More Information [optional]
+* **Trigger Thinking:** Thinking is enabled by including the `<|think|>` token at the start of the system prompt. To disable thinking, remove the token.   
+* **Standard Generation:** When thinking is enabled, the model will output its internal reasoning followed by the final answer using this structure:  
+  `<|channel>thought\n`**[Internal reasoning]**`<channel|>`  
+* **Disabled Thinking Behavior:** For all models except for the E2B and E4B variants, if thinking is disabled, the model will still generate the tags but with an empty thought block:  
+  `<|channel>thought\n<channel|>`**[Final answer]**
 
-[More Information Needed]
+> [!Note]
+> Note that many libraries like Transformers and llama.cpp handle the complexities of the chat template for you.
 
-## Model Card Authors [optional]
+### 3. Multi-Turn Conversations
 
-[More Information Needed]
+* **No Thinking Content in History**: In multi-turn conversations, the historical model output should only include the final response. Thoughts from previous model turns must *not be added* before the next user turn begins.
 
-## Model Card Contact
+### 4. Modality order
 
-[More Information Needed]
+* For optimal performance with multimodal inputs, place image and/or audio content **before** the text in your prompt. 
+
+### 5. Variable Image Resolution
+
+Aside from variable aspect ratios, Gemma 4 supports variable image resolution through a configurable visual token budget, which controls how many tokens are used to represent an image. A higher token budget preserves more visual detail at the cost of additional compute, while a lower budget enables faster inference for tasks that don't require fine-grained understanding.
+
+* The supported token budgets are: **70**, **140**, **280**, **560**, and **1120**.  
+  * Use *lower budgets* for classification, captioning, or video understanding, where faster inference and processing many frames outweigh fine-grained detail.   
+  * Use *higher budgets* for tasks like OCR, document parsing, or reading small text.
+
+### 6. Audio
+
+Use the following prompt structures for audio processing:
+
+* **Audio Speech Recognition (ASR)**
+
+```text
+Transcribe the following speech segment in {LANGUAGE} into {LANGUAGE} text.
+
+Follow these specific instructions for formatting the answer:
+* Only output the transcription, with no newlines.
+* When transcribing numbers, write the digits, i.e. write 1.7 and not one point seven, and write 3 instead of three.
+```
+
+* **Automatic Speech Translation (AST)**
+
+```text
+Transcribe the following speech segment in {SOURCE_LANGUAGE}, then translate it into {TARGET_LANGUAGE}.
+When formatting the answer, first output the transcription in {SOURCE_LANGUAGE}, then one newline, then output the string '{TARGET_LANGUAGE}: ', then the translation in {TARGET_LANGUAGE}.
+```
+
+### 7. Audio and Video Length
+
+All models support image inputs and can process videos as frames whereas the E2B and E4B models also support audio inputs. Audio supports a maximum length of 30 seconds. Video supports a maximum of 60 seconds assuming the images are processed at one frame per second.
+
+## 🙏 Acknowledgements
+
+  - **Google**: For providing an exceptional open weights model. Read more about Gemma 4 on the [Google Innovation Blog](https://blog.google/innovation-and-ai/technology/developers-tools/gemma-4/).
+  - **Unsloth**: For assembling ready-to-use, cutting-edge fine-tuning environments that make this work possible.
+  - **Crownelius**: For creating and sharing his awesome Opus reasoning dataset with the community.
+
+
+## 📖 Citation
+
+If you use this model in your research or projects, please cite:
+
+```bibtex
+@misc{teichai_gemma4_31b_opus_distilled_v2,
+  title        = {Gemma-4-31B-it-Claude-Opus-Distill-v2},
+  author       = {TeichAI},
+  year         = {2026},
+  publisher    = {Hugging Face},
+  howpublished = {\url{https://huggingface.co/TeichAI/gemma-4-31B-it-Claude-Opus-Distill-v2}}
+}
+```