| --- |
| license: llama3.3 |
| base_model: |
| - shb777/Llama-3.3-8B-Instruct-128K |
| pipeline_tag: text-generation |
| tags: |
| - heretic |
| - uncensored |
| - decensored |
| - abliterated |
| --- |
| This is a **Llama-3.3-8B-Instruct-128K** fine-tune, produced through P-E-W's [Heretic](https://github.com/p-e-w/heretic) (v1.2.0) abliteration engine with [Magnitude-Preserving Orthogonal Ablation](https://github.com/p-e-w/heretic/pull/52) enabled. |
|
|
| **Note:** Model exhibits overt non-compliance (divergence, changing focus, reinterpretation, and rarely argumentation). An effort was made to target model-unique refusals, overt non-compliance, and disclaimer/warning attachments. |
|
|
| --- |
| <img src="https://img.shields.io/badge/RENEGADE_CHAPTER-PAPERWITCH-B85ADB?style=flat-square&labelColor=101010" align="right" width="300"> |
|
|
| **Heretication Results** |
|
|
| | Score Metric | Value | Parameter | Value | |
| | :--- | :--- | :--- | :--- | |
| | **Refusals** | 0/100 | **direction_index** | 11.17 | |
| | **KL Divergence** | 0.0448 | **attn.o_proj.max_weight** | 1.92 | |
| | **Initial Refusals** | 102/104 | **attn.o_proj.max_weight_position** | 6.82 | |
| ||| **attn.o_proj.min_weight** | 1.77 | |
| ||| **attn.o_proj.min_weight_distance** | 23.82 | |
| ||| **mlp.down_proj.max_weight** | 0.85 | |
| ||| **mlp.down_proj.max_weight_position** | 7.03 | |
| ||| **mlp.down_proj.min_weight** | 0.77 | |
| ||| **mlp.down_proj.min_weight_distance** | 28.52 | |
| |
| --- |
| **Appendix** |
| |
| > Empty system prompt. |
| |
| > Previous attempt: [Click Here](https://huggingface.co/MuXodious/Llama-3.3-8B-Instruct-128K-absolute-heresy) |
| |
| <img src="https://huggingface.co/MuXodious/Llama-3.3-8B-Instruct-128K-PaperWitch-heresy/resolve/main/Llama-3.3-8B-Instruct-128K.gif" alt="PaCMAP projection"/> |
| |
| Trial 196 was the optimal choice, picked 192. Additional trials can be run. |
| ``` |
| Restoring model from trial 196... |
| * Parameters: |
| * direction_index = 10.72 |
| * attn.o_proj.max_weight = 1.87 |
| * attn.o_proj.max_weight_position = 20.92 |
| * attn.o_proj.min_weight = 1.76 |
| * attn.o_proj.min_weight_distance = 16.32 |
| * mlp.down_proj.max_weight = 0.78 |
| * mlp.down_proj.max_weight_position = 6.49 |
| * mlp.down_proj.min_weight = 0.54 |
| * mlp.down_proj.min_weight_distance = 13.90 |
| ``` |
| |
| ``` |
| » [Trial 192] Refusals: 0/104, KL divergence: 0.0448 |
| [Trial 199] Refusals: 2/104, KL divergence: 0.0398 |
| [Trial 196] Refusals: 5/104, KL divergence: 0.0273 |
| [Trial 141] Refusals: 21/104, KL divergence: 0.0207 |
| [Trial 101] Refusals: 22/104, KL divergence: 0.0205 |
| [Trial 205] Refusals: 37/104, KL divergence: 0.0132 |
| [Trial 213] Refusals: 58/104, KL divergence: 0.0124 |
| [Trial 131] Refusals: 72/104, KL divergence: 0.0088 |
| [Trial 214] Refusals: 81/104, KL divergence: 0.0080 |
| [Trial 52] Refusals: 83/104, KL divergence: 0.0065 |
| [Trial 18] Refusals: 88/104, KL divergence: 0.0057 |
| [Trial 332] Refusals: 92/104, KL divergence: 0.0057 |
| [Trial 68] Refusals: 94/104, KL divergence: 0.0048 |
| [Trial 37] Refusals: 98/104, KL divergence: 0.0043 |
| [Trial 28] Refusals: 99/104, KL divergence: 0.0022 |
| [Trial 313] Refusals: 100/104, KL divergence: 0.0020 |
| [Trial 20] Refusals: 101/104, KL divergence: 0.0015 |
| [Trial 178] Refusals: 102/104, KL divergence: 0.0004 |
| ``` |
| |
| --- |
| # Llama 3.3 8B 128K Instruct (Fixed) |
| |
| > [!IMPORTANT] |
| > Original [allura-forge/Llama-3.3-8B-Instruct](https://huggingface.co/allura-forge/Llama-3.3-8B-Instruct), Thanks! |
| |
| > [!TIP] |
| > [imatrix GGUF's by mradermacher (Recommended)](https://huggingface.co/mradermacher/Llama-3.3-8B-Instruct-128K-i1-GGUF) |
| > |
| > [static GGUF's](https://huggingface.co/shb777/Llama-3.3-8B-Instruct-128K-GGUF) |
| > |
| > [Evals](https://huggingface.co/datasets/shb777/Llama-3.3-8B-Instruct-128K-Evals) |
| |
| Additional Fixes: |
| - Added `rope_scaling` |
| - Added chat template (Unsloth) in tokenizer config |
| - Updated generation config |
| - Enabled full context length |
