Hugging Face's logo

OccultAI
/
Qliphoth-12B-v1.2

Text Generation
Transformers
Safetensors
NeMo
mistral
mergekit
Merge
qliphoth
conversational
text-generation-inference
Model card Files
xet
 Community
1
Naphula commited on
Commit
d8880c1
·
verified ·
1 Parent(s): f2b45d4

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +286 -1
README.md CHANGED
@@ -113,4 +113,289 @@ out_dtype: bfloat16
113
  tokenizer:
114
  source: base
115
  chat_template: "chatml"
116
- ```
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
113
  tokenizer:
114
  source: base
115
  chat_template: "chatml"
116
+ ```
117
+
118
+ ---
119
+
120
+ ## Notes for v1 and 1.1
121
+
122
+ **Qliphoth 12B** has some refusals and may require jailbreaks or ablation to fully uncensor.
123
+
124
+ Both `Mistral Tekken` and `ChatML` chat templates are supported and may produce different results, so it's recommended to use one of those.
125
+
126
+ The model is very creative and produces highly varied, verbose output even at low temps. A `karcher` merge was tested using the same donors and was found to be less creative in comparison.
127
+
128
+ ## Version 1 vs 1.1
129
+ In my simple tests, [v1](https://huggingface.co/OccultAI/Qliphoth-12B-v1) had a distinct, visceral style and seemed to prefer `Mistral Tekken`, while [v1.1](https://huggingface.co/OccultAI/Qliphoth-12B-v1.1) was more clinical and detached, and did better with `ChatML` template.
130
+
131
+ Both versions performed well and were tested with [Q0 Bench](https://huggingface.co/Naphula/Q0_Bench), where v1.1 scored about 3000 points higher. They were also tested with the new [MiniBARD](https://huggingface.co/spaces/Naphula/MiniBARD) (Benchmark for Aesthetics, Roleplay & Depth). Again here, v1.1 outperformed v1. **Either version is great**, so test them both if you have time, although I may slightly prefer the style of v1.
132
+
133
+ ![BARD](https://cdn-uploads.huggingface.co/production/uploads/68e840caa318194c44ec2a04/NtsvJTHaEdvmRuaBSXHSn.png)
134
+
135
+ ### ⚙️ Configuration
136
+ The following YAML configuration was used to produce this model:
137
+
138
+ <details><summary>Qliphoth 12B v1 yaml</summary>
139
+
140
+ ```yaml
141
+ architecture: MistralForCausalLM
142
+ base_model: B:/12B/IntervitensInc--Mistral-Nemo-Base-2407-chatml
143
+ models:
144
+ - model: B:/12B/SicariusSicariiStuff--Impish_Bloodmoon_12B
145
+ parameters:
146
+ pinocchio: 0.0
147
+ - model: B:/12B/NeverSleep--Lumimaid-v0.2-12B
148
+ parameters:
149
+ pinocchio: 0.0
150
+ - model: B:/12B/KOOWEEYUS--BlackSheep-RP-12B
151
+ parameters:
152
+ pinocchio: 0.0
153
+ - model: B:/12B/KOOWEEYUS--BlackSheep-RP-12B # x2 influence for the apprentice
154
+ parameters:
155
+ pinocchio: 0.0
156
+ - model: B:/12B/SuperbEmphasis--MN-12b-RP-Ink-RP-Longform
157
+ parameters:
158
+ pinocchio: 0.0
159
+ - model: B:/12B/TheDrummer--Rocinante-X-12B-v1
160
+ parameters:
161
+ pinocchio: 0.0
162
+ - model: B:/12B/WokeAI--Tankie-DPE-12b-SFT
163
+ parameters:
164
+ pinocchio: 0.0
165
+ - model: B:/12B/WokeAI--Tankie-DPE-12B-SFT-v2 # pinocchio
166
+ parameters:
167
+ pinocchio: 1.0
168
+ - model: B:/12B/XeyonAI--Mistral-Helcyon-Mercury-12b-v3.2
169
+ parameters:
170
+ pinocchio: 0.0
171
+ - model: B:/12B/anthracite-org--magnum-v4-12b
172
+ parameters:
173
+ pinocchio: 0.0
174
+ - model: B:/12B/dphn--dolphin-2.9.3-mistral-nemo-12b
175
+ parameters:
176
+ pinocchio: 0.0
177
+ - model: B:/12B/Edens-Gate--nemo-erebus-lora-2152/nemo-erebus-lora-2152
178
+ parameters:
179
+ pinocchio: 0.0
180
+ - model: B:/12B/Epiculous--Azure_Dusk-v0.2
181
+ parameters:
182
+ pinocchio: 0.0
183
+ - model: B:/12B/Epiculous--Crimson_Dawn-v0.2
184
+ parameters:
185
+ pinocchio: 0.0
186
+ - model: B:/12B/Fizzarolli--MN-12b-Rosier-v1
187
+ parameters:
188
+ pinocchio: 0.0
189
+ - model: B:/12B/HumanLLMs--Human-Like-Mistral-Nemo-Instruct-2407
190
+ parameters:
191
+ pinocchio: 0.0
192
+ - model: B:/12B/IIEleven11--Kalypso
193
+ parameters:
194
+ pinocchio: 0.0
195
+ - model: B:/12B/Lambent--Arsenic-Shahrazad-12B-v4.1
196
+ parameters:
197
+ pinocchio: 0.0
198
+ - model: B:/12B/LatitudeGames--Wayfarer-2-12B
199
+ parameters:
200
+ pinocchio: 0.0
201
+ - model: B:/12B/PocketDoc--Dans-DangerousWinds-V1.1.0-12b
202
+ parameters:
203
+ pinocchio: 0.0
204
+ - model: B:/12B/PocketDoc--Dans-SakuraKaze-V1.0.0-12b
205
+ parameters:
206
+ pinocchio: 0.0
207
+ - model: B:/12B/PygmalionAI--Pygmalion-3-12B
208
+ parameters:
209
+ pinocchio: 0.0
210
+ - model: B:/12B/sleepdeprived3--Christian-Bible-Expert-v2.0-12B
211
+ parameters:
212
+ pinocchio: 0.0
213
+ - model: B:/12B/rAIfle--Questionable-MN-bf16
214
+ parameters:
215
+ pinocchio: 0.0
216
+ - model: B:/12B/jtatman--mistral_nemo_12b_reasoning_psychology_lora/mistral_nemo_12b_reasoning_psychology_lora
217
+ parameters:
218
+ pinocchio: 0.0
219
+ - model: B:/12B/LatitudeGames--Muse-12B
220
+ parameters:
221
+ pinocchio: 0.0
222
+ - model: B:/12B/allura-org--Tlacuilo-12B
223
+ parameters:
224
+ pinocchio: 0.0
225
+ - model: B:/12B/ChaoticNeutrals--Mag-Mell-Reasoner-12B
226
+ parameters:
227
+ pinocchio: 0.0
228
+ - model: A:/LLM/.cache/13B/taozi555--MN-12B-Mag-Mell-R1-KTO
229
+ parameters:
230
+ pinocchio: 0.0
231
+ - model: A:/LLM/.cache/13B/UniLLMer--GslayerKaa
232
+ parameters:
233
+ pinocchio: 0.0
234
+ merge_method: qliphoth
235
+ dtype: float32
236
+ out_dtype: bfloat16
237
+ tokenizer:
238
+ source: base
239
+ chat_template: "chatml"
240
+ ```
241
+
242
+ </details>
243
+
244
+ <details><summary>Qliphoth 12B v1.1 yaml</summary>
245
+
246
+ ```yaml
247
+ architecture: MistralForCausalLM
248
+ base_model: B:/12B/IntervitensInc--Mistral-Nemo-Base-2407-chatml
249
+ models:
250
+ - model: B:/12B/SicariusSicariiStuff--Impish_Bloodmoon_12B
251
+ parameters:
252
+ pinocchio: 0.0
253
+ - model: B:/12B/NeverSleep--Lumimaid-v0.2-12B
254
+ parameters:
255
+ pinocchio: 0.0
256
+ - model: B:/12B/KOOWEEYUS--BlackSheep-RP-12B
257
+ parameters:
258
+ pinocchio: 0.0
259
+ - model: B:/12B/nothingiisreal--MN-12B-Celeste-V1.9
260
+ parameters:
261
+ pinocchio: 0.0
262
+ - model: B:/12B/SuperbEmphasis--MN-12b-RP-Ink-RP-Longform
263
+ parameters:
264
+ pinocchio: 0.0
265
+ - model: B:/12B/TheDrummer--Rocinante-X-12B-v1
266
+ parameters:
267
+ pinocchio: 0.0
268
+ - model: B:/12B/WokeAI--Tankie-DPE-12b-SFT
269
+ parameters:
270
+ pinocchio: 0.0
271
+ - model: B:/12B/WokeAI--Tankie-DPE-12B-SFT-v2 # pinocchio
272
+ parameters:
273
+ pinocchio: 1.0
274
+ - model: B:/12B/XeyonAI--Mistral-Helcyon-Mercury-12b-v3.2
275
+ parameters:
276
+ pinocchio: 0.0
277
+ - model: B:/12B/anthracite-org--magnum-v4-12b
278
+ parameters:
279
+ pinocchio: 0.0
280
+ - model: B:/12B/dphn--dolphin-2.9.3-mistral-nemo-12b
281
+ parameters:
282
+ pinocchio: 0.0
283
+ - model: B:/12B/Edens-Gate--nemo-erebus-lora-2152/nemo-erebus-lora-2152
284
+ parameters:
285
+ pinocchio: 0.0
286
+ - model: B:/12B/Epiculous--Azure_Dusk-v0.2
287
+ parameters:
288
+ pinocchio: 0.0
289
+ - model: B:/12B/Epiculous--Crimson_Dawn-v0.2
290
+ parameters:
291
+ pinocchio: 0.0
292
+ - model: B:/12B/Fizzarolli--MN-12b-Rosier-v1
293
+ parameters:
294
+ pinocchio: 0.0
295
+ - model: B:/12B/HumanLLMs--Human-Like-Mistral-Nemo-Instruct-2407
296
+ parameters:
297
+ pinocchio: 0.0
298
+ - model: B:/12B/IIEleven11--Kalypso
299
+ parameters:
300
+ pinocchio: 0.0
301
+ - model: B:/12B/Lambent--Arsenic-Shahrazad-12B-v4.3.2
302
+ parameters:
303
+ pinocchio: 0.0
304
+ - model: B:/12B/Lambent--Arsenic-Shahrazad-12B-v4.1
305
+ parameters:
306
+ pinocchio: 0.0
307
+ - model: B:/12B/LatitudeGames--Wayfarer-2-12B
308
+ parameters:
309
+ pinocchio: 0.0
310
+ - model: B:/12B/PocketDoc--Dans-DangerousWinds-V1.1.0-12b
311
+ parameters:
312
+ pinocchio: 0.0
313
+ - model: B:/12B/PocketDoc--Dans-SakuraKaze-V1.0.0-12b
314
+ parameters:
315
+ pinocchio: 0.0
316
+ - model: B:/12B/PygmalionAI--Pygmalion-3-12B
317
+ parameters:
318
+ pinocchio: 0.0
319
+ - model: B:/12B/sleepdeprived3--Christian-Bible-Expert-v2.0-12B
320
+ parameters:
321
+ pinocchio: 0.0
322
+ - model: B:/12B/rAIfle--Questionable-MN-bf16
323
+ parameters:
324
+ pinocchio: 0.0
325
+ - model: B:/12B/jtatman--mistral_nemo_12b_reasoning_psychology_lora/mistral_nemo_12b_reasoning_psychology_lora
326
+ parameters:
327
+ pinocchio: 0.0
328
+ - model: B:/12B/LatitudeGames--Muse-12B
329
+ parameters:
330
+ pinocchio: 0.0
331
+ - model: B:/12B/allura-org--Tlacuilo-12B
332
+ parameters:
333
+ pinocchio: 0.0
334
+ - model: B:/12B/ChaoticNeutrals--Mag-Mell-Reasoner-12B
335
+ parameters:
336
+ pinocchio: 0.0
337
+ - model: A:/LLM/.cache/13B/taozi555--MN-12B-Mag-Mell-R1-KTO
338
+ parameters:
339
+ pinocchio: 0.0
340
+ - model: A:/LLM/.cache/13B/UniLLMer--GslayerKaa
341
+ parameters:
342
+ pinocchio: 0.0
343
+ - model: B:/12B/Retreatcost--Evertide-RX-12B
344
+ parameters:
345
+ pinocchio: 0.0
346
+ merge_method: qliphoth
347
+ dtype: float32
348
+ out_dtype: bfloat16
349
+ tokenizer:
350
+ source: base
351
+ chat_template: "chatml"
352
+ ```
353
+
354
+ </details>
355
+
356
+ To fix tokenizer issues while retaining enhanced intelligence of the chatml base:
357
+
358
+ 1. I first merged the model using
359
+ ```yaml
360
+ base_model: B:/12B/IntervitensInc--Mistral-Nemo-Base-2407-chatml
361
+ tokenizer:
362
+ source: base
363
+ chat_template: "chatml"
364
+ ```
365
+
366
+ 2. I then merged it again using
367
+ ```yaml
368
+ base_model: B:/12B/mistralai--Mistral-Nemo-Instruct-2407
369
+ tokenizer:
370
+ source: union
371
+ chat_template: auto
372
+ ```
373
+
374
+ 3. I then had to modify `mergekit/tokenizer/embed.py` to allow for the passthrough merge
375
+ ```py
376
+ token_configs[token] = TokenEmbeddingConfig(source=ZeroEmbedding(kind="zero"))
377
+
378
+ ) -> torch.Tensor:
379
+ if isinstance(cfg.source, ZeroEmbedding):
380
+ first_tensor = next(iter(tensors.values()))
381
+ embed = torch.zeros(
382
+ first_tensor.shape[1],
383
+ dtype=first_tensor.dtype,
384
+ device=first_tensor.device
385
+ )
386
+ ```
387
+
388
+ 4. I then ran another merge to fix tokenizer issues
389
+ ```yaml
390
+ merge_method: passthrough
391
+ slices:
392
+ - sources:
393
+ - model: B:\12B\Stage1Base
394
+ layer_range: [0, 40]
395
+ tokenizer_source: B:\12B\Stage2Union
396
+ chat_template: auto
397
+ ```
398
+
399
+ For some reason, this process produced significantly smarter output than just using `base_model: B:/12B/mistralai--Mistral-Nemo-Instruct-2407`.
400
+
401
+ **This passthrough process using base as base_model was replicated for v1.2**