Initial GPTQ int4 upload

.gitattributes

@@ -33,3 +33,4 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 *.zip filter=lfs diff=lfs merge=lfs -text
 *.zst filter=lfs diff=lfs merge=lfs -text
 *tfevents* filter=lfs diff=lfs merge=lfs -text
+tokenizer.json filter=lfs diff=lfs merge=lfs -text

README.md

@@ -0,0 +1,92 @@
+---
+license: apache-2.0
+base_model: talkie-lm/talkie-1930-13b-it
+tags:
+  - gptq
+  - 4-bit
+  - quantized
+  - instruction-tuned
+  - vintage-language-model
+  - chat
+language:
+  - en
+pipeline_tag: text-generation
+---
+
+# talkie-1930-13b-it — GPTQ int4
+
+A 4-bit GPTQ quantization of [`talkie-lm/talkie-1930-13b-it`](https://huggingface.co/talkie-lm/talkie-1930-13b-it) — the **instruction-tuned** variant of talkie 13B, by Alec Radford, Nick Levine, and David Duvenaud.
+
+The base model was trained on 260B tokens of pre-1931 English. This IT variant was further fine-tuned on a custom instruction-following dataset built entirely from pre-1931 reference works (etiquette manuals, letter-writing manuals, encyclopedias, poetry collections), then refined with online DPO. The result: a chatbot that answers in **early-20th-century formal English** and has no knowledge of anything after 1930.
+
+This quantization shrinks the model from ~24.7 GB (bf16) to ~7.4 GB (int4), so it fits comfortably on a single 16 GB consumer GPU.
+
+## Use it
+
+```python
+from gptqmodel import GPTQModel
+import talkie_hf.talkie_qmodel  # registers TalkieQModel
+
+model = GPTQModel.load("dtestnyrr/talkie-1930-13b-it-gptq-int4", trust_remote_code=True)
+
+# Use the talkie chat template
+prompt = "<|user|>Write a brief letter declining a dinner invitation.<|end|><|assistant|>"
+ids = model.tokenizer(prompt, return_tensors="pt").input_ids.cuda()
+out = model.generate(input_ids=ids, max_new_tokens=300, do_sample=True, temperature=0.7)
+print(model.tokenizer.decode(out[0], skip_special_tokens=False))
+```
+
+Stop generation at any of: `<|end|>`, `<|user|>`, `<|assistant|>`, `<|system|>`, `<|endoftext|>`.
+
+## Chat template
+
+```
+<|system|>{optional system prompt}<|end|>
+<|user|>{user message 1}<|end|>
+<|assistant|>{model reply 1}<|end|>
+<|user|>{user message 2}<|end|>
+<|assistant|>
+```
+
+End the prompt with `<|assistant|>` (no trailing `<|end|>`) to signal the model should generate.
+
+## Quantization details
+
+Same recipe as the base-variant int4 release ([dtestnyrr/talkie-1930-13b-base-gptq-int4](https://huggingface.co/dtestnyrr/talkie-1930-13b-base-gptq-int4)):
+
+| Parameter | Value |
+|---|---|
+| Method | GPTQ |
+| Bits | 4 |
+| Group size | 128 |
+| Activation order | False |
+| Symmetric | True |
+| Effective bits per weight | ~4.29 BPW |
+| Calibration corpus | 256 × 2048-token windows from 8 pre-1931 Project Gutenberg classics |
+| Quantization framework | [GPTQModel](https://github.com/ModelCloud/GPTQModel) v6.0.3 |
+
+## What it's good at
+
+- Period-correct prose in any genre (letters, essays, sermons, news editorials, fiction)
+- Encyclopedia-style explanations of anything pre-1930
+- 1900-1930s-style poetry in proper meter and rhyme
+- Edwardian / Georgian etiquette
+- "Predicting" the future from a 1930 vantage point
+
+## What it can't do
+
+- Knowledge of anything post-1930 (no WWII, no computers, no internet, no DNA structure, no antibiotics beyond very early research, etc.)
+- Modern slang or casual register — it will respond formally regardless of how casually you address it
+- Technical assistance with modern tools, languages, or frameworks
+- Knowledge of people who became famous after 1930
+
+## Architecture
+
+Custom decoder-only transformer (40 layers, 5120 hidden, 40 heads × 128 head_dim, 13,696 SwiGLU intermediate, vocab 65,540 = 65,536 BPE merges + 4 chat special tokens). Uses RoPE θ=10⁶, F.rms_norm everywhere, QK-norm, and per-residual learnable gain modules. See [`modeling_talkie.py`](modeling_talkie.py) for the full HF-compatible port.
+
+## License & attribution
+
+Apache 2.0. Original model credit:
+- Authors: Alec Radford, Nick Levine, David Duvenaud
+- Project: https://talkie-lm.com/
+- Reference code: https://github.com/talkie-lm/talkie

config.json

@@ -0,0 +1,73 @@
+{
+  "architectures": [
+    "TalkieForCausalLM"
+  ],
+  "auto_map": {
+    "AutoConfig": "configuration_talkie.TalkieConfig",
+    "AutoModelForCausalLM": "modeling_talkie.TalkieForCausalLM"
+  },
+  "dtype": "bfloat16",
+  "eos_token": "<|endoftext|>",
+  "eos_token_id": 262143,
+  "head_dim": 128,
+  "hidden_size": 5120,
+  "intermediate_size": 13696,
+  "max_position_embeddings": 2048,
+  "model_type": "talkie",
+  "num_attention_heads": 40,
+  "num_hidden_layers": 40,
+  "pad_token_id": 262143,
+  "quantization_config": {
+    "bits": 4,
+    "checkpoint_format": "gptq",
+    "desc_act": false,
+    "format": "gptq",
+    "group_size": 128,
+    "lm_head": false,
+    "meta": {
+      "act_group_aware": true,
+      "auto_forward_data_parallel": true,
+      "damp_auto_increment": 0.01,
+      "damp_percent": 0.05,
+      "fallback": {
+        "smooth": null,
+        "strategy": "rtn",
+        "threshold": "0.5%"
+      },
+      "foem": null,
+      "gc_mode": "interval",
+      "gptaq": null,
+      "hessian": {
+        "chunk_bytes": null,
+        "chunk_size": null,
+        "staging_dtype": "float32"
+      },
+      "mock_quantization": false,
+      "mse": 0.0,
+      "offload_to_disk": true,
+      "offload_to_disk_path": "./gptqmodel_offload/ctrizrxh-tetrojst/",
+      "pack_impl": "cpu",
+      "quantizer": [
+        "gptqmodel:6.0.3"
+      ],
+      "static_groups": false,
+      "true_sequential": true,
+      "uri": "https://github.com/modelcloud/gptqmodel",
+      "vram_strategy": "exclusive",
+      "wait_for_submodule_finalizers": false
+    },
+    "method": "gptq",
+    "pack_dtype": "int32",
+    "quant_method": "gptq",
+    "sym": true
+  },
+  "rope_parameters": {
+    "rope_theta": 1000000.0,
+    "rope_type": "default"
+  },
+  "rope_theta": 1000000.0,
+  "tie_word_embeddings": false,
+  "transformers_version": "5.6.2",
+  "use_cache": false,
+  "vocab_size": 65540
+}

configuration_talkie.py

@@ -0,0 +1,28 @@
+from transformers import PretrainedConfig
+
+
+class TalkieConfig(PretrainedConfig):
+    model_type = "talkie"
+
+    def __init__(
+        self,
+        vocab_size: int = 65536,
+        hidden_size: int = 5120,
+        num_hidden_layers: int = 40,
+        num_attention_heads: int = 40,
+        head_dim: int = 128,
+        intermediate_size: int = 13696,
+        max_position_embeddings: int = 2048,
+        rope_theta: float = 1_000_000.0,
+        tie_word_embeddings: bool = False,
+        **kwargs,
+    ):
+        self.vocab_size = vocab_size
+        self.hidden_size = hidden_size
+        self.num_hidden_layers = num_hidden_layers
+        self.num_attention_heads = num_attention_heads
+        self.head_dim = head_dim
+        self.intermediate_size = intermediate_size
+        self.max_position_embeddings = max_position_embeddings
+        self.rope_theta = rope_theta
+        super().__init__(tie_word_embeddings=tie_word_embeddings, **kwargs)

generation_config.json

@@ -0,0 +1,8 @@
+{
+  "_from_model_config": true,
+  "do_sample": true,
+  "output_attentions": false,
+  "output_hidden_states": false,
+  "transformers_version": "5.6.2",
+  "use_cache": false
+}

model-00001-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d168496e38c3848ee7002557e440c7f57b3469a3d67da11e5a4caaeb649c54b8
+size 4293839946

model-00002-of-00002.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:2d77720b8c0dd6c059680823acdc70988afa2324650c8eaabf7735688928a410
+size 3606506944

modeling_talkie.py

@@ -0,0 +1,253 @@
+"""HuggingFace-compatible port of TalkieModel.
+
+Mirrors talkie/src/talkie/model.py exactly:
+- F.rms_norm everywhere (no learnable RMSNorm scale)
+- RoPE with base=1e6
+- Attention with QK-norm and per-head HeadGain on Q
+- SwiGLU MLP
+- Per-layer ActGain on attn / mlp / embed_skip residuals
+- WeightGain on the lm_head matrix
+
+Linear projections are renamed to Llama conventions (q_proj, k_proj, v_proj,
+o_proj, gate_proj, up_proj, down_proj) so GPTQModel auto-detects them.
+"""
+
+from __future__ import annotations
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from transformers import PreTrainedModel
+from transformers.generation import GenerationMixin
+from transformers.modeling_outputs import CausalLMOutput
+
+from .configuration_talkie import TalkieConfig
+
+
+def _apply_rotary_emb(x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+    # x: [B, T, H, D]; cos/sin: [1, T, 1, D/2]
+    d = x.shape[-1] // 2
+    x1, x2 = x[..., :d], x[..., d:]
+    y1 = x1 * cos + x2 * sin
+    y2 = -x1 * sin + x2 * cos
+    return torch.cat([y1, y2], dim=-1).type_as(x)
+
+
+class HeadGain(nn.Module):
+    def __init__(self, n_head: int):
+        super().__init__()
+        self.head_g = nn.Parameter(torch.ones(n_head))
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return x * self.head_g.type_as(x).view(1, 1, -1, 1)
+
+
+class WeightGain(nn.Module):
+    def __init__(self):
+        super().__init__()
+        self.w_g = nn.Parameter(torch.ones(1))
+
+    def forward(self, w: torch.Tensor) -> torch.Tensor:
+        return w * self.w_g.type_as(w)
+
+
+class ActGain(nn.Module):
+    def __init__(self, init_value: float):
+        super().__init__()
+        self.a_g = nn.Parameter(torch.ones(1) * init_value)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return x * self.a_g.type_as(x)
+
+
+class TalkieAttention(nn.Module):
+    def __init__(self, config: TalkieConfig):
+        super().__init__()
+        self.n_head = config.num_attention_heads
+        self.head_dim = config.head_dim
+        h = config.hidden_size
+        self.q_proj = nn.Linear(h, h, bias=False)
+        self.k_proj = nn.Linear(h, h, bias=False)
+        self.v_proj = nn.Linear(h, h, bias=False)
+        self.o_proj = nn.Linear(h, h, bias=False)
+        self.head_gain = HeadGain(config.num_attention_heads)
+
+    def forward(self, x: torch.Tensor, cos: torch.Tensor, sin: torch.Tensor) -> torch.Tensor:
+        bsz, seq_len, _ = x.size()
+        q = self.q_proj(x).view(bsz, seq_len, self.n_head, self.head_dim)
+        k = self.k_proj(x).view(bsz, seq_len, self.n_head, self.head_dim)
+        v = self.v_proj(x).view(bsz, seq_len, self.n_head, self.head_dim)
+
+        q = _apply_rotary_emb(q, cos, sin)
+        k = _apply_rotary_emb(k, cos, sin)
+        q = F.rms_norm(q, (q.size(-1),))
+        k = F.rms_norm(k, (k.size(-1),))
+        q = self.head_gain(q)
+
+        # SDPA expects [B, H, T, D]
+        y = F.scaled_dot_product_attention(
+            q.transpose(1, 2), k.transpose(1, 2), v.transpose(1, 2), is_causal=True
+        )
+        y = y.transpose(1, 2).contiguous().view(bsz, seq_len, -1)
+        return self.o_proj(y)
+
+
+class TalkieMLP(nn.Module):
+    def __init__(self, config: TalkieConfig):
+        super().__init__()
+        self.gate_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.up_proj = nn.Linear(config.hidden_size, config.intermediate_size, bias=False)
+        self.down_proj = nn.Linear(config.intermediate_size, config.hidden_size, bias=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        return self.down_proj(F.silu(self.gate_proj(x)) * self.up_proj(x))
+
+
+class TalkieDecoderLayer(nn.Module):
+    def __init__(self, config: TalkieConfig):
+        super().__init__()
+        self.self_attn = TalkieAttention(config)
+        self.attn_gain = ActGain((2 * config.num_hidden_layers) ** -0.5)
+        self.mlp = TalkieMLP(config)
+        self.mlp_gain = ActGain((2 * config.num_hidden_layers) ** -0.5)
+        self.embed_skip = ActGain(0.0)
+
+    def forward(
+        self,
+        hidden_states: torch.Tensor,
+        e_x: torch.Tensor = None,
+        cos: torch.Tensor = None,
+        sin: torch.Tensor = None,
+        **kwargs,  # absorb attention_mask / position_ids / etc that HF tooling injects
+    ) -> torch.Tensor:
+        x = hidden_states
+        x = x + self.attn_gain(self.self_attn(F.rms_norm(x, (x.shape[-1],)), cos, sin))
+        x = x + self.mlp_gain(self.mlp(F.rms_norm(x, (x.shape[-1],))))
+        x = x + self.embed_skip(e_x)
+        return x
+
+
+class TalkiePreTrainedModel(PreTrainedModel):
+    config_class = TalkieConfig
+    base_model_prefix = "model"
+    supports_gradient_checkpointing = False
+    _no_split_modules = ["TalkieDecoderLayer"]
+
+    def _init_weights(self, module):
+        if isinstance(module, nn.Linear):
+            module.weight.data.normal_(mean=0.0, std=0.02)
+            if module.bias is not None:
+                module.bias.data.zero_()
+        elif isinstance(module, nn.Embedding):
+            module.weight.data.normal_(mean=0.0, std=0.02)
+
+
+class TalkieModel(TalkiePreTrainedModel):
+    def __init__(self, config: TalkieConfig):
+        super().__init__(config)
+        self.embed_tokens = nn.Embedding(config.vocab_size, config.hidden_size)
+        self.layers = nn.ModuleList(
+            [TalkieDecoderLayer(config) for _ in range(config.num_hidden_layers)]
+        )
+        # cos/sin are computed lazily in forward — see _rope. Avoid register_buffer
+        # so HF's meta-init / low_cpu_mem_usage loading path does not leave us
+        # holding meta tensors that we then try to slice (which raises).
+        self._rope_cache: tuple[torch.Tensor, torch.Tensor, torch.device, torch.dtype, int] | None = None
+        self.post_init()
+
+    @staticmethod
+    def _build_rope(
+        seq_len: int, head_dim: int, base: float, device, dtype
+    ) -> tuple[torch.Tensor, torch.Tensor]:
+        ch = torch.arange(0, head_dim, 2, dtype=torch.float32, device=device)
+        inv_freq = 1.0 / (base ** (ch / head_dim))
+        t = torch.arange(seq_len, dtype=torch.float32, device=device)
+        freqs = torch.outer(t, inv_freq)
+        cos, sin = freqs.cos().to(dtype), freqs.sin().to(dtype)
+        return cos[None, :, None, :], sin[None, :, None, :]
+
+    def _rope(self, seq_len: int, device, dtype) -> tuple[torch.Tensor, torch.Tensor]:
+        cache = self._rope_cache
+        if (cache is None or cache[2] != device or cache[3] != dtype or cache[4] < seq_len):
+            cap = max(seq_len, self.config.max_position_embeddings)
+            cos, sin = self._build_rope(cap, self.config.head_dim, self.config.rope_theta, device, dtype)
+            self._rope_cache = (cos, sin, device, dtype, cap)
+        cos, sin, _, _, _ = self._rope_cache
+        return cos[:, :seq_len], sin[:, :seq_len]
+
+    def forward(self, input_ids: torch.LongTensor, **kwargs) -> torch.Tensor:
+        _, seq_len = input_ids.shape
+        x = self.embed_tokens(input_ids)
+        x = F.rms_norm(x, (x.shape[-1],))
+        e_x = x  # post-RMSNorm input embeddings; reused as embed_skip source at every layer
+        cos, sin = self._rope(seq_len, x.device, x.dtype)
+        for layer in self.layers:
+            # Pass e_x/cos/sin as kwargs so HF tooling (GPTQModel etc) captures
+            # and replays them per-sample when iterating layers individually.
+            x = layer(x, e_x=e_x, cos=cos, sin=sin)
+        x = F.rms_norm(x, (x.shape[-1],))
+        return x
+
+
+class TalkieForCausalLM(TalkiePreTrainedModel, GenerationMixin):
+    _tied_weights_keys = []
+    _supports_cache_class = False
+    _supports_static_cache = False
+    # Talkie has no KV cache implementation — every generate step recomputes
+    # the full sequence. Mirror the reference talkie inference behavior.
+
+    def __init__(self, config: TalkieConfig):
+        super().__init__(config)
+        # Force use_cache=False so HF generate doesn't try to feed only the
+        # last token via past_key_values (which we don't support).
+        config.use_cache = False
+        self.model = TalkieModel(config)
+        self.lm_head = nn.Linear(config.hidden_size, config.vocab_size, bias=False)
+        self.lm_head_gain = WeightGain()
+        self.post_init()
+        # Belt-and-suspenders: also force the generation_config to not cache.
+        if hasattr(self, "generation_config") and self.generation_config is not None:
+            self.generation_config.use_cache = False
+
+    def prepare_inputs_for_generation(self, input_ids, **kwargs):
+        # Strip past_key_values and always feed the full sequence — talkie
+        # has no incremental state.
+        kwargs.pop("past_key_values", None)
+        kwargs.pop("cache_position", None)
+        kwargs["use_cache"] = False
+        return {"input_ids": input_ids, **kwargs}
+
+    def get_input_embeddings(self):
+        return self.model.embed_tokens
+
+    def set_input_embeddings(self, value):
+        self.model.embed_tokens = value
+
+    def get_output_embeddings(self):
+        return self.lm_head
+
+    def set_output_embeddings(self, new_embeddings):
+        self.lm_head = new_embeddings
+
+    def forward(
+        self,
+        input_ids: torch.LongTensor = None,
+        attention_mask: torch.Tensor = None,  # accepted but unused (causal-only)
+        labels: torch.LongTensor = None,
+        **kwargs,
+    ) -> CausalLMOutput:
+        hidden = self.model(input_ids)
+        # WeightGain is scalar-broadcast over the lm_head matrix, so applying
+        # it on the linear's output is mathematically identical to pre-scaling
+        # the weight (and avoids a cross-module tensor passing pattern that
+        # confuses accelerate's device-map hooks).
+        logits = self.lm_head(hidden).float() * self.lm_head_gain.w_g.float()
+        loss = None
+        if labels is not None:
+            shift_logits = logits[..., :-1, :].contiguous()
+            shift_labels = labels[..., 1:].contiguous()
+            loss = F.cross_entropy(
+                shift_logits.view(-1, shift_logits.size(-1)),
+                shift_labels.view(-1),
+            )
+        return CausalLMOutput(loss=loss, logits=logits)

quant_log.csv

@@ -0,0 +1,281 @@
+layer,module,loss,samples,damp,time
+0,mlp.gate_proj,0.0003081795,0.05000,17.913
+0,mlp.up_proj,0.0000542289,0.05000,18.174
+0,mlp.down_proj,0.0019961544,0.05000,21.241
+0,self_attn.k_proj,0.0000270423,0.05000,27.808
+0,self_attn.v_proj,0.0000235055,0.05000,27.895
+0,self_attn.o_proj,0.0000084061,0.05000,27.915
+0,self_attn.q_proj,0.0000267042,0.05000,27.942
+1,mlp.gate_proj,0.0003360091,0.05000,17.789
+1,mlp.up_proj,0.0000520824,0.05000,17.855
+1,mlp.down_proj,0.0012718793,0.05000,21.678
+1,self_attn.k_proj,0.0000274328,0.05000,28.150
+1,self_attn.q_proj,0.0000271052,0.05000,28.208
+1,self_attn.v_proj,0.0000236766,0.05000,28.278
+1,self_attn.o_proj,0.0000074347,0.05000,28.287
+2,mlp.up_proj,0.0000493462,0.05000,16.887
+2,mlp.gate_proj,0.0003627762,0.05000,17.242
+2,mlp.down_proj,0.0006479784,0.05000,20.663
+2,self_attn.v_proj,0.0000240699,0.05000,27.454
+2,self_attn.k_proj,0.0000277159,0.05000,27.778
+2,self_attn.q_proj,0.0000266389,0.05000,27.838
+2,self_attn.o_proj,0.0000097000,0.05000,27.889
+3,mlp.up_proj,0.0000229121,0.05000,17.016
+3,mlp.gate_proj,0.0000504172,0.05000,17.101
+3,mlp.down_proj,0.0000309956,0.05000,20.434
+3,self_attn.v_proj,0.0000148267,0.05000,28.005
+3,self_attn.o_proj,0.0000113674,0.05000,28.217
+3,self_attn.k_proj,0.0000155096,0.05000,28.254
+3,self_attn.q_proj,0.0000150627,0.05000,28.260
+4,mlp.up_proj,0.0000312758,0.05000,16.093
+4,mlp.gate_proj,0.0000512521,0.05000,17.032
+4,mlp.down_proj,0.0000542724,0.05000,20.156
+4,self_attn.v_proj,0.0000170589,0.05000,27.938
+4,self_attn.q_proj,0.0000153016,0.05000,28.085
+4,self_attn.k_proj,0.0000155928,0.05000,28.107
+4,self_attn.o_proj,0.0000104716,0.05000,28.149
+5,mlp.gate_proj,0.0000609030,0.05000,16.785
+5,mlp.up_proj,0.0000393444,0.05000,16.951
+5,mlp.down_proj,0.0000239930,0.05000,20.505
+5,self_attn.v_proj,0.0000176692,0.05000,28.079
+5,self_attn.o_proj,0.0000058350,0.05000,28.123
+5,self_attn.q_proj,0.0000156301,0.05000,28.158
+5,self_attn.k_proj,0.0000160085,0.05000,28.205
+6,mlp.gate_proj,0.0000515375,0.05000,17.278
+6,mlp.up_proj,0.0000411530,0.05000,17.776
+6,mlp.down_proj,0.0000603061,0.05000,20.828
+6,self_attn.o_proj,0.0000069052,0.05000,27.807
+6,self_attn.v_proj,0.0000171747,0.05000,28.234
+6,self_attn.k_proj,0.0000153144,0.05000,28.444
+6,self_attn.q_proj,0.0000151436,0.05000,28.471
+7,mlp.gate_proj,0.0000500004,0.05000,17.776
+7,mlp.up_proj,0.0000463201,0.05000,17.916
+7,mlp.down_proj,0.0000528303,0.05000,21.097
+7,self_attn.o_proj,0.0000056215,0.05000,28.227
+7,self_attn.q_proj,0.0000153945,0.05000,28.349
+7,self_attn.v_proj,0.0000174461,0.05000,28.433
+7,self_attn.k_proj,0.0000159118,0.05000,28.465
+8,mlp.gate_proj,0.0000503716,0.05000,17.055
+8,mlp.up_proj,0.0000486011,0.05000,17.603
+8,mlp.down_proj,0.0000455072,0.05000,21.122
+8,self_attn.k_proj,0.0000179451,0.05000,28.025
+8,self_attn.v_proj,0.0000188796,0.05000,28.335
+8,self_attn.q_proj,0.0000175658,0.05000,28.493
+8,self_attn.o_proj,0.0000047503,0.05000,28.501
+9,mlp.up_proj,0.0000556839,0.05000,17.929
+9,mlp.gate_proj,0.0000614202,0.05000,18.665
+9,mlp.down_proj,0.0000311427,0.05000,21.902
+9,self_attn.o_proj,0.0000064314,0.05000,27.886
+9,self_attn.q_proj,0.0000206849,0.05000,28.458
+9,self_attn.k_proj,0.0000209098,0.05000,28.482
+9,self_attn.v_proj,0.0000219568,0.05000,28.514
+10,mlp.gate_proj,0.0000767580,0.05000,16.931
+10,mlp.up_proj,0.0000679417,0.05000,17.664
+10,mlp.down_proj,0.0000403810,0.05000,20.917
+10,self_attn.k_proj,0.0000250125,0.05000,27.692
+10,self_attn.v_proj,0.0000258458,0.05000,28.023
+10,self_attn.o_proj,0.0000067075,0.05000,28.249
+10,self_attn.q_proj,0.0000249492,0.05000,28.268
+11,mlp.gate_proj,0.0000920226,0.05000,17.943
+11,mlp.up_proj,0.0000770476,0.05000,18.249
+11,mlp.down_proj,0.0000358957,0.05000,21.946
+11,self_attn.k_proj,0.0000284748,0.05000,27.930
+11,self_attn.q_proj,0.0000282358,0.05000,28.008
+11,self_attn.o_proj,0.0000082858,0.05000,28.214
+11,self_attn.v_proj,0.0000290026,0.05000,28.240
+12,mlp.up_proj,0.0000864709,0.05000,17.018
+12,mlp.gate_proj,0.0001036938,0.05000,17.838
+12,mlp.down_proj,0.0000380611,0.05000,21.033
+12,self_attn.q_proj,0.0000315321,0.05000,28.575
+12,self_attn.v_proj,0.0000327975,0.05000,28.766
+12,self_attn.k_proj,0.0000316397,0.05000,28.826
+12,self_attn.o_proj,0.0000083556,0.05000,28.853
+13,mlp.up_proj,0.0000923181,0.05000,17.830
+13,mlp.gate_proj,0.0001100664,0.05000,17.955
+13,mlp.down_proj,0.0000405574,0.05000,21.390
+13,self_attn.k_proj,0.0000350662,0.05000,28.578
+13,self_attn.v_proj,0.0000353049,0.05000,28.654
+13,self_attn.o_proj,0.0000075047,0.05000,28.673
+13,self_attn.q_proj,0.0000347006,0.05000,28.691
+14,mlp.gate_proj,0.0001040181,0.05000,18.228
+14,mlp.up_proj,0.0000936042,0.05000,18.411
+14,mlp.down_proj,0.0355678611,0.05000,22.067
+14,self_attn.q_proj,0.0000348256,0.05000,28.670
+14,self_attn.k_proj,0.0000353750,0.05000,28.753
+14,self_attn.v_proj,0.0000369457,0.05000,28.777
+14,self_attn.o_proj,0.0000068864,0.05000,28.853
+15,mlp.up_proj,0.0000964519,0.05000,17.509
+15,mlp.gate_proj,0.0001058904,0.05000,17.618
+15,mlp.down_proj,0.0000436096,0.05000,21.055
+15,self_attn.q_proj,0.0000349336,0.05000,28.645
+15,self_attn.o_proj,0.0000072430,0.05000,28.764
+15,self_attn.k_proj,0.0000358868,0.05000,28.789
+15,self_attn.v_proj,0.0000370624,0.05000,28.815
+16,mlp.up_proj,0.0001003270,0.05000,17.369
+16,mlp.gate_proj,0.0001031801,0.05000,17.872
+16,mlp.down_proj,0.0000429029,0.05000,21.587
+16,self_attn.k_proj,0.0000367004,0.05000,28.719
+16,self_attn.o_proj,0.0000060289,0.05000,28.823
+16,self_attn.v_proj,0.0000392006,0.05000,28.886
+16,self_attn.q_proj,0.0000359511,0.05000,28.907
+17,mlp.gate_proj,0.0001028387,0.05000,18.211
+17,mlp.up_proj,0.0001017818,0.05000,18.231
+17,mlp.down_proj,0.0000387011,0.05000,21.611
+17,self_attn.k_proj,0.0000379290,0.05000,28.602
+17,self_attn.v_proj,0.0000402245,0.05000,29.007
+17,self_attn.q_proj,0.0000368098,0.05000,29.019
+17,self_attn.o_proj,0.0000049005,0.05000,29.060
+18,mlp.gate_proj,0.0001001292,0.05000,17.687
+18,mlp.up_proj,0.0000996311,0.05000,17.958
+18,mlp.down_proj,0.0000380027,0.05000,21.696
+18,self_attn.k_proj,0.0000374850,0.05000,28.734
+18,self_attn.v_proj,0.0000388674,0.05000,28.870
+18,self_attn.o_proj,0.0000077347,0.05000,29.026
+18,self_attn.q_proj,0.0000370627,0.05000,29.042
+19,mlp.gate_proj,0.0000938521,0.05000,17.762
+19,mlp.up_proj,0.0000959272,0.05000,18.039
+19,mlp.down_proj,0.0000342812,0.05000,21.534
+19,self_attn.o_proj,0.0000068072,0.05000,28.265
+19,self_attn.q_proj,0.0000348771,0.05000,28.741
+19,self_attn.k_proj,0.0000353500,0.05000,28.802
+19,self_attn.v_proj,0.0000368596,0.05000,28.824
+20,mlp.gate_proj,0.0000895623,0.05000,17.897
+20,mlp.up_proj,0.0000947136,0.05000,18.148
+20,mlp.down_proj,0.0000322166,0.05000,21.563
+20,self_attn.k_proj,0.0000337982,0.05000,28.953
+20,self_attn.v_proj,0.0000363387,0.05000,29.284
+20,self_attn.q_proj,0.0000332785,0.05000,29.303
+20,self_attn.o_proj,0.0000063673,0.05000,29.339
+21,mlp.gate_proj,0.0000896193,0.05000,17.650
+21,mlp.up_proj,0.0000960996,0.05000,18.170
+21,mlp.down_proj,0.0000317455,0.05000,21.673
+21,self_attn.k_proj,0.0000347393,0.05000,29.186
+21,self_attn.o_proj,0.0000045270,0.05000,29.212
+21,self_attn.v_proj,0.0000374702,0.05000,29.263
+21,self_attn.q_proj,0.0000340738,0.05000,29.362
+22,mlp.up_proj,0.0000987218,0.05000,17.802
+22,mlp.gate_proj,0.0000916783,0.05000,17.819
+22,mlp.down_proj,0.0000354047,0.05000,21.465
+22,self_attn.v_proj,0.0000384820,0.05000,29.241
+22,self_attn.k_proj,0.0000358335,0.05000,29.278
+22,self_attn.o_proj,0.0000055582,0.05000,29.375
+22,self_attn.q_proj,0.0000351940,0.05000,29.393
+23,mlp.gate_proj,0.0000988934,0.05000,18.062
+23,mlp.up_proj,0.0001052049,0.05000,18.496
+23,mlp.down_proj,0.0000408503,0.05000,21.829
+23,self_attn.v_proj,0.0000397907,0.05000,29.463
+23,self_attn.k_proj,0.0000372553,0.05000,29.484
+23,self_attn.q_proj,0.0000363780,0.05000,29.582
+23,self_attn.o_proj,0.0000048959,0.05000,29.589
+24,mlp.gate_proj,0.0001048375,0.05000,18.129
+24,mlp.up_proj,0.0001094447,0.05000,18.787
+24,mlp.down_proj,0.0000439513,0.05000,22.425
+24,self_attn.v_proj,0.0000426328,0.05000,29.455
+24,self_attn.k_proj,0.0000398516,0.05000,29.500
+24,self_attn.o_proj,0.0000068015,0.05000,29.571
+24,self_attn.q_proj,0.0000387764,0.05000,29.642
+25,mlp.gate_proj,0.0001101101,0.05000,18.471
+25,mlp.up_proj,0.0001150894,0.05000,18.507
+25,mlp.down_proj,0.0000502667,0.05000,21.972
+25,self_attn.k_proj,0.0000414529,0.05000,29.407
+25,self_attn.q_proj,0.0000402841,0.05000,29.522
+25,self_attn.v_proj,0.0000460898,0.05000,29.677
+25,self_attn.o_proj,0.0000049322,0.05000,29.679
+26,mlp.up_proj,0.0001176010,0.05000,18.294
+26,mlp.gate_proj,0.0001157366,0.05000,18.715
+26,mlp.down_proj,0.0000451706,0.05000,22.165
+26,self_attn.v_proj,0.0000473667,0.05000,29.338
+26,self_attn.q_proj,0.0000415147,0.05000,29.424
+26,self_attn.k_proj,0.0000431917,0.05000,29.524
+26,self_attn.o_proj,0.0000047934,0.05000,29.531
+27,mlp.up_proj,0.0001159971,0.05000,18.210
+27,mlp.gate_proj,0.0001153119,0.05000,18.384
+27,mlp.down_proj,0.0000445333,0.05000,21.814
+27,self_attn.q_proj,0.0000421407,0.05000,29.062
+27,self_attn.k_proj,0.0000436215,0.05000,29.213
+27,self_attn.v_proj,0.0000475841,0.05000,29.256
+27,self_attn.o_proj,0.0000046830,0.05000,29.273
+28,mlp.gate_proj,0.0001169334,0.05000,17.645
+28,mlp.up_proj,0.0001143606,0.05000,18.008
+28,mlp.down_proj,0.0000337466,0.05000,21.847
+28,self_attn.o_proj,0.0000032019,0.05000,27.434
+28,self_attn.k_proj,0.0000432929,0.05000,27.719
+28,self_attn.v_proj,0.0000462728,0.05000,28.007
+28,self_attn.q_proj,0.0000419060,0.05000,28.151
+29,mlp.up_proj,0.0001137857,0.05000,18.228
+29,mlp.gate_proj,0.0001192500,0.05000,18.240
+29,mlp.down_proj,0.0000252381,0.05000,21.628
+29,self_attn.o_proj,0.0000022613,0.05000,28.345
+29,self_attn.v_proj,0.0000449540,0.05000,28.618
+29,self_attn.q_proj,0.0000412867,0.05000,28.819
+29,self_attn.k_proj,0.0000426163,0.05000,28.846
+30,mlp.up_proj,0.0001078411,0.05000,17.656
+30,mlp.gate_proj,0.0001135521,0.05000,17.932
+30,mlp.down_proj,0.0000231547,0.05000,21.266
+30,self_attn.k_proj,0.0000411614,0.05000,28.448
+30,self_attn.o_proj,0.0000030266,0.05000,28.505
+30,self_attn.v_proj,0.0000445966,0.05000,28.653
+30,self_attn.q_proj,0.0000403358,0.05000,28.710
+31,mlp.up_proj,0.0001045471,0.05000,17.233
+31,mlp.gate_proj,0.0001093158,0.05000,17.635
+31,mlp.down_proj,0.0000235892,0.05000,21.389
+31,self_attn.k_proj,0.0000396851,0.05000,29.104
+31,self_attn.o_proj,0.0000026403,0.05000,29.284
+31,self_attn.q_proj,0.0000388012,0.05000,29.338
+31,self_attn.v_proj,0.0000428868,0.05000,29.353
+32,mlp.gate_proj,0.0001087208,0.05000,17.920
+32,mlp.up_proj,0.0001044456,0.05000,18.058
+32,mlp.down_proj,0.0000240254,0.05000,21.339
+32,self_attn.q_proj,0.0000378783,0.05000,28.779
+32,self_attn.v_proj,0.0000425838,0.05000,28.862
+32,self_attn.k_proj,0.0000395236,0.05000,29.008
+32,self_attn.o_proj,0.0000025328,0.05000,29.019
+33,mlp.gate_proj,0.0001147145,0.05000,17.565
+33,mlp.up_proj,0.0001067313,0.05000,17.897
+33,mlp.down_proj,0.0000227866,0.05000,21.379
+33,self_attn.q_proj,0.0000378068,0.05000,29.419
+33,self_attn.o_proj,0.0000020157,0.05000,29.626
+33,self_attn.v_proj,0.0000429422,0.05000,29.716
+33,self_attn.k_proj,0.0000400440,0.05000,29.738
+34,mlp.up_proj,0.0000997575,0.05000,17.211
+34,mlp.gate_proj,0.0001061515,0.05000,18.252
+34,mlp.down_proj,0.0000202997,0.05000,21.364
+34,self_attn.v_proj,0.0000405380,0.05000,27.850
+34,self_attn.q_proj,0.0000365120,0.05000,28.540
+34,self_attn.o_proj,0.0000024578,0.05000,28.644
+34,self_attn.k_proj,0.0000380477,0.05000,28.668
+35,mlp.gate_proj,0.0000984036,0.05000,18.177
+35,mlp.up_proj,0.0000936725,0.05000,18.248
+35,mlp.down_proj,0.0000177211,0.05000,21.932
+35,self_attn.q_proj,0.0000328877,0.05000,27.925
+35,self_attn.v_proj,0.0000387808,0.05000,28.576
+35,self_attn.k_proj,0.0000347784,0.05000,28.628
+35,self_attn.o_proj,0.0000015525,0.05000,28.713
+36,mlp.gate_proj,0.0000940234,0.05000,18.255
+36,mlp.up_proj,0.0000899822,0.05000,18.335
+36,mlp.down_proj,0.0000180931,0.05000,21.839
+36,self_attn.k_proj,0.0000334674,0.05000,29.467
+36,self_attn.q_proj,0.0000326385,0.05000,29.651
+36,self_attn.o_proj,0.0000020520,0.05000,29.665
+36,self_attn.v_proj,0.0000368466,0.05000,29.685
+37,mlp.gate_proj,0.0000945316,0.05000,19.052
+37,mlp.up_proj,0.0000905586,0.05000,19.627
+37,mlp.down_proj,0.0000239953,0.05000,23.262
+37,self_attn.k_proj,0.0000336605,0.05000,29.512
+37,self_attn.o_proj,0.0000015146,0.05000,29.655
+37,self_attn.q_proj,0.0000327701,0.05000,29.723
+37,self_attn.v_proj,0.0000377369,0.05000,29.729
+38,mlp.up_proj,0.0000923737,0.05000,24.743
+38,mlp.gate_proj,0.0000938228,0.05000,24.822
+38,mlp.down_proj,0.0000328694,0.05000,28.743
+38,self_attn.k_proj,0.0000334626,0.05000,29.978
+38,self_attn.v_proj,0.0000377836,0.05000,30.020
+38,self_attn.q_proj,0.0000331712,0.05000,30.023
+38,self_attn.o_proj,0.0000035293,0.05000,30.060
+39,mlp.gate_proj,0.0000792578,0.05000,24.641
+39,mlp.up_proj,0.0000821907,0.05000,24.913
+39,mlp.down_proj,0.0000582872,0.05000,28.858
+39,self_attn.v_proj,0.0000294953,0.05000,29.453
+39,self_attn.o_proj,0.0000085167,0.05000,29.570
+39,self_attn.q_proj,0.0000305774,0.05000,29.662
+39,self_attn.k_proj,0.0000303001,0.05000,29.671

quantize_config.json

@@ -0,0 +1,44 @@
+{
+  "bits": 4,
+  "group_size": 128,
+  "desc_act": false,
+  "lm_head": false,
+  "method": "gptq",
+  "quant_method": "gptq",
+  "format": "gptq",
+  "checkpoint_format": "gptq",
+  "pack_dtype": "int32",
+  "meta": {
+    "quantizer": [
+      "gptqmodel:6.0.3"
+    ],
+    "uri": "https://github.com/modelcloud/gptqmodel",
+    "damp_percent": 0.05,
+    "damp_auto_increment": 0.01,
+    "static_groups": false,
+    "true_sequential": true,
+    "mse": 0.0,
+    "gptaq": null,
+    "foem": null,
+    "act_group_aware": true,
+    "fallback": {
+      "strategy": "rtn",
+      "threshold": "0.5%",
+      "smooth": null
+    },
+    "offload_to_disk": true,
+    "offload_to_disk_path": "./gptqmodel_offload/ctrizrxh-tetrojst/",
+    "pack_impl": "cpu",
+    "gc_mode": "interval",
+    "wait_for_submodule_finalizers": false,
+    "auto_forward_data_parallel": true,
+    "vram_strategy": "exclusive",
+    "mock_quantization": false,
+    "hessian": {
+      "chunk_size": null,
+      "chunk_bytes": null,
+      "staging_dtype": "float32"
+    }
+  },
+  "sym": true
+}

talkie_qmodel.py

@@ -0,0 +1,42 @@
+"""GPTQModel adapter for the Talkie architecture.
+
+Importing this module registers TalkieQModel under model_type='talkie' in
+GPTQModel's MODEL_MAP, so `GPTQModel.load(...)` and `GPTQModel.from_quantized(...)`
+work without manual configuration.
+
+Auto-detect produces the same module_tree, so this is purely for the from_quantized
+path (which doesn't run auto-detect — module_tree must be a class attribute).
+"""
+
+from __future__ import annotations
+
+from gptqmodel.models.base import BaseQModel
+from gptqmodel.models.auto import MODEL_MAP, SUPPORTED_MODELS
+
+
+class TalkieQModel(BaseQModel):
+    # talkie uses functional F.rms_norm with no learnable scale, so there's no
+    # named pre-lm-head normalization module. Empty string disables that hook.
+    pre_lm_head_norm_module = ""
+
+    # Module tree maps GPTQModel's iteration onto our TalkieDecoderLayer:
+    #   model.layers.{i}.self_attn.{q_proj,k_proj,v_proj,o_proj}
+    #   model.layers.{i}.mlp.{gate_proj,up_proj,down_proj}
+    # Suffix :0/:1 declares quantization grouping order — q/k/v share input
+    # (the post-attn-rmsnorm hidden state), o has a different input (SDPA output).
+    # Same for gate/up sharing input vs down. Mirrors LlamaQModel.
+    module_tree = [
+        "model",
+        "layers",
+        "#",
+        {
+            "self_attn": ("q_proj:0", "k_proj:0", "v_proj:0", "o_proj:1"),
+            "mlp": ("gate_proj:0", "up_proj:0", "down_proj:1"),
+        },
+    ]
+
+
+# Register under model_type='talkie' so GPTQModel.load auto-routes to us.
+MODEL_MAP["talkie"] = TalkieQModel
+if "talkie" not in SUPPORTED_MODELS:
+    SUPPORTED_MODELS.append("talkie")

tokenizer.json

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:250b431ae085034dccfb9a4bf13db6bd1f8c375bb7ed499efd57e7c94ca64a3a
+size 43187818

tokenizer_config.json

@@ -0,0 +1,13 @@
+{
+  "backend": "tokenizers",
+  "bos_token": null,
+  "clean_up_tokenization_spaces": false,
+  "eos_token": "<|endoftext|>",
+  "is_local": true,
+  "local_files_only": false,
+  "model_max_length": 1000000000000000019884624838656,
+  "pad_token": "<|endoftext|>",
+  "tokenizer_class": "TokenizersBackendFast",
+  "unk_token": null,
+  "_commit_hash": null
+}