joseluissaorin/talkie-1930-13b-it-mlx-q8

talkie-1930-13b-it MLX (8-bit quantized)

This is a 8-bit MLX-quantized port of talkie-lm/talkie-1930-13b-it — a 13B language model trained on pre-1931 English text — for use on Apple Silicon (M1/M2/M3/M4) via MLX.

• ~13.4 GB on disk, fits comfortably in 24 GB of unified memory.
• ~20 tok/s decode on an M4 Max.
• The original CUDA-only package needs ≥28 GB of VRAM. This makes it run on any modern Mac.

Usage

Use the talkie-mlx runtime:

git clone https://github.com/joseluissaorin/talkie-mlx.git
cd talkie-mlx
pip install -r requirements.txt

hf download joseluissaorin/talkie-1930-13b-it-mlx-q8 \
  --local-dir ~/models/talkie-1930-13b-it-mlx-q8

python run.py generate \
  --model ~/models/talkie-1930-13b-it-mlx-q8 \
  --prompt "Write a brief letter from London, dated July 1925, describing the weather and news of the day." \
  --max-tokens 200 --temperature 0.7 --top-p 0.9

Sample output:

Dear Friend, It is raining to-day, and the roads are heavy. The weather is very unsettled. The news of the day is the report that the King is ill, and that his life is in danger. Believe me, Yours sincerely, JOHN BROWN.

For interactive chat:

python run.py chat --model ~/models/talkie-1930-13b-it-mlx-q8

Quantization details

• 8-bit weight-only quantization via mx.quantize, group size 64.
• Quantized: every nn.Linear (attn_query, attn_key, attn_value, attn_resid, mlp_gate, mlp_linear, mlp_resid) and the token embedding.
• Not quantized: lm_head (kept in bf16, ~700 MB), per-head/-layer scalar gains, and the lm_head_gain scalar.
• 7 sharded safetensors files plus model.safetensors.index.json.

Files

config.json                        # architecture + quantization params
vocab.txt                          # tiktoken BPE merges
model.safetensors.index.json       # shard map
model-00001-of-00007.safetensors   # ~2.4 GB (embedding + first blocks)
model-0000{2..6}-of-00007.safetensors   # ~2.1 GB each
model-00007-of-00007.safetensors   # ~0.5 GB

Model architecture

40-layer decoder-only transformer:

• 40 heads, head_dim 128 (MHA, no GQA).
• RoPE with base 1_000_000, NeoX split-half rotation (with the upstream's specific sign convention).
• Per-head QK rms-norm after RoPE, learned per-head gain on q.
• SwiGLU MLP, n_mlp = 13696.
• Embedding skip: post-norm token embedding added to every block via a learned per-layer gain.
• Per-block attn_gain / mlp_gain (init (2·n_layer)^−0.5), global lm_head_gain.
• Vocab 65540 (base vocab 65536 + 4 IT special tokens), context 2048.

Limitations

• Pre-1931 only: the model has no knowledge of anything after that date and will give incorrect or anachronistic answers about modern events.
• 2048 context (upstream limit).
• 4-bit quality: good for completion-style tasks; for tricky reasoning prefer the 4-bit version or run bf16 on a Mac with ≥48 GB.

Credits and license

All credit for the model and the underlying research goes to talkie-lm. This is an unofficial port.

Licensed under Apache License 2.0, matching upstream. The original talkie-lm/talkie-1930-13b-it is also Apache 2.0.