Gensyn/open-1b-sft-mlx-bf16

open-1b-sft (MLX, bf16)

open-1b is a 1.6 billion parameter decoder-only language model pretrained on 400B tokens. It is the first language model whose training can be independently verified: it is released with its complete pretraining dataset, training and evaluation code, intermediate checkpoints at 100-step intervals, and a canonical state hash for every one of the 80,957 optimizer steps that produced it.

This repository is an MLX conversion of Gensyn/open-1b-sft — the supervised fine-tune (SFT) of the family — for Apple Silicon, runnable with mlx-lm and LM Studio with no custom code. The verifiable artefact is the reference checkpoint; this conversion is for convenient local inference and is not itself covered by the published state hashes.

The open-1b family

Model	What it is
Gensyn/open-1b-base	Pretraining checkpoint at step 80,957, 400B tokens
Gensyn/open-1b-midtrained-93B	Midtrained checkpoint: continued pretraining of the base model on 93B additional tokens
Gensyn/open-1b-sft	Supervised fine-tune on allenai/tulu-3-sft-olmo-2-mixture-0225, the chat-capable model
Gensyn/open-1b-sft-mlx-bf16	This repo: open-1b-sft converted to MLX (bf16)

Model sources

• Verification record: https://open1b.gensyn.ai
• Tech report: open1b.gensyn.ai/open1b-tech-report.pdf
• Training code, configs and RepOps: https://github.com/gensyn-ai/open-transformers
• Audit tool: https://github.com/gensyn-ai/pretraining-audit-cli

How to use

With mlx-lm:

pip install mlx-lm
python -m mlx_lm.chat --model Gensyn/open-1b-sft-mlx-bf16

from mlx_lm import load, generate

model, tokenizer = load("Gensyn/open-1b-sft-mlx-bf16")
messages = [{"role": "user", "content": "Explain verifiable training in one paragraph."}]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True, tokenize=False)
print(generate(model, tokenizer, prompt, max_tokens=256))

In LM Studio, search for Gensyn/open-1b-sft-mlx-bf16, download it and load it. LM Studio samples at temperature > 0 by default and ignores generation_config.json; we recommend setting Temperature to 0 in the model's settings (see Limitations). mlx-lm already decodes greedily by default.

Conversion details

Weights. These are the bf16 master weights of open-1b-sft. The model was trained with int8 W8A8 quantization-aware training (LSQ); the reference transformers port emulates that int8 grid by default, but native MLX has no equivalent, so this conversion drops the learned weight_scale tensors and runs plain bf16 GEMMs — equivalent to the reference with quantized_forward=False. Greedy outputs match that reference token for token on our probes. Expect small differences from the reference's default int8-emulated forward.

Architecture as a built-in mlx-lm type. mlx-lm has no open1b architecture, and apps like LM Studio refuse checkpoints that need custom model code. The weights are therefore re-expressed as mlx-lm's built-in step3p5 architecture — which is why config.json says "model_type": "step3p5". This is still open-1b, not a Step model. Every rewrite is exact:

open-1b feature	expressed as
embedding RMSNorm	baked into the embedding table (it depends only on the token)
interleaved-pair RoPE	q_proj/k_proj rows permuted within each head → half-split RoPE (exact, because the QK-norm before RoPE has no gain)
gain-free per-head QK-norm	q_norm / k_norm with unit weights
fused gate/up projection	separate gate_proj / up_proj
dense SwiGLU MLP	moe_layers_enum: "-1" (no MoE layers), no attention gate

The one approximation: open-1b's sliding-window layers (19 of 24) use a window aligned to 32-token blocks (each query sees 481–512 keys), while mlx-lm uses a plain 512-key window. Outputs are identical up to 481 tokens of context and near-identical beyond it (per-token loss on a 1.5k-token document unchanged to three decimals).

Tokenizer. Identical to the reference except for one removed BPE merge ("Ċ" + "Ċ" → "ĊĊ", id 408). A \n\n at the very end of an encoded string (exactly where the chat template's generation prompt ends) otherwise becomes a token the model never saw there, producing empty replies. The reference repo fixes this with custom tokenizer code; removing the merge fixes it for any standard tokenizer consumer. Text where \n\n is followed by content tokenizes exactly as in training.

Conversion records: post-conversion-fixes.json (what changed here, with hashes) and provenance/ (the upstream converter, its hash-pinned manifest and metadata history).

Chat format

<|start_header|>{{role}}<|end_header|>

{{content}}<|eot|>

Special tokens <|start_header|> (id 2), <|end_header|> (id 3) and <|eot|> (id 4, the eos). No BOS prefix. The generation prompt ends in <|start_header|>assistant<|end_header|>\n\n, exactly as in training.

Limitations

This is an SFT-only model (no RLHF or preference tuning) at 1.6B parameters: a helpful but lightly-tuned research assistant, not a production chat model. It has a learned tendency to hedge on simple factual questions ("As of my knowledge cutoff…", "your question is underspecified…"). The tendency is stronger under sampling and on casually phrased prompts. For example, What is the capital of France? gets a plain answer, while what is the capital of france hedges even with greedy decoding. This is a property of the trained model: the reference transformers checkpoint behaves the same way. It is not caused by the conversion.

License

Apache 2.0 — see LICENSE.