TernaryBananaMind-10M

TernaryBananaMind-10M is a compact decoder-only base language model from BananaMind. Its inference checkpoint stores projection and output weights as packed ternary values and input embeddings as signed 8-bit values. The intended model ID is BananaMind/TernaryBananaMind-10M.

The model has 8,428,032 logical weights, a 4,096-token context window, and a custom 2,048-token digit-aware byte-level BPE tokenizer. Its model.safetensors file is 2,175,676 bytes (about 2.07 bits per logical weight, including scales, float normalization weights, and file overhead). The model name is a size class; the exact logical weight count is 8.43M.

Model Details

Field	Value
Architecture	BananaMind 2 style decoder-only Transformer
Logical weights	8,428,032
Layers	10
Hidden size	256
Intermediate size	704
Attention heads	4
KV heads	2
Head dimension	64
Attention	Grouped-query attention with QK norm
MLP	SwiGLU
Position embeddings	RoPE, theta 100,000
Normalization	RMSNorm, epsilon 1e-6
Vocabulary size	2,048
Context length	4,096
Input embeddings	Separate from output head; signed 8-bit with one scale per row
Projection and output weights	Ternary (-1, 0, +1), five values per byte
Activations in linear layers	Quantized to 8 bits during inference
Checkpoint	model.safetensors, 2,175,676 bytes
HF architecture	BananaAllForCausalLM
HF model type	bananaall

We have trained TernaryBananaMind-10M on our BananaAll training framework. See it at https://github.com/BananaMind/BananaAll/ to train your own model simply.

The packed weights are decoded to temporary tensors for matrix multiplication. 2.07 bits per weight describes checkpoint storage, not arithmetic precision or peak inference memory. Packed weights are registered as buffers, so sum(p.numel() for p in model.parameters()) reports only the 6,656 trainable normalization parameters. Use the logical weight count above for model-size comparisons.

Tokenizer

The custom 2k byte-level BPE tokenizer isolates digits during pre-tokenization.

Special token	ID
<|pad|>	0
<|bos|>	1
<|eos|>	2
<|unk|>	3

Training

The model was trained on 2 billion tokens of FineWeb-Edu with the BananaAll training framework. It was trained on a RTX Pro 6000 by Molab and the script was exported as a notebook. The included training_args.bin records the settings below. Its maximum-step value is a configured limit, not a separately verified final step count.

Setting	Recorded value
Maximum optimizer steps	61,036
Per-device micro batch	8 sequences
Gradient accumulation	2
Optimizer	PyTorch fused AdamW
Betas	0.9, 0.999
Peak learning rate	0.0018
Learning-rate schedule	Cosine
Warmup steps	1,831
Weight decay	0.01
Gradient clipping	1.0
BF16 training	Enabled
PyTorch compile	Enabled
Seed	37

Training took 2 hours.

Evaluation

The lm_eval and BananaMind Base Bench 1.1 scores below were supplied for the current packed checkpoint. Harness version, dtype, and runtime settings can affect results.

These scores have been evaluated via our BananaAll framework.

Standard benchmarks (lm_eval)

Benchmark	Acc	Acc norm	Samples
PIQA	54.46%	53.54%	1,838
ARC Easy	30.98%	31.57%	2,376
ARC Challenge	18.09%	21.08%	1,172
HellaSwag	26.94%	27.76%	10,042
Four-task mean	—	33.49%	—

BananaMind Base Bench 1.1

The reported official complete run scored 37.14% accuracy (130/350), 35.95% weighted accuracy, and 904 overall Elo.

Category	Elo	Accuracy	Weighted accuracy
Language completion	963	58.00%	58.28%
Commonsense	828	32.00%	33.12%
World knowledge	869	36.00%	38.24%
Context tracking	832	32.00%	28.34%
Quantitative	785	20.00%	18.30%
Logical reasoning	1,043	46.00%	42.32%
Code completion	1,000	36.00%	37.26%

ArithMark 3.0

The official benchmark script was run against this local packed checkpoint on all 1,000 examples with its default batch size and context limit, using CUDA and BF16. The model loaded without missing or unexpected checkpoint keys.

Metric	Score
Raw continuation accuracy	31.10% (311/1,000)
Length-normalized accuracy	31.20% (312/1,000)

Dataset SHA-256: bf8ab1a5193d52cdf0e05ff0b3ca226bdfcf416cb6e75562dcbe72e7e4559435.

Intelligence Index

The Intelligence Index is 4.90. Each component is adjusted for its chance floor with N(score, chance) = 100 × (score - chance) / (100 - chance). The calculation uses the length-normalized scores above. Combined ARC is the mean of ARC Easy and ARC Challenge before chance adjustment.

Component	Score used	Chance floor	Adjusted score	Weight
HellaSwag	27.76%	25%	3.68	1.00
Combined ARC	26.325%	25%	1.77	1.00
PIQA	53.54%	50%	7.08	1.00
ArithMark 3.0	31.20%	25%	8.27	0.65

(3.68 + 1.7667 + 7.08 + 0.65 × 8.2667) / 3.65 = 4.9041, rounded to 4.90.

Comparison with BananaMind-2-Nano

Property	TernaryBananaMind-10M	BananaMind-2-Nano
Logical weights	8,428,032	9,968,128
Vocabulary size	2,048	8,192
Intermediate size	704	768
Input/output embeddings	Separate; input is 8-bit	Tied
Training tokens seen	2B FineWeb-Edu	30B across four datasets

Shared lm_eval benchmark	TernaryBananaMind-10M acc norm	BananaMind-2-Nano acc norm	Difference
PIQA	53.54%	55.98%	-2.44 points
ARC Easy	31.57%	36.20%	-4.63 points
ARC Challenge	21.08%	23.38%	-2.30 points
HellaSwag	27.76%	27.50%	+0.26 points
Four-task mean	33.49%	35.77%	-2.28 points

The two models differ in vocabulary, MLP width, training token count, and weight format, so these scores are a model-level comparison rather than an isolated measure of quantization.

Additional metric	TernaryBananaMind-10M	BananaMind-2-Nano	Difference
Intelligence Index	4.90	8.01	-3.11
ArithMark 3.0 acc norm	31.20%	33.70%	-2.50 points
BananaMind Base Bench 1.1 Elo	904	917	-13
BananaMind Base Bench 1.1 accuracy	37.14% (130/350)	40.86% (143/350)	-3.72 points
BananaMind Base Bench 1.1 weighted accuracy	35.95%	37.52%	-1.57 points

BananaMind Base Bench 1.1 category comparison

Category	Ternary Elo	Nano Elo	Ternary accuracy	Nano accuracy	Ternary weighted	Nano weighted
Language completion	963	1,147	58.00%	80.00%	58.28%	80.25%
Commonsense	828	865	32.00%	38.00%	33.12%	37.85%
World knowledge	869	895	36.00%	46.00%	38.24%	41.69%
Context tracking	832	804	32.00%	26.00%	28.34%	25.16%
Quantitative	785	881	20.00%	32.00%	18.30%	28.08%
Logical reasoning	1,043	1,043	46.00%	46.00%	42.32%	42.32%
Code completion	1,000	827	36.00%	18.00%	37.26%	18.15%

Usage

This model uses custom architecture code. Load it with trust_remote_code=True.

pip install -U transformers safetensors torch

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "BananaMind/TernaryBananaMind-10M"
device = "cuda" if torch.cuda.is_available() else "cpu"
dtype = (
    torch.bfloat16
    if device == "cuda" and torch.cuda.is_bf16_supported()
    else torch.float32
)

tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    trust_remote_code=True,
    dtype=dtype,
).to(device).eval()

inputs = tokenizer("The color of the sky is", return_tensors="pt").to(device)
with torch.inference_mode():
    output = model.generate(
        **inputs,
        max_new_tokens=96,
        do_sample=True,
        temperature=0.7,
        top_p=0.9,
        pad_token_id=tokenizer.eos_token_id,
        eos_token_id=tokenizer.eos_token_id,
    )

print(tokenizer.decode(output[0], skip_special_tokens=True))

License

Apache 2.0.