---
language:
- en
base_model:
- ni-co-la-s/gemmeh-it
tags:
- quantization
---

# Have some GGUFs idk

Wow, my first quantizations! Thanks GPT 5.6 Luna for helping me with this (i'm an idiot).
These GGUF files were converted from the original Hugging Face checkpoint using the Gemmeh-compatible [llama.cpp](https://github.com/Ni-co-la-s/llama.cpp-gemmeh) implementation.

To actually do anything with these files, you need **[this fork of llama.cpp.](https://github.com/Ni-co-la-s/llama.cpp-gemmeh)**

Also have some graphs :D

![kld_vs_size](https://cdn-uploads.huggingface.co/production/uploads/694742331f2408791d8e1472/IVV6upfGMhQRgVdNGR0eX.png)

![ppl_vs_size](https://cdn-uploads.huggingface.co/production/uploads/694742331f2408791d8e1472/ykb0bVfWh6z8qcz_VPp31.png)

---

# Gemmeh-IT 1B

Gemmeh-IT is the instruction-tuned version of [ni-co-la-s/gemmeh](https://huggingface.co/ni-co-la-s/gemmeh), a 1.1B-parameter decoder-only transformer trained for educational purposes from scratch on 20B tokens of pre-2024 [FineWeb-Edu](https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu), then LoRA-finetuned on [OpenHermes](https://huggingface.co/datasets/teknium/OpenHermes-2.5) for chat.

Architecture is Gemma 3-inspired without sliding-window attention. Custom 32k SentencePiece BPE tokenizer trained on the same corpus.

For the base model, see [ni-co-la-s/gemmeh](https://huggingface.co/ni-co-la-s/gemmeh).
For the GGUF, see [ni-co-la-s/gemmeh-it-GGUF](https://huggingface.co/ni-co-la-s/gemmeh-it-GGUF).


## Usage

This model uses custom modeling code, so `trust_remote_code=True` is required.

```python
import torch
from huggingface_hub import hf_hub_download
from sentencepiece import SentencePieceProcessor
from transformers import AutoModelForCausalLM

repo_id = "ni-co-la-s/gemmeh-it"

model = AutoModelForCausalLM.from_pretrained(
    "ni-co-la-s/gemmeh-it",
    trust_remote_code=True,
    torch_dtype="bfloat16",
)
model.eval()
print("Model loaded")

# Load tokenizer
sp_path = hf_hub_download(
    repo_id="ni-co-la-s/gemmeh-it",
    filename="tokenizer.model",
    token=True,
)
sp = SentencePieceProcessor()
sp.Load(sp_path)
print("Tokenizer loaded")

# Test generation
def generate(question, max_new_tokens=40, temperature=0.0):
    prompt = (
        f"<start_of_turn>user\n{question}\n<end_of_turn>\n"
        "<start_of_turn>model\n"
    )
    ids = sp.Encode(prompt, out_type=int)
    input_ids = torch.tensor([ids], dtype=torch.long)
    with torch.no_grad():
        for _ in range(max_new_tokens):
            out = model(input_ids)
            next_logits = out.logits[0, -1, :]
            if temperature == 0:
                next_id = next_logits.argmax().item()
            else:
                probs = torch.softmax(next_logits / temperature, dim=-1)
                next_id = torch.multinomial(probs, 1).item()
            if next_id == sp.eos_id():
                break
            input_ids = torch.cat([input_ids, torch.tensor([[next_id]])], dim=1)
    return sp.Decode(input_ids[0][len(ids):].tolist())

print(generate("What is the capital of France?", max_new_tokens=40, temperature=0.0))
```

## Training details

| | |
|---|---|
| Parameters | 1.1B |
| Architecture | Gemma 3-inspired |
| Vocab | 32,768 (SentencePiece BPE, English-only) |
| Context | 4,096 |
| Pretraining data | FineWeb-Edu sample, 20B tokens |
| Knowledge cutoff | Pre-2024 (intentional) |
| Finetuning | LoRA rank 16 on OpenHermes (250M assistant tokens) |

## Benchmarks

Evaluated through the BF16 GGUF served via llama.cpp with [lm-eval](https://github.com/EleutherAI/lm-evaluation-harness). Gemma 3 1B IT numbers obtained locally by running [unsloth/gemma-3-1b-it-GGUF at Q8_0](https://huggingface.co/unsloth/gemma-3-1b-it-GGUF) through the same pipeline.

| Benchmark | Metric | Gemmeh 1B (base) | Gemmeh-IT 1B | Gemma 3 1B IT (local) |
|---|---|---:|---:|---:|
| PIQA | 0-shot | 70.2 | 71.4 | **72.8** |
| ARC-Challenge | 25-shot | 38.4 | **40.4** | 40.3 |
| ARC-Easy | 0-shot | 57.3 | 57.6 | **63.4** |
| WinoGrande | 5-shot | 52.2 | 54.0 | **55.1** |
| TruthfulQA | mc2, 0-shot | 37.8 | **44.8** | 38.9 |

Trained on roughly 10-100x less data than the references, with a much smaller (32k vs 262k) vocabulary, and no distillation.


## Limitations

- Smaller and less benchmark-competitive than similarly-sized models trained on more data.
- English only.
- 4,096 token context.
- Pre-2024 knowledge only.