sizzlebop/tinybrainbot-350mV3-instruct-GGUF

TinyBrainBot 350M V3 Instruct GGUF

This repository provides GGUF quantizations for nkthebass/tinybrainbot-350mV3-instruct.

TinyBrainBot 350M V3 Instruct is a 348M-parameter decoder-only model pretrained from scratch on 22.7B tokens and instruction-tuned across 2,000 SFT steps (131M tokens) with warm-up stable decay (WSD). It answers direct factual questions, summarizes short texts, and provides succinct explanations while fitting in extremely compact compute budgets.

All GGUF files were converted from the original safetensors weights using llama.cpp at native F16 precision, followed by standard k-quant quantization.

---

Available Files and Quantizations

File	Quant Type	Size	Description / Recommendation
tinybrainbot-350mV3-instruct-F16.gguf	F16	665.25 MB	Unquantized full precision base conversion. Highest fidelity reference weights.
tinybrainbot-350mV3-instruct-Q8_0.gguf	Q8_0	353.85 MB	Near-lossless 8-bit quantization. Recommended for best quality evaluation.
tinybrainbot-350mV3-instruct-Q6_K.gguf	Q6_K	335.87 MB	High quality retention with minimal degradation. Excellent accuracy/memory balance.
tinybrainbot-350mV3-instruct-Q5_K_M.gguf	Q5_K_M	262.20 MB	Good balance between memory consumption and text generation fidelity.
tinybrainbot-350mV3-instruct-Q4_K_M.gguf	Q4_K_M	243.48 MB	Fast, lightweight 4-bit quant. Recommended default for edge devices and mobile setups.
tinybrainbot-350mV3-instruct-Q3_K_M.gguf	Q3_K_M	210.46 MB	Compact footprint when memory headroom is strictly limited.
tinybrainbot-350mV3-instruct-Q2_K.gguf	Q2_K	195.49 MB	Maximum compression for ultra-constrained low-RAM environments.

---

Model Architecture Details

• Parameters: ~348M (348,342,912)
• Architecture: Decoder-only transformer (LlamaForCausalLM)
• Layers: 22
• Hidden Size: 1152
• Attention Heads: 18 query heads / 6 KV heads (GQA), head dimension 64
• Intermediate Size: 3072
• Context Length: 2048 tokens (SFT trained at 1024)
• Vocabulary Size: 32,000 (tbb-32k-v2, tied word embeddings)
• RoPE Theta: 10,000.0

---

Prompt Format and Chat Template

The TinyBrainBot V3 chat template is space-separated, not newline-separated, with an essential leading space:

<|user|> {prompt} <|end|> <|assistant|>

Constructing prompts with extra newlines degrades generation output.

Example Single-turn Prompt:

 <|user|> What is the boiling point of water at sea level? <|end|> <|assistant|>

Example Multi-turn Prompt:

 <|user|> Hello! Who are you? <|end|> <|assistant|> I am TinyBrainBot, a compact language model. <|end|> <|user|> Can you explain what photosynthesis is? <|end|> <|assistant|>

---

Quickstart Guide

1. llama.cpp

Run interactive instruction generation with llama-cli:

llama-cli -m ./tinybrainbot-350mV3-instruct-Q4_K_M.gguf \
    -p " <|user|> Explain quantum computing in three sentences. <|end|> <|assistant|>" \
    -n 256 \
    --temp 0.7 \
    --top-p 0.9 \
    --repeat-penalty 1.1

Launch a local OpenAI-compatible inference server:

llama-server -m ./tinybrainbot-350mV3-instruct-Q4_K_M.gguf \
    --port 8080 \
    -c 2048

2. Ollama

Create a Modelfile:

FROM ./tinybrainbot-350mV3-instruct-Q4_K_M.gguf
TEMPLATE " <|user|> {{ .Prompt }} <|end|> <|assistant|>"
PARAMETER stop "<|end|>"
PARAMETER temperature 0.7
PARAMETER top_p 0.9

Build and run:

ollama create tinybrainbot-instruct -f Modelfile
ollama run tinybrainbot-instruct "What causes the northern lights?"

3. LM Studio

1. Place the .gguf file in your LM Studio models folder.
2. Select tinybrainbot-350mV3-instruct-Q4_K_M.gguf.
3. Set the prompt template format:
   • System prompt: (Leave empty)
   • User prefix: <|user|>
   • User suffix: <|end|>
   • Assistant prefix: <|assistant|>
   • Stop strings: <|end|>