sensix-zo/sensix-paite-4b-nitro-16bit

Sensix Paite 4B Nitro (16-bit) - Rapid Test Build

This model is the "Nitro" test build of the Sensix Paite AI development pipeline. It is an experimental Continued Pre-Training (CPT) model built on top of the Gemma 3 4B-IT architecture.

This model is completely UNGATED and open to the public for testing and experimentation.

The primary objective of this specific "Nitro" release is rapid iteration. To achieve incredibly fast training times and test the viability of our dataset pipeline, this model was trained with a strict 512 Max Token limit. It is designed for developers and researchers who want to test the baseline Paite vocabulary acquisition without downloading massive context-heavy models.

Training Parameters: Built for Speed

This model was trained using Unsloth on a high-performance GPU environment. The context length was intentionally bottlenecked to allow for rapid epochs and quick evaluation of the dataset's effectiveness.

• Base Model: unsloth/gemma-3-4b-it
• Context Length: 512 tokens (Crucial: Do not expect this model to remember deep, multi-page context)
• Learning Rate: 2e-4 (Aggressive for vocab acquisition)
• LoRA Config: r=64, alpha=64
• Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Epochs: 1

Dataset Strategy: Avoiding "Preacher Bias"

Even though this is a rapid test build, data quality remains the priority. To prevent this model from developing a repetitive "robotic" tone or archaic "Preacher Bias," it was trained exclusively on the PERFECT_PAITE_DATA.jsonl dataset, chunked to fit the 512-token limit.

• Included Data: Modern news articles, contemporary essays, parallel dictionaries, and conversational paragraphs.
• Excluded Data: The Paite Bible and isolated short-sentence fragments were strictly removed to ensure the model learns modern, fluid, and logical conversational reasoning.

Technical Implementation: The Hard Merge

Standard Unsloth merging functions (save_pretrained_merged) have been known to cause weight-scrambling bugs in Gemma 3 architectures, often resulting in gibberish outputs (the "Calcium/Blades" bug).

To guarantee stability during testing, this model was fused using an Official Hard Merge (model.merge_and_unload()).

• The LoRA adapter weights are physically baked into the base model.
• The model is saved in full 16-bit precision (bfloat16).
• It operates seamlessly in standard inference environments.

Usage & Testing

Because this model is ungated, anyone can plug it into their workflow immediately. While it has a short memory (512 tokens), it inherits the underlying instruction chat template of the Gemma 3 IT model and is great for testing short translation tasks or basic completion.

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "sensix-zo/sensix-paite-4b-nitro-16bit"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto"
)

# Example usage for text generation
messages = [
    {"role": "user", "content": "Paite pau in thulim khat hon gelh in."}
]

inputs = tokenizer.apply_chat_template(messages, add_generation_prompt=True, return_tensors="pt").to("cuda")

# Note: Keep max_new_tokens reasonable since the model was trained on max 512 length
outputs = model.generate(**inputs, max_new_tokens=256, do_sample=True, temperature=0.7)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

1. Short Memory: Due to the 512 max token training limit, the model will struggle with long prompts or deep multi-turn conversations. It is best used for single-shot queries, short translations, and vocabulary testing.
2. Phase 1 Only: This model has only undergone Continued Pre-Training (CPT). Its conversational personality is not yet fully aligned.
3. Experimental: This is a rapid-iteration build meant to validate datasets, not a finalized production model.