com-junkawasaki/oka-gemma3-4b-lora

Oka Gemma 3 4B LoRA

Oka is an experimental GFTD multimodal model family named after mathematician Kiyoshi Oka. This repository publishes the LoRA adapter only for an Oka identity and routing smoke model on top of google/gemma-3-4b-it.

This is not a full standalone model. To use it, you need access to the base model google/gemma-3-4b-it and must comply with the base model license and access requirements.

Model Details

• Base model: google/gemma-3-4b-it
• Adapter type: PEFT LoRA
• Training precision: BF16
• LoRA rank: 8
• LoRA alpha: 16
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
• Trainable parameters: 16,394,240 / 4,316,473,712 total parameters
• Training run: oka-gemma3-4b-lora-identity-200step-20260509
• Hardware: RunPod H100 NVL Secure pod

Intended Use

This adapter is intended for Oka development smoke tests: identity, model-card behavior, routing language, and same-size Gemma 3 4B throughput comparison. It is suitable for internal experimentation and reproducibility of the Oka training path.

It is not a production safety-tuned assistant, not a medical/legal/financial model, and not a complete multimodal trunk.

Training Summary

The adapter was trained for 200 optimizer steps on a small synthetic Oka identity dataset. The run completed with:

{
  "run_id": "oka-gemma3-4b-lora-identity-200step-20260509",
  "model_id": "google/gemma-3-4b-it",
  "output_dir": "/workspace/oka-gemma3-4b-lora-identity-200step-20260509",
  "train_runtime_sec": 339.54392671585083,
  "metrics": {
    "train_runtime": 313.397,
    "train_samples_per_second": 2.553,
    "train_steps_per_second": 0.638,
    "total_flos": 4473449644032000.0,
    "train_loss": 0.27960334181785584,
    "epoch": 2.0
  }
}

A local H100 benchmark showed:

• Base google/gemma-3-4b-it: 107 generated tokens at 24.967 tok/s
• Oka LoRA unmerged: 107 generated tokens at 14.101 tok/s
• Oka LoRA merged into the base trunk: 121 generated tokens at 24.293 tok/s

The merged adapter therefore preserved near-baseline same-size Gemma 3 4B throughput in this short prompt benchmark while producing Oka-specific identity behavior.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel
import torch

base_model = "google/gemma-3-4b-it"
adapter = "com-junkawasaki/oka-gemma3-4b-lora"

tokenizer = AutoTokenizer.from_pretrained(adapter)
model = AutoModelForCausalLM.from_pretrained(
    base_model,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
model = PeftModel.from_pretrained(model, adapter)

prompt = "What is the Oka model? Answer in one concise paragraph."
inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(**inputs, max_new_tokens=128, do_sample=False)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

For deployment, merge the adapter before throughput-sensitive serving:

model = model.merge_and_unload()

Limitations

• This is a very small adapter trained on synthetic identity examples, not a broad instruction tune.
• It does not add true audio, image, video, 3D, geospatial, tabular, document, or time-series perception by itself.
• It inherits the limitations and safety profile of the base Gemma model.
• It has not passed a production eval gate or safety review.

Provenance

The training and benchmark results are recorded in the Oka ADR in the GFTD application repository: 90-docs/adr/2605092345-runpod-l40s-fp8-multimodal-model-design.md.