Instructions to use google/gemma-2-2b-it with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use google/gemma-2-2b-it with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="google/gemma-2-2b-it")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("google/gemma-2-2b-it")
model = AutoModelForCausalLM.from_pretrained("google/gemma-2-2b-it", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use google/gemma-2-2b-it with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "google/gemma-2-2b-it"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-2-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/google/gemma-2-2b-it

SGLang

How to use google/gemma-2-2b-it with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "google/gemma-2-2b-it" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-2-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "google/gemma-2-2b-it" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "google/gemma-2-2b-it",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use google/gemma-2-2b-it with Docker Model Runner:
```
docker model run hf.co/google/gemma-2-2b-it
```

[Research / Experiment] Native Dual-Regime Gemma: Sub-35ms Motor Reflexes & Zero-Cost KV-Cache Handoff on Gemma-2-2B

#84

by savhascelik - opened 12 days ago

Discussion

savhascelik

12 days ago

Hi everyone,

We wanted to share an open-source experimental architecture we've developed on top of Google's Gemma-2-2B-IT: Native Dual-Regime Inference.

The Problem: Tool-Calling Latency & Conversational Drift

In agentic workflows, spending 35–50 autoregressive tokens solely to select an action or tool introduces latency bottlenecks (600–900ms on GPU), wastes compute, and frequently triggers parsing failures due to natural language preambles.

The Solution: Native Dual-Regime (Zero Fine-Tuning)

Inspired by the System 1 motor reflex concept from TypeSafe AI's Jev, we built a 100% native execution pipeline without auxiliary heads or fine-tuning:

System 1 (Reflex Phase): Evaluates bounded candidate actions in a single forward pass by slicing logits directly from the native language modeling head ($W_{lm}$). Discrete decisions resolve in <35ms consuming exactly 0 tokens.

System 2 (KV-Cache Handoff): If arguments are required, the precomputed Key-Value attention matrices (past_key_values) are handed off directly to causal generation. Prefill latency drops to 0
ms, and prefix-clamping eliminates conversational chatter.

Empirical Benchmark Summary (40 Real-World Kaggle Scenarios)

We evaluated four regimes on identical Kaggle T4 GPU environments under strict Python standard json.loads() validation (zero regex tolerance):

Architecture	Action Accuracy	Strict Valid JSON Rate	Mean Latency	Speedup	Tokens Burned	Token Savings
Baseline Causal LM (Control)	90.0% (36/40)	62.5% (25/40)	22,598 ms	1.0x (Base)	1,485	0% (Base)
Cascaded Dual-Regime	92.5% (37/40)	75.0% (30/40)	7,100 ms	3.18x	132	-91.1%
Speculative Dual-Regime ($\tau=0.75$)	92.5% (37/40)	100.0% (40/40)	5,448 ms	4.15x	23	-98.5%
Continuous Cognitive Anchoring	92.5% (37/40)	62.5% (25/40)	9,572 ms	2.36x	277	-81.3%

Key Takeaways & Scope:

Hallucination Suppression: System 1 logit slicing eliminated tool hallucinations (e.g. baseline generated a non-existent set_urgent_meeting tool, whereas Dual-Regime locked the correct candidate).
Early Aborts: In the Speculative regime, 38/40 tasks (95%) were resolved with 0 generated tokens.
Experimental Scope: These findings represent an initial proof-of-concept on 40 curated scenarios using Gemma-2-2B. We believe this provides a strong architectural signal that scaling this mechanism to larger models (e.g. Gemma-2-9B / 27B) will deliver even sharper calibration and robust agent performance.

All code, reproducible Kaggle notebooks, and raw evaluation datasets are open-sourced:
👉 Repository: https://github.com/savhascelik/dual-head-gemma

We would love to hear feedback and thoughts from the Gemma and open-source agent community!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment