Model card: Cactus Hybrid quickstart + collection/GitHub links

README.md

@@ -9,20 +9,53 @@ tags:
   - custom_generate
 ---
 
-# gemma-4-e2b-it-hybrid
+# Cactus Hybrid — Gemma 4 E2B
 
-`Cactus-Compute/gemma-4-e2b-it-hybrid` is **google/gemma-4-E2B-it plus a handoff probe**: a
-small head (weight prefix `handoff_probe.*`) that scores every generation with
+A small, on-device model is fast and private, but sometimes wrong. At Cactus we
+post-train models to *know when they are wrong*: we ship probes inside the
+checkpoint that score every answer with a **confidence** between 0 and 1,
+returned as structured data (never parsed out of the answer text). Answer
+on-device when confidence is high; re-route to a bigger model when it's low —
+`0.85` is a good threshold:
 
+```python
+if confidence < 0.85:
+    answer = ask_a_bigger_model(prompt)
 ```
-confidence = 1 - p_wrong
+
+This repo is **google/gemma-4-E2B-it plus the handoff probe**: a small head
+(weight prefix `handoff_probe.*`) that scores every generation with
+`confidence = 1 - p_wrong`. The base weights are byte-identical to the stock
+checkpoint (same keys); the repo adds eleven probe tensors, a remote-code model
+class and a `custom_generate` recipe. **Stock engine commands work unchanged** —
+you only add `--trust-remote-code` / `trust_remote_code=True`.
+
+## Quickstart
+
+```python
+# pip install "transformers>=5.5.4,<5.6" torch   (5.14+ segfaults on this checkpoint)
+import torch
+from transformers import AutoModelForCausalLM, AutoTokenizer
+
+model_id = "Cactus-Compute/gemma-4-e2b-it-hybrid"
+device = "cuda" if torch.cuda.is_available() else "mps" if torch.backends.mps.is_available() else "cpu"
+
+tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)
+model = AutoModelForCausalLM.from_pretrained(model_id, trust_remote_code=True, dtype="auto").to(device)
+
+messages = [{"role": "user", "content": "What is the capital of France?"}]
+inputs = tokenizer.apply_chat_template(
+    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
+).to(device)
+out = model.generate(**inputs, return_confidence=True, max_new_tokens=512)
+
+print(tokenizer.decode(out.sequences[0][inputs["input_ids"].shape[-1]:], skip_special_tokens=True))
+print("confidence:", out.confidence)
 ```
 
-so a client can decide when to keep the on-device answer and when to hand off to
-a cloud model. The base weights are byte-identical to the stock checkpoint (same
-keys); the repo adds eleven probe tensors, a remote-code model class and a
-`custom_generate` recipe. **Stock engine commands work unchanged** — you only add
-`--trust-remote-code` / `trust_remote_code=True`.
+Load the model with an explicit `.to(device)`, not `device_map="auto"`: the
+probe scores generations outside the module `forward()` path, so weights that
+accelerate offloads (left on the `meta` device) crash the confidence read.
 
 ## Serving with stock transformers
 
@@ -62,34 +95,19 @@ trailer is emitted in both streaming and non-streaming modes.
 - The trailer's token ids are appended to the returned sequences, so reported
   completion token counts include the trailer (a handful of tokens).
 
-## Python usage
+## More Python APIs
 
 ```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-
-model = AutoModelForCausalLM.from_pretrained(
-    "Cactus-Compute/gemma-4-e2b-it-hybrid", trust_remote_code=True, dtype="auto"
-)
-tokenizer = AutoTokenizer.from_pretrained("Cactus-Compute/gemma-4-e2b-it-hybrid")
-inputs = tokenizer.apply_chat_template(
-    [{"role": "user", "content": "Explain why this contract clause is risky."}],
-    add_generation_prompt=True, return_tensors="pt",
-).to(model.device)
-
 # Structured API: clean sequences + raw float (no in-band trailer).
 sequences, confidence = model.generate_with_confidence(inputs, max_new_tokens=512)
 print(confidence)                     # e.g. 0.7812
 print(model.last_confidence)          # same value
 
 # Stock generate (custom_generate recipe): plain tensor + in-band trailer.
-sequences = model.generate(inputs, max_new_tokens=512)
-
-# Or opt into a rich return with the raw float:
-out = model.generate(inputs, max_new_tokens=512, return_confidence=True)
-out.sequences, out.confidence, out.trailer_text
+sequences = model.generate(**inputs, max_new_tokens=512)
 
 # Suppress the trailer while keeping stock behavior:
-sequences = model.generate(inputs, max_new_tokens=512, emit_trailer=False)
+sequences = model.generate(**inputs, max_new_tokens=512, emit_trailer=False)
 ```
 
 ## Probe contract
@@ -107,14 +125,6 @@ sequences = model.generate(inputs, max_new_tokens=512, emit_trailer=False)
 - Capture uses a forward hook that keeps only one `[1, 1536]` row per decode
   step — full hidden-state stacks are never materialized.
 
-## MLX (Apple Silicon)
-
-The repo ships a single-file `mlx-lm` model (`gemma_4_e2b_it_hybrid.py`),
-referenced by `"model_file"` in `config.json` (mlx-lm >= 0.30.1). After
-generation, read `model.last_confidence` (or `model.confidence(num_tokens=N)`).
-An mlx-lm `Model` cannot inject tokens into the stream, so there is **no
-in-band trailer** on MLX today — confidence is Python-API only there.
-
 ## Repo contents
 
 | File | Purpose |
@@ -125,5 +135,18 @@ in-band trailer** on MLX today — confidence is Python-API only there.
 | `model*.safetensors` | base weights (identical keys) + `handoff_probe.*` tensors |
 | `gemma_4_e2b_it_hybrid.py` | single-file `mlx-lm` model, wired via config.json's `model_file` |
 
-Requires `transformers>=5,<6`. Gemma model use remains subject to the Gemma
-terms.
+## All formats
+
+All Cactus Hybrid builds live in the
+[Cactus Hybrid collection](https://huggingface.co/collections/Cactus-Compute/cactus-hybrid-6a60da4551074db058e8bb64):
+[Transformers](https://huggingface.co/Cactus-Compute/gemma-4-e2b-it-hybrid) ·
+[GGUF / llama.cpp](https://huggingface.co/Cactus-Compute/gemma-4-e2b-it-hybrid-GGUF) ·
+[MLX](https://huggingface.co/Cactus-Compute/gemma-4-e2b-it-hybrid-mlx) ·
+[Cactus engine](https://huggingface.co/Cactus-Compute/gemma-4-E2B-it).
+Copy-paste quickstarts for every engine:
+[github.com/cactus-compute/cactus-hybrid](https://github.com/cactus-compute/cactus-hybrid).
+
+## License
+
+Gemma is provided under and subject to the Gemma Terms of Use. This derivative
+includes the Cactus handoff probe head.