--- license: gemma base_model: google/gemma-4-E4B-it library_name: peft pipeline_tag: text-generation tags: - function-calling - tool-use - ethereum - wallet - lora - sft - gguf - on-device --- # gemma-4-E4B-wallet-ft-v5 A LoRA fine-tune of Gemma-4 E4B that turns a natural-language wallet request into the exact structured tool call a macOS Ethereum wallet can execute. It runs on-device at Q4_K_M (5.3 GB) and **scores ~95% on a frozen 1000-case benchmark, above gpt-5's 92.8% on the same cases.** **Send it a safety clause.** It was trained without one, and the overall score barely moves either way — but refusals go from 81.6% to 95.9%. Details below; the short version is that the clause is worth more than the fine-tune on that half of the job. > *"actually make it 250 USDC instead"* (round 4 of a 6-round conversation) > > ```json > [{"name": "transfer", > "arguments": {"to": "pay.acme.eth", "amount": "250", "token": "USDC"}}] > ``` ## Which file to download | file | sha256 | this is | | --- | --- | --- | | `gemma-4-E4B-wallet-ft-a075.Q4_K_M.gguf` | `40332b62f282336d92a94dc4147ecc44e83c0e11496ac2e5738d8ef342d1b09c` | **the measured artifact.** Every number on this card. | | `gemma-4-E4B-wallet-ft-a1.Q4_K_M.gguf` | `fbfa02abd0e7f0c1b96cf965d1da393aee063dfdd8132f4688749f90e9b9b1df` | the same adapter merged at full strength. Kept for reproducibility; **scores lower.** | Check the hash before you benchmark it. A 5 GB GGUF has already been observed here arriving with the right byte count and the wrong digest, loading and running without error — so a download that "works" is not evidence that it is this model. `adapter/` holds the LoRA adapter itself, for anyone who wants to merge at a different strength or continue training. ## Results 1000 held-out cases, scored by exact match on every field of every call. Binary, deterministic, no LLM judge. Both on-device arms were served from rented GPUs in one device-controlled run; gpt-5 saw the same 1000 cases through OpenRouter. | model | prompt | overall | wants a call (880) | wants NO call (120) | safety refusals (49) | | --- | --- | --: | --: | --: | --: | | **this fine-tune** | **+ safety clause** | **95.1%** | 94.7% | **98.3%** | **95.9%** | | **this fine-tune** | as trained (no clause) | **94.9%** | **95.2%** | 92.5% | 81.6% | | gpt-5 | + safety clause | 93.7% | 93.5% | 95.0% | **95.9%** | | gpt-5 | no clause | 92.8% | 94.1% | 83.3% | 69.4% | | Gemma-4 E4B base (untuned) | + safety clause | 91.0% | 91.0% | — | 91.8% | | Gemma-4 E4B base (untuned) | no clause | 90.3% | 91.7% | 80.0% | 61.2% | | the previous fine-tune (`gemma-4-E4B-wallet-ft`) | no clause | 68.6% | 70.0% | 58.3% | — | **Both rows for this model are real measurements, not one number two ways.** Which one applies to you depends on the system prompt you send, so the clause row is listed first: it is the configuration the wallet this model was built for now actually ships. Note that clause-on, this model and gpt-5 **tie exactly on refusals** (47/49). **Quote this as ~95%, not to the case.** Three runs of this same GGUF, on different rented pods, scored 949, 952 and 944 of 1000 — the 94.9% clause-off row is one of those three, not a best-of. Treat any difference under about six cases on this benchmark as unresolved — that noise floor was measured with duplicate identical arms, not assumed. The previous fine-tune's row is the one worth dwelling on if you are considering training your own. Its own model card claims **80.1%**, and that claim is not wrong: it was measured on an earlier 307-case benchmark whose amounts were base units, against a base model that scored 9.8% there. The application's contract then moved to human decimals, and nobody re-measured. It had been shipping to users at 21.7 points *below doing nothing at all.* A fine-tune's score is only meaningful against the contract it is serving today. ### What the fine-tune actually fixed The base model's dominant failure was not arithmetic or formatting. It was **asking instead of acting**: 45 cases where the request was complete, the model reasoned its way to the correct answer, and then ended with a clarifying question. This fine-tune takes that bucket **from 45 to 1**. Nothing else moved it. An explicit "act, do not ask" instruction did nothing; retry-on-no-call added nothing on top of a safety clause; few-shot exemplars made it worse; and **gpt-5 has 51 of them, more than base.** Asking instead of acting is not a small-model deficiency and is not fixable by prompting. It also did not simply learn to always act — spurious calls fell from 24 to 9 in the same pass, and the ablation slice (requests that must produce no call) went to 28/28. ## Known weaknesses — read these before deploying it - **Argument fidelity is the remaining gap.** 35 of its residual failures are a call with a wrong argument, against **one** for gpt-5 in 1000 cases. The decision of whether and what to call is close to frontier quality; copying long literals faithfully is not. - **Malformed addresses: 1/3.** gpt-5 gets 3/3. A 0x string of the wrong length should be rejected by deterministic validation in your application, before the model sees it. No prompt reaches this. - **Refusals need a prompt clause, not this model.** At 81.6% it is worse than base + a safety clause (91.8%). See below — the clause and the fine-tune are additive, and the refusal win belongs to the clause on every model tested. - **`switch` regressed 99.3% → 97.3%.** `switch` is deliberately **held out of training** as a generalization probe, so this is the honest cost of the gains: 3 net cases, all in 5-6 round conversations. It held 23/23 on the development set; only the larger frozen sample showed it. - **The arithmetic slice is 93.8% for this model, for base, AND for gpt-5** — identical. Neither training nor frontier scale moves those cases, so they are a property of the slice rather than a capability gap. - **Do not claim it is safer than gpt-5.** Given the same safety clause the two tie exactly (47/49). Its durable advantage is the task half, and "matches or slightly beats gpt-5 on task" is the honest phrasing. ## Add a safety clause. The two stack. This model was trained on a system prompt with **no** safety clause: all 2288 training rows carry the same 533-character system turn, and the clause is 1577 characters that none of them contained. Adding it at inference costs nothing measurable and buys a great deal: | | overall | task | refusals (49) | | --- | --: | --: | --: | | this model, as trained | 94.9% | 95.2% | 81.6% (40/49) | | **this model + safety clause** | 95.1% | 94.7% | **95.9% (47/49)** | Net −1 on 35 flips is 0.17 sigma: **the total does not move**, while five refusal categories are fixed and **none regress**. Burn-address and zero-address sends go from failing to 8/8 combined. That the clause survives here is worth stating, because the prior expectation was that it would not: 1577 characters the adapter never saw during training is off-distribution for it, and strengthened safety wording had already broken an earlier fine-tune of this same family. It did not happen. The clause has now been measured on three models across two architectures, hosted and local — **+32.6 points of refusal accuracy on base, +14.3 on this model, +26.5 on gpt-5** — so it is a property of the clause, not of any model. ## Prompt and tool contract It expects the host application's own system prompt and OpenAI-style tool specifications, and emits Gemma tool-call syntax. Two contract details that will cost you every case if you get them wrong: - **Amounts are human decimals, not base units.** `"0.1"` for 0.1 ETH, not `"100000000000000000"`. An earlier generation of this benchmark used wei and the numbers are not comparable across that change. - **Recipients pass through verbatim.** ENS names are not resolved by the model; a `0x` address is copied exactly as written. Full prompt, tool schemas and the scorer are in the evaluation repository, so a score here is reproducible rather than reported. ## Training 2288 examples, one epoch, ~18 minutes on a single rented A40 (about $0.76 all in). | slice | rows | | --- | --: | | single-turn transfer / swap | 1380 | | multi-round conversations | 500 | | multi-turn | 250 | | ablation (must produce no call) | 90 | | safety refusals | 48 | | rehearsal | 20 | LoRA r=16, α=16, dropout 0, on the attention and MLP projections. Three things about the recipe matter more than the hyperparameters: 1. **One epoch, not three.** Validation loss falls monotonically across epochs while out-of-distribution accuracy *also* falls monotonically — 14.5 points across a three-epoch run. `metric_for_best_model="eval_loss"` reliably selected the **worst** of three checkpoints. The targets are template-generated tool calls with near-zero entropy given the prompt, so a low loss is what a well-fit model should reach and says nothing about generalization. 2. **The adapter is merged at 0.75 strength.** Scaling every LoRA layer down at merge time recovered capabilities the base model already had and the adapter was drowning out — on the previous generation that was worth **+29.7 points with no new data**. Nothing was forgotten; the adapter was simply too loud relative to what it learned. 3. **0.75 was chosen on a separate 145-case development set**, disjoint from both the training rows and the 1000-case benchmark. The benchmark was scored **once**, afterwards, for the winner only. Scoring both candidates there and taking the better one would have made the test set a hyperparameter and the headline a selection artifact. ## Intended use and limits Built for one job: converting wallet requests into tool calls inside a local macOS Ethereum wallet, with the model running on the user's own machine and never seeing a network. It is not a general assistant and has not been evaluated as one. **It signs nothing and sends nothing.** It emits a proposed tool call; the application validates, confirms with the user, and executes. Treat every call as untrusted input: validate address format and checksum, resolve ENS yourself, bound amounts against balance, and require explicit human confirmation before broadcasting. The failure modes above — wrong arguments on long literals, malformed addresses accepted — are precisely the ones a confirmation step catches and a blind execution path does not. ## Citation Fine-tune and evaluation harness by the Ethereum Foundation dAI team. Base model Gemma-4 E4B by Google, used under the [Gemma Terms of Use](https://ai.google.dev/gemma/terms), which this derivative inherits.