gabrielfior commited on
Commit
92a2821
·
verified ·
1 Parent(s): e2bf07d

Lead with the clause-on configuration: that is the prompt the wallet now sends

Browse files
Files changed (1) hide show
  1. README.md +24 -10
README.md CHANGED
@@ -21,6 +21,10 @@ exact structured tool call a macOS Ethereum wallet can execute. It runs on-devic
21
  Q4_K_M (5.3 GB) and **scores ~95% on a frozen 1000-case benchmark, above gpt-5's
22
  92.8% on the same cases.**
23
 
 
 
 
 
24
  > *"actually make it 250 USDC instead"* (round 4 of a 6-round conversation)
25
  >
26
  > ```json
@@ -48,16 +52,24 @@ strength or continue training.
48
  deterministic, no LLM judge. Both on-device arms were served from rented GPUs in one
49
  device-controlled run; gpt-5 saw the same 1000 cases through OpenRouter.
50
 
51
- | model | overall | wants a call (880) | wants NO call (120) | safety refusals (49) |
52
- | --- | --: | --: | --: | --: |
53
- | **this fine-tune** | **94.9%** | **95.2%** | 92.5% | 81.6% |
54
- | gpt-5 | 92.8% | 94.1% | 83.3% | 69.4% |
55
- | Gemma-4 E4B base (untuned) | 90.3% | 91.7% | 80.0% | 61.2% |
56
- | the previous fine-tune (`gemma-4-E4B-wallet-ft`) | 68.6% | 70.0% | 58.3% | |
 
 
 
 
 
 
 
 
57
 
58
  **Quote this as ~95%, not to the case.** Three runs of this same GGUF, on different
59
- rented pods, scored 949, 952 and 944 of 1000. The 94.9% above is one of them, not a
60
- best-of. Treat any difference under about six cases on this benchmark as unresolved —
61
  that noise floor was measured with duplicate identical arms, not assumed.
62
 
63
  The previous fine-tune's row is the one worth dwelling on if you are considering
@@ -108,8 +120,10 @@ same pass, and the ablation slice (requests that must produce no call) went to 2
108
 
109
  ## Add a safety clause. The two stack.
110
 
111
- This model was trained on a system prompt with **no** safety clause. Adding one at
112
- inference costs nothing measurable and buys a great deal:
 
 
113
 
114
  | | overall | task | refusals (49) |
115
  | --- | --: | --: | --: |
 
21
  Q4_K_M (5.3 GB) and **scores ~95% on a frozen 1000-case benchmark, above gpt-5's
22
  92.8% on the same cases.**
23
 
24
+ **Send it a safety clause.** It was trained without one, and the overall score barely
25
+ moves either way — but refusals go from 81.6% to 95.9%. Details below; the short version
26
+ is that the clause is worth more than the fine-tune on that half of the job.
27
+
28
  > *"actually make it 250 USDC instead"* (round 4 of a 6-round conversation)
29
  >
30
  > ```json
 
52
  deterministic, no LLM judge. Both on-device arms were served from rented GPUs in one
53
  device-controlled run; gpt-5 saw the same 1000 cases through OpenRouter.
54
 
55
+ | model | prompt | overall | wants a call (880) | wants NO call (120) | safety refusals (49) |
56
+ | --- | --- | --: | --: | --: | --: |
57
+ | **this fine-tune** | **+ safety clause** | **95.1%** | 94.7% | **98.3%** | **95.9%** |
58
+ | **this fine-tune** | as trained (no clause) | **94.9%** | **95.2%** | 92.5% | 81.6% |
59
+ | gpt-5 | + safety clause | 93.7% | 93.5% | 95.0% | **95.9%** |
60
+ | gpt-5 | no clause | 92.8% | 94.1% | 83.3% | 69.4% |
61
+ | Gemma-4 E4B base (untuned) | + safety clause | 91.0% | 91.0% | — | 91.8% |
62
+ | Gemma-4 E4B base (untuned) | no clause | 90.3% | 91.7% | 80.0% | 61.2% |
63
+ | the previous fine-tune (`gemma-4-E4B-wallet-ft`) | no clause | 68.6% | 70.0% | 58.3% | — |
64
+
65
+ **Both rows for this model are real measurements, not one number two ways.** Which one
66
+ applies to you depends on the system prompt you send, so the clause row is listed first:
67
+ it is the configuration the wallet this model was built for now actually ships. Note that
68
+ clause-on, this model and gpt-5 **tie exactly on refusals** (47/49).
69
 
70
  **Quote this as ~95%, not to the case.** Three runs of this same GGUF, on different
71
+ rented pods, scored 949, 952 and 944 of 1000 the 94.9% clause-off row is one of those
72
+ three, not a best-of. Treat any difference under about six cases on this benchmark as unresolved —
73
  that noise floor was measured with duplicate identical arms, not assumed.
74
 
75
  The previous fine-tune's row is the one worth dwelling on if you are considering
 
120
 
121
  ## Add a safety clause. The two stack.
122
 
123
+ This model was trained on a system prompt with **no** safety clause: all 2288 training
124
+ rows carry the same 533-character system turn, and the clause is 1577 characters that
125
+ none of them contained. Adding it at inference costs nothing measurable and buys a great
126
+ deal:
127
 
128
  | | overall | task | refusals (49) |
129
  | --- | --: | --: | --: |