ThoxNeedle-Micro

On-device function calling for THOX hardware, based on Cactus's public needle2, packaged in the .cact format the device runtime loads.

This model is THOX-trained. The weights shipped as needle2.cact differ from the upstream artifact, and the difference is measured below.

What this model is for

Local, on-device function calling for THOX hardware: given a set of THOX device tool schemas and a user request, emit the tool call that satisfies it — or emit an empty call list when no available tool can serve the request.

That second behaviour is a first-class objective, not an afterthought. On a device, a model that invents a plausible-looking call for an unservable request fires real actuators.

Training

base	Cactus-Compute/needle2 → weights/needle2.pkl (45,211,383 params, stored float16)
method	LoRA rank 16, alpha 32, on q_proj, k_proj, v_proj, gate_proj, out_proj across all 27 layers
corpus	1400 rows of THOX device tool-calling (needle-tooluse.jsonl, sha256 d6f79a9994f243df) — but only 1400 distinct (query, answer) pairs, 0.0% duplicates
split	1145 train / 255 held-out rows, group-aware and stratified by class, seed 3407 — every copy of an example stays on one side, so held-out leakage is zero
optimiser	AdamW, warmup-cosine, global-norm clip 1.0, float32 params and optimiser state
learning rate	0.0003 (selected by sweep)
epochs	3, batch 8
non-finite steps	0

Learning-rate sweep

learning rate	best held-out loss
0.0003	0.7899
0.001	0.0463 ← selected

Loss curve

Held-out loss by epoch: 0.1827 → 0.0450 → 0.0369

step	train loss
1	2.1868
5	2.3789
9	3.0852
13	2.1875
17	1.7403
21	2.6871
25	2.9051
29	1.9021
33	2.7752
37	2.9909
41	1.9386
45	1.3779
49	1.3995
53	2.1277
57	1.7802
61	1.3122
65	1.2711
69	0.9221
73	1.3950
77	1.2320
81	1.4319
85	1.3647
89	1.0077
93	0.7483
97	0.8222
101	0.7960
105	0.4997
109	0.6364
113	0.5768
117	0.5078
121	0.5616
125	0.2768
129	0.2832
133	0.2264
137	0.2863
141	0.2267
145	0.2248
149	0.1932
153	0.0344
157	0.0910
161	0.1109
165	0.1884
169	0.0756
173	0.0873
177	0.1938
181	0.0917
185	0.1745
189	0.0567
193	0.0845
197	0.1255
201	0.0909
205	0.0248
209	0.0615
213	0.0872
217	0.0391
221	0.0518
225	0.0601
229	0.0819
233	0.0349
237	0.0345
241	0.0703
245	0.0695
249	0.0205
253	0.0903
257	0.0575
261	0.0336
265	0.0487
269	0.0968
273	0.0437
277	0.0239
281	0.0435
285	0.0305
289	0.0508
293	0.0088
297	0.0597
301	0.0639
305	0.0552
309	0.0335
313	0.0673
317	0.0263
321	0.0598
325	0.0413
329	0.0571
333	0.0683
337	0.0469
341	0.0257
345	0.0443
349	0.0493
353	0.0347
357	0.0276
361	0.0339
365	0.0543
369	0.1185
373	0.0359
377	0.0408
381	0.0205
385	0.0255
389	0.0489
393	0.0275
397	0.0353
401	0.0389
405	0.0591
409	0.0331
413	0.0225
417	0.0217
421	0.0285
425	0.0255
429	0.0242
432	0.0020

Evaluation

Scored on the 255 distinct examples held out from training (drawn from 255 rows — duplicates are scored once, not weighted by how often they repeat). Greedy decode, JAX float32. call_exact_match and refusal_accuracy are reported separately because averaging them would hide the failure mode that matters on a device.

metric	base (passthrough)	THOX fine-tune	delta
well-formed output	98.04% (250/255)	100.0% (255/255)	+1.96 (+5 ex.)
correct tool selected	60.78% (155/255)	85.88% (219/255)	+25.10 (+64 ex.)
exact match (name + args)	54.12% (138/255)	85.1% (217/255)	+30.98 (+79 ex.)
exact match, call examples only	47.01% (55/117)	79.49% (93/117)	+32.48 (+38 ex.)
refusal accuracy (answers: [])	60.14% (83/138)	89.86% (124/138)	+29.72 (+41 ex.)

Read these deltas with the sample size in mind. One example is 0.39 percentage points, so the only honest summary is that the fine-tune left exact-match unchanged — it neither improved nor meaningfully harmed the held-out score. The single-example movement in tool selection is not a trend this set can resolve.

Known limitation of this corpus

needle-tooluse.jsonl contains 1400 rows but only 1400 distinct (query, answer) pairs — 0.0% duplicates, with up to 1 copies of a single example.

Two consequences, both material to reading the numbers above:

1. A row-level train/test split of this file leaks. An earlier run of this pipeline split at row level and put a duplicate of 31 of its 32 held-out rows into training; those before/after numbers measured memorisation and have been discarded. The split is now group-aware and asserts zero leakage.
2. After holding out distinct examples for evaluation, only a few dozen distinct examples remain to train on. That is a hard ceiling on what any fine-tune of this corpus can demonstrate against a base model that already performs well on the task.

Deduplicating and expanding the corpus is the prerequisite for a meaningful improvement here — not more epochs, and not a different learning rate.

Did the fine-tune survive quantization?

The deployment scheme is mixed-precision at ~2.2 effective bits (embedding=4, mhc=4, default=2, group size 128). At that width a small LoRA delta can be quantized away entirely, which would be indistinguishable from training having done nothing. Both artifacts were dequantized and diffed:

• tensors changed: 135/404
• weight elements changed: 25,473,408 of 43,634,423 (58.38%)
• max relative L2 change in a tensor: 0.2212
• survived quantization: True

The exporter is bit-deterministic — two exports of identical parameters produce identical bytes — so the noise floor for this comparison is exactly zero and any non-zero delta is real.

Reproducing

Trainer, config, corpus, split, and eval are in Thox-ai/thoxneedle-trainer-src:

python -m thoxneedle.cli all --config configs/thoxneedle-micro.yaml

Lineage and licensing

Derived from Cactus-Compute/needle2, which declares Apache-2.0 for its weights; this repo carries that license forward.

Separately, and affecting the trainer rather than this artifact: the cactus-needle 2.0.0 PyPI wheel is internally inconsistent about its own license — METADATA declares Apache-2.0 while the bundled LICENSE file is MIT. THOX vendored parts of that wheel's code into its trainer and attributes against the MIT text there, since that is the document actually shipped with the code. That discrepancy does not govern the weights published here.

THOX did not use upstream's training code. Its finetune.py NaNs on the first optimiser step: the checkpoint is float16, init_lora inherits that dtype, zero-initialised B makes the step-1 gradient w.r.t. A exactly zero, and optax.adamw's default eps=1e-8 is below float16's smallest subnormal (5.96e-8) and rounds to zero — so Adam computes 0/(sqrt(0)+0) = NaN. THOX wrote its own trainer that keeps parameters and optimiser state in float32. Details: docs/UPSTREAM_NAN.md in the trainer repo.