| --- |
| base_model: ornith-ai/Ornith-1.5-9B |
| library_name: peft |
| license: mit |
| tags: [lora, spreadsheetbench, tool-use, agent] |
| --- |
| |
| # Ornith 1.5 9B, LoRA fine-tune for SpreadsheetBench agent trajectories |
|
|
| LoRA adapter for `ornith-ai/Ornith-1.5-9B`, trained by Ambit Labs for the Ylookup x Encode hackathon (SpreadsheetBench). Code, harness, evaluation and the full write-up: https://github.com/Ambit-Labs/hacks-enyl-research (README, "How the Ornith 9B was fine-tuned"). |
|
|
| ## Data |
|
|
| 282 multi-turn tool-call trajectories (system prompt, `task <id>` user turn, assistant tool calls, tool outputs truncated to 6000 characters, ending at the `submit` call) produced by the same agent harness driven by DeepSeek v4 flash on the SpreadsheetBench verified-400 set, keeping only tasks the evaluator scored as passing. 58 failing tasks and 60 passing tasks were held out. Golden workbooks were never in the training text; they only selected which trajectories to keep. |
|
|
| ## Recipe |
|
|
| LoRA on all linear projections, rank 32, alpha 64, dropout 0.05; bf16; max sequence length 32768, no packing; 2 epochs; learning rate 1e-4 cosine; batch 1 with 8-step gradient accumulation (72 optimizer steps); gradient checkpointing; loss on assistant tokens only via a custom collator (the chat template has no generation tags). TRL 1.12, PEFT 0.20, transformers 5.16, torch 2.14, one H100, 97 minutes. Train loss 0.58 to 0.55. |
|
|
| ## Serving |
|
|
| Merge into the base with PEFT `merge_and_unload()` and serve with vLLM 0.28: `--tool-call-parser qwen3_xml --reasoning-parser qwen3 --enable-auto-tool-choice --max-model-len 32768`, thinking off through `chat_template_kwargs: {"enable_thinking": false}`, temperature 0. |
|
|
| ## Results |
|
|
| Held out (118 tasks): base 9B 45, this adapter 29, DeepSeek-driven harness 82. Full 400: 0.34 pass rate (the base 9B run of the same 400 at higher concurrency scored 0.19). The adapter makes the model quit the tool loop earlier than the base; the repo README explains the likely causes and the next round. |
|
|
