--- license: apache-2.0 base_model: meta-models/Muse-Glimmer-30B library_name: peft pipeline_tag: text-generation tags: [lora, tool-calling, function-calling, agentic, muse-glimmer] datasets: [Agent-Ark/Toucan-1.5M] --- # Muse-Glimmer-30B — agentic tool-calling LoRA A LoRA adapter that makes **Muse-Glimmer-30B** reliably *commit* to a tool call. ## Why this finetune is needed Muse-Glimmer-30B is sold as an agentic model, and it is not short of knowledge about the tools you hand it. Probe the stock model on held-out agentic trajectories and it will name the correct function **verbatim in its own reasoning** — and then keep deliberating past the point where an agent loop needed a call. Here is the stock model, mid-thought, on a task whose gold action is `databutton-submit_app_requirements`: > *"We need to submit app requirements via databutton-submit_app_requirements. > Need name, pitch, sp…"* Correct tool, correctly named, never emitted. The generation budget ran out first. That is not a knowledge gap, it is a **decisiveness and format gap**, and it is the single thing standing between this model and a working agent loop. On 200 held-out examples the stock model produced a well-formed call only 98.3% of the time when one was required. Supervised finetuning on real trajectories is the direct fix, and it costs 191.7M trainable parameters — 0.64% of the model. ## Results | Toucan held-out (in-distribution), n=200 | stock | + adapter | change | |---|---|---|---| | named the right tool anywhere in its output | 92.4% | 91.6% | -0.8pp (no significant change) | | emitted a well-formed call when one was needed | 98.3% | 90.8% | ↓ -7.6pp worse | | tool-name accuracy | 85.7% | 88.2% | +2.5pp (no significant change) | | argument exact match | 61.3% | **72.3%** | ↑ +10.9pp better | | schema valid (name exists, required args present) | 97.5% | 90.8% | ↓ -6.7pp worse | | abstained when no tool applies | 85.2% | **100.0%** | ↑ +14.8pp better | | hit the generation ceiling mid-answer | 1.7% | 1.7% | — +0.0pp | | BFCL v3 (**not trained on**), n=120/subset | stock | + adapter | change | |---|---|---|---| | **overall AST accuracy** | 81.2% | **85.2%** | ↑ +4.0pp better | | live irrelevance | 77.5% | **83.3%** | ↑ +5.8pp better | | live multiple | 78.3% | 78.3% | — +0.0pp | | live simple | 72.5% | **76.7%** | ↑ +4.2pp better | | multiple | 81.7% | **90.0%** | ↑ +8.3pp better | | simple | 92.5% | **95.8%** | ↑ +3.3pp better | BFCL v3 is the honest test here. Every Toucan trajectory was generated by MiniMax-M2.5, so gains against Toucan gold are partly gains at imitating that model. BFCL was never trained on. ## Does it break anything? A tool-calling finetune damages a general model in one characteristic way: the model starts emitting call syntax when **no tools were offered at all**. That failure is invisible to every metric above, so it is measured directly — GSM8K prompts with `tools=None`, counting any `= 5.15 is required: `muse_glimmer` is not a registered model # type before that, and the checkpoint carries no remote code. base = AutoModelForImageTextToText.from_pretrained( "meta-models/Muse-Glimmer-30B", dtype="bfloat16", device_map="auto") model = PeftModel.from_pretrained(base, "PursuitOfDataScience/Muse-Glimmer-30B-ToolCall-LoRA") tok = AutoTokenizer.from_pretrained("meta-models/Muse-Glimmer-30B") msgs = [{"role": "user", "content": "What's the weather in Chicago?"}] tools = [{"type": "function", "function": { "name": "get_weather", "description": "Current weather for a city.", "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}}}] ids = tok.apply_chat_template(msgs, tools=tools, add_generation_prompt=True, return_tensors="pt").to(model.device) print(tok.decode(model.generate(ids, max_new_tokens=256)[0, ids.shape[1]:])) ``` The model answers in Muse-Glimmer's native ATEM syntax, on the `to=` recipient channel — the adapter does not change the interface, only the reliability of using it. ## Training | | | |---|---| | Data | 44,800 examples: 40,000 Toucan-1.5M trajectories + 4,800 manufactured abstention hard negatives | | Steps | 700 x 24 examples/step | | Hardware | 3x H100 NVL 94GB, 669 GPU-minutes over 14 one-hour slices | | LoRA | r=32, alpha=64, dropout=0.05, text tower only | | Targets | `model\.language_model\.layers\.\d+\.(self_attn\.(q|k|v|o)_proj|mlp\.(gate|up|down)_proj)$` | | LR | 0.0001 cosine to 1e-05, warmup 40, grad-clip 1.0 | | Sequence | max 4096 tokens, micro-batches capped at 6144 padded tokens | | Loss | assistant spans only, masked by re-rendering each message prefix | The perception encoder and vision adapter are **frozen**. This corpus is text; adapting a ViT on text trajectories damages the multimodal path for nothing. ## Data preparation Three defects in the raw corpus are filtered, because each teaches the opposite of the goal: - **7.4%** of records leak Toucan's own `{...}` JSON into assistant *prose* — a third syntax this model's template never uses. Lifted into structured calls. - **4.2%** of gold calls name a tool that was never offered; 114 are literally named `unknown`. Dropped — supervising those teaches exactly the hallucination the finetune is meant to remove. - `thinking_cot` interleaves the agent's planning with `[Tool Simulation — …]` blocks, which are the *data generator* inventing tool results. Only the agent passes are kept; an agent must never be trained to hallucinate observations. Validation is deduplicated against training by normalised opening prompt. Toucan's subsets overlap by construction — `single-turn-diversify` rewrites the same seeds as `single-turn-original` — so distinct uuids are not distinct problems, and a naive split leaks **4.6%** of validation. The abstention hard negatives are manufactured from training data at zero labelling cost: take a trajectory, delete the tool it needs, keep the question, and expect an honest refusal. Over-calling is the failure that actually hurts a deployed agent, and Toucan's own `irrelevant` subset is small. ## Provenance and licensing - Base model **Muse-Glimmer-30B**, Apache-2.0. Meta also publishes a `USAGE_POLICY.md` acceptable-use policy which travels with any derivative. - Training data **Toucan-1.5M**, Apache-2.0. - Toucan is **synthetic**: every trajectory used here was generated by **MiniMax-M2.5**. The behaviour this adapter installs is distilled from that model's outputs and inherits its habits. BFCL is reported precisely because it is *not* from that distribution. ## Limitations - Trained for 700 steps on 16,800 examples. Validation loss was still falling when the step budget ran out (0.5038 at step 150 -> 0.4753 at step 700), so this is not a converged finetune. - The tool-calling evaluation scores the **first** call of a trajectory. It does not measure multi-turn task completion. - Muse-Glimmer is multimodal. The vision tower was frozen, but the language tower was adapted and **image understanding was not re-measured**. Treat the multimodal path as untested after this adapter. - Single seed. Small differences between arms are not meaningful; the paired test above is reported for exactly that reason. - Sampling error is roughly ±6.9 points per cell at these n.