Add tool-calling wrapper files

7f5a0c1 8 months ago

14.2 kB

FluxEM + Qwen3-4B Tool-Calling Benchmark

This package implements a comprehensive benchmark system that demonstrates how using FluxEM's 11 deterministic domain tools with Qwen3-4B (MLX-based) significantly outperforms baseline LLM performance.

Overview

The key insight from FluxEM experimentation is that neural networks cannot learn arithmetic or domain computations from sparse embeddings. However, FluxEM's 11 domains are perfect, deterministic symbolic tools. This benchmark demonstrates that:

LLM + Deterministic Tools >> LLM Learning Embeddings Alone

The 11 FluxEM Domains

Domain	Description	Tools Available
Arithmetic	`fluxem.compute()` for exact arithmetic	5 operations
Physics	Unit conversion, dimensional analysis	2 tools
Chemistry	Molecular weight, stoichiometry	1 tool
Biology	DNA analysis, transcription	3 tools
Mathematics	Vector/matrices, dot products	2 tools
Music	Pitch class sets, chords, scales	2 tools
Geometry	Points, distances, transforms	3 tools
Graphs	Connectivity, shortest paths	2 tools
Sets	Union, intersection, complement	4 tools
Logic	Tautology checking, validity	1 tool
Number Theory	Primality, GCD, modular arithmetic	2 tools

Project Structure

experiments/qwen3_toolcalling/
├── __init__.py                    # Package initialization
├── tool_registry.py               # Registry of all 11 FluxEM tools
├── qwen3_wrapper.py               # Qwen3-4B MLX wrapper with tool-calling
├── benchmark_data.py              # Test prompts for all domains (40-55 total)
├── evaluator.py                    # Metrics calculation and comparison
├── run_benchmark.py                # Main orchestration script with CLI
├── visualize_results.py           # Plotting and report generation
└── README.md                       # This file

Installation

Prerequisites

FluxEM (already installed)
```
pip install fluxem
```
MLX Backend (for Apple Silicon)
```
pip install mlx
```
FluxEM automatically uses MLX when fluxem.backend.set_backend(fluxem.BackendType.MLX) is called.
Visualization Dependencies (optional)
```
pip install matplotlib
```

Model Setup

For this benchmark, we recommend using Qwen3-4B MLX models for optimal Apple Silicon performance:

Download Qwen3-4B-Instruct-MLX from Hugging Face:

# Visit: https://huggingface.co/Qwen/Qwen3-4B
# Download the MLX quantized model

The model should be in a location like:
- ~/.mlx/models/Qwen/Qwen3-4B-Instruct-MLX/
- ~/.cache/huggingface/hub/Qwen/Qwen3-4B-Instruct-MLX/

Usage

Quick Start

# Run full benchmark on all 11 domains
python experiments/qwen3_toolcalling/run_benchmark.py \
    --model-path ~/.mlx/models/Qwen/Qwen3-4B-Instruct-MLX \
    --save-results \
    --verbose

# Run quick test on 3 domains
python experiments/qwen3_toolcalling/run_benchmark.py \
    --quick-test \
    --verbose

Command-Line Options

Option	Description	Default
`--model-path`	Path to Qwen3-4B MLX model	`~/.mlx/models/Qwen/Qwen3-4B-Instruct-MLX`
`--domains`	Specific domains to test (all 11)	all domains
`--no-baseline`	Skip baseline evaluation	False
`--output`	Output directory	`experiments/qwen3_toolcalling/results`
`--verbose`	Print detailed information	False
`--quick-test`	Run quick test (3 samples/domain)	False
`--save-results`	Save results to JSON	False

Benchmark Methodology

Evaluation Process

For each test query, the system:

Domain Detection: LLM identifies which FluxEM domain applies
Tool Selection: Calls appropriate FluxEM function for the domain
Tool Execution: FluxEM computes deterministic result
Response Generation: LLM formulates answer using tool result
Baseline Comparison: Same query processed without tools

Metrics Tracked

Domain Detection Accuracy: % of correct domain identifications
Tool Success Rate: % of successful tool executions
Answer Accuracy: % of correct final answers (tool-calling only)
Tool Execution Time: Average time for FluxEM computation
Baseline Response Time: Average time for LLM without tools
Improvement Ratio: (Tool Accuracy) / (Baseline Accuracy)

Expected Results

Based on FluxEM research (experiments/PROMPT_TOOL_CALLING_DEMO.md):

Domain	Baseline Accuracy	Tool-Calling Accuracy	Expected Improvement
Arithmetic	0-5%	100%	20x+
Physics	10-20%	95%+	5-10x
Chemistry	15-25%	90%+	4-6x
Biology	20-30%	90%+	3-5x
Mathematics	10-20%	95%+	5-10x
Music	25-35%	85%+	2.5-4x
Geometry	15-25%	90%+	4-6x
Graphs	10-20%	85%+	4-9x
Sets	30-40%	90%+	2.5-3x
Logic	35-45%	85%+	2-3x
Number Theory	15-25%	90%+	4-9x

Overall Expected: ~95%+ tool-calling accuracy vs ~20-40% baseline accuracy → 2.4x to 4.8x overall improvement

Understanding the Results

Domain Detection

The system uses LLM-based domain detection with pattern-based fallback:

LLM Detection (Qwen3-4B loaded):

# Prompt: "Given this question, which domain applies? Options: arithmetic, physics, chemistry, ..."
response = qwen3_wrapper.detect_domain("What is 54 * 44?")
# Result: "arithmetic"

Pattern-Based Fallback (no MLX): Simple keyword matching for domain keywords (e.g., "calculate" → arithmetic, "dna" → biology).

Tool Calling Workflow

flowchart LR
    User[User Query] --> DomainDetector
    DomainDetector -->|Correct Domain|
    DomainDetector --> ToolRegistry
    ToolRegistry -->|FluxEM Tools|
    FluxEM Tools -->|Exact Result|
    Exact Result --> ResponseGenerator
    ResponseGenerator -->|Final Answer|
    
    DomainDetector -.-> BaselineLLM[Baseline Response]
    BaselineLLM -->|Final Baseline Answer|

Key Components:

Tool Registry (tool_registry.py)
- Maps 11 FluxEM domains to 23+ callable functions
- Each tool includes description, input/output format, examples
Qwen3 Wrapper (qwen3_wrapper.py)
- Loads Qwen3-4B MLX model with Apple Silicon optimization
- Detects domain from query using LLM or pattern matching
- Executes appropriate FluxEM tool
- Generates responses incorporating tool results
- Fallback to baseline mode when tools unavailable
Benchmark Data (benchmark_data.py)
- 40-55 test prompts across all 11 domains
- 3-5 prompts per domain
- Ground truth expected answers
- Descriptions of what's being tested
Evaluator (evaluator.py)
- Compares tool-calling vs baseline responses
- Calculates accuracy, domain detection, tool success
- Tracks timing metrics
- Generates comprehensive reports
Visualizer (visualize_results.py)
- Creates comparison plots (accuracy bar charts, time comparison)
- Generates improvement heatmaps
- Produces markdown reports

Domain-Specific Examples

Example 1: Arithmetic

User: "What is 54 * 44?"

Step 1: Domain Detection
LLM detects: "arithmetic"

Step 2: Tool Execution
FluxEM Tool: arithmetic.compute("54 * 44")
Result: 2376.0

Step 3: Response Generation
LLM: "54 * 44 = 2376.0"

Baseline (no tool):
LLM: "The answer is 2376."

Example 2: Biology

User: "What's the GC content of GATTACA?"

Step 1: Domain Detection
LLM detects: "biology"

Step 2: Tool Execution
FluxEM Tool: biology_gc_content("GATTACA")
Result: 0.42857 (42.9% GC content)

Step 3: Response Generation
LLM: "The GC content of GATTACA is approximately 42.9%, calculated as 3 GC nucleotides out of 7 total."

Baseline (no tool):
LLM: "GATTACA has GC content of approximately 42-857%, which is around 3 GC nucleotides."

Example 3: Sets

User: "What is the union of {1, 2, 3} and {2, 3, 4}?"

Step 1: Domain Detection
LLM detects: "sets"

Step 2: Tool Execution
FluxEM Tool: sets_union([[1, 2, 3]], [[2, 3, 4]])
Result: [1, 2, 3, 4]

Step 3: Response Generation
LLM: "The union of these two sets is {1, 2, 3, 4}."

Baseline (no tool):
LLM: "The union includes 1, 2, 3, and also includes 4."

Benchmark Reports

Output Files

Running with --save-results generates:

benchmark_results_YYYYMMDD_HHMMSS.json: Full results in JSON format
- All prompts, tool calls, baseline responses, evaluations
- Timing information for each query
accuracy_comparison.png: Bar chart comparing tool-calling vs baseline accuracy
- Shows performance per domain
time_comparison.png: Grouped bar chart of response times
- Tool-calling time vs baseline time
improvement_heatmap.png: Heatmap showing improvement ratios
- Domain detection accuracy, tool success rate, answer accuracy
benchmark_report.md: Comprehensive markdown report
- Executive summary
- Domain-by-domain detailed results
- Key findings and recommendations

Interpreting the Reports

Improvement Ratio:

1.0 = Infinite improvement (baseline gets 0% accuracy)
2.0x = Tool-calling is 2x better than baseline
>2.0x = Strong improvement

Accuracy Heatmap:

Yellow/Green cells = High improvement (>2x)
Orange cells = Moderate improvement (1.5-2x)
Red cells = Minimal or no improvement (<1.5x)

Domain Detection Accuracy:

Target: >90% for reliable tool selection
Below 80% = LLM struggles with domain identification
May need prompt engineering or fine-tuning for difficult domains

Technical Architecture

Tool Registry Design

The 23+ tools are organized by domain with:

ToolDescription(
    name="arithmetic",
    function=lambda expr: fluxem.compute(expr),
    description="Evaluates arithmetic expressions with 100% accuracy",
    input_format="Arithmetic expression as string",
    output_format="Numeric result as float",
    example="arithmetic.compute('54 * 44') returns 2376.0",
    domain="arithmetic"
)

Qwen3 Integration

MLX Backend:

Uses fluxem.backend.BackendType.MLX
Apple Silicon GPU acceleration for FluxEM operations
Quantized models for memory efficiency

Domain Detection Strategy:

LLM-based (with Qwen3-4B MLX): Use model to select domain
Fallback: Pattern-based keyword matching (no MLX needed)
Hybrid: LLM detection with pattern validation

Prompt Engineering:

# Domain detection prompt (shown to LLM)
"Given this user question, determine which domain from the following list applies:
Options: arithmetic, physics, chemistry, biology, math, music, geometry, graphs, sets, logic, number_theory

If the question doesn't clearly fit any single domain, respond with 'none'.
Answer with just the domain name (lowercase), nothing else."

Response Generation

The system generates responses with tool usage information:

[Tool Usage]
Domain: arithmetic
Tool Execution Time: 23.5ms
Total Time: 68.7ms

54 * 44 = 2376.0

Troubleshooting

Common Issues

Model Loading Errors

Error: Model not loaded. Please check model path.

Solution: Verify MLX model path or run without MLX (uses fallback pattern-based detection).

Tool Execution Failures

Error: Tool call failed: ValueError: Invalid unit string

Solution: System gracefully falls back to baseline response when tools fail.

Domain Detection Issues

LLM detected 'none' for clear arithmetic question

Solution: The system attempts pattern-based fallback. Consider adding few-shot examples to improve detection.

Visualization Issues

Warning: matplotlib not available. Plots will be skipped.

Solution: Install matplotlib: pip install matplotlib for plots.

Performance Tips

For Best Results

Use MLX Backend: Apple Silicon optimization provides 10-100x speedup
Warm Start: Run a few warm-up queries before benchmarking
Temperature Settings: Use 0.6 for thinking mode (Qwen3 default)
Max Tokens: Keep at 2048 (default) for consistent comparison
Clear Caches: Between test runs, consider restarting for clean measurements

Debugging

Add --verbose flag to see:

Domain detection decisions
Tool execution details
Timing breakdown
Error messages

Extending the System

Adding New Domains

To add a new domain to FluxEM:

Create Encoder: Implement in fluxem/domains/<new_domain>/
Add Tool: Register in tool_registry.py
Add Tests: Add prompts to benchmark_data.py
Update Wrapper: Add parsing logic in qwen3_wrapper.py
Update Evaluator: Add domain-specific comparison logic

Adding New Tools

To add a new FluxEM tool to an existing domain:

Implement Function: Add method to domain encoder
Register Tool: Add new tool in tool_registry.py
Add Test Cases: Create benchmark prompts
Update Parsing: Add query extraction in qwen3_wrapper.py

Research Questions

This benchmark system enables investigation of:

Domain Detection Accuracy: How well does the LLM identify when to use which tool?
Tool-Calling Efficiency: Does the LLM successfully invoke tools?
Answer Quality: Are FluxEM tools producing accurate results?
Performance Trade-offs: What's the overhead of tool-calling vs speed?
Domain-Specific Performance: Which domains benefit most from tools?

Citation

If you find this work helpful, please cite:

@misc{fluxem-tool-calling-qwen3-4b-benchmark,
      title={FluxEM Tool-Calling with Qwen3-4B},
      author={FluxEM Project},
      year={2026},
      url={https://github.com/Hmbown/FluxEM},
      eprint={2700.01},
      archivePrefix={arXiv},
      primaryClass={cs.CL},
}

License

MIT License - See FluxEM project root.

Contact

For questions or issues:

GitHub: https://github.com/Hmbown/FluxEM/issues
FluxEM Documentation: See docs/ directory in project root