Instructions to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- llama.cpp
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with llama.cpp:
Install (macOS, Linux)
curl -LsSf https://llama.app/install.sh | sh # Start a local OpenAI-compatible server with a web UI: llama serve -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M # Run inference directly in the terminal: llama cli -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp # Start a local OpenAI-compatible server with a web UI: llama serve -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M # Run inference directly in the terminal: llama cli -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Use pre-built binary
# Download pre-built binary from: # https://github.com/ggerganov/llama.cpp/releases # Start a local OpenAI-compatible server with a web UI: ./llama-server -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M # Run inference directly in the terminal: ./llama-cli -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Build from source code
git clone https://github.com/ggerganov/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-server llama-cli # Start a local OpenAI-compatible server with a web UI: ./build/bin/llama-server -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M # Run inference directly in the terminal: ./build/bin/llama-cli -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Use Docker
docker model run hf.co/oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
- LM Studio
- Jan
- vLLM
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with vLLM:
Install from pip and serve model
# Install vLLM from pip: pip install vllm # Start the vLLM server: vllm serve "oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama" # Call the server using curl (OpenAI-compatible API): curl -X POST "http://localhost:8000/v1/chat/completions" \ -H "Content-Type: application/json" \ --data '{ "model": "oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'Use Docker
docker model run hf.co/oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
- Ollama
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with Ollama:
ollama run hf.co/oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
- Unsloth Desktop
- Pi
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with Pi:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Configure the model in Pi
# Install Pi: npm install -g @earendil-works/pi-coding-agent # Add to ~/.pi/agent/models.json: { "providers": { "llama-cpp": { "baseUrl": "http://localhost:8080/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M" } ] } } }Run Pi
# Start Pi in your project directory: pi
- Docker Model Runner
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with Docker Model Runner:
docker model run hf.co/oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
- Lemonade
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with Lemonade:
Pull the model
# Download Lemonade from https://lemonade-server.ai/ lemonade pull oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Run and chat with the model
lemonade run user.Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama-Q4_K_M
List all available models
lemonade list
- Hermes Agent
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with Hermes Agent:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Configure Hermes
# Install Hermes: curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup # Point Hermes at the local server: hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:8080/v1 hermes config set model.default oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Run Hermes
hermes
- Atomic Chat
- OpenClaw
How to use oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama with OpenClaw:
Start the llama.cpp server
# Install llama.cpp: brew install llama.cpp # Start a local OpenAI-compatible server: llama serve -hf oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M
Configure OpenClaw
# Install OpenClaw: npm install -g openclaw@latest # Register the local server and set it as the default model: openclaw onboard --non-interactive --mode local \ --auth-choice custom-api-key \ --custom-base-url http://127.0.0.1:8080/v1 \ --custom-model-id "oktayd/Qwen3.6-35B-v2-MoE-Ablit-Heretic-Uncensor-Hermes-MTP-Vision-Llama:Q4_K_M" \ --custom-provider-id llama-cpp \ --custom-compatibility openai \ --custom-text-input \ --accept-risk \ --skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Explain MoE and inherited stages; add per-test benchmark dropdowns and laptop capability roadmap
Browse files- BENCHMARK-TEST-COUNTS.json +1362 -0
- LAPTOP-CAPABILITY-PLAN.md +51 -0
- README.md +241 -1
- SHA256SUMS +3 -1
|
@@ -0,0 +1,1362 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
{
|
| 2 |
+
"schema_version": "q36.public-strict-counts.v1",
|
| 3 |
+
"source_sha256": "9aef69af26b03d3b26a9250a1f6c21fdd4972a63522806d7724454b66b820b17",
|
| 4 |
+
"scoring": "Original strict evaluator, not semantic regrade. Ungraded and incomplete may overlap.",
|
| 5 |
+
"runs": [
|
| 6 |
+
{
|
| 7 |
+
"run": "q36-final-q4_k_m-llamacpp-h200",
|
| 8 |
+
"model": "Q36",
|
| 9 |
+
"device": "H200",
|
| 10 |
+
"benchmarks": [
|
| 11 |
+
{
|
| 12 |
+
"benchmark": "ARC-Challenge",
|
| 13 |
+
"planned": 10,
|
| 14 |
+
"attempted": 10,
|
| 15 |
+
"completed": 8,
|
| 16 |
+
"scored": 8,
|
| 17 |
+
"passed": 8,
|
| 18 |
+
"ungraded": 2,
|
| 19 |
+
"not_run": 0,
|
| 20 |
+
"incomplete": 2
|
| 21 |
+
},
|
| 22 |
+
{
|
| 23 |
+
"benchmark": "BBH",
|
| 24 |
+
"planned": 23,
|
| 25 |
+
"attempted": 23,
|
| 26 |
+
"completed": 20,
|
| 27 |
+
"scored": 20,
|
| 28 |
+
"passed": 3,
|
| 29 |
+
"ungraded": 3,
|
| 30 |
+
"not_run": 0,
|
| 31 |
+
"incomplete": 3
|
| 32 |
+
},
|
| 33 |
+
{
|
| 34 |
+
"benchmark": "ChartQA",
|
| 35 |
+
"planned": 5,
|
| 36 |
+
"attempted": 5,
|
| 37 |
+
"completed": 5,
|
| 38 |
+
"scored": 5,
|
| 39 |
+
"passed": 4,
|
| 40 |
+
"ungraded": 0,
|
| 41 |
+
"not_run": 0,
|
| 42 |
+
"incomplete": 0
|
| 43 |
+
},
|
| 44 |
+
{
|
| 45 |
+
"benchmark": "GPQA-Diamond",
|
| 46 |
+
"planned": 5,
|
| 47 |
+
"attempted": 5,
|
| 48 |
+
"completed": 5,
|
| 49 |
+
"scored": 5,
|
| 50 |
+
"passed": 2,
|
| 51 |
+
"ungraded": 0,
|
| 52 |
+
"not_run": 0,
|
| 53 |
+
"incomplete": 0
|
| 54 |
+
},
|
| 55 |
+
{
|
| 56 |
+
"benchmark": "GSM8K",
|
| 57 |
+
"planned": 10,
|
| 58 |
+
"attempted": 10,
|
| 59 |
+
"completed": 9,
|
| 60 |
+
"scored": 9,
|
| 61 |
+
"passed": 5,
|
| 62 |
+
"ungraded": 1,
|
| 63 |
+
"not_run": 0,
|
| 64 |
+
"incomplete": 1
|
| 65 |
+
},
|
| 66 |
+
{
|
| 67 |
+
"benchmark": "HumanEval-Plus",
|
| 68 |
+
"planned": 5,
|
| 69 |
+
"attempted": 5,
|
| 70 |
+
"completed": 5,
|
| 71 |
+
"scored": 0,
|
| 72 |
+
"passed": 0,
|
| 73 |
+
"ungraded": 5,
|
| 74 |
+
"not_run": 0,
|
| 75 |
+
"incomplete": 0
|
| 76 |
+
},
|
| 77 |
+
{
|
| 78 |
+
"benchmark": "IFEval",
|
| 79 |
+
"planned": 10,
|
| 80 |
+
"attempted": 10,
|
| 81 |
+
"completed": 6,
|
| 82 |
+
"scored": 0,
|
| 83 |
+
"passed": 0,
|
| 84 |
+
"ungraded": 10,
|
| 85 |
+
"not_run": 0,
|
| 86 |
+
"incomplete": 4
|
| 87 |
+
},
|
| 88 |
+
{
|
| 89 |
+
"benchmark": "LiveCodeBench",
|
| 90 |
+
"planned": 9,
|
| 91 |
+
"attempted": 9,
|
| 92 |
+
"completed": 9,
|
| 93 |
+
"scored": 0,
|
| 94 |
+
"passed": 0,
|
| 95 |
+
"ungraded": 9,
|
| 96 |
+
"not_run": 0,
|
| 97 |
+
"incomplete": 0
|
| 98 |
+
},
|
| 99 |
+
{
|
| 100 |
+
"benchmark": "MATH-Level-5",
|
| 101 |
+
"planned": 10,
|
| 102 |
+
"attempted": 10,
|
| 103 |
+
"completed": 8,
|
| 104 |
+
"scored": 0,
|
| 105 |
+
"passed": 0,
|
| 106 |
+
"ungraded": 10,
|
| 107 |
+
"not_run": 0,
|
| 108 |
+
"incomplete": 2
|
| 109 |
+
},
|
| 110 |
+
{
|
| 111 |
+
"benchmark": "MBPP-Plus",
|
| 112 |
+
"planned": 5,
|
| 113 |
+
"attempted": 5,
|
| 114 |
+
"completed": 5,
|
| 115 |
+
"scored": 0,
|
| 116 |
+
"passed": 0,
|
| 117 |
+
"ungraded": 5,
|
| 118 |
+
"not_run": 0,
|
| 119 |
+
"incomplete": 0
|
| 120 |
+
},
|
| 121 |
+
{
|
| 122 |
+
"benchmark": "MMLU-Pro",
|
| 123 |
+
"planned": 1,
|
| 124 |
+
"attempted": 1,
|
| 125 |
+
"completed": 1,
|
| 126 |
+
"scored": 1,
|
| 127 |
+
"passed": 0,
|
| 128 |
+
"ungraded": 0,
|
| 129 |
+
"not_run": 0,
|
| 130 |
+
"incomplete": 0
|
| 131 |
+
},
|
| 132 |
+
{
|
| 133 |
+
"benchmark": "MMMU",
|
| 134 |
+
"planned": 6,
|
| 135 |
+
"attempted": 6,
|
| 136 |
+
"completed": 6,
|
| 137 |
+
"scored": 6,
|
| 138 |
+
"passed": 4,
|
| 139 |
+
"ungraded": 0,
|
| 140 |
+
"not_run": 0,
|
| 141 |
+
"incomplete": 0
|
| 142 |
+
},
|
| 143 |
+
{
|
| 144 |
+
"benchmark": "MathVista",
|
| 145 |
+
"planned": 5,
|
| 146 |
+
"attempted": 5,
|
| 147 |
+
"completed": 5,
|
| 148 |
+
"scored": 5,
|
| 149 |
+
"passed": 2,
|
| 150 |
+
"ungraded": 0,
|
| 151 |
+
"not_run": 0,
|
| 152 |
+
"incomplete": 0
|
| 153 |
+
},
|
| 154 |
+
{
|
| 155 |
+
"benchmark": "Q36-Agent-Function-Calling",
|
| 156 |
+
"planned": 5,
|
| 157 |
+
"attempted": 5,
|
| 158 |
+
"completed": 5,
|
| 159 |
+
"scored": 5,
|
| 160 |
+
"passed": 0,
|
| 161 |
+
"ungraded": 0,
|
| 162 |
+
"not_run": 0,
|
| 163 |
+
"incomplete": 0
|
| 164 |
+
},
|
| 165 |
+
{
|
| 166 |
+
"benchmark": "Q36-Answer-Termination-No-Looping",
|
| 167 |
+
"planned": 10,
|
| 168 |
+
"attempted": 10,
|
| 169 |
+
"completed": 10,
|
| 170 |
+
"scored": 10,
|
| 171 |
+
"passed": 7,
|
| 172 |
+
"ungraded": 0,
|
| 173 |
+
"not_run": 0,
|
| 174 |
+
"incomplete": 0
|
| 175 |
+
},
|
| 176 |
+
{
|
| 177 |
+
"benchmark": "Q36-Benign-Compliance-No-Overrefusal",
|
| 178 |
+
"planned": 20,
|
| 179 |
+
"attempted": 20,
|
| 180 |
+
"completed": 18,
|
| 181 |
+
"scored": 0,
|
| 182 |
+
"passed": 0,
|
| 183 |
+
"ungraded": 20,
|
| 184 |
+
"not_run": 0,
|
| 185 |
+
"incomplete": 2
|
| 186 |
+
},
|
| 187 |
+
{
|
| 188 |
+
"benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
|
| 189 |
+
"planned": 10,
|
| 190 |
+
"attempted": 10,
|
| 191 |
+
"completed": 10,
|
| 192 |
+
"scored": 10,
|
| 193 |
+
"passed": 10,
|
| 194 |
+
"ungraded": 0,
|
| 195 |
+
"not_run": 0,
|
| 196 |
+
"incomplete": 0
|
| 197 |
+
},
|
| 198 |
+
{
|
| 199 |
+
"benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
|
| 200 |
+
"planned": 10,
|
| 201 |
+
"attempted": 10,
|
| 202 |
+
"completed": 8,
|
| 203 |
+
"scored": 0,
|
| 204 |
+
"passed": 0,
|
| 205 |
+
"ungraded": 10,
|
| 206 |
+
"not_run": 0,
|
| 207 |
+
"incomplete": 2
|
| 208 |
+
},
|
| 209 |
+
{
|
| 210 |
+
"benchmark": "Q36-Direct-Style-and-Personality",
|
| 211 |
+
"planned": 10,
|
| 212 |
+
"attempted": 10,
|
| 213 |
+
"completed": 9,
|
| 214 |
+
"scored": 0,
|
| 215 |
+
"passed": 0,
|
| 216 |
+
"ungraded": 10,
|
| 217 |
+
"not_run": 0,
|
| 218 |
+
"incomplete": 1
|
| 219 |
+
},
|
| 220 |
+
{
|
| 221 |
+
"benchmark": "Q36-Hermes-Tool-Format",
|
| 222 |
+
"planned": 5,
|
| 223 |
+
"attempted": 5,
|
| 224 |
+
"completed": 5,
|
| 225 |
+
"scored": 5,
|
| 226 |
+
"passed": 5,
|
| 227 |
+
"ungraded": 0,
|
| 228 |
+
"not_run": 0,
|
| 229 |
+
"incomplete": 0
|
| 230 |
+
},
|
| 231 |
+
{
|
| 232 |
+
"benchmark": "Q36-JSON-Schema",
|
| 233 |
+
"planned": 5,
|
| 234 |
+
"attempted": 5,
|
| 235 |
+
"completed": 5,
|
| 236 |
+
"scored": 5,
|
| 237 |
+
"passed": 0,
|
| 238 |
+
"ungraded": 0,
|
| 239 |
+
"not_run": 0,
|
| 240 |
+
"incomplete": 0
|
| 241 |
+
},
|
| 242 |
+
{
|
| 243 |
+
"benchmark": "Q36-Legal-Alternatives-and-Boundaries",
|
| 244 |
+
"planned": 10,
|
| 245 |
+
"attempted": 10,
|
| 246 |
+
"completed": 10,
|
| 247 |
+
"scored": 0,
|
| 248 |
+
"passed": 0,
|
| 249 |
+
"ungraded": 10,
|
| 250 |
+
"not_run": 0,
|
| 251 |
+
"incomplete": 0
|
| 252 |
+
},
|
| 253 |
+
{
|
| 254 |
+
"benchmark": "Q36-Output-Integrity",
|
| 255 |
+
"planned": 5,
|
| 256 |
+
"attempted": 5,
|
| 257 |
+
"completed": 5,
|
| 258 |
+
"scored": 5,
|
| 259 |
+
"passed": 5,
|
| 260 |
+
"ungraded": 0,
|
| 261 |
+
"not_run": 0,
|
| 262 |
+
"incomplete": 0
|
| 263 |
+
},
|
| 264 |
+
{
|
| 265 |
+
"benchmark": "TruthfulQA",
|
| 266 |
+
"planned": 10,
|
| 267 |
+
"attempted": 10,
|
| 268 |
+
"completed": 9,
|
| 269 |
+
"scored": 9,
|
| 270 |
+
"passed": 6,
|
| 271 |
+
"ungraded": 1,
|
| 272 |
+
"not_run": 0,
|
| 273 |
+
"incomplete": 1
|
| 274 |
+
}
|
| 275 |
+
]
|
| 276 |
+
},
|
| 277 |
+
{
|
| 278 |
+
"run": "q36-final-q4_k_m-llamacpp-rtx5090",
|
| 279 |
+
"model": "Q36",
|
| 280 |
+
"device": "RTX 5090",
|
| 281 |
+
"benchmarks": [
|
| 282 |
+
{
|
| 283 |
+
"benchmark": "ARC-Challenge",
|
| 284 |
+
"planned": 10,
|
| 285 |
+
"attempted": 10,
|
| 286 |
+
"completed": 8,
|
| 287 |
+
"scored": 8,
|
| 288 |
+
"passed": 8,
|
| 289 |
+
"ungraded": 2,
|
| 290 |
+
"not_run": 0,
|
| 291 |
+
"incomplete": 2
|
| 292 |
+
},
|
| 293 |
+
{
|
| 294 |
+
"benchmark": "BBH",
|
| 295 |
+
"planned": 23,
|
| 296 |
+
"attempted": 23,
|
| 297 |
+
"completed": 20,
|
| 298 |
+
"scored": 20,
|
| 299 |
+
"passed": 3,
|
| 300 |
+
"ungraded": 3,
|
| 301 |
+
"not_run": 0,
|
| 302 |
+
"incomplete": 3
|
| 303 |
+
},
|
| 304 |
+
{
|
| 305 |
+
"benchmark": "ChartQA",
|
| 306 |
+
"planned": 5,
|
| 307 |
+
"attempted": 5,
|
| 308 |
+
"completed": 5,
|
| 309 |
+
"scored": 5,
|
| 310 |
+
"passed": 4,
|
| 311 |
+
"ungraded": 0,
|
| 312 |
+
"not_run": 0,
|
| 313 |
+
"incomplete": 0
|
| 314 |
+
},
|
| 315 |
+
{
|
| 316 |
+
"benchmark": "GPQA-Diamond",
|
| 317 |
+
"planned": 5,
|
| 318 |
+
"attempted": 5,
|
| 319 |
+
"completed": 5,
|
| 320 |
+
"scored": 5,
|
| 321 |
+
"passed": 1,
|
| 322 |
+
"ungraded": 0,
|
| 323 |
+
"not_run": 0,
|
| 324 |
+
"incomplete": 0
|
| 325 |
+
},
|
| 326 |
+
{
|
| 327 |
+
"benchmark": "GSM8K",
|
| 328 |
+
"planned": 10,
|
| 329 |
+
"attempted": 10,
|
| 330 |
+
"completed": 10,
|
| 331 |
+
"scored": 10,
|
| 332 |
+
"passed": 7,
|
| 333 |
+
"ungraded": 0,
|
| 334 |
+
"not_run": 0,
|
| 335 |
+
"incomplete": 0
|
| 336 |
+
},
|
| 337 |
+
{
|
| 338 |
+
"benchmark": "HumanEval-Plus",
|
| 339 |
+
"planned": 5,
|
| 340 |
+
"attempted": 5,
|
| 341 |
+
"completed": 4,
|
| 342 |
+
"scored": 0,
|
| 343 |
+
"passed": 0,
|
| 344 |
+
"ungraded": 5,
|
| 345 |
+
"not_run": 0,
|
| 346 |
+
"incomplete": 1
|
| 347 |
+
},
|
| 348 |
+
{
|
| 349 |
+
"benchmark": "IFEval",
|
| 350 |
+
"planned": 10,
|
| 351 |
+
"attempted": 10,
|
| 352 |
+
"completed": 8,
|
| 353 |
+
"scored": 0,
|
| 354 |
+
"passed": 0,
|
| 355 |
+
"ungraded": 10,
|
| 356 |
+
"not_run": 0,
|
| 357 |
+
"incomplete": 2
|
| 358 |
+
},
|
| 359 |
+
{
|
| 360 |
+
"benchmark": "LiveCodeBench",
|
| 361 |
+
"planned": 9,
|
| 362 |
+
"attempted": 9,
|
| 363 |
+
"completed": 9,
|
| 364 |
+
"scored": 0,
|
| 365 |
+
"passed": 0,
|
| 366 |
+
"ungraded": 9,
|
| 367 |
+
"not_run": 0,
|
| 368 |
+
"incomplete": 0
|
| 369 |
+
},
|
| 370 |
+
{
|
| 371 |
+
"benchmark": "MATH-Level-5",
|
| 372 |
+
"planned": 10,
|
| 373 |
+
"attempted": 10,
|
| 374 |
+
"completed": 9,
|
| 375 |
+
"scored": 0,
|
| 376 |
+
"passed": 0,
|
| 377 |
+
"ungraded": 10,
|
| 378 |
+
"not_run": 0,
|
| 379 |
+
"incomplete": 1
|
| 380 |
+
},
|
| 381 |
+
{
|
| 382 |
+
"benchmark": "MBPP-Plus",
|
| 383 |
+
"planned": 5,
|
| 384 |
+
"attempted": 5,
|
| 385 |
+
"completed": 5,
|
| 386 |
+
"scored": 0,
|
| 387 |
+
"passed": 0,
|
| 388 |
+
"ungraded": 5,
|
| 389 |
+
"not_run": 0,
|
| 390 |
+
"incomplete": 0
|
| 391 |
+
},
|
| 392 |
+
{
|
| 393 |
+
"benchmark": "MMLU-Pro",
|
| 394 |
+
"planned": 1,
|
| 395 |
+
"attempted": 1,
|
| 396 |
+
"completed": 1,
|
| 397 |
+
"scored": 1,
|
| 398 |
+
"passed": 0,
|
| 399 |
+
"ungraded": 0,
|
| 400 |
+
"not_run": 0,
|
| 401 |
+
"incomplete": 0
|
| 402 |
+
},
|
| 403 |
+
{
|
| 404 |
+
"benchmark": "MMMU",
|
| 405 |
+
"planned": 6,
|
| 406 |
+
"attempted": 6,
|
| 407 |
+
"completed": 6,
|
| 408 |
+
"scored": 6,
|
| 409 |
+
"passed": 4,
|
| 410 |
+
"ungraded": 0,
|
| 411 |
+
"not_run": 0,
|
| 412 |
+
"incomplete": 0
|
| 413 |
+
},
|
| 414 |
+
{
|
| 415 |
+
"benchmark": "MathVista",
|
| 416 |
+
"planned": 5,
|
| 417 |
+
"attempted": 5,
|
| 418 |
+
"completed": 5,
|
| 419 |
+
"scored": 5,
|
| 420 |
+
"passed": 2,
|
| 421 |
+
"ungraded": 0,
|
| 422 |
+
"not_run": 0,
|
| 423 |
+
"incomplete": 0
|
| 424 |
+
},
|
| 425 |
+
{
|
| 426 |
+
"benchmark": "Q36-Agent-Function-Calling",
|
| 427 |
+
"planned": 5,
|
| 428 |
+
"attempted": 5,
|
| 429 |
+
"completed": 5,
|
| 430 |
+
"scored": 5,
|
| 431 |
+
"passed": 0,
|
| 432 |
+
"ungraded": 0,
|
| 433 |
+
"not_run": 0,
|
| 434 |
+
"incomplete": 0
|
| 435 |
+
},
|
| 436 |
+
{
|
| 437 |
+
"benchmark": "Q36-Answer-Termination-No-Looping",
|
| 438 |
+
"planned": 10,
|
| 439 |
+
"attempted": 10,
|
| 440 |
+
"completed": 10,
|
| 441 |
+
"scored": 10,
|
| 442 |
+
"passed": 9,
|
| 443 |
+
"ungraded": 0,
|
| 444 |
+
"not_run": 0,
|
| 445 |
+
"incomplete": 0
|
| 446 |
+
},
|
| 447 |
+
{
|
| 448 |
+
"benchmark": "Q36-Benign-Compliance-No-Overrefusal",
|
| 449 |
+
"planned": 20,
|
| 450 |
+
"attempted": 20,
|
| 451 |
+
"completed": 18,
|
| 452 |
+
"scored": 0,
|
| 453 |
+
"passed": 0,
|
| 454 |
+
"ungraded": 20,
|
| 455 |
+
"not_run": 0,
|
| 456 |
+
"incomplete": 2
|
| 457 |
+
},
|
| 458 |
+
{
|
| 459 |
+
"benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
|
| 460 |
+
"planned": 10,
|
| 461 |
+
"attempted": 10,
|
| 462 |
+
"completed": 10,
|
| 463 |
+
"scored": 10,
|
| 464 |
+
"passed": 10,
|
| 465 |
+
"ungraded": 0,
|
| 466 |
+
"not_run": 0,
|
| 467 |
+
"incomplete": 0
|
| 468 |
+
},
|
| 469 |
+
{
|
| 470 |
+
"benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
|
| 471 |
+
"planned": 10,
|
| 472 |
+
"attempted": 10,
|
| 473 |
+
"completed": 8,
|
| 474 |
+
"scored": 0,
|
| 475 |
+
"passed": 0,
|
| 476 |
+
"ungraded": 10,
|
| 477 |
+
"not_run": 0,
|
| 478 |
+
"incomplete": 2
|
| 479 |
+
},
|
| 480 |
+
{
|
| 481 |
+
"benchmark": "Q36-Direct-Style-and-Personality",
|
| 482 |
+
"planned": 10,
|
| 483 |
+
"attempted": 10,
|
| 484 |
+
"completed": 10,
|
| 485 |
+
"scored": 0,
|
| 486 |
+
"passed": 0,
|
| 487 |
+
"ungraded": 10,
|
| 488 |
+
"not_run": 0,
|
| 489 |
+
"incomplete": 0
|
| 490 |
+
},
|
| 491 |
+
{
|
| 492 |
+
"benchmark": "Q36-Hermes-Tool-Format",
|
| 493 |
+
"planned": 5,
|
| 494 |
+
"attempted": 5,
|
| 495 |
+
"completed": 5,
|
| 496 |
+
"scored": 5,
|
| 497 |
+
"passed": 3,
|
| 498 |
+
"ungraded": 0,
|
| 499 |
+
"not_run": 0,
|
| 500 |
+
"incomplete": 0
|
| 501 |
+
},
|
| 502 |
+
{
|
| 503 |
+
"benchmark": "Q36-JSON-Schema",
|
| 504 |
+
"planned": 5,
|
| 505 |
+
"attempted": 5,
|
| 506 |
+
"completed": 5,
|
| 507 |
+
"scored": 5,
|
| 508 |
+
"passed": 0,
|
| 509 |
+
"ungraded": 0,
|
| 510 |
+
"not_run": 0,
|
| 511 |
+
"incomplete": 0
|
| 512 |
+
},
|
| 513 |
+
{
|
| 514 |
+
"benchmark": "Q36-Legal-Alternatives-and-Boundaries",
|
| 515 |
+
"planned": 10,
|
| 516 |
+
"attempted": 10,
|
| 517 |
+
"completed": 10,
|
| 518 |
+
"scored": 0,
|
| 519 |
+
"passed": 0,
|
| 520 |
+
"ungraded": 10,
|
| 521 |
+
"not_run": 0,
|
| 522 |
+
"incomplete": 0
|
| 523 |
+
},
|
| 524 |
+
{
|
| 525 |
+
"benchmark": "Q36-Output-Integrity",
|
| 526 |
+
"planned": 5,
|
| 527 |
+
"attempted": 5,
|
| 528 |
+
"completed": 5,
|
| 529 |
+
"scored": 5,
|
| 530 |
+
"passed": 5,
|
| 531 |
+
"ungraded": 0,
|
| 532 |
+
"not_run": 0,
|
| 533 |
+
"incomplete": 0
|
| 534 |
+
},
|
| 535 |
+
{
|
| 536 |
+
"benchmark": "TruthfulQA",
|
| 537 |
+
"planned": 10,
|
| 538 |
+
"attempted": 10,
|
| 539 |
+
"completed": 9,
|
| 540 |
+
"scored": 9,
|
| 541 |
+
"passed": 6,
|
| 542 |
+
"ungraded": 1,
|
| 543 |
+
"not_run": 0,
|
| 544 |
+
"incomplete": 1
|
| 545 |
+
}
|
| 546 |
+
]
|
| 547 |
+
},
|
| 548 |
+
{
|
| 549 |
+
"run": "huihui-original-q4_k_m-llamacpp-rtx5090",
|
| 550 |
+
"model": "Huihui",
|
| 551 |
+
"device": "RTX 5090",
|
| 552 |
+
"benchmarks": [
|
| 553 |
+
{
|
| 554 |
+
"benchmark": "ARC-Challenge",
|
| 555 |
+
"planned": 10,
|
| 556 |
+
"attempted": 10,
|
| 557 |
+
"completed": 10,
|
| 558 |
+
"scored": 10,
|
| 559 |
+
"passed": 10,
|
| 560 |
+
"ungraded": 0,
|
| 561 |
+
"not_run": 0,
|
| 562 |
+
"incomplete": 0
|
| 563 |
+
},
|
| 564 |
+
{
|
| 565 |
+
"benchmark": "BBH",
|
| 566 |
+
"planned": 23,
|
| 567 |
+
"attempted": 23,
|
| 568 |
+
"completed": 23,
|
| 569 |
+
"scored": 23,
|
| 570 |
+
"passed": 16,
|
| 571 |
+
"ungraded": 0,
|
| 572 |
+
"not_run": 0,
|
| 573 |
+
"incomplete": 0
|
| 574 |
+
},
|
| 575 |
+
{
|
| 576 |
+
"benchmark": "ChartQA",
|
| 577 |
+
"planned": 5,
|
| 578 |
+
"attempted": 5,
|
| 579 |
+
"completed": 5,
|
| 580 |
+
"scored": 5,
|
| 581 |
+
"passed": 5,
|
| 582 |
+
"ungraded": 0,
|
| 583 |
+
"not_run": 0,
|
| 584 |
+
"incomplete": 0
|
| 585 |
+
},
|
| 586 |
+
{
|
| 587 |
+
"benchmark": "GPQA-Diamond",
|
| 588 |
+
"planned": 5,
|
| 589 |
+
"attempted": 5,
|
| 590 |
+
"completed": 5,
|
| 591 |
+
"scored": 5,
|
| 592 |
+
"passed": 5,
|
| 593 |
+
"ungraded": 0,
|
| 594 |
+
"not_run": 0,
|
| 595 |
+
"incomplete": 0
|
| 596 |
+
},
|
| 597 |
+
{
|
| 598 |
+
"benchmark": "GSM8K",
|
| 599 |
+
"planned": 10,
|
| 600 |
+
"attempted": 10,
|
| 601 |
+
"completed": 10,
|
| 602 |
+
"scored": 10,
|
| 603 |
+
"passed": 7,
|
| 604 |
+
"ungraded": 0,
|
| 605 |
+
"not_run": 0,
|
| 606 |
+
"incomplete": 0
|
| 607 |
+
},
|
| 608 |
+
{
|
| 609 |
+
"benchmark": "HumanEval-Plus",
|
| 610 |
+
"planned": 5,
|
| 611 |
+
"attempted": 5,
|
| 612 |
+
"completed": 5,
|
| 613 |
+
"scored": 0,
|
| 614 |
+
"passed": 0,
|
| 615 |
+
"ungraded": 5,
|
| 616 |
+
"not_run": 0,
|
| 617 |
+
"incomplete": 0
|
| 618 |
+
},
|
| 619 |
+
{
|
| 620 |
+
"benchmark": "IFEval",
|
| 621 |
+
"planned": 10,
|
| 622 |
+
"attempted": 10,
|
| 623 |
+
"completed": 10,
|
| 624 |
+
"scored": 0,
|
| 625 |
+
"passed": 0,
|
| 626 |
+
"ungraded": 10,
|
| 627 |
+
"not_run": 0,
|
| 628 |
+
"incomplete": 0
|
| 629 |
+
},
|
| 630 |
+
{
|
| 631 |
+
"benchmark": "LiveCodeBench",
|
| 632 |
+
"planned": 9,
|
| 633 |
+
"attempted": 9,
|
| 634 |
+
"completed": 4,
|
| 635 |
+
"scored": 0,
|
| 636 |
+
"passed": 0,
|
| 637 |
+
"ungraded": 9,
|
| 638 |
+
"not_run": 0,
|
| 639 |
+
"incomplete": 5
|
| 640 |
+
},
|
| 641 |
+
{
|
| 642 |
+
"benchmark": "MATH-Level-5",
|
| 643 |
+
"planned": 10,
|
| 644 |
+
"attempted": 10,
|
| 645 |
+
"completed": 8,
|
| 646 |
+
"scored": 0,
|
| 647 |
+
"passed": 0,
|
| 648 |
+
"ungraded": 10,
|
| 649 |
+
"not_run": 0,
|
| 650 |
+
"incomplete": 2
|
| 651 |
+
},
|
| 652 |
+
{
|
| 653 |
+
"benchmark": "MBPP-Plus",
|
| 654 |
+
"planned": 5,
|
| 655 |
+
"attempted": 5,
|
| 656 |
+
"completed": 5,
|
| 657 |
+
"scored": 0,
|
| 658 |
+
"passed": 0,
|
| 659 |
+
"ungraded": 5,
|
| 660 |
+
"not_run": 0,
|
| 661 |
+
"incomplete": 0
|
| 662 |
+
},
|
| 663 |
+
{
|
| 664 |
+
"benchmark": "MMLU-Pro",
|
| 665 |
+
"planned": 1,
|
| 666 |
+
"attempted": 1,
|
| 667 |
+
"completed": 1,
|
| 668 |
+
"scored": 1,
|
| 669 |
+
"passed": 0,
|
| 670 |
+
"ungraded": 0,
|
| 671 |
+
"not_run": 0,
|
| 672 |
+
"incomplete": 0
|
| 673 |
+
},
|
| 674 |
+
{
|
| 675 |
+
"benchmark": "MMMU",
|
| 676 |
+
"planned": 6,
|
| 677 |
+
"attempted": 6,
|
| 678 |
+
"completed": 6,
|
| 679 |
+
"scored": 6,
|
| 680 |
+
"passed": 4,
|
| 681 |
+
"ungraded": 0,
|
| 682 |
+
"not_run": 0,
|
| 683 |
+
"incomplete": 0
|
| 684 |
+
},
|
| 685 |
+
{
|
| 686 |
+
"benchmark": "MathVista",
|
| 687 |
+
"planned": 5,
|
| 688 |
+
"attempted": 5,
|
| 689 |
+
"completed": 5,
|
| 690 |
+
"scored": 5,
|
| 691 |
+
"passed": 3,
|
| 692 |
+
"ungraded": 0,
|
| 693 |
+
"not_run": 0,
|
| 694 |
+
"incomplete": 0
|
| 695 |
+
},
|
| 696 |
+
{
|
| 697 |
+
"benchmark": "Q36-Agent-Function-Calling",
|
| 698 |
+
"planned": 5,
|
| 699 |
+
"attempted": 5,
|
| 700 |
+
"completed": 5,
|
| 701 |
+
"scored": 5,
|
| 702 |
+
"passed": 0,
|
| 703 |
+
"ungraded": 0,
|
| 704 |
+
"not_run": 0,
|
| 705 |
+
"incomplete": 0
|
| 706 |
+
},
|
| 707 |
+
{
|
| 708 |
+
"benchmark": "Q36-Answer-Termination-No-Looping",
|
| 709 |
+
"planned": 10,
|
| 710 |
+
"attempted": 10,
|
| 711 |
+
"completed": 10,
|
| 712 |
+
"scored": 10,
|
| 713 |
+
"passed": 10,
|
| 714 |
+
"ungraded": 0,
|
| 715 |
+
"not_run": 0,
|
| 716 |
+
"incomplete": 0
|
| 717 |
+
},
|
| 718 |
+
{
|
| 719 |
+
"benchmark": "Q36-Benign-Compliance-No-Overrefusal",
|
| 720 |
+
"planned": 20,
|
| 721 |
+
"attempted": 20,
|
| 722 |
+
"completed": 20,
|
| 723 |
+
"scored": 0,
|
| 724 |
+
"passed": 0,
|
| 725 |
+
"ungraded": 20,
|
| 726 |
+
"not_run": 0,
|
| 727 |
+
"incomplete": 0
|
| 728 |
+
},
|
| 729 |
+
{
|
| 730 |
+
"benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
|
| 731 |
+
"planned": 10,
|
| 732 |
+
"attempted": 10,
|
| 733 |
+
"completed": 10,
|
| 734 |
+
"scored": 10,
|
| 735 |
+
"passed": 10,
|
| 736 |
+
"ungraded": 0,
|
| 737 |
+
"not_run": 0,
|
| 738 |
+
"incomplete": 0
|
| 739 |
+
},
|
| 740 |
+
{
|
| 741 |
+
"benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
|
| 742 |
+
"planned": 10,
|
| 743 |
+
"attempted": 10,
|
| 744 |
+
"completed": 10,
|
| 745 |
+
"scored": 0,
|
| 746 |
+
"passed": 0,
|
| 747 |
+
"ungraded": 10,
|
| 748 |
+
"not_run": 0,
|
| 749 |
+
"incomplete": 0
|
| 750 |
+
},
|
| 751 |
+
{
|
| 752 |
+
"benchmark": "Q36-Direct-Style-and-Personality",
|
| 753 |
+
"planned": 10,
|
| 754 |
+
"attempted": 10,
|
| 755 |
+
"completed": 10,
|
| 756 |
+
"scored": 0,
|
| 757 |
+
"passed": 0,
|
| 758 |
+
"ungraded": 10,
|
| 759 |
+
"not_run": 0,
|
| 760 |
+
"incomplete": 0
|
| 761 |
+
},
|
| 762 |
+
{
|
| 763 |
+
"benchmark": "Q36-Hermes-Tool-Format",
|
| 764 |
+
"planned": 5,
|
| 765 |
+
"attempted": 5,
|
| 766 |
+
"completed": 5,
|
| 767 |
+
"scored": 5,
|
| 768 |
+
"passed": 4,
|
| 769 |
+
"ungraded": 0,
|
| 770 |
+
"not_run": 0,
|
| 771 |
+
"incomplete": 0
|
| 772 |
+
},
|
| 773 |
+
{
|
| 774 |
+
"benchmark": "Q36-JSON-Schema",
|
| 775 |
+
"planned": 5,
|
| 776 |
+
"attempted": 5,
|
| 777 |
+
"completed": 5,
|
| 778 |
+
"scored": 5,
|
| 779 |
+
"passed": 2,
|
| 780 |
+
"ungraded": 0,
|
| 781 |
+
"not_run": 0,
|
| 782 |
+
"incomplete": 0
|
| 783 |
+
},
|
| 784 |
+
{
|
| 785 |
+
"benchmark": "Q36-Legal-Alternatives-and-Boundaries",
|
| 786 |
+
"planned": 10,
|
| 787 |
+
"attempted": 10,
|
| 788 |
+
"completed": 10,
|
| 789 |
+
"scored": 0,
|
| 790 |
+
"passed": 0,
|
| 791 |
+
"ungraded": 10,
|
| 792 |
+
"not_run": 0,
|
| 793 |
+
"incomplete": 0
|
| 794 |
+
},
|
| 795 |
+
{
|
| 796 |
+
"benchmark": "Q36-Output-Integrity",
|
| 797 |
+
"planned": 5,
|
| 798 |
+
"attempted": 5,
|
| 799 |
+
"completed": 5,
|
| 800 |
+
"scored": 5,
|
| 801 |
+
"passed": 5,
|
| 802 |
+
"ungraded": 0,
|
| 803 |
+
"not_run": 0,
|
| 804 |
+
"incomplete": 0
|
| 805 |
+
},
|
| 806 |
+
{
|
| 807 |
+
"benchmark": "TruthfulQA",
|
| 808 |
+
"planned": 10,
|
| 809 |
+
"attempted": 10,
|
| 810 |
+
"completed": 9,
|
| 811 |
+
"scored": 9,
|
| 812 |
+
"passed": 6,
|
| 813 |
+
"ungraded": 1,
|
| 814 |
+
"not_run": 0,
|
| 815 |
+
"incomplete": 1
|
| 816 |
+
}
|
| 817 |
+
]
|
| 818 |
+
},
|
| 819 |
+
{
|
| 820 |
+
"run": "q36-final-q4_k_m-llamacpp-rtx2000ada-mixed24",
|
| 821 |
+
"model": "Q36",
|
| 822 |
+
"device": "RTX 2000 Ada (CPU+GPU)",
|
| 823 |
+
"benchmarks": [
|
| 824 |
+
{
|
| 825 |
+
"benchmark": "ARC-Challenge",
|
| 826 |
+
"planned": 10,
|
| 827 |
+
"attempted": 10,
|
| 828 |
+
"completed": 7,
|
| 829 |
+
"scored": 7,
|
| 830 |
+
"passed": 7,
|
| 831 |
+
"ungraded": 3,
|
| 832 |
+
"not_run": 0,
|
| 833 |
+
"incomplete": 3
|
| 834 |
+
},
|
| 835 |
+
{
|
| 836 |
+
"benchmark": "BBH",
|
| 837 |
+
"planned": 23,
|
| 838 |
+
"attempted": 23,
|
| 839 |
+
"completed": 20,
|
| 840 |
+
"scored": 20,
|
| 841 |
+
"passed": 3,
|
| 842 |
+
"ungraded": 3,
|
| 843 |
+
"not_run": 0,
|
| 844 |
+
"incomplete": 3
|
| 845 |
+
},
|
| 846 |
+
{
|
| 847 |
+
"benchmark": "ChartQA",
|
| 848 |
+
"planned": 5,
|
| 849 |
+
"attempted": 5,
|
| 850 |
+
"completed": 5,
|
| 851 |
+
"scored": 5,
|
| 852 |
+
"passed": 4,
|
| 853 |
+
"ungraded": 0,
|
| 854 |
+
"not_run": 0,
|
| 855 |
+
"incomplete": 0
|
| 856 |
+
},
|
| 857 |
+
{
|
| 858 |
+
"benchmark": "GPQA-Diamond",
|
| 859 |
+
"planned": 5,
|
| 860 |
+
"attempted": 5,
|
| 861 |
+
"completed": 5,
|
| 862 |
+
"scored": 5,
|
| 863 |
+
"passed": 2,
|
| 864 |
+
"ungraded": 0,
|
| 865 |
+
"not_run": 0,
|
| 866 |
+
"incomplete": 0
|
| 867 |
+
},
|
| 868 |
+
{
|
| 869 |
+
"benchmark": "GSM8K",
|
| 870 |
+
"planned": 10,
|
| 871 |
+
"attempted": 10,
|
| 872 |
+
"completed": 10,
|
| 873 |
+
"scored": 10,
|
| 874 |
+
"passed": 8,
|
| 875 |
+
"ungraded": 0,
|
| 876 |
+
"not_run": 0,
|
| 877 |
+
"incomplete": 0
|
| 878 |
+
},
|
| 879 |
+
{
|
| 880 |
+
"benchmark": "HumanEval-Plus",
|
| 881 |
+
"planned": 5,
|
| 882 |
+
"attempted": 5,
|
| 883 |
+
"completed": 5,
|
| 884 |
+
"scored": 0,
|
| 885 |
+
"passed": 0,
|
| 886 |
+
"ungraded": 5,
|
| 887 |
+
"not_run": 0,
|
| 888 |
+
"incomplete": 0
|
| 889 |
+
},
|
| 890 |
+
{
|
| 891 |
+
"benchmark": "IFEval",
|
| 892 |
+
"planned": 10,
|
| 893 |
+
"attempted": 10,
|
| 894 |
+
"completed": 9,
|
| 895 |
+
"scored": 0,
|
| 896 |
+
"passed": 0,
|
| 897 |
+
"ungraded": 10,
|
| 898 |
+
"not_run": 0,
|
| 899 |
+
"incomplete": 1
|
| 900 |
+
},
|
| 901 |
+
{
|
| 902 |
+
"benchmark": "LiveCodeBench",
|
| 903 |
+
"planned": 9,
|
| 904 |
+
"attempted": 9,
|
| 905 |
+
"completed": 9,
|
| 906 |
+
"scored": 0,
|
| 907 |
+
"passed": 0,
|
| 908 |
+
"ungraded": 9,
|
| 909 |
+
"not_run": 0,
|
| 910 |
+
"incomplete": 0
|
| 911 |
+
},
|
| 912 |
+
{
|
| 913 |
+
"benchmark": "MATH-Level-5",
|
| 914 |
+
"planned": 10,
|
| 915 |
+
"attempted": 10,
|
| 916 |
+
"completed": 9,
|
| 917 |
+
"scored": 0,
|
| 918 |
+
"passed": 0,
|
| 919 |
+
"ungraded": 10,
|
| 920 |
+
"not_run": 0,
|
| 921 |
+
"incomplete": 1
|
| 922 |
+
},
|
| 923 |
+
{
|
| 924 |
+
"benchmark": "MBPP-Plus",
|
| 925 |
+
"planned": 5,
|
| 926 |
+
"attempted": 5,
|
| 927 |
+
"completed": 5,
|
| 928 |
+
"scored": 0,
|
| 929 |
+
"passed": 0,
|
| 930 |
+
"ungraded": 5,
|
| 931 |
+
"not_run": 0,
|
| 932 |
+
"incomplete": 0
|
| 933 |
+
},
|
| 934 |
+
{
|
| 935 |
+
"benchmark": "MMLU-Pro",
|
| 936 |
+
"planned": 1,
|
| 937 |
+
"attempted": 1,
|
| 938 |
+
"completed": 1,
|
| 939 |
+
"scored": 1,
|
| 940 |
+
"passed": 0,
|
| 941 |
+
"ungraded": 0,
|
| 942 |
+
"not_run": 0,
|
| 943 |
+
"incomplete": 0
|
| 944 |
+
},
|
| 945 |
+
{
|
| 946 |
+
"benchmark": "MMMU",
|
| 947 |
+
"planned": 6,
|
| 948 |
+
"attempted": 6,
|
| 949 |
+
"completed": 6,
|
| 950 |
+
"scored": 6,
|
| 951 |
+
"passed": 1,
|
| 952 |
+
"ungraded": 0,
|
| 953 |
+
"not_run": 0,
|
| 954 |
+
"incomplete": 0
|
| 955 |
+
},
|
| 956 |
+
{
|
| 957 |
+
"benchmark": "MathVista",
|
| 958 |
+
"planned": 5,
|
| 959 |
+
"attempted": 5,
|
| 960 |
+
"completed": 5,
|
| 961 |
+
"scored": 5,
|
| 962 |
+
"passed": 3,
|
| 963 |
+
"ungraded": 0,
|
| 964 |
+
"not_run": 0,
|
| 965 |
+
"incomplete": 0
|
| 966 |
+
},
|
| 967 |
+
{
|
| 968 |
+
"benchmark": "Q36-Agent-Function-Calling",
|
| 969 |
+
"planned": 5,
|
| 970 |
+
"attempted": 5,
|
| 971 |
+
"completed": 5,
|
| 972 |
+
"scored": 5,
|
| 973 |
+
"passed": 0,
|
| 974 |
+
"ungraded": 0,
|
| 975 |
+
"not_run": 0,
|
| 976 |
+
"incomplete": 0
|
| 977 |
+
},
|
| 978 |
+
{
|
| 979 |
+
"benchmark": "Q36-Answer-Termination-No-Looping",
|
| 980 |
+
"planned": 10,
|
| 981 |
+
"attempted": 10,
|
| 982 |
+
"completed": 10,
|
| 983 |
+
"scored": 10,
|
| 984 |
+
"passed": 8,
|
| 985 |
+
"ungraded": 0,
|
| 986 |
+
"not_run": 0,
|
| 987 |
+
"incomplete": 0
|
| 988 |
+
},
|
| 989 |
+
{
|
| 990 |
+
"benchmark": "Q36-Benign-Compliance-No-Overrefusal",
|
| 991 |
+
"planned": 20,
|
| 992 |
+
"attempted": 20,
|
| 993 |
+
"completed": 19,
|
| 994 |
+
"scored": 0,
|
| 995 |
+
"passed": 0,
|
| 996 |
+
"ungraded": 20,
|
| 997 |
+
"not_run": 0,
|
| 998 |
+
"incomplete": 1
|
| 999 |
+
},
|
| 1000 |
+
{
|
| 1001 |
+
"benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
|
| 1002 |
+
"planned": 10,
|
| 1003 |
+
"attempted": 10,
|
| 1004 |
+
"completed": 10,
|
| 1005 |
+
"scored": 10,
|
| 1006 |
+
"passed": 9,
|
| 1007 |
+
"ungraded": 0,
|
| 1008 |
+
"not_run": 0,
|
| 1009 |
+
"incomplete": 0
|
| 1010 |
+
},
|
| 1011 |
+
{
|
| 1012 |
+
"benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
|
| 1013 |
+
"planned": 10,
|
| 1014 |
+
"attempted": 10,
|
| 1015 |
+
"completed": 9,
|
| 1016 |
+
"scored": 0,
|
| 1017 |
+
"passed": 0,
|
| 1018 |
+
"ungraded": 10,
|
| 1019 |
+
"not_run": 0,
|
| 1020 |
+
"incomplete": 1
|
| 1021 |
+
},
|
| 1022 |
+
{
|
| 1023 |
+
"benchmark": "Q36-Direct-Style-and-Personality",
|
| 1024 |
+
"planned": 10,
|
| 1025 |
+
"attempted": 10,
|
| 1026 |
+
"completed": 10,
|
| 1027 |
+
"scored": 0,
|
| 1028 |
+
"passed": 0,
|
| 1029 |
+
"ungraded": 10,
|
| 1030 |
+
"not_run": 0,
|
| 1031 |
+
"incomplete": 0
|
| 1032 |
+
},
|
| 1033 |
+
{
|
| 1034 |
+
"benchmark": "Q36-Hermes-Tool-Format",
|
| 1035 |
+
"planned": 5,
|
| 1036 |
+
"attempted": 5,
|
| 1037 |
+
"completed": 5,
|
| 1038 |
+
"scored": 5,
|
| 1039 |
+
"passed": 5,
|
| 1040 |
+
"ungraded": 0,
|
| 1041 |
+
"not_run": 0,
|
| 1042 |
+
"incomplete": 0
|
| 1043 |
+
},
|
| 1044 |
+
{
|
| 1045 |
+
"benchmark": "Q36-JSON-Schema",
|
| 1046 |
+
"planned": 5,
|
| 1047 |
+
"attempted": 5,
|
| 1048 |
+
"completed": 5,
|
| 1049 |
+
"scored": 5,
|
| 1050 |
+
"passed": 0,
|
| 1051 |
+
"ungraded": 0,
|
| 1052 |
+
"not_run": 0,
|
| 1053 |
+
"incomplete": 0
|
| 1054 |
+
},
|
| 1055 |
+
{
|
| 1056 |
+
"benchmark": "Q36-Legal-Alternatives-and-Boundaries",
|
| 1057 |
+
"planned": 10,
|
| 1058 |
+
"attempted": 10,
|
| 1059 |
+
"completed": 10,
|
| 1060 |
+
"scored": 0,
|
| 1061 |
+
"passed": 0,
|
| 1062 |
+
"ungraded": 10,
|
| 1063 |
+
"not_run": 0,
|
| 1064 |
+
"incomplete": 0
|
| 1065 |
+
},
|
| 1066 |
+
{
|
| 1067 |
+
"benchmark": "Q36-Output-Integrity",
|
| 1068 |
+
"planned": 5,
|
| 1069 |
+
"attempted": 5,
|
| 1070 |
+
"completed": 5,
|
| 1071 |
+
"scored": 5,
|
| 1072 |
+
"passed": 5,
|
| 1073 |
+
"ungraded": 0,
|
| 1074 |
+
"not_run": 0,
|
| 1075 |
+
"incomplete": 0
|
| 1076 |
+
},
|
| 1077 |
+
{
|
| 1078 |
+
"benchmark": "TruthfulQA",
|
| 1079 |
+
"planned": 10,
|
| 1080 |
+
"attempted": 10,
|
| 1081 |
+
"completed": 10,
|
| 1082 |
+
"scored": 10,
|
| 1083 |
+
"passed": 6,
|
| 1084 |
+
"ungraded": 0,
|
| 1085 |
+
"not_run": 0,
|
| 1086 |
+
"incomplete": 0
|
| 1087 |
+
}
|
| 1088 |
+
]
|
| 1089 |
+
},
|
| 1090 |
+
{
|
| 1091 |
+
"run": "huihui-original-q4_k_m-llamacpp-rtx2000ada-mixed24",
|
| 1092 |
+
"model": "Huihui",
|
| 1093 |
+
"device": "RTX 2000 Ada (CPU+GPU)",
|
| 1094 |
+
"benchmarks": [
|
| 1095 |
+
{
|
| 1096 |
+
"benchmark": "ARC-Challenge",
|
| 1097 |
+
"planned": 10,
|
| 1098 |
+
"attempted": 10,
|
| 1099 |
+
"completed": 10,
|
| 1100 |
+
"scored": 10,
|
| 1101 |
+
"passed": 10,
|
| 1102 |
+
"ungraded": 0,
|
| 1103 |
+
"not_run": 0,
|
| 1104 |
+
"incomplete": 0
|
| 1105 |
+
},
|
| 1106 |
+
{
|
| 1107 |
+
"benchmark": "BBH",
|
| 1108 |
+
"planned": 23,
|
| 1109 |
+
"attempted": 11,
|
| 1110 |
+
"completed": 10,
|
| 1111 |
+
"scored": 10,
|
| 1112 |
+
"passed": 6,
|
| 1113 |
+
"ungraded": 1,
|
| 1114 |
+
"not_run": 12,
|
| 1115 |
+
"incomplete": 1
|
| 1116 |
+
},
|
| 1117 |
+
{
|
| 1118 |
+
"benchmark": "ChartQA",
|
| 1119 |
+
"planned": 5,
|
| 1120 |
+
"attempted": 5,
|
| 1121 |
+
"completed": 5,
|
| 1122 |
+
"scored": 5,
|
| 1123 |
+
"passed": 5,
|
| 1124 |
+
"ungraded": 0,
|
| 1125 |
+
"not_run": 0,
|
| 1126 |
+
"incomplete": 0
|
| 1127 |
+
},
|
| 1128 |
+
{
|
| 1129 |
+
"benchmark": "GPQA-Diamond",
|
| 1130 |
+
"planned": 5,
|
| 1131 |
+
"attempted": 5,
|
| 1132 |
+
"completed": 3,
|
| 1133 |
+
"scored": 3,
|
| 1134 |
+
"passed": 3,
|
| 1135 |
+
"ungraded": 2,
|
| 1136 |
+
"not_run": 0,
|
| 1137 |
+
"incomplete": 2
|
| 1138 |
+
},
|
| 1139 |
+
{
|
| 1140 |
+
"benchmark": "GSM8K",
|
| 1141 |
+
"planned": 10,
|
| 1142 |
+
"attempted": 10,
|
| 1143 |
+
"completed": 10,
|
| 1144 |
+
"scored": 10,
|
| 1145 |
+
"passed": 7,
|
| 1146 |
+
"ungraded": 0,
|
| 1147 |
+
"not_run": 0,
|
| 1148 |
+
"incomplete": 0
|
| 1149 |
+
},
|
| 1150 |
+
{
|
| 1151 |
+
"benchmark": "HumanEval-Plus",
|
| 1152 |
+
"planned": 5,
|
| 1153 |
+
"attempted": 5,
|
| 1154 |
+
"completed": 5,
|
| 1155 |
+
"scored": 0,
|
| 1156 |
+
"passed": 0,
|
| 1157 |
+
"ungraded": 5,
|
| 1158 |
+
"not_run": 0,
|
| 1159 |
+
"incomplete": 0
|
| 1160 |
+
},
|
| 1161 |
+
{
|
| 1162 |
+
"benchmark": "IFEval",
|
| 1163 |
+
"planned": 10,
|
| 1164 |
+
"attempted": 10,
|
| 1165 |
+
"completed": 10,
|
| 1166 |
+
"scored": 0,
|
| 1167 |
+
"passed": 0,
|
| 1168 |
+
"ungraded": 10,
|
| 1169 |
+
"not_run": 0,
|
| 1170 |
+
"incomplete": 0
|
| 1171 |
+
},
|
| 1172 |
+
{
|
| 1173 |
+
"benchmark": "LiveCodeBench",
|
| 1174 |
+
"planned": 9,
|
| 1175 |
+
"attempted": 9,
|
| 1176 |
+
"completed": 5,
|
| 1177 |
+
"scored": 0,
|
| 1178 |
+
"passed": 0,
|
| 1179 |
+
"ungraded": 9,
|
| 1180 |
+
"not_run": 0,
|
| 1181 |
+
"incomplete": 4
|
| 1182 |
+
},
|
| 1183 |
+
{
|
| 1184 |
+
"benchmark": "MATH-Level-5",
|
| 1185 |
+
"planned": 10,
|
| 1186 |
+
"attempted": 10,
|
| 1187 |
+
"completed": 5,
|
| 1188 |
+
"scored": 0,
|
| 1189 |
+
"passed": 0,
|
| 1190 |
+
"ungraded": 10,
|
| 1191 |
+
"not_run": 0,
|
| 1192 |
+
"incomplete": 5
|
| 1193 |
+
},
|
| 1194 |
+
{
|
| 1195 |
+
"benchmark": "MBPP-Plus",
|
| 1196 |
+
"planned": 5,
|
| 1197 |
+
"attempted": 5,
|
| 1198 |
+
"completed": 5,
|
| 1199 |
+
"scored": 0,
|
| 1200 |
+
"passed": 0,
|
| 1201 |
+
"ungraded": 5,
|
| 1202 |
+
"not_run": 0,
|
| 1203 |
+
"incomplete": 0
|
| 1204 |
+
},
|
| 1205 |
+
{
|
| 1206 |
+
"benchmark": "MMLU-Pro",
|
| 1207 |
+
"planned": 1,
|
| 1208 |
+
"attempted": 1,
|
| 1209 |
+
"completed": 0,
|
| 1210 |
+
"scored": 0,
|
| 1211 |
+
"passed": 0,
|
| 1212 |
+
"ungraded": 1,
|
| 1213 |
+
"not_run": 0,
|
| 1214 |
+
"incomplete": 1
|
| 1215 |
+
},
|
| 1216 |
+
{
|
| 1217 |
+
"benchmark": "MMMU",
|
| 1218 |
+
"planned": 6,
|
| 1219 |
+
"attempted": 6,
|
| 1220 |
+
"completed": 6,
|
| 1221 |
+
"scored": 6,
|
| 1222 |
+
"passed": 4,
|
| 1223 |
+
"ungraded": 0,
|
| 1224 |
+
"not_run": 0,
|
| 1225 |
+
"incomplete": 0
|
| 1226 |
+
},
|
| 1227 |
+
{
|
| 1228 |
+
"benchmark": "MathVista",
|
| 1229 |
+
"planned": 5,
|
| 1230 |
+
"attempted": 5,
|
| 1231 |
+
"completed": 5,
|
| 1232 |
+
"scored": 5,
|
| 1233 |
+
"passed": 3,
|
| 1234 |
+
"ungraded": 0,
|
| 1235 |
+
"not_run": 0,
|
| 1236 |
+
"incomplete": 0
|
| 1237 |
+
},
|
| 1238 |
+
{
|
| 1239 |
+
"benchmark": "Q36-Agent-Function-Calling",
|
| 1240 |
+
"planned": 5,
|
| 1241 |
+
"attempted": 5,
|
| 1242 |
+
"completed": 5,
|
| 1243 |
+
"scored": 5,
|
| 1244 |
+
"passed": 0,
|
| 1245 |
+
"ungraded": 0,
|
| 1246 |
+
"not_run": 0,
|
| 1247 |
+
"incomplete": 0
|
| 1248 |
+
},
|
| 1249 |
+
{
|
| 1250 |
+
"benchmark": "Q36-Answer-Termination-No-Looping",
|
| 1251 |
+
"planned": 10,
|
| 1252 |
+
"attempted": 10,
|
| 1253 |
+
"completed": 10,
|
| 1254 |
+
"scored": 10,
|
| 1255 |
+
"passed": 10,
|
| 1256 |
+
"ungraded": 0,
|
| 1257 |
+
"not_run": 0,
|
| 1258 |
+
"incomplete": 0
|
| 1259 |
+
},
|
| 1260 |
+
{
|
| 1261 |
+
"benchmark": "Q36-Benign-Compliance-No-Overrefusal",
|
| 1262 |
+
"planned": 20,
|
| 1263 |
+
"attempted": 10,
|
| 1264 |
+
"completed": 10,
|
| 1265 |
+
"scored": 0,
|
| 1266 |
+
"passed": 0,
|
| 1267 |
+
"ungraded": 10,
|
| 1268 |
+
"not_run": 10,
|
| 1269 |
+
"incomplete": 0
|
| 1270 |
+
},
|
| 1271 |
+
{
|
| 1272 |
+
"benchmark": "Q36-CAPTCHA-Detection-and-Handoff",
|
| 1273 |
+
"planned": 10,
|
| 1274 |
+
"attempted": 10,
|
| 1275 |
+
"completed": 10,
|
| 1276 |
+
"scored": 10,
|
| 1277 |
+
"passed": 10,
|
| 1278 |
+
"ungraded": 0,
|
| 1279 |
+
"not_run": 0,
|
| 1280 |
+
"incomplete": 0
|
| 1281 |
+
},
|
| 1282 |
+
{
|
| 1283 |
+
"benchmark": "Q36-Contradiction-and-Anti-Sycophancy",
|
| 1284 |
+
"planned": 10,
|
| 1285 |
+
"attempted": 10,
|
| 1286 |
+
"completed": 10,
|
| 1287 |
+
"scored": 0,
|
| 1288 |
+
"passed": 0,
|
| 1289 |
+
"ungraded": 10,
|
| 1290 |
+
"not_run": 0,
|
| 1291 |
+
"incomplete": 0
|
| 1292 |
+
},
|
| 1293 |
+
{
|
| 1294 |
+
"benchmark": "Q36-Direct-Style-and-Personality",
|
| 1295 |
+
"planned": 10,
|
| 1296 |
+
"attempted": 10,
|
| 1297 |
+
"completed": 10,
|
| 1298 |
+
"scored": 0,
|
| 1299 |
+
"passed": 0,
|
| 1300 |
+
"ungraded": 10,
|
| 1301 |
+
"not_run": 0,
|
| 1302 |
+
"incomplete": 0
|
| 1303 |
+
},
|
| 1304 |
+
{
|
| 1305 |
+
"benchmark": "Q36-Hermes-Tool-Format",
|
| 1306 |
+
"planned": 5,
|
| 1307 |
+
"attempted": 5,
|
| 1308 |
+
"completed": 5,
|
| 1309 |
+
"scored": 5,
|
| 1310 |
+
"passed": 3,
|
| 1311 |
+
"ungraded": 0,
|
| 1312 |
+
"not_run": 0,
|
| 1313 |
+
"incomplete": 0
|
| 1314 |
+
},
|
| 1315 |
+
{
|
| 1316 |
+
"benchmark": "Q36-JSON-Schema",
|
| 1317 |
+
"planned": 5,
|
| 1318 |
+
"attempted": 5,
|
| 1319 |
+
"completed": 5,
|
| 1320 |
+
"scored": 5,
|
| 1321 |
+
"passed": 3,
|
| 1322 |
+
"ungraded": 0,
|
| 1323 |
+
"not_run": 0,
|
| 1324 |
+
"incomplete": 0
|
| 1325 |
+
},
|
| 1326 |
+
{
|
| 1327 |
+
"benchmark": "Q36-Legal-Alternatives-and-Boundaries",
|
| 1328 |
+
"planned": 10,
|
| 1329 |
+
"attempted": 10,
|
| 1330 |
+
"completed": 10,
|
| 1331 |
+
"scored": 0,
|
| 1332 |
+
"passed": 0,
|
| 1333 |
+
"ungraded": 10,
|
| 1334 |
+
"not_run": 0,
|
| 1335 |
+
"incomplete": 0
|
| 1336 |
+
},
|
| 1337 |
+
{
|
| 1338 |
+
"benchmark": "Q36-Output-Integrity",
|
| 1339 |
+
"planned": 5,
|
| 1340 |
+
"attempted": 5,
|
| 1341 |
+
"completed": 5,
|
| 1342 |
+
"scored": 5,
|
| 1343 |
+
"passed": 5,
|
| 1344 |
+
"ungraded": 0,
|
| 1345 |
+
"not_run": 0,
|
| 1346 |
+
"incomplete": 0
|
| 1347 |
+
},
|
| 1348 |
+
{
|
| 1349 |
+
"benchmark": "TruthfulQA",
|
| 1350 |
+
"planned": 10,
|
| 1351 |
+
"attempted": 10,
|
| 1352 |
+
"completed": 10,
|
| 1353 |
+
"scored": 10,
|
| 1354 |
+
"passed": 7,
|
| 1355 |
+
"ungraded": 0,
|
| 1356 |
+
"not_run": 0,
|
| 1357 |
+
"incomplete": 0
|
| 1358 |
+
}
|
| 1359 |
+
]
|
| 1360 |
+
}
|
| 1361 |
+
]
|
| 1362 |
+
}
|
|
@@ -0,0 +1,51 @@
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 1 |
+
# Laptop assistant: from quick tests to useful workflows
|
| 2 |
+
|
| 3 |
+
**Status: local preparation in progress; no laptop winner selected yet.** This is a deployment roadmap, not a claim that the full integration already works.
|
| 4 |
+
|
| 5 |
+
## π» Coding and agent skills β what we will actually test
|
| 6 |
+
|
| 7 |
+
| Track | Cases | Success evidence | Current status |
|
| 8 |
+
|---|---:|---|---|
|
| 9 |
+
| HumanEval+ | 5 | Generated functions pass the selected task's base and extended tests in an isolated container | Existing outputs; execution/grading must be reconciled |
|
| 10 |
+
| MBPP+ | 5 | Correct results on base and extended tests, including edge cases | Existing outputs; execution/grading must be reconciled |
|
| 11 |
+
| LiveCodeBench | 9 | Program passes task tests within sandbox limits | Existing outputs; execution/grading must be reconciled |
|
| 12 |
+
| Q36 bug fixes | 5 | Reproduce a failing test, apply a patch, pass regression tests | New tasks to prepare |
|
| 13 |
+
| Q36 small app / website tasks | 5 | Build succeeds; required UI behavior passes browser checks | New tasks to prepare |
|
| 14 |
+
| Tool choice and arguments | 5 | Correct tool and schema-valid arguments; no invented tool result | Local smoke fixtures prepared separately |
|
| 15 |
+
| Multi-step planning | 5 | Respect dependencies and reach a verifiable final state | New interactive fixtures to prepare |
|
| 16 |
+
| Recovery from tool errors | 5 | Respond to a simulated failure, choose a valid alternative, verify recovery | New interactive fixtures to prepare |
|
| 17 |
+
| Memory and conflicting evidence | 5 | Retrieve the correct note, cite it, distinguish updated from stale facts | New isolated test vault to prepare |
|
| 18 |
+
| Stopping and task completion | 5 | Stop after success; report missing prerequisites instead of inventing completion | New interactive fixtures to prepare |
|
| 19 |
+
| Transfer to changed constraints | 5 | Solve held-out variants after a changed rule or environment | New tasks to prepare |
|
| 20 |
+
|
| 21 |
+
**64 case slots in this extension**, including 19 existing coding slots. These are compact diagnostics, not full leaderboard scores. The agent tasks measure useful components of general problem solving, **not proof of AGI**. Existing Hermes-format and JSON-schema diagnostics remain in the broad suite and are not silently counted twice.
|
| 22 |
+
|
| 23 |
+
### How a capability earns a card entry
|
| 24 |
+
|
| 25 |
+
Freeze tasks and evaluators before comparing runtime settings. Count one primary attempt per case. Report **passes / executed-and-graded cases**, task coverage, failures and ungraded cases together. Publish results even when they are weak; only describe a workflow as demonstrated when its actual end-to-end task passes. A successful JSON tool-call test alone does not establish autonomous coding or desktop control.
|
| 26 |
+
|
| 27 |
+
Generated code runs in a disposable container without network access, host credentials or user-folder mounts, with CPU, RAM, process and time limits. Host tools and paid services are not available to benchmark prompts.
|
| 28 |
+
|
| 29 |
+
## β‘ Pick a laptop runtime
|
| 30 |
+
|
| 31 |
+
1. Smoke-test **IQ4_XS, Q3_K_M, IQ2_M and IQ1_M**; exclude the BF16 server package. Start with IQ2_M to check loading and the harness.
|
| 32 |
+
2. Compare **Ollama and llama.cpp**, each with a documented baseline and an optimized profile: 16 primary configurations. FreeToken is an additional experimental backend only if a compatible build actually runs.
|
| 33 |
+
3. Use the same prompts, image inputs, context, output cap and thinking setting. Tune only on a separate warm-up set, then freeze settings.
|
| 34 |
+
4. Record answer quality separately from prefill/decode tokens/s, end-to-end speed, time to first visible output, CPU/RAM/VRAM, temperatures, timeouts and incomplete outputs.
|
| 35 |
+
5. Select a measured quality/performance/memory compromise, not simply the fastest token rate. An OOM or unsupported backend is a technical failure, not a wrong answer.
|
| 36 |
+
|
| 37 |
+
The target laptop has **32 GB RAM and an RTX 4060 Laptop GPU with 8 GB VRAM**. These 35B-weight models need CPU/RAM offload. MoE reduces active computation, not the need to store the expert weights.
|
| 38 |
+
|
| 39 |
+
## π οΈ Hermes, routing and personal memory
|
| 40 |
+
|
| 41 |
+
After selecting the runtime, test it as a local endpoint behind the existing Hermes/router installation. Keep the previous route available for rollback. First use a dedicated test workspace and synthetic notes; connect the real Obsidian vault only after the test flow passes.
|
| 42 |
+
|
| 43 |
+
Useful scenarios to validate:
|
| 44 |
+
|
| 45 |
+
- **Code companion:** inspect a sample repository, fix one bug, run tests and summarize the patch.
|
| 46 |
+
- **Project memory:** find a note, cite its source and use its latest decision in a response.
|
| 47 |
+
- **Research notebook:** turn supplied material into a draft note with provenance, then ask before writing into the real vault.
|
| 48 |
+
- **Task recovery:** recognize an unavailable tool, ask for what is missing or use an explicitly allowed alternative.
|
| 49 |
+
- **Private local assistant:** use the configured local route and verify that external fallback is disabled for private inputs.
|
| 50 |
+
|
| 51 |
+
Obsidian storage, retrieval/indexing, memory updates and routing belong to the host application. They are not built into the model weights. Exact endpoint instructions and working configuration examples will be added after local validation; no production routing has been changed by this document.
|
|
@@ -12,7 +12,7 @@ tags:
|
|
| 12 |
- abliterated
|
| 13 |
- heretic
|
| 14 |
- uncensor
|
| 15 |
-
- hermes
|
| 16 |
- mtp
|
| 17 |
- coding
|
| 18 |
- tool-calling
|
|
@@ -46,6 +46,12 @@ Meet your local AI companion for ideas, code, images and conversation. **Qwen3.6
|
|
| 46 |
|
| 47 |
These are things to try, not promises of perfect results. It can make mistakes or repeat itself; check important answers. Adult-oriented training is included (**18+**).
|
| 48 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 49 |
## πΆοΈ Street knowledge. Business mind. Your style.
|
| 50 |
|
| 51 |
The personality direction is **direct, sharp-witted and business-minded**: a street-smart conversation partner with humor, creative confidence and room for disagreement. The training mix includes internet culture, slang, practical business topics and personality-oriented conversations. Think less formal textbook, more an opinionated partner for brainstorming, writing and exploring alternatives.
|
|
@@ -148,6 +154,219 @@ Huihui performed better on the shared automatically assessable RTX 5090 subset;
|
|
| 148 |
|
| 149 |
</details>
|
| 150 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 151 |
## π License, lineage & credits
|
| 152 |
|
| 153 |
[Apache-2.0 license](LICENSE). Dataset sources and their own license information are linked in [DATASETS.md](DATASETS.md); model licensing does not relicense the source datasets. Thanks to the Qwen team, the inherited model and dataset authors, and the Soup, PEFT, Transformers, llama.cpp and Ollama projects.
|
|
@@ -156,6 +375,27 @@ Huihui performed better on the shared automatically assessable RTX 5090 subset;
|
|
| 156 |
|
| 157 |
These names do not imply affiliation or a promise of unrestricted behavior. Full provenance and validation are retained, including the historical Opus4.7-labelled source. No universal superiority, guaranteed compliance or removal of memorization is claimed.
|
| 158 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 159 |
## π Related collection
|
| 160 |
|
| 161 |
The previous release remains separate: [Qwen3.6 Opus4.7 Heretic Hermes Agent β Editions](https://huggingface.co/collections/oktayd/qwen36-opus47-heretic-hermes-agent-editions-6a8d09f6c2eb42ed1b112184).
|
|
|
|
| 12 |
- abliterated
|
| 13 |
- heretic
|
| 14 |
- uncensor
|
| 15 |
+
- hermes-lineage
|
| 16 |
- mtp
|
| 17 |
- coding
|
| 18 |
- tool-calling
|
|
|
|
| 46 |
|
| 47 |
These are things to try, not promises of perfect results. It can make mistakes or repeat itself; check important answers. Adult-oriented training is included (**18+**).
|
| 48 |
|
| 49 |
+
## π§ A team of experts inside one model
|
| 50 |
+
|
| 51 |
+
**MoE means Mixture of Experts.** Think of a team of specialists: for each token, a router chooses which expert networks should contribute. This model has about **35 billion parameters in total**, with roughly **3 billion active per token**; its configuration selects **8 of 256 routed experts**.
|
| 52 |
+
|
| 53 |
+
That saves computation compared with activating every expert at once. It does **not** turn a 35B download into a 3B-sized model: the expert weights still need disk space and accessible RAM/VRAM. Quantized editions make local use more practical. The experts are learned networks, not separate installed apps or named profession-specific agents.
|
| 54 |
+
|
| 55 |
## πΆοΈ Street knowledge. Business mind. Your style.
|
| 56 |
|
| 57 |
The personality direction is **direct, sharp-witted and business-minded**: a street-smart conversation partner with humor, creative confidence and room for disagreement. The training mix includes internet culture, slang, practical business topics and personality-oriented conversations. Think less formal textbook, more an opinionated partner for brainstorming, writing and exploring alternatives.
|
|
|
|
| 154 |
|
| 155 |
</details>
|
| 156 |
|
| 157 |
+
<!-- q36-capability-expansion:start -->
|
| 158 |
+
## π Broad benchmark β tests and passes
|
| 159 |
+
|
| 160 |
+
Expand a device to inspect every test family. **Passes / graded** use the original strict evaluator, including format-sensitive checks. Ungraded answers are not failures or passes. Incomplete is a separate delivery flag and can overlap with ungraded. These are small local subsets, not official leaderboard scores.
|
| 161 |
+
|
| 162 |
+
<details>
|
| 163 |
+
<summary>Q36 Β· H200 β per-test breakdown</summary>
|
| 164 |
+
|
| 165 |
+
| Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
|
| 166 |
+
|---|---:|---:|---:|---:|---:|
|
| 167 |
+
| ARC-Challenge | 10 / 10 | 8 / 8 | 2 | 2 | 0 |
|
| 168 |
+
| BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
|
| 169 |
+
| ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
|
| 170 |
+
| GPQA-Diamond | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
|
| 171 |
+
| GSM8K | 10 / 10 | 5 / 9 | 1 | 1 | 0 |
|
| 172 |
+
| HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 173 |
+
| IFEval | 10 / 10 | 0 / 0 | 10 | 4 | 0 |
|
| 174 |
+
| LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
|
| 175 |
+
| MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
|
| 176 |
+
| MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 177 |
+
| MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
|
| 178 |
+
| MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
|
| 179 |
+
| MathVista | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
|
| 180 |
+
| Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 181 |
+
| Q36-Answer-Termination-No-Looping | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
|
| 182 |
+
| Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 2 | 0 |
|
| 183 |
+
| Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 184 |
+
| Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
|
| 185 |
+
| Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
|
| 186 |
+
| Q36-Hermes-Tool-Format | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 187 |
+
| Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 188 |
+
| Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 189 |
+
| Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 190 |
+
| TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
|
| 191 |
+
|
| 192 |
+
</details>
|
| 193 |
+
|
| 194 |
+
<details>
|
| 195 |
+
<summary>Q36 Β· RTX 5090 β per-test breakdown</summary>
|
| 196 |
+
|
| 197 |
+
| Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
|
| 198 |
+
|---|---:|---:|---:|---:|---:|
|
| 199 |
+
| ARC-Challenge | 10 / 10 | 8 / 8 | 2 | 2 | 0 |
|
| 200 |
+
| BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
|
| 201 |
+
| ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
|
| 202 |
+
| GPQA-Diamond | 5 / 5 | 1 / 5 | 0 | 0 | 0 |
|
| 203 |
+
| GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
|
| 204 |
+
| HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 1 | 0 |
|
| 205 |
+
| IFEval | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
|
| 206 |
+
| LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
|
| 207 |
+
| MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
|
| 208 |
+
| MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 209 |
+
| MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
|
| 210 |
+
| MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
|
| 211 |
+
| MathVista | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
|
| 212 |
+
| Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 213 |
+
| Q36-Answer-Termination-No-Looping | 10 / 10 | 9 / 10 | 0 | 0 | 0 |
|
| 214 |
+
| Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 2 | 0 |
|
| 215 |
+
| Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 216 |
+
| Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
|
| 217 |
+
| Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 218 |
+
| Q36-Hermes-Tool-Format | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
|
| 219 |
+
| Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 220 |
+
| Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 221 |
+
| Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 222 |
+
| TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
|
| 223 |
+
|
| 224 |
+
</details>
|
| 225 |
+
|
| 226 |
+
<details>
|
| 227 |
+
<summary>Huihui Β· RTX 5090 β per-test breakdown</summary>
|
| 228 |
+
|
| 229 |
+
| Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
|
| 230 |
+
|---|---:|---:|---:|---:|---:|
|
| 231 |
+
| ARC-Challenge | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 232 |
+
| BBH | 23 / 23 | 16 / 23 | 0 | 0 | 0 |
|
| 233 |
+
| ChartQA | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 234 |
+
| GPQA-Diamond | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 235 |
+
| GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
|
| 236 |
+
| HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 237 |
+
| IFEval | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 238 |
+
| LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 5 | 0 |
|
| 239 |
+
| MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 2 | 0 |
|
| 240 |
+
| MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 241 |
+
| MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
|
| 242 |
+
| MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
|
| 243 |
+
| MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
|
| 244 |
+
| Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 245 |
+
| Q36-Answer-Termination-No-Looping | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 246 |
+
| Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 0 | 0 |
|
| 247 |
+
| Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 248 |
+
| Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 249 |
+
| Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 250 |
+
| Q36-Hermes-Tool-Format | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
|
| 251 |
+
| Q36-JSON-Schema | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
|
| 252 |
+
| Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 253 |
+
| Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 254 |
+
| TruthfulQA | 10 / 10 | 6 / 9 | 1 | 1 | 0 |
|
| 255 |
+
|
| 256 |
+
</details>
|
| 257 |
+
|
| 258 |
+
<details>
|
| 259 |
+
<summary>Q36 Β· RTX 2000 Ada (CPU+GPU) β per-test breakdown</summary>
|
| 260 |
+
|
| 261 |
+
| Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
|
| 262 |
+
|---|---:|---:|---:|---:|---:|
|
| 263 |
+
| ARC-Challenge | 10 / 10 | 7 / 7 | 3 | 3 | 0 |
|
| 264 |
+
| BBH | 23 / 23 | 3 / 20 | 3 | 3 | 0 |
|
| 265 |
+
| ChartQA | 5 / 5 | 4 / 5 | 0 | 0 | 0 |
|
| 266 |
+
| GPQA-Diamond | 5 / 5 | 2 / 5 | 0 | 0 | 0 |
|
| 267 |
+
| GSM8K | 10 / 10 | 8 / 10 | 0 | 0 | 0 |
|
| 268 |
+
| HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 269 |
+
| IFEval | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
|
| 270 |
+
| LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 0 | 0 |
|
| 271 |
+
| MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
|
| 272 |
+
| MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 273 |
+
| MMLU-Pro | 1 / 1 | 0 / 1 | 0 | 0 | 0 |
|
| 274 |
+
| MMMU | 6 / 6 | 1 / 6 | 0 | 0 | 0 |
|
| 275 |
+
| MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
|
| 276 |
+
| Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 277 |
+
| Q36-Answer-Termination-No-Looping | 10 / 10 | 8 / 10 | 0 | 0 | 0 |
|
| 278 |
+
| Q36-Benign-Compliance-No-Overrefusal | 20 / 20 | 0 / 0 | 20 | 1 | 0 |
|
| 279 |
+
| Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 9 / 10 | 0 | 0 | 0 |
|
| 280 |
+
| Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 1 | 0 |
|
| 281 |
+
| Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 282 |
+
| Q36-Hermes-Tool-Format | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 283 |
+
| Q36-JSON-Schema | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 284 |
+
| Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 285 |
+
| Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 286 |
+
| TruthfulQA | 10 / 10 | 6 / 10 | 0 | 0 | 0 |
|
| 287 |
+
|
| 288 |
+
</details>
|
| 289 |
+
|
| 290 |
+
<details>
|
| 291 |
+
<summary>Huihui Β· RTX 2000 Ada (CPU+GPU) β per-test breakdown</summary>
|
| 292 |
+
|
| 293 |
+
| Test family | Attempted / planned | Passes / graded | Ungraded | Incomplete | Not run |
|
| 294 |
+
|---|---:|---:|---:|---:|---:|
|
| 295 |
+
| ARC-Challenge | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 296 |
+
| BBH | 11 / 23 | 6 / 10 | 1 | 1 | 12 |
|
| 297 |
+
| ChartQA | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 298 |
+
| GPQA-Diamond | 5 / 5 | 3 / 3 | 2 | 2 | 0 |
|
| 299 |
+
| GSM8K | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
|
| 300 |
+
| HumanEval-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 301 |
+
| IFEval | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 302 |
+
| LiveCodeBench | 9 / 9 | 0 / 0 | 9 | 4 | 0 |
|
| 303 |
+
| MATH-Level-5 | 10 / 10 | 0 / 0 | 10 | 5 | 0 |
|
| 304 |
+
| MBPP-Plus | 5 / 5 | 0 / 0 | 5 | 0 | 0 |
|
| 305 |
+
| MMLU-Pro | 1 / 1 | 0 / 0 | 1 | 1 | 0 |
|
| 306 |
+
| MMMU | 6 / 6 | 4 / 6 | 0 | 0 | 0 |
|
| 307 |
+
| MathVista | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
|
| 308 |
+
| Q36-Agent-Function-Calling | 5 / 5 | 0 / 5 | 0 | 0 | 0 |
|
| 309 |
+
| Q36-Answer-Termination-No-Looping | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 310 |
+
| Q36-Benign-Compliance-No-Overrefusal | 10 / 20 | 0 / 0 | 10 | 0 | 10 |
|
| 311 |
+
| Q36-CAPTCHA-Detection-and-Handoff | 10 / 10 | 10 / 10 | 0 | 0 | 0 |
|
| 312 |
+
| Q36-Contradiction-and-Anti-Sycophancy | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 313 |
+
| Q36-Direct-Style-and-Personality | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 314 |
+
| Q36-Hermes-Tool-Format | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
|
| 315 |
+
| Q36-JSON-Schema | 5 / 5 | 3 / 5 | 0 | 0 | 0 |
|
| 316 |
+
| Q36-Legal-Alternatives-and-Boundaries | 10 / 10 | 0 / 0 | 10 | 0 | 0 |
|
| 317 |
+
| Q36-Output-Integrity | 5 / 5 | 5 / 5 | 0 | 0 | 0 |
|
| 318 |
+
| TruthfulQA | 10 / 10 | 7 / 10 | 0 | 0 | 0 |
|
| 319 |
+
|
| 320 |
+
</details>
|
| 321 |
+
|
| 322 |
+
[Machine-readable counts](BENCHMARK-TEST-COUNTS.json). Code generation is not a pass until the relevant execution tests have been graded.
|
| 323 |
+
|
| 324 |
+
## π Hard benchmarks on the roadmap
|
| 325 |
+
|
| 326 |
+
All **25 families** below are registered from the [Qwen3.8-27B card](https://huggingface.co/Qwen/Qwen3.8-27B). The matching official adapters and datasets are not yet fully prepared, and **no local Qwen3.8 baseline has been run**. Existing short similarly named diagnostics do not substitute for those runs.
|
| 327 |
+
|
| 328 |
+
<details>
|
| 329 |
+
<summary>Coding, reasoning, agent and visual benchmarks β complete planned list</summary>
|
| 330 |
+
|
| 331 |
+
| Benchmark | Planned local cases | Preparation status |
|
| 332 |
+
|---|---:|---|
|
| 333 |
+
| Terminal Bench 2.1 (Terminus) | 5 | Adapter/data preparation pending |
|
| 334 |
+
| SWE-bench Pro | 5 | Adapter/data preparation pending |
|
| 335 |
+
| NL2Repo-Bench | 5 | Adapter/data preparation pending |
|
| 336 |
+
| DeepSWE 1.1 | 5 | Adapter/data preparation pending |
|
| 337 |
+
| QwenSWEBench | TBD | Internal release/access needed |
|
| 338 |
+
| CoWorkBench | TBD | Internal release/access needed |
|
| 339 |
+
| JobBench | 5 | Adapter/data preparation pending |
|
| 340 |
+
| Agents' Last Exam | 5 | Adapter/data preparation pending |
|
| 341 |
+
| IFBench | 20 | Adapter/data preparation pending |
|
| 342 |
+
| GPQA Diamond | 20 | Adapter/data preparation pending |
|
| 343 |
+
| HLE | 10 | Adapter/data preparation pending |
|
| 344 |
+
| LiveCodeBench v6 | 10 | Adapter/data preparation pending |
|
| 345 |
+
| OSWorld-Verified | 5 | Adapter/data preparation pending |
|
| 346 |
+
| WebArena-Verified | 5 | Adapter/data preparation pending |
|
| 347 |
+
| AndroidWorld | 5 | Adapter/data preparation pending |
|
| 348 |
+
| RecreationBench | TBD | Internal release/access needed |
|
| 349 |
+
| ClawEval-MM | 5 | Adapter/data preparation pending |
|
| 350 |
+
| SWE-MM | 5 | Adapter/data preparation pending |
|
| 351 |
+
| Vision2Web | 5 | Adapter/data preparation pending |
|
| 352 |
+
| MathVision | 10 | Adapter/data preparation pending |
|
| 353 |
+
| BabyVision | 10 | Adapter/data preparation pending |
|
| 354 |
+
| CharXiv (RQ) | 10 | Adapter/data preparation pending |
|
| 355 |
+
| OmniDocBench 1.5 | 10 | Adapter/data preparation pending |
|
| 356 |
+
| RealWorldQA | 20 | Adapter/data preparation pending |
|
| 357 |
+
| ERQA | 10 | Adapter/data preparation pending |
|
| 358 |
+
|
| 359 |
+
</details>
|
| 360 |
+
|
| 361 |
+
Comparison graphics will follow measured results, with our local samples and publisher-reported scores kept clearly separate. Different harnesses, budgets and trial counts will not be presented as a head-to-head win. [Full protocol and references](BENCHMARK-PLAN.md).
|
| 362 |
+
|
| 363 |
+
## π» Coding, agents and your laptop
|
| 364 |
+
|
| 365 |
+
The next local checks cover executable coding tests, bug fixes, tool calls, planning, recovery, memory and stopping at the right time. **64 case slots** are specified, including 19 existing coding slots. They are compact skill diagnostics, not proof of AGI. Results and failures will both be reported; new capability claims require actual task-level evidence.
|
| 366 |
+
|
| 367 |
+
[π οΈ Laptop test plan, Hermes routing and Obsidian workflows](LAPTOP-CAPABILITY-PLAN.md). The runtime comparison and full memory integration are **in preparation**, not yet validated. The four device quickstarts above remain separate from these future end-to-end checks.
|
| 368 |
+
<!-- q36-capability-expansion:end -->
|
| 369 |
+
|
| 370 |
## π License, lineage & credits
|
| 371 |
|
| 372 |
[Apache-2.0 license](LICENSE). Dataset sources and their own license information are linked in [DATASETS.md](DATASETS.md); model licensing does not relicense the source datasets. Thanks to the Qwen team, the inherited model and dataset authors, and the Soup, PEFT, Transformers, llama.cpp and Ollama projects.
|
|
|
|
| 375 |
|
| 376 |
These names do not imply affiliation or a promise of unrestricted behavior. Full provenance and validation are retained, including the historical Opus4.7-labelled source. No universal superiority, guaranteed compliance or removal of memorization is claimed.
|
| 377 |
|
| 378 |
+
<details>
|
| 379 |
+
<summary>𧬠What Ablit, Heretic, Uncensor, Hermes, MTP and Vision mean</summary>
|
| 380 |
+
|
| 381 |
+
This release builds on the [previous project's model card](https://huggingface.co/oktayd/Q36-35B-A3B-Opus4.7-Ablit-Heretic-OBLITERATUS-Hermes-MTP-Vision-FT), which records the following stages. These are inherited stages, not new operations performed while packaging this release.
|
| 382 |
+
|
| 383 |
+
| Name / stage | What it contributes |
|
| 384 |
+
|---|---|
|
| 385 |
+
| **Qwen3.6 / MoE** | The underlying language-and-vision architecture and mixture-of-experts backbone. |
|
| 386 |
+
| **Reasoning-distilled lineage** | The earlier card traces a lordx64 derivative followed by the huihui-ai derivative. The historical Opus4.7 label describes inherited reasoning-distillation provenance, not inclusion of proprietary Claude weights. |
|
| 387 |
+
| **Ablit / abliterated** | The huihui-ai source underwent a weight-modification stage aimed at reducing refusal behavior. It is not a separate tool library. |
|
| 388 |
+
| **Heretic** | A subsequent custom, fused-MoE-aware modification stage documented by the previous release. It is distinct from the earlier abliteration. |
|
| 389 |
+
| **OBLITERATUS Nuclear / Uncensor** | Another separately recorded inherited modification stage aimed at reducing refusal behavior. **Uncensor** is the concise release-name label; OBLITERATUS remains part of the provenance. |
|
| 390 |
+
| **Hermes** | Tool-oriented supervised training: function-call structure, coding, terminal/file/repository workflows and multi-tool coordination, using [Hermes Function Calling](https://huggingface.co/datasets/NousResearch/hermes-function-calling-v1) and [Hermes Agent reasoning traces](https://huggingface.co/datasets/lambda/hermes-agent-reasoning-traces). Your host application still supplies, authorizes and executes tools. |
|
| 391 |
+
| **v2 fine-tuning** | Three additional Soup/PEFT training runs with 34,000 record uses spanning knowledge, instructions, coding, personality and selected vision data. See [datasets and counts](DATASETS.md). |
|
| 392 |
+
| **MTP** | Preserved multi-token-prediction tensors. Preservation is not evidence that speculative decoding is enabled or faster in your runtime. |
|
| 393 |
+
| **Vision** | Image-input architecture; GGUF runtimes also need the matching projector. It does not itself supply browser control, memory or a 3D engine. |
|
| 394 |
+
|
| 395 |
+
The old card reports **23,220 training and 1,179 validation examples** for its own earlier SFT stage; those are separate from the current release's 34,000 record uses. It also records protection of 333 vision tensors and 19 MTP tensors in that earlier build. These historical checks are not new laptop benchmark results.
|
| 396 |
+
|
| 397 |
+
</details>
|
| 398 |
+
|
| 399 |
## π Related collection
|
| 400 |
|
| 401 |
The previous release remains separate: [Qwen3.6 Opus4.7 Heretic Hermes Agent β Editions](https://huggingface.co/collections/oktayd/qwen36-opus47-heretic-hermes-agent-editions-6a8d09f6c2eb42ed1b112184).
|
|
@@ -2,9 +2,11 @@
|
|
| 2 |
807d54b88902798393fc104c3ca80d3e424d64270e8d1e1a34a44d4574f52c7e BENCHMARK-DIAGNOSTIC.json
|
| 3 |
0e712dd8e4bc65088e170826e98bdc440fb19a2cfdb7613c643e9dabb624ecd8 BENCHMARK-EXTENSION-QWEN38.json
|
| 4 |
67d070e3b12bcd185626054c81cdf4d528a9a1af26774050e1d50be370b5ee14 BENCHMARK-PLAN.md
|
|
|
|
| 5 |
5c1a6fa0bd2d21e400b4c84001abec8085e3f5d7fbe5d217e7a0af7f32f42c6d DATASETS.md
|
| 6 |
b483301314df6773850288b6291d49d082a7f816f7266d8cd62ef803b4f4e154 DEVICE-QUICKSTART.md
|
| 7 |
5fa5bdc4687e5ed723af58eb436f545cc62589e5395d5041722639f03bb5ec9b LAPTOP-BENCHMARK-MATRIX.json
|
|
|
|
| 8 |
ad3521db2dbcdf0b2ff300349d312c0fb44a1ebccc84571712602c512bf4a733 Qwen3.6-35B-v2-IQ1_M.gguf
|
| 9 |
61a1a351bdc64ad14616277b0f7c630abe297733a29d96659da3bb7b369c769d Qwen3.6-35B-v2-IQ2_M.gguf
|
| 10 |
80b45e93018023c6a9098cdee83f2a36ccc57dc490625cba2b783c6b3eb832cc Qwen3.6-35B-v2-IQ4_XS.gguf
|
|
@@ -12,7 +14,7 @@ ad3521db2dbcdf0b2ff300349d312c0fb44a1ebccc84571712602c512bf4a733 Qwen3.6-35B-v2
|
|
| 12 |
3cc8cd0bad0c37fba8e038f68c089cdf2e5f55619d48487fdd9bfff0686ef830 Qwen3.6-35B-v2-Q4_K_M.gguf
|
| 13 |
a3a5b8afaf96d6d4c9dfb257e54406a04a8363791f980647f788c67ff3966615 Qwen3.6-35B-v2-Q5_K_M.gguf
|
| 14 |
71099584272a77b22c4aa1152271bf7d85595e1b869ffc883440a9d0e225c073 Qwen3.6-35B-v2-Q8_0.gguf
|
| 15 |
-
|
| 16 |
a69838e3728d2aa08941191fce1d7c5af6f0703a9a4444e8d6f0ef18cb4ecf8a RELEASE-NAMING.json
|
| 17 |
d431e82d0ab961c261e525fc08fc3e64c0916fbd4d3f52b839b3d2eda0766338 assets/qwen3.6-35b-v2-banner.png
|
| 18 |
fc5f718f531acdfe2e45bb09655d40f525eab939f9a3ab501287758f901cb3ad mmproj-Qwen3.6-35B-v2-F16.gguf
|
|
|
|
| 2 |
807d54b88902798393fc104c3ca80d3e424d64270e8d1e1a34a44d4574f52c7e BENCHMARK-DIAGNOSTIC.json
|
| 3 |
0e712dd8e4bc65088e170826e98bdc440fb19a2cfdb7613c643e9dabb624ecd8 BENCHMARK-EXTENSION-QWEN38.json
|
| 4 |
67d070e3b12bcd185626054c81cdf4d528a9a1af26774050e1d50be370b5ee14 BENCHMARK-PLAN.md
|
| 5 |
+
cfd2a06c82f3f41ceaf8c333fb0ee0e38247d0c9419084c0501d5c2430bb7b68 BENCHMARK-TEST-COUNTS.json
|
| 6 |
5c1a6fa0bd2d21e400b4c84001abec8085e3f5d7fbe5d217e7a0af7f32f42c6d DATASETS.md
|
| 7 |
b483301314df6773850288b6291d49d082a7f816f7266d8cd62ef803b4f4e154 DEVICE-QUICKSTART.md
|
| 8 |
5fa5bdc4687e5ed723af58eb436f545cc62589e5395d5041722639f03bb5ec9b LAPTOP-BENCHMARK-MATRIX.json
|
| 9 |
+
42f57a1314669908fc20a4c657e32aafe4bedc9085508a31d1784cc3efd6c1c4 LAPTOP-CAPABILITY-PLAN.md
|
| 10 |
ad3521db2dbcdf0b2ff300349d312c0fb44a1ebccc84571712602c512bf4a733 Qwen3.6-35B-v2-IQ1_M.gguf
|
| 11 |
61a1a351bdc64ad14616277b0f7c630abe297733a29d96659da3bb7b369c769d Qwen3.6-35B-v2-IQ2_M.gguf
|
| 12 |
80b45e93018023c6a9098cdee83f2a36ccc57dc490625cba2b783c6b3eb832cc Qwen3.6-35B-v2-IQ4_XS.gguf
|
|
|
|
| 14 |
3cc8cd0bad0c37fba8e038f68c089cdf2e5f55619d48487fdd9bfff0686ef830 Qwen3.6-35B-v2-Q4_K_M.gguf
|
| 15 |
a3a5b8afaf96d6d4c9dfb257e54406a04a8363791f980647f788c67ff3966615 Qwen3.6-35B-v2-Q5_K_M.gguf
|
| 16 |
71099584272a77b22c4aa1152271bf7d85595e1b869ffc883440a9d0e225c073 Qwen3.6-35B-v2-Q8_0.gguf
|
| 17 |
+
2a5701e733ab7b19f4b073fa13ff2e44d45eef7a54c6ad52679f940249031bab README.md
|
| 18 |
a69838e3728d2aa08941191fce1d7c5af6f0703a9a4444e8d6f0ef18cb4ecf8a RELEASE-NAMING.json
|
| 19 |
d431e82d0ab961c261e525fc08fc3e64c0916fbd4d3f52b839b3d2eda0766338 assets/qwen3.6-35b-v2-banner.png
|
| 20 |
fc5f718f531acdfe2e45bb09655d40f525eab939f9a3ab501287758f901cb3ad mmproj-Qwen3.6-35B-v2-F16.gguf
|