POCKET vs Bonsai Β· CPU
β
47
BONSAI vs POCKET β 35B MoE out-runs the top 1-bit 27B on CPU
Joonhee Han
hinow1
AI & ML interests
vidraft.net cio
Recent Activity
liked a Space 5 days ago
FINAL-Bench/POCKET-35B-CPU liked a model 5 days ago
FINAL-Bench/Darwin-35B-A3B-Opus liked a Space 5 days ago
quantid/homeOrganizations
None yet
liked a
Space 5 days ago
liked a
model 5 days ago
liked a
Space 5 days ago
liked a
model 5 days ago
reacted to SeaWolf-AI's post with β€οΈ about 1 month ago
Post
3755
We opened a benchmark for drug property prediction tools. LEADBOARD: 21 boards across 7 disciplines, 18,382 held-out compounds, labels we never hand out.
Two numbers we hit while building it are the reason it exists.
First. Split the hERG cardiotoxicity data at random and you get AUROC 0.818. Split it by first-report year instead and you get 0.606. Same molecules, same fingerprints, same learner, same hyperparameters. The only thing that changed was where the line went, and the score moved 0.211. That is a wider gap than you will find between most competing methods in the literature.
Second. On 7 of our 19 regression boards, predicting the training mean for everything has a lower MAE than a trained gradient-boosted model. hERG is one of them, 0.599 against 0.589. The trained model loses.
So every board publishes its homework before anyone submits. Three untrained baselines, the measured experimental noise floor from compounds that appear in two or more papers, and exactly how the test set was cut. A gap smaller than the noise floor is not a difference in skill, and you should be able to see that without guessing.
Entering is simple. Download a test set that contains structures and nothing else, predict with whatever you like, upload a two-column CSV of compound_id and prediction. Trained model, physics engine, LLM, rule of thumb. We do not care what is inside. We measure the output.
Post: https://huggingface.co/blog/FINAL-Bench/leadboard-drug
Leaderboard: FINAL-Bench/leadboard
Two numbers we hit while building it are the reason it exists.
First. Split the hERG cardiotoxicity data at random and you get AUROC 0.818. Split it by first-report year instead and you get 0.606. Same molecules, same fingerprints, same learner, same hyperparameters. The only thing that changed was where the line went, and the score moved 0.211. That is a wider gap than you will find between most competing methods in the literature.
Second. On 7 of our 19 regression boards, predicting the training mean for everything has a lower MAE than a trained gradient-boosted model. hERG is one of them, 0.599 against 0.589. The trained model loses.
So every board publishes its homework before anyone submits. Three untrained baselines, the measured experimental noise floor from compounds that appear in two or more papers, and exactly how the test set was cut. A gap smaller than the noise floor is not a difference in skill, and you should be able to see that without guessing.
Entering is simple. Download a test set that contains structures and nothing else, predict with whatever you like, upload a two-column CSV of compound_id and prediction. Trained model, physics engine, LLM, rule of thumb. We do not care what is inside. We measure the output.
Post: https://huggingface.co/blog/FINAL-Bench/leadboard-drug
Leaderboard: FINAL-Bench/leadboard
- 7 replies
liked a
Space about 2 months ago
reacted to SeaWolf-AI's post with β€οΈ 2 months ago
Post
3444
We wrote up our run in The Fast Gemma Challenge β as vidraft-darwin β and wanted to share the recipe. π
gemma-challenge/gemma-dashboard
Verified result: 510.58 TPS at PPL 2.3930 on a single A10G (fw188-ctk49-n64-patchbridge, re-run & VERIFIED). Honest note: on raw TPS there are faster runs (535+), but those went over the PPL bar and didn't verify β what we're proud of is the fastest result that keeps quality.
The recipe is already open, so we explained each piece: sliding-window W188, CTK49 kernel tuning, noprecache (honest, verifiable measurement), and an N64 synthetic warmup bridge that shrinks the publicβprivate gap (~15 TPS), plus INT4 + MTP K=7 + CUDA-graph capture. One rule: only stack quality-neutral speedups.
Huge thanks to @firfir-cast , @gemma-slayer , @chiku-inu , @kenyan-duma , @dixie-flatline and everyone who shared their experiments. Full write-up
π
https://huggingface.co/blog/FINAL-Bench/fast-gemma
gemma-challenge/gemma-dashboard
Verified result: 510.58 TPS at PPL 2.3930 on a single A10G (fw188-ctk49-n64-patchbridge, re-run & VERIFIED). Honest note: on raw TPS there are faster runs (535+), but those went over the PPL bar and didn't verify β what we're proud of is the fastest result that keeps quality.
The recipe is already open, so we explained each piece: sliding-window W188, CTK49 kernel tuning, noprecache (honest, verifiable measurement), and an N64 synthetic warmup bridge that shrinks the publicβprivate gap (~15 TPS), plus INT4 + MTP K=7 + CUDA-graph capture. One rule: only stack quality-neutral speedups.
Huge thanks to @firfir-cast , @gemma-slayer , @chiku-inu , @kenyan-duma , @dixie-flatline and everyone who shared their experiments. Full write-up
π
https://huggingface.co/blog/FINAL-Bench/fast-gemma
- 8 replies
reacted to SeaWolf-AI's post with π€πβ€οΈ 2 months ago
Post
4125
π± POCKET β a 35-billion-parameter model that runs on your iPhone, and on your PC with no GPU
We're releasing POCKET, VIDRAFT's flagship Darwin-36B-Opus compressed for on-device use. No fork, no CUDA, no cloud β it runs on stock llama.cpp. It's a sparse Mixture-of-Experts model (256 experts, only 8 active per token), so the file can be large while the work per token stays small. That's what lets a 35B model run on a phone, and generate fast on a CPU with no graphics card.
Measured (POCKET-35B IQ1_M vs Bonsai-27B Q1_0):
β’ CPU generate (Xeon, 16 threads): 27.0 vs 10.1 tok/s β 2.69Γ faster
β’ GPU generate (H100): 197 vs 89 tok/s β 2.22Γ faster
β’ GPU prompt processing (H100): 753 vs 1816 β 0.41Γ (Bonsai wins this one β MoE prefill wakes every expert, so sparsity stops helping there. We say so.)
β’ Quality (HellaSwag, 400 q): 61.0% vs 60.0% β a tie (confidence intervals overlap)
On a real consumer laptop β MacBook M3 Pro (18 GB) β POCKET wins every axis, prompt processing included:
β’ Metal generate: 25.4 vs 12.8 β 1.99Γ
β’ CPU generate: 13.8 vs 4.4 β 3.13Γ
β’ Metal prompt: 240.7 vs 73.4 β 3.28Γ
One more quiet fact: the same-size, quality-oriented rival Ternary-Bonsai-27B (7.2 GB) fails to load in upstream llama.cpp at all β it needs the PrismML fork. POCKET runs on the tools you already have: LM Studio, Ollama, PocketPal, MLX.
π Full story (tech, measurements, recipes): https://huggingface.co/blog/FINAL-Bench/pocket
Models:
π¦ POCKET-35B-GGUF (PC / server, no GPU): FINAL-Bench/POCKET-35B-GGUF
π°π· POCKET-KR-GGUF (Android): FINAL-Bench/POCKET-KR-GGUF
π POCKET-KR-MLX (iPhone / Mac): FINAL-Bench/POCKET-KR-MLX
π POCKET-EN-GGUF (English phone / PC): FINAL-Bench/POCKET-EN-GGUF
π₯οΈ Live demo (answering on a CPU, no GPU): FINAL-Bench/POCKET-35B-CPU
π Collection: FINAL-Bench/pocket-models-6a618ee5d23eafb7e185a5c6
We're releasing POCKET, VIDRAFT's flagship Darwin-36B-Opus compressed for on-device use. No fork, no CUDA, no cloud β it runs on stock llama.cpp. It's a sparse Mixture-of-Experts model (256 experts, only 8 active per token), so the file can be large while the work per token stays small. That's what lets a 35B model run on a phone, and generate fast on a CPU with no graphics card.
Measured (POCKET-35B IQ1_M vs Bonsai-27B Q1_0):
β’ CPU generate (Xeon, 16 threads): 27.0 vs 10.1 tok/s β 2.69Γ faster
β’ GPU generate (H100): 197 vs 89 tok/s β 2.22Γ faster
β’ GPU prompt processing (H100): 753 vs 1816 β 0.41Γ (Bonsai wins this one β MoE prefill wakes every expert, so sparsity stops helping there. We say so.)
β’ Quality (HellaSwag, 400 q): 61.0% vs 60.0% β a tie (confidence intervals overlap)
On a real consumer laptop β MacBook M3 Pro (18 GB) β POCKET wins every axis, prompt processing included:
β’ Metal generate: 25.4 vs 12.8 β 1.99Γ
β’ CPU generate: 13.8 vs 4.4 β 3.13Γ
β’ Metal prompt: 240.7 vs 73.4 β 3.28Γ
One more quiet fact: the same-size, quality-oriented rival Ternary-Bonsai-27B (7.2 GB) fails to load in upstream llama.cpp at all β it needs the PrismML fork. POCKET runs on the tools you already have: LM Studio, Ollama, PocketPal, MLX.
π Full story (tech, measurements, recipes): https://huggingface.co/blog/FINAL-Bench/pocket
Models:
π¦ POCKET-35B-GGUF (PC / server, no GPU): FINAL-Bench/POCKET-35B-GGUF
π°π· POCKET-KR-GGUF (Android): FINAL-Bench/POCKET-KR-GGUF
π POCKET-KR-MLX (iPhone / Mac): FINAL-Bench/POCKET-KR-MLX
π POCKET-EN-GGUF (English phone / PC): FINAL-Bench/POCKET-EN-GGUF
π₯οΈ Live demo (answering on a CPU, no GPU): FINAL-Bench/POCKET-35B-CPU
π Collection: FINAL-Bench/pocket-models-6a618ee5d23eafb7e185a5c6
- 5 replies