The Grux Local scoreboard scores small open models on Grux's own jobs: calling the right tool with the right details, saying so when no tool can do what was asked, ignoring instructions hidden in pasted text, fixing dictation, sorting mail, titling chats and summarizing meetings. Each score is a count of tasks done right on Grux's held-out sets of 464 and 466 tasks, written separately and never used in training and scored by Grux's own parsers, with a 95% interval beside it. On set v1, on one NVIDIA H200 GPU in the cloud, rented by the hour, qwen3.5:9b got 396 of 464 right (81.8% to 88.3%), qwen3.5:4b 392 (80.9% to 87.5%) and mimo-v2.6:9b 367 (75.2% to 82.6%). Every model in the headline table is open, licensed Apache-2.0 and MIT, and every number links its raw JSON in the public grux-local repository at commit 3b6512e. The scoreboard measures how well a model does the jobs Grux hands it. It is not a measure of general ability, and it ranks no model against a hosted one.
Machine: one NVIDIA H200 GPU in the cloud, rented by the hour. Runtime: Ollama 0.40.16, llamacpp runner, thinking off, a 16,384 token context, one request at a time. Each model is the stock Q4_K_M build from the Ollama library, pulled by the machine that scored it. A stock model is scored again only when its build changes, so each row carries the date it was scored, and every row in a table shares one runtime and one GPU. Rows are ordered by tasks right on each set.
| Model | Licence | Scored | Right | Share, 95% interval | Hidden instructions ignored | Raw |
|---|---|---|---|---|---|---|
| qwen3.5:9b, 9.0B | Apache-2.0 | 2026-11-10 | 396 of 464 | 85.3%, 81.8% to 88.3% | 57 of 70 | JSON |
| qwen3.5:4b, 4.2B | Apache-2.0 | 2026-11-10 | 392 of 464 | 84.5%, 80.9% to 87.5% | 51 of 70 | JSON |
| mimo-v2.6:9b, 9.0B | MIT | 2026-11-10 | 367 of 464 | 79.1%, 75.2% to 82.6% | 59 of 70 | JSON |
| Model | Licence | Scored | Right | Share, 95% interval | Hidden-instruction cases passed | Raw |
|---|---|---|---|---|---|---|
| qwen3.5:4b, 4.2B | Apache-2.0 | 2026-11-10 | 362 of 466 | 77.7%, 73.7% to 81.2% | 8 of 70 | JSON |
| qwen3.5:9b, 9.0B | Apache-2.0 | 2026-11-10 | 357 of 466 | 76.6%, 72.6% to 80.2% | 10 of 70 | JSON |
| mimo-v2.6:9b, 9.0B | MIT | 2026-11-10 | 354 of 466 | 76.0%, 71.9% to 79.6% | 24 of 70 | JSON |
Two intervals that do not overlap mark a gap this set can see. Two that overlap may still differ, and the raw files carry every task family for anyone who wants to look closer.
7c3f2a9ff3316ecc, and set v2 is 466 tasks, hash da6577c7178a0963. Both are pinned in recipe/check_prompts.py, and a result joins this page only when both hashes and both sizes match.Grux Local is our own model, trained for exactly these jobs, so it is expected to score higher than the stock build it starts from. These tables show what training did, each on one Mac, and they are not a ranking against any other model. Each tag is pulled as gruxai.com/grux/<tag>. They are read from results/v1.json, measured with Ollama on Apple silicon, and never mixed with the weekly cloud table.
| Model | Trained for Grux | Measured | Right | Share, 95% interval | Hidden instructions ignored |
|---|---|---|---|---|---|
| qwen3.5:2b | no, stock | 2026-10-23 | 295 of 464 | 63.6%, 59.1% to 67.8% | 43 of 70 |
| grux-local:2b | yes | 2026-10-23 | 435 of 464 | 93.8%, 91.2% to 95.6% | 66 of 70 |
| qwen3.5:4b | no, stock | 2026-10-22 | 396 of 464 | 85.3%, 81.8% to 88.3% | 55 of 70 |
| grux-local:4b | yes | 2026-10-23 | 454 of 464 | 97.8%, 96.1% to 98.8% | 65 of 70 |
| grux-local:9b | yes | 2026-10-23 | 458 of 464 | 98.7%, 97.2% to 99.4% | 66 of 70 |
Ollama 0.40.8, MLX runner, one request at a time, thinking off as Grux sends it. Raw JSON: v1.json.
| Model | Trained for Grux | Measured | Right | Share, 95% interval | Hidden instructions ignored |
|---|---|---|---|---|---|
| grux-local:4b | yes | 2026-10-23 | 456 of 464 | 98.3%, 96.6% to 99.1% | 66 of 70 |
| grux-local:4b-v2 | yes | 2026-10-23 | 455 of 464 | 98.1%, 96.4% to 99.0% | 69 of 70 |
| qwen3.5:9b | no, stock | 2026-10-23 | 400 of 464 | 86.2%, 82.8% to 89.1% | 65 of 70 |
| grux-local:9b | yes | 2026-10-23 | 458 of 464 | 98.7%, 97.2% to 99.4% | 66 of 70 |
Ollama 0.40.8, MLX runner, one request at a time, thinking off as Grux sends it. Raw JSON: v1.json.
| Model | Trained for Grux | Measured | Right | Share, 95% interval | Hidden-instruction cases passed |
|---|---|---|---|---|---|
| grux-local:4b | yes | 2026-10-23 | 398 of 466 | 85.4%, 81.9% to 88.3% | 10 of 70 |
| grux-local:4b-v2 | yes | 2026-10-23 | 442 of 466 | 94.9%, 92.5% to 96.5% | 53 of 70 |
Ollama 0.40.8, MLX runner, one request at a time, thinking off as Grux sends it. Raw JSON: v1.json.
The same Ollama tag, qwen3.5:4b, got 392 of 464 in the weekly table and 396 of 464 on the Mac mini, Apple M4, 16 GB. Same tasks, but a different machine, Ollama version and runner, and so a different build of the weights, which is why no table on this page mixes machines.
The same Ollama tag, qwen3.5:9b, got 396 of 464 in the weekly table and 400 of 464 on the MacBook Pro, Apple M4 Pro, 48 GB. Same tasks, but a different machine, Ollama version and runner, and so a different build of the weights, which is why no table on this page mixes machines.
All 3 weekly result files at commit 3b6512e, newest first. A file is never edited once it is published; a new score is a new file.
| Scored | Model | Set v1, of 464 | Set v2, of 466 | Raw |
|---|---|---|---|---|
| 2026-11-10 | qwen3.5:9b | 396 | 357 | JSON |
| 2026-11-10 | qwen3.5:4b | 392 | 362 | JSON |
| 2026-11-10 | mimo-v2.6:9b | 367 | 354 | JSON |
Each week the Grux Local lane looks for new open models at Mac sizes and scores each one it can run on both held-out sets, beside the stock Qwen 3.5 rows it last scored. Once its result files are published to the grux-local repository, this page is rebuilt from them at one pinned commit, and the rebuild refuses any file that did not finish, whose set hash or size does not match, whose task families do not add up to its total, whose stored interval differs from a recompute, or whose model has no open licence.