Pull the largest model your Mac has memory for, then check it can do the job you need done. On Apple silicon that usually means a 4B build on a 16 GB Mac and a 9B from 24 GB. The Ollama library tells you how big a download is; it does not tell you whether a model calls the right tool with the right details, says so when no tool fits, or notices when pasted text tries to steer it. Grux scores small open models from that library on its own held-out set of 464 tasks, written separately and never used in training, and publishes each count with a 95% interval and the raw JSON behind it. The tables below join the two halves: what a pull costs, and how many of those tasks it got right, with the date, the machine and the runner beside it. 3 library builds are scored on the weekly run, from 3.3 GB to 6.6 GB.
Each row is the stock build pulled from the Ollama library with ollama pull, by the machine that scored it, on held-out set v1. The size on disk is what the build itself reports. Rows are ordered by tasks right, and the date links the raw result file it came from.
| Model | Size | Disk | Licence | Right of 464 | Hidden | Raw |
|---|---|---|---|---|---|---|
qwen3.5:9b | 9.0B, Q4_K_M | 6.6 GB | Apache-2.0 | 396 of 464 | 57 of 70 | 2026-11-10 |
qwen3.5:4b | 4.2B, Q4_K_M | 3.3 GB | Apache-2.0 | 392 of 464 | 51 of 70 | 2026-11-10 |
mimo-v2.6:9b | 9.0B, Q4_K_M | 6.6 GB | MIT | 367 of 464 | 59 of 70 | 2026-11-10 |
Runner for every row in this table: Ollama 0.40.16, llamacpp runner, thinking off, a 16,384 token context, one request at a time, on the one NVIDIA H200 GPU in the cloud, rented by the hour.
Every build in this table carries the same context length, 262,144 tokens, so it is not a column here.
These are the same library tags pulled and scored on Apple silicon, on held-out set v1. They are kept in their own table because another machine and runner means another build of the weights, and two machines never share a table here. The counts differ from the table above, which is the point of keeping them apart.
| Model | Machine | Right of 464 | Hidden | Measured | Raw |
|---|---|---|---|---|---|
qwen3.5:2b | Mac mini, Apple M4, 16 GB | 295 of 464 | 43 of 70 | 2026-10-23 | JSON |
qwen3.5:4b | Mac mini, Apple M4, 16 GB | 396 of 464 | 55 of 70 | 2026-10-22 | JSON |
Ollama 0.40.8, MLX runner, one request at a time, thinking off as Grux sends it
| Model | Machine | Right of 464 | Hidden | Measured | Raw |
|---|---|---|---|---|---|
qwen3.5:9b | MacBook Pro, Apple M4 Pro, 48 GB | 400 of 464 | 65 of 70 | 2026-10-23 | JSON |
Ollama 0.40.8, MLX runner, one request at a time, thinking off as Grux sends it
A pull is weights plus whatever loads them, and the loader is part of the result, so every count on this page names the runner that produced it.
The Grux Local 4B is the one model here scored on the same Mac mini, Apple M4, 16 GB through more than one runner.
| Runner | Model | Machine | Right of 464 | Hidden | Measured | Raw |
|---|---|---|---|---|---|---|
| Ollama 0.40.8, MLX runner | grux-local:4b | Mac mini, Apple M4, 16 GB | 454 of 464 | 65 of 70 | 2026-10-23 | JSON |
| mlx-lm 0.32.2 | grux-local:4b | 16 GB Mac mini (Apple M4) | 458 of 464 | 69 of 70 | 2026-11-15 | not published, see the hash below |
The same model, packaged for each runner, on the same Mac and the same 464 tasks: 4 tasks and 4 hidden-instruction cases apart, one run each. The run behind the second row hashes to 9edfdfbe03da34a0. That is why every number on this page names its runner, and why a score read anywhere else is only about the runner it was measured on.
A score belongs to a runner, not to a model name. Ask which runner a number came from before you compare it with another one.
| Claim | Source |
|---|---|
| Every score in the library and Mac tables is a count on Grux's held-out set, with its raw JSON; the second runner's row names the hash of its run instead | Grux Local scoreboard |
| Both held-out sets are pinned by hash and size at the commit the rows came from | recipe/check_prompts.py |
| The Mac rows and their task families are in the published result file | results/v1.json |
| The library itself, with every model and its download size | ollama.com/library |