2026-10-25 · Mac · Apple M5 Max
A model scorecard round on M5 Max · Apple M5 Max (integrated GPU) · 128 GB: 5 models across 1 task set. Every round is listed in the model scorecard, with its results side by side in one table.
M5 Max · Apple M5 Max (integrated GPU) · 128 GB · llama-cpp + mlx · 3 trials per task · 2026-10-25 · gezel 4f2c5491d9a1cbf6cdd645e708dd674a22615d15 · catalog 0.3.0
Complete five-model core comparison: 165 trials, 110 raw passes, 55 task failures; 158 artifact reviews and seven missing outputs. Bolt uses llama.cpp Metal; Gemma and Qwen use MLX. Four advisory rubrics remain separate. Bolt decode is request-derived; MLX output is an active-stream estimate and prefill is unavailable. Task observations include failed attempts and recovery, not admission probes. Known framework and evaluator defects remain in the frozen raw results; a gate pass does not certify artifact quality. See the October 25 M5 Max Bolt comparison JSON download for all judgments, performance coverage and 32 findings. Earlier old-build and interrupted trials are excluded.
General capability
| Model | Size | Tasks passed |
|---|---|---|
| Gemma 4 (E4B, Q4) | small | 26/33 (79%) |
| Nemotron 3.5 Bolt (E4B, Q4) | small | 26/33 (79%) |
| Gemma 4 (12B, Q4) | medium | 23/33 (70%) |
| Qwen 3.5 (9B, Q4) | small | 21/33 (64%) |
| Gemma 4 (E2B, Q4) | tiny | 14/33 (42%) |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 (E4B, Q4) | Frozen run · MLX sampled decode estimate | 33 | 79.00 (55.00–103.00; n=33) | — | 2.3 (0.5–19.4; n=33) | 8.11 (7.98–17.15; n=33) | — |
| Nemotron 3.5 Bolt (E4B, Q4) | Frozen run · llama.cpp request timings | 33 | 72.00 (59.18–90.14; n=33) | 2,513.96 (1,977.37–3,492.57; n=33) | 2.6 (0.5–25.9; n=33) | 10.04 (8.60–19.80; n=33) | — |
| Gemma 4 (12B, Q4) | Frozen run · MLX sampled decode estimate | 33 | 38.00 (30.00–49.00; n=33) | — | 4.2 (0.7–19.2; n=33) | 10.67 (10.31–14.21; n=33) | — |
| Qwen 3.5 (9B, Q4) | Frozen run · MLX sampled decode estimate | 33 | 63.50 (30.00–87.00; n=30) | — | 2.6 (0.7–44.5; n=33) | 7.97 (7.20–20.80; n=33) | — |
| Gemma 4 (E2B, Q4) | Frozen run · MLX sampled decode estimate | 33 | 115.50 (96.00–136.00; n=32) | — | 1.7 (0.5–19.9; n=33) | 7.11 (7.03–10.38; n=33) | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | Code quality | Data quality | Document quality | HTML quality | No artifact | Excluded reviews |
|---|---|---|---|---|---|---|---|
| Gemma 4 (E4B, Q4) | Frozen run · MLX sampled decode estimate | 4.84/10 (8 pieces) | 4.50/10 (3 pieces) | 4.02/10 (12 pieces) | 2.56/10 (9 pieces) | 1 | 0 |
| Nemotron 3.5 Bolt (E4B, Q4) | Frozen run · llama.cpp request timings | 4.78/10 (9 pieces) | 4.00/10 (3 pieces) | 4.17/10 (12 pieces) | 2.58/10 (6 pieces) | 3 | 0 |
| Gemma 4 (12B, Q4) | Frozen run · MLX sampled decode estimate | 4.50/10 (9 pieces) | 4.75/10 (2 pieces) | 4.08/10 (12 pieces) | 3.19/10 (9 pieces) | 1 | 0 |
| Qwen 3.5 (9B, Q4) | Frozen run · MLX sampled decode estimate | 3.67/10 (9 pieces) | 2.63/10 (2 pieces) | 4.06/10 (12 pieces) | 2.78/10 (9 pieces) | 1 | 0 |
| Gemma 4 (E2B, Q4) | Frozen run · MLX sampled decode estimate | 4.14/10 (9 pieces) | 2.50/10 (3 pieces) | 3.73/10 (12 pieces) | 2.25/10 (8 pieces) | 1 | 0 |
Tasks in this set: 11.
October 25 M5 Max Bolt comparison and review data
The five-model core campaign completed all 165 trials: 11 scenarios, three repeats, five models. Bolt and Gemma E4B each passed 26/33, Gemma E2B 14/33, Gemma 12B 23/33 and Qwen 9B 21/33. Every trial was reviewed, including failed outputs: 158 artifacts have advisory scores and seven missing outputs have no invented score. HTML, code, data and document quality remain separate.
Download the scorecard data and full report JSON with judging and performance. The full report preserves exact review axes and rationales, per-trial timings, token coverage, fixed-score breakdowns, model identities and 32 documented framework/evaluator findings. Some gate passes contained incomplete or broken artifacts; some gates also rejected valid outputs. Raw verdicts remain unchanged, and these findings do not establish when a defect was introduced.
Bolt ran on llama.cpp Metal using NVIDIA's October 23 revision; the other models used MLX. The Task performance rows show task medians, ranges and sample counts, including failed attempts and recovery. Bolt throughput is request-derived; MLX decode is an active-stream estimate and MLX prefill is unavailable. Token streams have incomplete coverage, first-write latency is not completed-output latency, and process RSS is not GPU telemetry. Download notes and validation explain the frozen comparison, measurement limits and excluded older trials.