2026-11-03 · Mac · Apple M5 Max
A focused rerun on M5 Max · Apple M5 Max (integrated GPU) · 128 GB: 2 models across 2 task sets. Every round is listed in the model scorecard, with its results side by side in one table.
M5 Max · Apple M5 Max (integrated GPU) · 128 GB · llama-cpp + mlx · focused subset · 1 trial per task · 2026-11-03 · gezel d0750db1712b45c304d3ba772e996f01a1da1b30 · catalog 0.3.5
User-curtailed exploratory comparison: 72 matched trials, 18 pairs per model, one repetition over core and productivity; developer and complex-work were not run. Both models passed 16/18 on MLX and 18/18 on llama.cpp; equivalence within ±5 percentage points is not established. The passing, unpaired PowerPoint trial has a preserved fixture-identity flag and remains in downloads only. All four recorded failures remain unchanged, including one diagnosed gate false-positive. Native 0.1.55; generalist auto; Gilde 0.3.5 with the staged Gemma MLX assistant drafter candidate, not an installed catalog release. Observed MLX prefill is recovered from rounded, partial first-token chat events, not aggregate request throughput or pure prefill time. Engine quantizations and speculation differ. Raw scores, judgments, performance coverage and paired uncertainty are in the downloads.
General capability
| Model | Size | Tasks passed | Context | Memory used |
|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp (kv: f16) | 12B | 11/11 (some tasks run once — count not rate) | 256K | 18.8 GB |
| Qwen 3.8 (27B, Q4) · llama.cpp (kv: f16) | 27B | 11/11 (some tasks run once — count not rate) | 256K | 39.0 GB |
| Qwen 3.8 (27B, Q4) · MLX | 27B | 10/11 (some tasks run once — count not rate) | — | — |
| Gemma 4 (12B, Q4) · MLX | 12B | 9/11 (some tasks run once — count not rate) | — | — |
Task performance
Task measurements include failures within each stated period or comparison subset. Measurement methods and exclusions are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change. Observed prefill uses partial chat-event rates summarized within each trial. It is not a duration-weighted aggregate of every request, and does not recover pure prefill time.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 11 | 81.72 (63.36–123.39; n=11) | 1,101.53 (853.20–1,519.39; n=11) | 1.4 (0.4–4.6; n=11) | 18.84 (13.70–28.44; n=11) | — |
| Qwen 3.8 (27B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 11 | 35.85 (27.64–50.85; n=11) | 449.19 (395.51–630.37; n=11) | 1.9 (0.6–19.7; n=11) | 38.95 (35.79–51.01; n=11) | — |
| Qwen 3.8 (27B, Q4) · MLX | Matched subset · native 0.1.55 | 11 | 34.00 (26.00–45.00; n=11) | 574.00 (493.00–801.00; n=11) observed | 2.0 (1.0–47.8; n=11) | 17.21 (17.13–20.02; n=11) | — |
| Gemma 4 (12B, Q4) · MLX | Matched subset · native 0.1.55 | 11 | 63.00 (47.00–80.00; n=11) | 1,341.50 (1,046.00–1,963.00; n=11) observed | 2.6 (0.5–10.8; n=11) | 10.07 (9.87–18.06; n=11) | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | HTML quality | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 4.50/10 (3 pieces) | 0 | 0 | 8 |
| Qwen 3.8 (27B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 4.75/10 (3 pieces) | 0 | 0 | 8 |
| Qwen 3.8 (27B, Q4) · MLX | Matched subset · native 0.1.55 | 6.08/10 (3 pieces) | 0 | 0 | 8 |
| Gemma 4 (12B, Q4) · MLX | Matched subset · native 0.1.55 | 3.92/10 (3 pieces) | 0 | 0 | 8 |
Tasks in this set: 11.
Office and knowledge work
| Model | Size | Tasks passed | Context | Memory used |
|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp (kv: f16) | 12B | 7/7 (some tasks run once — count not rate) | 256K | 18.8 GB |
| Gemma 4 (12B, Q4) · MLX | 12B | 7/7 (some tasks run once — count not rate) | — | — |
| Qwen 3.8 (27B, Q4) · llama.cpp (kv: f16) | 27B | 7/7 (some tasks run once — count not rate) | 256K | 38.6 GB |
| Qwen 3.8 (27B, Q4) · MLX | 27B | 6/7 (some tasks run once — count not rate) | — | — |
Task performance
Task measurements include failures within each stated period or comparison subset. Measurement methods and exclusions are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change. Observed prefill uses partial chat-event rates summarized within each trial. It is not a duration-weighted aggregate of every request, and does not recover pure prefill time.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 7 | 85.05 (75.04–105.02; n=7) | 1,101.83 (931.24–1,277.21; n=7) | 2.0 (0.5–2.9; n=7) | 18.78 (14.14–24.56; n=7) | — |
| Gemma 4 (12B, Q4) · MLX | Matched subset · native 0.1.55 | 7 | 57.00 (51.00–68.00; n=7) | 1,222.00 (1,028.00–1,615.50; n=7) observed | 2.5 (0.6–6.3; n=7) | 9.96 (9.89–10.81; n=7) | — |
| Qwen 3.8 (27B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 7 | 33.73 (23.55–53.75; n=7) | 440.62 (401.65–612.30; n=7) | 2.9 (1.2–9.4; n=7) | 38.56 (35.83–44.83; n=7) | — |
| Qwen 3.8 (27B, Q4) · MLX | Matched subset · native 0.1.55 | 7 | 34.00 (25.00–49.00; n=6) | 743.00 (522.00–943.00; n=7) observed | 2.2 (1.0–21.1; n=7) | 17.29 (17.15–17.58; n=7) | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-spreadsheet-model | docblocks-theme-roundtrip | meeting-followup | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 4.50/10 (1 pieces) | 4.00/10 (1 pieces) | 6.50/10 (1 pieces) | 0 | 0 | 4 |
| Gemma 4 (12B, Q4) · MLX | Matched subset · native 0.1.55 | 4.50/10 (1 pieces) | 5.67/10 (1 pieces) | 5.25/10 (1 pieces) | 0 | 0 | 4 |
| Qwen 3.8 (27B, Q4) · llama.cpp | Matched subset · native 0.1.55 | 5.50/10 (1 pieces) | 4.00/10 (1 pieces) | 6.75/10 (1 pieces) | 0 | 0 | 4 |
| Qwen 3.8 (27B, Q4) · MLX | Matched subset · native 0.1.55 | 4.50/10 (1 pieces) | 4.67/10 (1 pieces) | 6.75/10 (1 pieces) | 0 | 0 | 4 |
Tasks in this set: 7.
November 3 M5 Max condensed engine comparison
This follow-up compared Qwen 3.8 27B Q4 and Gemma 4 12B Q4 on MLX and llama.cpp after fixes, using native 0.1.55 on an Apple M5 Max with 128 GB of unified memory. The user shortened the original 516-trial plan for time. The comparison contains 72 matched trials: 18 pairs per model, one repetition, across core and productivity. Developer and complex-work were not run. This chronological subset was selected after outcomes were visible, so the findings are exploratory.
Both models passed 16/18 on MLX and 18/18 on llama.cpp. Practical equivalence within the user-selected ±5 percentage points is not established. The observed difference is −11.11 pp; the conservative paired 95% interval is [−38.09, +20.66] pp and the scenario-cluster bootstrap interval is [−26.32, 0.00] pp for each model. These data also do not establish that the underlying gap exceeds 5 pp. The paired analysis documents assumptions, unavailable original pairs, sensitivity and nominal/adjusted tests.
Median recorded decode was 34.00 tok/s on MLX versus 35.66 on llama.cpp for Qwen, and 59.00 versus 84.12 for Gemma. Gemma's roughly 30% lower MLX rate here does not reproduce the earlier twofold gap. These are complete integration measurements with different estimators, quantizations and speculative decoding, not an isolated engine benchmark or causal test of a fix. Gemma MLX used a staged assistant drafter candidate; Gilde remained 0.3.5.
MLX prefill was recovered from first-token diagnostics: 121 rounded rate observations in the 36 included MLX trials. The medians of trial medians are 604.75 tok/s for Qwen and 1332.25 for Gemma. Tables mark these rates as observed: they cover partial provider turns and are not equivalent to llama.cpp's duration-weighted aggregate. Exact MLX prefill times remain unknown. The download preserves all 126 observations across all 37 recorded MLX trials, their source hashes and coverage limits.
All four recorded failures remain unchanged. The diagnoses identify an image-tool workflow failure, unresolved duplicate data, a section-placement failure, and a conflict-synthesis gate false-positive. The last unpaired Gemma MLX PowerPoint trial passed its gate but encountered a fixture-identity flag; it is excluded from the matched comparison and retained in downloads. There are 25 artifact reviews, including that excluded trial's review; 24 enter the displayed rubric means. The other 48 trials have no applicable rubric. Fixed scores and advisory quality remain separate.
Download the report, paired analysis, performance CSV, performance data, MLX prefill observations, and judge data. The download index also links all 73 trial records, postmortems, provenance, verification and the scope amendment. The earlier native-0.1.54 round remains a separate record.