2026-10-28 · Mac · Apple M5 Max
A model scorecard round on M5 Max · Apple M5 Max (integrated GPU) · 128 GB: 2 models across 4 task sets. Every round is listed in the model scorecard, with its results side by side in one table.
M5 Max · Apple M5 Max (integrated GPU) · 128 GB · llama-cpp + mlx · 3 trials per task · 2026-10-28 · gezel 22fec99111a5 · catalog 0.3.1
Completed M5 Max 128 GB comparison: 516 attempts, 349 passes, 165 model-classified failures and two preserved infrastructure exclusions. Every model/engine/scenario/suite has three repetitions. Native 0.1.54, generalist auto. Task measurements include all attempts; 215 bounded artifact reviews retain separate rubrics, including 80 failed trials. Twelve incompatible A/B specifications and one missing artifact receive no numeric score. MLX observed prefill was recovered from rounded first-token chat events for all 258 MLX trials; these partial rates are not aggregate request throughput or pure prefill time. macOS GPU utilization and MLX granted context remain unknown. Engine-specific quantization, speculative decoding, raw fixed scores, advisory quality, timing, throughput, memory and failure diagnoses are documented in the downloads.
General capability
| Model | Size | Tasks passed | Context | Memory used |
|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp (kv: f16) | 27B | 33/33 (100%) | 256K | 39.6 GB |
| Gemma 4 (12B, Q4) · llama.cpp (kv: f16) | 12B | 32/33 (97%) | 256K | 18.9 GB |
| Qwen 3.8 (27B, Q4) · MLX | 27B | 31/33 (94%) | — | — |
| Gemma 4 (12B, Q4) · MLX | 12B | 22/33 (67%) | — | — |
Task performance
Task measurements include failures within each stated period or comparison subset. Measurement methods and exclusions are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change. Observed prefill uses partial chat-event rates summarized within each trial. It is not a duration-weighted aggregate of every request, and does not recover pure prefill time.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 33 | 32.09 (24.14–56.38; n=33) | 422.30 (349.36–547.67; n=33) | 3.3 (0.7–20.0; n=33) | 39.60 (36.17–55.69; n=33) | — |
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 33 | 78.51 (56.29–125.57; n=33) | 999.81 (775.33–1,517.85; n=33) | 1.2 (0.5–10.1; n=33) | 18.95 (14.53–32.10; n=33) | — |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 33 | 33.00 (21.00–53.00; n=33) | 564.00 (397.00–805.50; n=33) observed | 2.2 (0.6–27.6; n=33) | 17.60 (17.42–26.97; n=33) | — |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 33 | 37.00 (32.00–44.00; n=33) | 1,146.00 (603.00–1,463.50; n=33) observed | 4.1 (0.6–27.4; n=33) | 10.43 (10.14–12.56; n=33) | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | HTML quality | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 5.28/10 (9 pieces) | 0 | 0 | 24 |
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 4.25/10 (9 pieces) | 0 | 0 | 24 |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 5.28/10 (9 pieces) | 0 | 0 | 24 |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 3.42/10 (9 pieces) | 0 | 0 | 24 |
Tasks in this set: 11.
Office and knowledge work
| Model | Size | Tasks passed | Context | Memory used |
|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp (kv: f16) | 12B | 33/39 (85%) | 256K | 24.6 GB |
| Qwen 3.8 (27B, Q4) · llama.cpp (kv: f16) | 27B | 32/39 (82%) | 256K | 42.3 GB |
| Qwen 3.8 (27B, Q4) · MLX | 27B | 29/39 (74%) | — | — |
| Gemma 4 (12B, Q4) · MLX | 12B | 27/39 (69%) | — | — |
Task performance
Task measurements include failures within each stated period or comparison subset. Measurement methods and exclusions are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change. Observed prefill uses partial chat-event rates summarized within each trial. It is not a duration-weighted aggregate of every request, and does not recover pure prefill time.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 39 | 75.49 (58.35–107.20; n=39) | 1,039.96 (858.18–1,327.01; n=39) | 3.3 (0.5–83.1; n=39) | 24.65 (14.54–33.04; n=39) | — |
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 39 | 29.53 (22.78–54.41; n=39) | 406.81 (342.75–615.57; n=39) | 6.7 (1.1–90.1; n=39) | 42.35 (36.16–50.13; n=39) | — |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 39 | 29.00 (14.00–48.00; n=37) | 574.00 (494.00–862.50; n=39) observed | 6.9 (0.9–90.3; n=39) | 17.75 (17.50–20.53; n=39) | — |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 39 | 36.00 (33.00–49.00; n=39) | 1,146.00 (893.00–1,921.00; n=39) observed | 8.8 (0.7–70.2; n=39) | 10.63 (10.05–18.76; n=39) | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-annotated-bibliography | craftbook-powerpoint-deck | craftbook-research-to-document | craftbook-spreadsheet-model | craftbook-week-plan | docblocks-theme-roundtrip | meeting-followup | wikipedia-research-brief | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 6.08/10 (3 pieces) | 7.00/10 (3 pieces) | 6.89/10 (3 pieces) | 4.83/10 (3 pieces) | 0.92/10 (3 pieces) | 4.33/10 (3 pieces) | 6.58/10 (3 pieces) | 3.33/10 (3 pieces) | 0 | 3 | 12 |
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 6.50/10 (3 pieces) | 7.17/10 (3 pieces) | 6.56/10 (3 pieces) | 6.67/10 (3 pieces) | 1.75/10 (3 pieces) | 7.00/10 (3 pieces) | 7.25/10 (3 pieces) | 6.58/10 (3 pieces) | 0 | 3 | 12 |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 6.25/10 (3 pieces) | 7.25/10 (3 pieces) | 7.00/10 (3 pieces) | 5.33/10 (3 pieces) | 2.83/10 (3 pieces) | 5.89/10 (3 pieces) | 7.17/10 (3 pieces) | 3.25/10 (3 pieces) | 0 | 3 | 12 |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 5.58/10 (3 pieces) | 7.17/10 (3 pieces) | 6.67/10 (3 pieces) | 4.50/10 (3 pieces) | 0.67/10 (3 pieces) | 3.11/10 (3 pieces) | 5.00/10 (3 pieces) | 3.58/10 (3 pieces) | 0 | 3 | 12 |
Tasks in this set: 13.
Engineering work
| Model | Size | Tasks passed | Context | Memory used |
|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp (kv: f16) | 27B | 20/30 (67%) | 256K | 43.3 GB |
| Qwen 3.8 (27B, Q4) · MLX | 27B | 19/30 (63%) | — | — |
| Gemma 4 (12B, Q4) · llama.cpp (kv: f16) | 12B | 12/30 (40%) | 256K | 25.6 GB |
| Gemma 4 (12B, Q4) · MLX | 12B | 12/30 (40%) | — | — |
Task performance
Task measurements include failures within each stated period or comparison subset. Measurement methods and exclusions are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change. Observed prefill uses partial chat-event rates summarized within each trial. It is not a duration-weighted aggregate of every request, and does not recover pure prefill time.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 30 | 27.18 (19.00–46.36; n=30) | 395.63 (237.83–557.47; n=30) | 16.4 (1.2–133.0; n=30) | 43.25 (36.53–50.35; n=30) | — |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 30 | 25.00 (16.00–41.00; n=30) | 512.00 (155.00–735.50; n=30) observed | 19.2 (1.7–113.5; n=30) | 17.84 (17.52–19.29; n=30) | — |
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 30 | 64.21 (38.28–97.17; n=30) | 873.75 (498.48–1,188.73; n=30) | 7.2 (0.9–76.1; n=30) | 25.60 (19.09–30.55; n=30) | — |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 30 | 37.50 (28.00–46.00; n=30) | 1,052.50 (231.00–1,534.50; n=30) observed | 11.6 (1.1–77.9; n=30) | 10.92 (9.97–12.65; n=30) | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-api-contract-review | craftbook-bug-fix-tdd | craftbook-code-review | craftbook-codemod-sweep | craftbook-deep-security-review | craftbook-refactor-module | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 6.17/10 (3 pieces) | 2.22/10 (3 pieces) | 7.17/10 (3 pieces) | 6.67/10 (3 pieces) | 7.33/10 (3 pieces) | 4.78/10 (3 pieces) | 0 | 0 | 12 |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 6.50/10 (3 pieces) | 4.33/10 (3 pieces) | 7.50/10 (3 pieces) | 7.00/10 (3 pieces) | 7.22/10 (3 pieces) | 4.11/10 (3 pieces) | 0 | 0 | 12 |
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 4.00/10 (3 pieces) | 3.89/10 (3 pieces) | 5.75/10 (2 pieces) | 4.56/10 (3 pieces) | 4.33/10 (3 pieces) | 1.56/10 (3 pieces) | 1 | 0 | 12 |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 3.17/10 (3 pieces) | 3.11/10 (3 pieces) | 5.50/10 (3 pieces) | 3.78/10 (3 pieces) | 3.89/10 (3 pieces) | 1.78/10 (3 pieces) | 0 | 0 | 12 |
Tasks in this set: 10.
Complex workflows
| Model | Size | Tasks passed | Context | Memory used |
|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp (kv: f16) | 27B | 19/27 (70%) | 256K | 48.3 GB |
| Qwen 3.8 (27B, Q4) · MLX | 27B | 13/26 (50%) | — | — |
| Gemma 4 (12B, Q4) · llama.cpp (kv: f16) | 12B | 9/26 (35%) | 256K | 27.3 GB |
| Gemma 4 (12B, Q4) · MLX | 12B | 6/27 (22%) | — | — |
Task performance
Task measurements include failures within each stated period or comparison subset. Measurement methods and exclusions are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change. Observed prefill uses partial chat-event rates summarized within each trial. It is not a duration-weighted aggregate of every request, and does not recover pure prefill time.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 27 | 23.82 (15.77–37.66; n=27) | 354.57 (236.28–455.05; n=27) | 26.6 (1.4–94.8; n=27) | 48.29 (36.70–52.05; n=27) | — |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 27 | 21.00 (9.50–38.00; n=27) | 495.00 (259.00–807.00; n=27) observed | 30.2 (0.9–178.9; n=27) | 18.65 (17.56–24.07; n=27) | — |
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 27 | 59.35 (44.39–100.31; n=27) | 923.57 (760.83–1,159.42; n=27) | 13.6 (1.4–76.7; n=27) | 27.29 (17.12–29.45; n=27) | — |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 27 | 36.00 (32.00–41.00; n=27) | 712.00 (461.50–1,058.00; n=27) observed | 15.2 (2.1–30.9; n=27) | 11.02 (10.16–13.92; n=27) | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-invoice-run | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) · llama.cpp | All attempts · native 0.1.54 | 6.40/10 (3 pieces) | 0 | 0 | 24 |
| Qwen 3.8 (27B, Q4) · MLX | All attempts · native 0.1.54 | 5.87/10 (3 pieces) | 0 | 0 | 24 |
| Gemma 4 (12B, Q4) · llama.cpp | All attempts · native 0.1.54 | 5.00/10 (3 pieces) | 0 | 0 | 24 |
| Gemma 4 (12B, Q4) · MLX | All attempts · native 0.1.54 | 4.93/10 (3 pieces) | 0 | 0 | 24 |
Tasks in this set: 9.
October 27–November 2 M5 Max dual-engine comparison
Qwen 3.8 27B Q4 and Gemma 4 12B Q4 completed 516 planned trials on an Apple M5 Max with 128 GB of unified memory, using MLX and llama.cpp with native 0.1.54. Core, productivity, developer and complex-work each have three repetitions per model, engine and scenario: 172 treatments in total. The campaign began October 27 local time (October 28 UTC) and finished November 2.
Qwen on llama.cpp passed 104/129 eligible trials, versus 92/128 on MLX. Gemma passed 86/128 on llama.cpp and 67/129 on MLX. The two original infrastructure exclusions remain excluded from capability scores; their results and performance remain in the download. No trial was replaced or reclassified for publication. The frozen fixed-score means were 8.26, 7.88, 7.85 and 7.19 respectively. Fixed scores and advisory artifact quality measure different things and remain separate.
There are 215 numeric artifact reviews, including 80 artifacts from failed trials. Twelve A/B readouts had incompatible judge specifications; one declared code-review artifact was missing. Neither group receives an invented numeric score. The other 288 trials had no applicable rubric. The downloads retain exact prompts, axes, scores and justifications. These were source reviews capped at 20 KiB, not full visual or interactive reviews; two pet-shop artifacts were truncated at that cap.
Gemma's median in-trial decode rate was 71.76 tokens/s on llama.cpp and 36 on MLX. Qwen's was 28.45 and 28 respectively. These engine estimates differ: MLX uses the median of reported pulses; llama.cpp divides generated tokens by decode time. MLX granted context is unknown, and macOS GPU utilization is unavailable. Performance includes failed and excluded trials, with sample counts, latency, token usage, process RSS and system memory.
MLX prefill rates were recovered on November 3 from 2,175 first-token diagnostics across all 258 original MLX trials. The medians of trial medians are 547 tok/s for Qwen and 1,084 tok/s for Gemma. These rounded, partial provider-turn observations are marked observed in the tables; they are not duration-weighted aggregate request throughput. Exact MLX prefill times remain unknown. The recovery data retains every observation and source hashes. The original trial archive, scores, exclusions and judgments were not changed.
Quantization and speculative decoding also differ. Qwen uses affine 4-bit MLX weights versus UD-Q4_K_M GGUF, with MTP and possible MLX grammar fallback. Gemma uses MXFP4 on MLX without observed MTP, versus qat UD-Q4_K_XL with an external Q4_0 MTP drafter on llama.cpp. The report records these conditions and diagnoses evaluator defects, timeouts, stalled chats and incomplete work.
Download the full report, failure evidence, performance data, judge data, and all trial evidence and postmortems. The download index also links the CSV, aggregate analysis, original runner report and validation records.