2026-10-16 · Mac · Apple M5 Max
A model scorecard round on M5 Max · Apple M5 Max (integrated GPU) · 128 GB: 4 models across 4 task sets. Every round is listed in the model scorecard, with its results side by side in one table.
M5 Max · Apple M5 Max (integrated GPU) · 128 GB · llama-cpp · 3 trials per task · 2026-10-16 · gezel bc4af5dc2649 · catalog 0.2.3
Completed arms only: LFM 2.5, Qwen 3.6, Qwen 3.8 and BTL4 each recorded all 129 trials (516 observations). Ornith is retained in a separate partial round, not mixed into this headline table. Four suites, 43 suite entries, three requested repetitions. Fixed classifications and scores are unchanged. No pooled qualitative judge score: see scorecard-details.json for exact per-task rubrics and 248 artifact reviews. Performance columns are arithmetic means of short admission probes (each under 256 decode tokens), not sustained in-trial speeds; prefer the companion in-trial distributions for throughput comparisons. Six frozen infrastructure exclusions remain recorded across the package. Models ran sequentially on one device, with model-specific frozen tuning.
General capability
| Model | Size | Tasks passed | Performance | Context | Memory used |
|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 33/33 (100%) | 36.7 tok/s output 501 tok/s prefill | 256K | 40.4 GB |
| Qwen 3.6 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 31/33 (94%) | 133.9 tok/s output 1,441 tok/s prefill | 256K | 29.6 GB |
| BTL-4 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 28/33 (85%) | 119.4 tok/s output 2,828 tok/s prefill | 256K | 28.1 GB |
| LFM2.5 (24B-A2B, Q4) (kv: f16) | 24B-A2B | 24/32 (75%) | 182.6 tok/s output 4,711 tok/s prefill | 128K | 17.7 GB |
Tasks in this set: 11.
Office and knowledge work
| Model | Size | Tasks passed | Performance | Context | Memory used |
|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 37/39 (95%) | 36.7 tok/s output 501 tok/s prefill | 256K | 43.7 GB |
| Qwen 3.6 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 35/39 (90%) | 133.9 tok/s output 1,441 tok/s prefill | 256K | 32.7 GB |
| BTL-4 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 31/39 (79%) | 119.4 tok/s output 2,828 tok/s prefill | 256K | 30.0 GB |
| LFM2.5 (24B-A2B, Q4) (kv: f16) | 24B-A2B | 28/39 (72%) | 182.6 tok/s output 4,711 tok/s prefill | 128K | 18.1 GB |
Tasks in this set: 13.
Engineering work
| Model | Size | Tasks passed | Performance | Context | Memory used |
|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 27/30 (90%) | 36.7 tok/s output 501 tok/s prefill | 256K | 45.4 GB |
| Qwen 3.6 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 26/30 (87%) | 133.9 tok/s output 1,441 tok/s prefill | 256K | 34.0 GB |
| BTL-4 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 25/30 (83%) | 119.4 tok/s output 2,828 tok/s prefill | 256K | 29.3 GB |
| LFM2.5 (24B-A2B, Q4) (kv: f16) | 24B-A2B | 18/30 (60%) | 182.6 tok/s output 4,711 tok/s prefill | 128K | 19.1 GB |
Tasks in this set: 10.
Complex workflows
| Model | Size | Tasks passed | Performance | Context | Memory used |
|---|---|---|---|---|---|
| Qwen 3.6 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 19/25 (76%) | 133.9 tok/s output 1,441 tok/s prefill | 256K | 38.0 GB |
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 19/27 (70%) | 36.7 tok/s output 501 tok/s prefill | 256K | 47.0 GB |
| BTL-4 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 14/26 (54%) | 119.4 tok/s output 2,828 tok/s prefill | 256K | 33.5 GB |
| LFM2.5 (24B-A2B, Q4) (kv: f16) | 24B-A2B | 7/27 (26%) | 182.6 tok/s output 4,711 tok/s prefill | 128K | 17.8 GB |
Tasks in this set: 9.
October 2026 Mac results and review data
This note also covers 2026-10-21 · Mac · Apple M5 Max.
The Apple M5 Max campaign closed at 601 recorded attempts: 476 passed, 119 failed their tasks, and six were excluded for infrastructure problems. LFM 2.5, Qwen 3.6, Qwen 3.8 and BTL4 each completed 129 attempts. Ornith completed 85 of 129 planned attempts; Nemotron and MiniCPM were not run.
The tables retain 600 attempts in two separate rounds: four complete model arms and the partial Ornith round. One additional failed Ornith attempt had comparison-identity drift and remains in the evidence download with its original classification, outside the comparison tables.
The scorecard data and full judging and performance data are available as JSON. The companion includes all 601 observations, 248 artifact reviews, each task's rubric and sample counts, and per-trial speed and memory measurements. Four declared artifacts were missing, 14 reviews had a mismatched rubric, and 335 tasks had no applicable declared rubric. No review is pending. Different task rubrics are kept separate, so this round has no combined quality score in the tables.
All nine admission probes generated fewer than 256 output tokens; Ornith's single probe generated 84. Treat the table's speeds as preliminary. The companion also records engine-derived speeds during the actual tasks and their sample counts. Download notes and validation details explain the populations and caveats.