2026-10-21 · Mac · Apple M5 Max
A focused rerun on M5 Max · Apple M5 Max (integrated GPU) · 128 GB: 1 model across 4 task sets. Every round is listed in the model scorecard, with its results side by side in one table.
M5 Max · Apple M5 Max (integrated GPU) · 128 GB · llama-cpp · focused subset · 3 trials per task · 2026-10-21 · gezel bc4af5dc2649 · catalog 0.2.3
PARTIAL OBSERVATIONS: Ornith recorded 85 of 129 planned trials; 84 stable-identity observations are included here. The final drift-flagged failure is retained in the 601-observation companion and omitted from these comparison cells. No third repetition was run; report counts, not rates. Nemotron and MiniCPM were not run. Four suites, 43 suite entries, three requested repetitions. Fixed classifications and scores are unchanged. No pooled qualitative judge score: see scorecard-details.json for exact per-task rubrics and 248 artifact reviews. Performance columns are arithmetic means of short admission probes (each under 256 decode tokens), not sustained in-trial speeds; prefer the companion in-trial distributions for throughput comparisons. Six frozen infrastructure exclusions remain recorded across the package. Models ran sequentially on one device, with model-specific frozen tuning.
General capability
| Model | Size | Tasks passed | Performance | Context | Memory used |
|---|---|---|---|---|---|
| Ornith 1.5 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 21/22 (some tasks run 2× — count not rate) | 190.6 tok/s output 2,619 tok/s prefill | 256K | 28.7 GB |
Tasks in this set: 11.
Office and knowledge work
| Model | Size | Tasks passed | Performance | Context | Memory used |
|---|---|---|---|---|---|
| Ornith 1.5 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 23/26 (some tasks run 2× — count not rate) | 190.6 tok/s output 2,619 tok/s prefill | 256K | 32.3 GB |
Tasks in this set: 13.
Engineering work
| Model | Size | Tasks passed | Performance | Context | Memory used |
|---|---|---|---|---|---|
| Ornith 1.5 (35B-A3B, Q4) (kv: f16) | 35B-A3B | 18/20 (some tasks run 2× — count not rate) | 190.6 tok/s output 2,619 tok/s prefill | 256K | 32.5 GB |
Tasks in this set: 10.
Complex workflows
Not published — some tasks could not be measured on this round: Ornith 1.5 (35B-A3B, Q4) (1 task(s) unmeasured)
Tasks in this set: 9.
October 2026 Mac results and review data
This note also covers 2026-10-16 · Mac · Apple M5 Max.
The Apple M5 Max campaign closed at 601 recorded attempts: 476 passed, 119 failed their tasks, and six were excluded for infrastructure problems. LFM 2.5, Qwen 3.6, Qwen 3.8 and BTL4 each completed 129 attempts. Ornith completed 85 of 129 planned attempts; Nemotron and MiniCPM were not run.
The tables retain 600 attempts in two separate rounds: four complete model arms and the partial Ornith round. One additional failed Ornith attempt had comparison-identity drift and remains in the evidence download with its original classification, outside the comparison tables.
The scorecard data and full judging and performance data are available as JSON. The companion includes all 601 observations, 248 artifact reviews, each task's rubric and sample counts, and per-trial speed and memory measurements. Four declared artifacts were missing, 14 reviews had a mismatched rubric, and 335 tasks had no applicable declared rubric. No review is pending. Different task rubrics are kept separate, so this round has no combined quality score in the tables.
All nine admission probes generated fewer than 256 output tokens; Ornith's single probe generated 84. Treat the table's speeds as preliminary. The companion also records engine-derived speeds during the actual tasks and their sample counts. Download notes and validation details explain the populations and caveats.