2026-10-28 · DGX Spark Class
A model scorecard round on Arm Cortex-X925 + Cortex-A725 (20 cores) · NVIDIA GB10 · 122 GB: 2 models across 4 task sets. Every round is listed in the model scorecard, with its results side by side in one table.
Arm Cortex-X925 + Cortex-A725 (20 cores) · NVIDIA GB10 · 122 GB · llama-cpp · 3 trials per task · 2026-10-28 · gezel 22fec991 · catalog 0.3.1
Qwen 3.8 27B Q4 and Gemma 4 12B Q4 on a DGX Spark (NVIDIA GB10, Linux) with native 0.1.54: llama.cpp on CUDA, with MTP speculative decoding on by default for both models. 258 trials, three per model and scenario, ran October 28–30, 2026; none was repeated or replaced, and every raw verdict is kept. Three failures are set aside as infrastructure: one Linux script-sandbox failure and two gates the model wrote itself, which the harness classed as setup failures. Evaluation homes had package scripts switched off, so bug-fix-tdd, codemod-sweep and refactor-module could not pass for either model; without those 18 trials the developer failure rate is 0/21 for Qwen and 7/21 for Gemma. docblocks-theme-roundtrip lacked its DocBlocks tools in all six trials, and the harness stopped two Qwen route-multi trials and one Qwen author-fanout trial while they were still progressing. Task speeds are engine-log medians over every attempt. Memory is not shown: the GB10 shares its memory with the GPU in a way the measurement cannot see. Artifact reviews are by Claude Sonnet 4.6, one column per rubric. The six A/B readout reviews used a mismatched rubric and seven reviews graded a starting or incidental page instead of the work, so those 13 are excluded.
Memory isn't shown for this round. This machine's graphics chip shares the computer's memory in a way our measurement can't see, so the number would read far below what the model really uses.
General capability
| Model | Size | Tasks passed | Context |
|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 33/33 (100%) | 256K |
| Gemma 4 (12B, Q4) (kv: f16) | 12B | 32/33 (97%) | 256K |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 33 | 30.63 (20.73–37.71; n=33) | 781.15 (733.93–816.27; n=33) | 2.2 (0.7–9.7; n=33) | — | — |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 33 | 79.52 (63.35–90.79; n=33) | 1,610.34 (1,169.66–1,697.62; n=33) | 1.2 (0.5–5.4; n=33) | — | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: anthropic-cli/claude-sonnet-4-6. Review identity and limitations are recorded with the evaluation.
| Model | Period | HTML quality | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 8.23/10 (9 pieces) | 0 | 0 | 24 |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 6.88/10 (9 pieces) | 0 | 0 | 24 |
Tasks in this set: 11.
Office and knowledge work
| Model | Size | Tasks passed | Context |
|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 34/39 (87%) | 256K |
| Gemma 4 (12B, Q4) (kv: f16) | 12B | 32/39 (82%) | 256K |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 39 | 29.38 (20.96–35.65; n=39) | 790.15 (719.00–816.82; n=39) | 6.1 (1.0–90.2; n=39) | — | — |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 39 | 76.90 (64.29–87.37; n=39) | 1,578.28 (1,456.91–1,798.06; n=39) | 3.0 (0.5–64.0; n=39) | — | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: anthropic-cli/claude-sonnet-4-6. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-annotated-bibliography | craftbook-powerpoint-deck | craftbook-research-to-document | craftbook-spreadsheet-model | craftbook-week-plan | docblocks-theme-roundtrip | meeting-followup | wikipedia-research-brief | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 8.80/10 (3 pieces) | 7.27/10 (3 pieces) | 8.70/10 (3 pieces) | 8.50/10 (3 pieces) | 8.77/10 (3 pieces) | 8.57/10 (3 pieces) | 8.70/10 (3 pieces) | 8.43/10 (3 pieces) | 0 | 3 | 12 |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 7.70/10 (3 pieces) | 7.37/10 (3 pieces) | 8.03/10 (3 pieces) | 8.00/10 (3 pieces) | 5.27/10 (3 pieces) | 4.77/10 (3 pieces) | 8.20/10 (3 pieces) | 7.43/10 (3 pieces) | 0 | 3 | 12 |
Tasks in this set: 13.
Engineering work
| Model | Size | Tasks passed | Context |
|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 21/30 (70%) | 256K |
| Gemma 4 (12B, Q4) (kv: f16) | 12B | 14/30 (47%) | 256K |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 30 | 28.59 (22.71–35.05; n=30) | 769.68 (593.52–810.47; n=30) | 12.7 (1.3–85.5; n=30) | — | — |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 30 | 78.30 (67.48–89.52; n=30) | 1,488.67 (1,142.14–1,706.31; n=30) | 6.5 (0.9–39.8; n=30) | — | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: anthropic-cli/claude-sonnet-4-6. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-api-contract-review | craftbook-bug-fix-tdd | craftbook-code-review | craftbook-codemod-sweep | craftbook-deep-security-review | craftbook-refactor-module | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 8.83/10 (3 pieces) | 6.67/10 (3 pieces) | 8.67/10 (3 pieces) | 7.57/10 (3 pieces) | 8.90/10 (3 pieces) | 6.90/10 (3 pieces) | 0 | 0 | 12 |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 6.83/10 (3 pieces) | 5.23/10 (3 pieces) | 7.67/10 (3 pieces) | 3.77/10 (3 pieces) | 7.37/10 (3 pieces) | 1.97/10 (3 pieces) | 0 | 0 | 12 |
Tasks in this set: 10.
Complex workflows
| Model | Size | Tasks passed | Context |
|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 18/25 (72%) | 256K |
| Gemma 4 (12B, Q4) (kv: f16) | 12B | 12/26 (46%) | 256K |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 27 | 24.82 (20.76–35.64; n=27) | 757.21 (633.61–807.13; n=27) | 19.8 (1.2–100.9; n=27) | — | — |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 27 | 69.77 (60.37–89.12; n=27) | 1,488.31 (1,105.60–1,656.97; n=27) | 9.4 (1.2–44.5; n=27) | — | — |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: anthropic-cli/claude-sonnet-4-6. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-invoice-run | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Native 0.1.54 · CUDA · all attempts | 6.47/10 (3 pieces) | 0 | 3 | 21 |
| Gemma 4 (12B, Q4) | Native 0.1.54 · CUDA · all attempts | 4.93/10 (3 pieces) | 0 | 4 | 20 |
Tasks in this set: 9.
October 28–30 DGX Spark results and review data
Qwen 3.8 27B Q4 and Gemma 4 12B Q4 completed all 258 trials on a DGX Spark (NVIDIA GB10, Linux) with native 0.1.54: three per model and scenario across the four task sets. Qwen passed 106 of 129 and Gemma 90 of 129. Three failures are set aside as infrastructure: a gap in Linux script sandboxing, and two gates the model wrote itself, which the harness classed as setup failures.
Read the engineering set with care. Package scripts were switched off in the evaluation homes, so three tasks that need a test run could not pass for either model. The table keeps those 18 failures: Qwen failed 9 of 30 engineering trials and Gemma 16 of 30. Excluding them, Qwen failed 0 of 21 and Gemma 7 of 21. Docblocks-theme-roundtrip lacked its tools in all six trials, and the harness stopped three Qwen complex-workflow trials while they were still progressing. The reviewed report explains each case.
Claude Sonnet 4.6 reviewed 121 artifacts after the sweep. The Artifact reviews tables use 108 of them, one column per rubric. They exclude 13 reviews that used the wrong rubric or graded a starting page instead of the work; the other 137 tasks had no rubric to apply. The Task performance tables give output speed, prefill speed and wall time from the engine's logs for every attempt. Both models used speculative decoding, which is on by default. Memory is left out because this machine's shared memory is invisible to the measurement.
Download the scorecard data, full results with judging and performance, reviewed report, and all trials. Download notes explain the set-aside trials, the review selection and how to check the data.