2026-10-29 · Windows ARM64 · NVIDIA RTX Spark N1X
A focused rerun on NVIDIA RTX Spark N1X (6144-core GPU, 20-core CPU) · NVIDIA RTX Spark N1X (6144-core Blackwell RTX GPU) · 23 GB: 2 models across 4 task sets. Every round is listed in the model scorecard, with its results side by side in one table.
NVIDIA RTX Spark N1X (6144-core GPU, 20-core CPU) · NVIDIA RTX Spark N1X (6144-core Blackwell RTX GPU) · 23 GB · llama-cpp · focused subset · 3 trials per task · 2026-10-29 · gezel 22fec99111a5 · catalog 0.3.1
Separate continuation: 83 previously unstarted attempts, 59 passes. Scripts ran without the Windows OS sandbox; Node permissions, capabilities and timeouts remained. One undefined parameter in a model-authored recipe was classified as an unevaluable gate; its raw infrastructure exclusion remains recorded with authoring-origin diagnostics. Six unstarted cells moved to the separately labeled final segment. Part of the 258-attempt Spark comparison across four task sets, three repetitions overall. Segments retain separate rounds and counts because neither execution condition has three repeats of every task. Native 0.1.54, llama.cpp CUDA, generalist auto. Task performance includes all attempts; artifact reviews retain separate rubrics. Source reviews can expose incorrect outputs accepted by automatic checks. The downloadable Spark report preserves all raw fixed scores, 119 artifact judgments, exact axes, exclusions, latency, tokens, RAM/GPU samples and review diagnostics.
System memory counts this computer's own memory only. What the model keeps on the separate graphics card isn't included.
General capability
| Model | Size | Tasks passed | Context | System memory |
|---|---|---|---|---|
| Gemma 4 (12B, Q4) (kv: f16) | 12B | 10/11 (some tasks run once — count not rate) | 256K | 9.1 GB |
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 10/11 (some tasks run once — count not rate) | 250K | 6.5 GB |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 11 | 70.64 (45.36–77.83; n=11) | 1,332.12 (876.10–1,416.84; n=11) | 2.6 (1.0–26.1; n=11) | 9.11 (4.83–11.75; n=11) | 13.21 (13.03–20.17; n=11) |
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 11 | 27.79 (14.36–34.01; n=11) | 573.22 (317.14–637.31; n=11) | 4.1 (1.4–38.6; n=11) | 6.55 (3.32–11.36; n=11) | 35.40 (35.23–35.65; n=11) |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | HTML quality | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 4.75/10 (3 pieces) | 0 | 0 | 8 |
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 5.83/10 (3 pieces) | 0 | 0 | 8 |
Tasks in this set: 11.
Office and knowledge work
| Model | Size | Tasks passed | Context | System memory |
|---|---|---|---|---|
| Gemma 4 (12B, Q4) (kv: f16) | 12B | 11/13 (some tasks run once — count not rate) | 256K | 9.8 GB |
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 11/13 (some tasks run once — count not rate) | 250K | 7.6 GB |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 13 | 69.04 (60.89–75.66; n=13) | 1,313.93 (992.52–1,589.97; n=13) | 5.2 (0.8–90.4; n=13) | 9.75 (4.83–11.18; n=13) | 13.35 (13.04–13.60; n=13) |
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 13 | 30.46 (22.80–36.51; n=13) | 615.99 (542.78–688.07; n=13) | 6.8 (1.9–38.7; n=13) | 7.57 (3.40–11.67; n=13) | 35.54 (35.23–35.92; n=13) |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-annotated-bibliography | craftbook-powerpoint-deck | craftbook-research-to-document | craftbook-spreadsheet-model | craftbook-week-plan | docblocks-theme-roundtrip | meeting-followup | wikipedia-research-brief | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 7.00/10 (1 pieces) | 8.50/10 (1 pieces) | 8.00/10 (1 pieces) | 4.50/10 (1 pieces) | 1.75/10 (1 pieces) | 2.33/10 (1 pieces) | 8.50/10 (1 pieces) | 6.00/10 (1 pieces) | 0 | 1 | 4 |
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 7.50/10 (1 pieces) | 8.50/10 (1 pieces) | 7.67/10 (1 pieces) | 6.00/10 (1 pieces) | — | 5.67/10 (1 pieces) | 8.50/10 (1 pieces) | 7.00/10 (1 pieces) | 1 | 1 | 4 |
Tasks in this set: 13.
Engineering work
| Model | Size | Tasks passed | Context | System memory |
|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) (kv: f16) | 27B | 6/10 (some tasks run once — count not rate) | 250K | 8.1 GB |
| Gemma 4 (12B, Q4) (kv: f16) | 12B | 4/10 (some tasks run once — count not rate) | 256K | 10.5 GB |
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 10 | 30.58 (25.46–40.91; n=10) | 657.90 (578.55–685.24; n=10) | 12.4 (2.0–38.1; n=10) | 8.11 (3.91–10.42; n=10) | 35.25 (35.11–35.28; n=10) |
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 10 | 76.07 (69.09–80.21; n=10) | 1,292.13 (1,140.10–1,490.38; n=10) | 8.4 (1.6–30.5; n=10) | 10.49 (8.68–11.38; n=10) | 13.06 (12.91–13.07; n=10) |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-api-contract-review | craftbook-bug-fix-tdd | craftbook-code-review | craftbook-codemod-sweep | craftbook-deep-security-review | craftbook-refactor-module | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 7.00/10 (1 pieces) | 2.67/10 (1 pieces) | 8.00/10 (1 pieces) | 8.33/10 (1 pieces) | 8.33/10 (1 pieces) | 0.67/10 (1 pieces) | 0 | 0 | 4 |
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 4.50/10 (1 pieces) | 1.67/10 (1 pieces) | 6.50/10 (1 pieces) | 2.67/10 (1 pieces) | 4.33/10 (1 pieces) | — | 1 | 0 | 4 |
Tasks in this set: 10.
Complex workflows
Task performance
Task measurements across all attempted trials, including failures. Measurement methods and limits are stated in period labels and round notes. Values are per-trial medians (minimum–maximum; sample count), not admission probes. Wall time includes tools. System RSS and GPU memory are separate peak measurements; do not add their medians. Periods remain separate because hardware performance can change.
| Model | Period | Trials | Output (tok/s) | Prefill (tok/s) | Wall time (min) | Peak system RSS (GiB) | Peak GPU memory (GiB) |
|---|---|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 8 | 66.32 (60.41–85.99; n=8) | 1,233.25 (645.59–1,401.94; n=8) | 12.6 (3.5–29.3; n=8) | 10.40 (9.09–11.74; n=8) | 13.08 (13.02–13.51; n=8) |
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 7 | 27.33 (22.54–38.00; n=7) | 639.87 (539.34–669.15; n=7) | 16.0 (2.4–48.0; n=7) | 8.70 (3.98–10.48; n=7) | 35.27 (35.25–35.50; n=7) |
Artifact reviews
Advisory means out of 10, with eligible artifact counts. Rubrics are separate and are not a combined model score. A dash means no eligible artifact was scored under that rubric. Missing artifacts have no score; excluded reviews do not enter these means. Not applicable means the task had no applicable qualitative rubric. Judge: codex-inline/current-session. Review identity and limitations are recorded with the evaluation.
| Model | Period | craftbook-invoice-run | No artifact | Excluded reviews | Not applicable |
|---|---|---|---|---|---|
| Gemma 4 (12B, Q4) | Separate continuation condition · all attempts | 6.20/10 (1 pieces) | 0 | 0 | 7 |
| Qwen 3.8 (27B, Q4) | Separate continuation condition · all attempts | 7.40/10 (1 pieces) | 0 | 0 | 6 |
Not published — some tasks could not be measured on this round: Gemma 4 (12B, Q4) (1 task(s) unmeasured); Qwen 3.8 (27B, Q4) (3 task(s) unmeasured)
Tasks in this set: 9.
October 28–30 Spark comparison and review data
This note also covers 2026-10-28 · Windows ARM64 · NVIDIA RTX Spark N1X, 2026-10-30 · Windows ARM64 · NVIDIA RTX Spark N1X.
Qwen 3.8 27B Q4 and Gemma 4 12B Q4 completed 258 planned attempts on Windows ARM64 with an NVIDIA RTX Spark N1X and native 0.1.54. The four task sets each have three repetitions per model and scenario across the complete campaign. The tables retain three separate segments: 169 original attempts, 83 continuation attempts, and six final attempts. These segments show counts, not percentages; the original and later script-isolation conditions are not pooled.
On the original condition, Qwen passed 65/84 eligible attempts and Gemma 52/84. Gemma's median engine decode speed was 66.15 tokens/s versus Qwen's 26.44. The two later segments used the same verified condition with the Windows script sandbox disabled while other script permissions remained. All raw outcomes and fixed scores are retained. Two raw infrastructure exclusions remain, including a model-authored unevaluable gate. The published table additionally sets aside the final Gemma fanout attempt because capacity admission blocked it before inference; the download preserves its original failure classification and explains the correction.
All 119 artifact judgments, covering 125 produced paths across 112 trials, are included in the download. The table uses one primary review per trial, keeps different rubrics separate, and excludes six primary A/B reviews whose rubric did not fit the output. Two declared artifacts were missing; 144 trials had no declared quality rubric. These were bounded source reviews, not complete interactive or visual checks. Some automatic passes contained incorrect results, including the final Gemma parameterized report.
Task performance includes every attempt within its own segment: engine-log decode and prefill, duration, and separately reported system RSS and GPU memory. The companion adds first-action and first-artifact timings, tokens, GPU utilization, sample coverage, exact review axes and justifications, per-trial postmortems, and infrastructure/evaluator findings. Local-model token evidence is unavailable for the admission-blocked attempt, rather than reported as zero.
Download the scorecard data, full judging and performance data, reviewed report, and all trials. Download notes and validation explain the preserved baselines, review selection and published attribution.