Spark Quality Index

What local models actually score on real local hardware. Every number below is a full 164-problem HumanEval+ run executed on an NVIDIA DGX Spark (GX10, 128 GB unified memory) — the same class of machine you can rent from us. No cloud numbers passed off as local performance, no cherry-picked subsets, and the runs that broke are listed right here instead of quietly deleted.

bare is the model alone, one attempt per problem — those numbers are verified and published below. The with-Chad comparison is being re-measured right now under a stricter rule; the next section explains exactly why and what changed.

Re-verification in progress

The with-Chad numbers are being re-measured

We hold this page to the standard in the footer, so here is the honest state of it. While reviewing how the with-Chad runs were scored, we found that Chad's repair step was allowed to read the benchmark's own grading tests when it retried a problem. That inflates a benchmark score and does not reflect what the tool does for you on your own code, so every previously published with-Chad and lift figure has been pulled until the runs are redone under a stricter rule: on retry, Chad may only see the examples in the problem statement, never the hidden grading tests.

The bare numbers below are unaffected — a bare run takes one attempt with no feedback of any kind — and they stand. New with-Chad numbers go back up here the moment the honest re-runs land. If you want to be told when they do, ask.

Bare baselines — verified

What each model scores on its own, on this hardware

One attempt per problem, no feedback, no retries. These are the honest floor: what the raw model does on a DGX Spark before any tooling. The with-Chad rows return here once the re-runs described above are complete.

Run	Pass	Rate	Avg tries	Wall time	Date
bare qwen2.5-coder:14b	136/164	82.9%	1.00	47m	2026-05-23
bare qwen3:14b	131/164	79.9%	1.00	283m	2026-05-23
bare gemma3:27b	124/164	75.6%	1.00	96m	2026-05-23
bare codestral:22b	120/164	73.2%	1.00	41m	2026-05-23
bare nemotron:70b	120/164	73.2%	1.00	144m	2026-07-08
bare mistral-nemo:12b	86/164	52.4%	1.00	20m	2026-07-05

HumanEval+ bare baselines. The with-Chad comparison is being re-measured under the stricter retry rule described above and will be republished when the honest runs land.

What we don't publish

Excluded and quarantined runs

A results page you can trust has to show its trash can, not just its trophies.

Quarantined: deepseek-coder-v2:16b (both bare and Chad, 2026-05-23) — 0/164 in under six minutes, which is the signature of a broken run, not a real score. It stays out of the tables until a clean re-run replaces it.

Excluded: six partial and smoke-test runs (3–10 problems each), used for development sanity checks. Small samples flatter whoever ran them; only complete 164-problem runs make this page.

Methodology

How these numbers are made

Suite: HumanEval+ (the EvalPlus extended test set), all 164 problems, no sampling. A problem passes only if every test passes.
Serving: Ollama on an NVIDIA DGX Spark GX10, model tags exactly as shown, Ollama's default quantization unless a row says otherwise.
Modes: bare = one attempt, no help. Chad w/ = Chad runs the model's code against tests and has the model repair failures, at most three attempts per problem. The with-Chad runs are being re-measured so that on retry Chad sees only the problem statement's own examples, never the hidden grading tests.
Grading safety: generated code is graded in an isolated worker with an 8-second per-test limit and a 180-second hard kill, so a pathological solution is recorded as a failure instead of stalling the run.
Reliability: every problem's result is written to disk the moment it finishes; interrupted runs resume exactly where they stopped; infrastructure failures are re-run rather than recorded as model failures. This page is generated from those raw result files, not typed in by hand.

Last updated 2026-07-08. New completed runs are added as they land. Want a model or quantization measured? Ask.