The Version-Drift Problem: Is a Published SAST 'Accuracy' Score Even Reproducible?

TL;DR — A SAST "accuracy" quoted against the OWASP Benchmark has at least three undisclosed degrees of freedom, and each one moves the number. Version: the suite ships as v1.1 (21,041 test cases) and v1.2 (2,740) — a 7.7× difference in the population that false-positive rate is computed over. Subset: the headline is an equal-weighted mean over 11 categories, so which categories you average changes it — a fixed reference detector that is perfect on the injection family and blind elsewhere scores 45.5 over all 11 categories but 100 over injection-only, a 54.5-point swing from subset choice alone. Weighting: that official macro-average gives every category 9.1% of the headline, so the smallest category (XPath injection, 35 cases) counts 14.4× more than it would under a case-weighted average, while the largest (SQLi, 504 cases) counts about half as much. Everything below is computed live from the public v1.2 ground truth. A score with no stated version, subset, or matching rule isn't a comparable quantity — it can't be re-derived.

Why this matters

Buyers, security teams and researchers compare SAST tools with a single quoted number — "N% precision," a "detection rate," a "Youden score" — against a public benchmark. But the benchmark's own scoring pipeline has documented degrees of freedom. If they aren't disclosed, two numbers can't be compared, and neither can be reproduced from scratch. This study quantifies, from the OWASP Benchmark's public ground truth, how far a single number moves purely from provenance. It's a statement about measurement, not about any tool — no scanner is named or scored here.

The scoring pipeline, and where it bends

The OWASP Benchmark scores a tool with the Youden index — (sensitivity + specificity) − 1, normalized to 0–100 — computed per category and then averaged equally over the 11 categories. The ground truth is a four-column CSV (test name, category, real-vulnerability true/false, CWE), read by the project's official BenchmarkUtils scorer through a config file. Because version, category subset, and CWE matching are configuration inputs — not fixed constants — the same detections can produce different headline numbers.

Knob 1 — Version: two suites, one name

The Benchmark exists in two published sizes:

Version	Test cases
v1.1	21,041
v1.2	2,740

An "OWASP Benchmark score" with no version stated is ambiguous between two populations that differ 7.7× in size — and false-positive rate, the specificity half of the Youden index, is a fraction computed over the (very different) count of non-vulnerable cases in each.

Knob 2 — Subset: an equal-weighted mean is a choice

The v1.2 ground truth has 2,740 cases across 11 categories, unevenly distributed:

Category	CWE	Cases	Real	Fake	Real %
sqli	89	504	272	232	54.0%
weakrand	330	493	218	275	44.2%
xss	79	455	246	209	54.1%
pathtraver	22	268	133	135	49.6%
cmdi	78	251	126	125	50.2%
crypto	327	246	130	116	52.8%
hash	328	236	129	107	54.7%
trustbound	501	126	83	43	65.9%
securecookie	614	67	36	31	53.7%
ldapi	90	59	27	32	45.8%
xpathi	643	35	15	20	42.9%

Because the headline averages the 11 categories equally, which categories are in the average changes it. Take a transparent reference detector — a neutral probe, not any product — that scores a perfect Youden 1.0 on the injection family (cmdi, sqli, xss, ldapi, xpathi) and 0.0 everywhere else. Its headline is:

• over all 11 categories: 5/11 = 45.5
• over the injection-only subset: 100.0

The identical detector reports a 54.5-point difference depending only on which categories the average includes. "Precision on injection" and "score on the full suite" are not the same claim, and a headline that states neither is not reconstructable.

Knob 3 — Weighting: macro vs. micro redistributes influence

The official mean is a macro-average (every category = 1/11 = 9.1% of the headline). A case-weighted micro-average would instead weight each category by its share of the 2,740 cases. On the real distribution those two are far apart:

• Macro gives every category 9.1%.
• Micro gives SQLi 18.4% but XPath injection 1.3%.

So under the official macro-average the 35-case XPath category counts 14.4× more toward the headline than it would under case-weighting, and the 504-case SQLi category counts about half as much. The same per-category results produce a different headline depending on which averaging the scorer is configured for.

Knob 4 — Matching: which CWE counts as a hit

Every category maps to a CWE (sqli→CWE-89, xss→CWE-79, cmdi→CWE-78, pathtraver→CWE-22, …). Whether a finding is matched to a test case by strict per-CWE identity or by a category-family mapping changes which detections count as true positives versus false positives — a documented configuration of the scorer, not a property of the tool. A tool that flags the right line with an adjacent CWE id scores differently under the two rules.

What the numbers say

• Base rates aren't uniform either: the real-vulnerability fraction ranges from 42.9% (xpathi) to 65.9% (trustbound), overall 51.6% — so per-category specificity/sensitivity sit on different denominators.
• Provenance dominates comparability. Version (7.7× population), subset (up to a 54.5-point swing for one detector), and weighting (14.4× category-influence gap) each move the number without any change to the underlying detections.
• Conclusion the data supports: a headline SAST score with no pinned benchmark version, no stated CWE subset, and no stated matching rule is not independently reproducible — two such numbers are not comparable.

Why we pin provenance (category-level, honest)

This is the empirical case for a reproducibility discipline, not a claim that any one tool scores higher. We publish our own OWASP-Java result with the version, the exact category set, and the matching rule attached, pinned to a release and a machine-readable benchmark.json, so a third party can re-run it and land on the same number. The point of this study is category-level: a score without stated provenance can't be reconstructed, so it can't be compared — which is exactly the gap a pinned-provenance methodology closes.

Limitations & honest caveats

• The reference detector is a neutral arithmetic probe, not a scanner and not a measurement of any tool's quality — it exists only to isolate how scoring configuration alone moves a headline.
• The v1.1 = 21,041 figure is the OWASP project's published case count; the drift computations in Knobs 2–4 are computed here from the v1.2 ground truth we pin below.
• We describe the matching knob rather than assign it a single number, because its magnitude depends on a real tool's CWE-attribution behavior, which we deliberately do not model.

Reproduce it yourself

git clone https://github.com/OWASP-Benchmark/BenchmarkJava
# Category counts, base rates, and macro-vs-micro weights come straight from the CSV:
python3 - <<'PY'
import csv
from collections import defaultdict
rows=list(csv.DictReader(open('BenchmarkJava/expectedresults-1.2.csv')))
cat=defaultdict(lambda:[0,0])
for r in rows:
    real = r[' real vulnerability'].strip().lower()=='true'
    cat[r[' category'].strip()][0 if real else 1]+=1
N=len(rows)
for c in sorted(cat):
    real,fake=cat[c]; n=real+fake
    print(f"{c:<13}{n:>5} macro={100/11:.1f}% micro={100*n/N:.1f}% real%={100*real/n:.1f}")
PY

The suite and its official scorer (OWASP BenchmarkUtils) are public; scoring is done by the project's own tool via a config that references the expectedresults CSV version. No scanner is named or scored in this study — the tool axis is a transparent reference probe.

FAQ

Is the OWASP Benchmark "broken"? No — it's a well-designed public suite. The point is that a number quoted from it is only meaningful with its version, category subset, and matching rule attached; the suite gives you all three, but headlines usually drop them.

Why does macro-averaging matter so much? Because the categories are uneven (35 to 504 cases), so equal weighting and case weighting distribute the headline's influence very differently — up to 14.4× for the smallest category.

Where does Shield fit? We publish per-benchmark, per-version, per-CWE numbers pinned to a release and a benchmark.json rather than one blended headline — this study is why. It evaluates no scanner, ours included.