Where static analysis is easy vs hard: a walk through 2,740 OWASP tests

TL;DR — The OWASP Benchmark is 2,740 labelled Java test cases across 11 vulnerability classes. Running our engine over all of them and reading the results by category reveals a clean rule: a scanner's accuracy tracks how local the bug's signal is. Weak-crypto bugs (one function call) score 100% precision; injection bugs (untrusted input traced to a sink) drop to 39–59% recall. Two classes scored a flat zero at first measurement because we shipped no rules for them — and the engine update that closed both turned into a natural experiment that confirmed the rule: perfectly-local insecure-cookie flags jumped straight to +1.000, while dataflow-hungry LDAP injection landed mid-table at +0.499, exactly where its injection siblings sit. We publish the whole scorecard because the shape is the interesting part.

Updated 2026-08-20: figures refreshed to the current measured run (2026-08-04, engine commit 515fff7, 341 rules — the release that added LDAP-injection and insecure-cookie detection). The original 2026-07-21 run, including its two zero-score classes, is kept in the narrative below because the before/after IS the finding.

The benchmark

The OWASP Benchmark v1.2 is the standard test suite for SAST tools: 2,740 Java cases, each labelled true (a real vulnerability) or false (a safe look-alike). The headline metric is the Benchmark Score = true-positive rate − false-positive rate (Youden's J) — it rewards catching real bugs and punishes crying wolf equally.

We measured Zennoxa Shield across the full suite (current run: commit 515fff7, 2026-08-04): Benchmark Score +0.582, precision 92.5%, recall 63.7%, FPR 5.5%. (Our first published run — commit 9d182dd, 2026-07-21 — scored +0.547 with two uncovered classes at zero; more on that below.) But the overall number hides the story. Here's every category, ranked:

Vulnerability class	Recall	Precision	Benchmark Score
Weak randomness	100.0%	100.0%	+1.000
Insecure cookie	100.0%	100.0%	+1.000
Weak cipher	80.8%	100.0%	+0.808
Weak hash	69.0%	100.0%	+0.690
Cross-site scripting	57.3%	92.2%	+0.516
LDAP injection	59.3%	84.2%	+0.499
XPath injection	46.7%	100.0%	+0.467
Command injection	57.9%	82.0%	+0.451
Path traversal	53.4%	85.5%	+0.445
SQL injection	39.0%	84.1%	+0.303
Trust boundary	47.0%	79.6%	+0.237

The pattern: locality predicts accuracy

Read that table top to bottom and a gradient appears.

Top — "syntactic" bugs (100% precision). Weak randomness, weak cipher, weak hash, insecure cookie. The signal is local: a single call or attribute — new Random(), DES, MessageDigest.getInstance("MD5"), a cookie created without setSecure(true). You don't need to know where any data came from; the presence of the API is the finding. So a scanner can be both complete and precise here — 100% precision on all of them. If a tool can't nail these, be worried.

Middle — injection (39–59% recall). XSS, LDAP injection, command injection, path traversal, SQL injection. These are not local: a bug exists only if untrusted input reaches a dangerous sink. The scanner has to trace data from a source (a request parameter) through assignments and helper methods to the sink (a query, a shell, a file path). Miss one hop and you miss the bug (lower recall); over-approximate and you flag safe code (lower precision). SQL injection is the hardest here at 39% recall — SQL gets built through the most indirection.

The former zeros — a natural experiment. In our first published run (2026-07-21) LDAP injection and insecure-cookie flags both scored a flat 0.0: we shipped no rules for them, and we said so — a Benchmark Score of exactly 0 means "didn't play", a coverage gap rather than a subtle failure. The engine release that followed added detection for both, and the two classes landed exactly where the locality rule predicts: insecure cookie went straight to +1.000 (it is a purely local attribute check, so shipping the rule buys the whole class), while LDAP injection arrived at +0.499 (it is an injection class needing source-to-sink dataflow, so it slots into the middle of the table next to command injection and path traversal). One engine change, two new data points, same gradient.

Why this matters if you're choosing or trusting a scanner

• A single headline number is misleading. "92% precision" is true for us overall, but per class it spans 100% on weak crypto down to 79.6% on trust boundary — and recall spans 100% down to 39%. Always ask for the per-class breakdown.
• Recall is where SAST quietly fails. Precision gets marketed; recall gets buried. On the hardest injection classes even a well-tuned engine catches well under half in this suite. SAST is a filter, not a guarantee — pair it with review and test coverage for the injection-heavy parts of your app.
• "Syntactic" classes are table stakes. If a scanner misses weak crypto/hash/ random, that's a red flag — those are the easy ones.

Honest limitations

• This is one benchmark, in Java. Real codebases have framework magic and cross-file flows the Benchmark's self-contained cases don't fully exercise; scores elsewhere will differ.
• We optimise precision-first, which caps recall by design — a tool tuned the other way would trade our low 5.5% FPR for higher recall and more noise.
• The 2026-07-21 zeros were real coverage gaps, not measurement artifacts — and closing them (2026-08-04 run) is documented above rather than silently rewritten.

Reproduce it

git clone --depth 1 https://github.com/OWASP-Benchmark/BenchmarkJava
OWASP_BENCH=$PWD/BenchmarkJava make bench-owasp

You get the same per-category scorecard. The full ranked numbers are in data.json, and the current measured run is always published at zennoxa.com/benchmark. For how other engines score on the same suite, see OWASP's own published scorecards — we cite theirs rather than measuring other tools ourselves.

---

Zennoxa Research publishes reproducible security data. This study reports only our own engine's results on a public dataset.