Every application security team knows the drill: a scanner returns hundreds of flagged vulnerabilities, and almost none of them turn out to matter. Large language models were supposed to fix that. So far, they haven't.
The False Positive Wall
Across tests on more than a dozen scanning tools, over 60% of flagged vulnerabilities turned out to be false positives, sat in code that could never actually be reached, or were low severity noise, according to Arshan Dabirsiaghi, CTO and co-founder of AI-powered AppSec startup Pixee. He plans to present the full results at Black Hat USA in August.
The core issue isn't that AI can't spot patterns. It's that foundational models are still generalists, trained to reason broadly rather than to understand a specific company's codebase, deployment setup, and risk tolerance. Without that context, a model will confidently flag a line of code as vulnerable and try to "fix" it, whether or not the flaw is real.
An Impossible Choice for AppSec Teams
Dabirsiaghi describes the dilemma facing most security teams bluntly: automated tools like Dependabot can bury engineers in constant pull requests, or a human reviewer can manually sort through which vulnerabilities out of the pile are genuinely exploitable. Neither option scales, especially as the volume of vulnerabilities assigned CVE identifiers keeps climbing while detection accuracy has barely moved.
Three types of context separate useful triage from noise, per Dabirsiaghi: organizational context, technical context, and the surrounding code context. His go-to example is MD5 hashing. Flag it at a blanket medium severity by default and you'll be wrong most of the time. Use MD5 to protect passwords and it's a serious finding. Use it purely for collision resistance outside a security path and it's a false positive. The same code pattern, two completely different risk levels, and only context tells you which.
Reachability Is the Real Lever
Beyond severity scoring, the single biggest reduction in noise comes from reachability analysis, essentially asking whether an attacker could ever actually execute the vulnerable code path in production. Research has repeatedly shown that a large share of flagged open source dependencies in real applications are never invoked at runtime, meaning teams have been patching code that was never at risk in the first place.
There's a second, quieter problem too: consistency. Run the same scan twice and an LLM can return two different verdicts on the same finding, calling something a false positive one pass and a true positive the next. That kind of inconsistency erodes trust fast, since a team that can't predict how a tool will classify a repeat finding has little reason to act on any single result.
Building the Harness, Not Just the Model
Dabirsiaghi's argument ultimately isn't anti-AI. It's that raw model output needs a deterministic "harness" around it: the right context injection, the right tooling, and guardrails that keep results stable across repeated runs. Without that scaffolding, AI-assisted scanning risks making the noise problem worse rather than better, since teams end up auto-merging fixes for issues that were never real.
That has real consequences beyond wasted engineering time. Over-fixing based on bad AI output degrades application quality and performance, and it teaches developers to distrust the scanner altogether, which is exactly the outcome AppSec teams were hoping AI would prevent.
The broader lesson for any team evaluating AI-powered scanning tools right now: ask not just how many vulnerabilities a model finds, but how much organizational and code context it was given before making that call, and whether it gives you the same answer twice.