We ran 2,300 pentests in 3 weeks. Here's what our AI finds

Over three weeks, our agents ran about 2,300 autonomous penetration tests against live web applications and source code. A typical run lasted around an hour, some pushed past three. This post is a breakdown of what came out the other side: how severe the findings were, what types showed up most, and where they diverge from what a traditional scanner reports.

Short version: roughly 37% of findings were high or critical, and the bugs that landed hardest were authorization and business-logic flaws, the kind a signature-based scanner rarely catches.

AI pentesting vs traditional scanners

A scanner matches patterns. It sees a library version, a header, or a response shape that looks wrong, and it files a finding. Whether the issue is actually reachable and exploitable in your running app is left for a human to check later.

That gap is expensive. In an August 3, 2026 survey reported by IT Security Guru, one practitioner described an AI tool that returned 300 findings, 250 of which turned out to be junk, including "potential SQLi that is not exploitable" and CVEs that do not exist. Across that survey, false-positive rate and signal quality were the top selection criterion, cited by 63% of respondents, ahead of cost. Academic work points the same direction: a 2026 benchmark, "Are Frontier LLMs Ready for Cybersecurity?" (arXiv:2605.23243), measured 10 to 50% false-positive rates for frontier models on white-box detection, and found that structured testing methodology, not raw model size, was the main lever for real detection.

Our agents work from the opposite end. A finding ships only when the agent reproduces it and captures the evidence: the request, the response, the state change. If it can't prove the bug, the bug doesn't get filed. That rule is the reason the numbers below are worth reading.

The data: severity distribution

We pulled a random sample of 1,200 findings from the run and graded them by severity.

Critical 11%High 26%Medium 30%Low 28%Info 5%Severity of a random 1,200-finding sample from the run. Critical and high, in blue, account for 37% of everything filed.

High and critical together come to about 37%. For an autonomous system running with no human in the loop during the test, that is a high hit rate of serious, reproducible issues rather than a pile of low-severity noise.

Top vulnerability classes: authorization, IDOR, SSRF, business logic

We categorized the same 1,200 findings by keyword analysis of their titles. Categories overlap, since one finding can touch several, so read these as themes rather than a clean partition.

Broken authn and access control42%Exposed secrets and configuration17%IDOR and object-level authz15%Information disclosure12%Injection (SQL, command, template)11%Open redirects5%Server-side request forgery3%Share of the 1,200-finding sample touching each theme. Themes overlap, so the shares don't sum to 100%.

Authorization dominates. Three anonymized examples from the run, each reproduced with a working exploit:

• Cross-tenant data access through unguarded server actions, letting one tenant read another tenant's data. Graded critical.
• An internal admin dashboard exposed with no authentication, leaking queue contents and operational stats. Graded high.
• A missing auth check on an API endpoint that returned another user's run data and stream. Graded high.

The most common critical finding in the run: a request that crosses a tenant boundary the application assumed nobody would test.

Every one of these maps to the OWASP API Security Top 10: broken object-level authorization (BOLA), broken authentication, and broken function-level authorization. They are logic and access-control failures, not memory-safety bugs or known-CVE matches.

Why scanners miss these

Authorization bugs need context a scanner does not have. "User A should not be able to read user B's invoice" is a statement about intent, roles, and tenancy. A pattern matcher has no model of who should access what. An agent does: it logs in as one user, performs an action, then replays the same request as a different user and compares the results. When the second user gets data they shouldn't, that's the finding, proven.

A scanner reports what the surface looks like. An agent walks the chain, one step at a time, until it reaches something worth reporting.

Business-logic and multi-step chains are the same story at a larger scale. An SSRF that looks minor on its own becomes critical once the agent reaches an internal endpoint and pulls credentials. A scanner sees one request at a time. An agent holds the whole chain in view and keeps going until it either proves impact or rules the path out.

The second reason is proof. Our agents don't submit a list of endpoints that "appear vulnerable." They reproduce the exploit and attach the evidence, which is why the 37% high-and-critical share reflects confirmed issues rather than guesses.

FAQ

What vulnerabilities can AI penetration testing find?

AI penetration testing most often finds:

• Broken authentication and access control, such as login bypasses and missing auth on API endpoints
• IDOR and cross-tenant access, where one user or tenant reads another's data
• Business-logic flaws that abuse an application's intended workflow
• Server-side request forgery (SSRF), including chains that reach internal services
• Exposed secrets and configuration, such as API keys and unauthenticated admin dashboards
• Injection, including SQL, command, and template injection
• Information disclosure through verbose errors and unprotected data

Does AI penetration testing have false positives?

It can, and independent research puts frontier-model false-positive rates between 10 and 50%. The mitigation is to require proof: our agents only file a finding after reproducing it and capturing the request and response, so unverified guesses don't reach the report.

Can AI penetration testing replace vulnerability scanners?

They do different jobs. Scanners are fast at breadth and known-CVE coverage. Autonomous agents add depth: authorization logic, multi-step chains, and exploit confirmation. Most teams run both.

What is IDOR, and why do scanners miss it?

IDOR (insecure direct object reference) is when an app lets a user access an object, a record, a file, an account, that isn't theirs by changing an identifier. Scanners miss it because deciding whether access is authorized requires knowing who should own that object, which a pattern rule can't infer. An agent tests it directly by acting as two different users and comparing what each can reach.

If a finding can't be reproduced, it doesn't ship. That single rule is what makes the 37% number mean something.

Point the same agents at your own stack: see first findings within 24 hours.