HN user

mens_rea

7 karma
Posts0
Comments7
View on HN
No posts found.

Deeply flawed paper for several reasons:

* Small data set of 2 runs (!!)

* Exaggerated claims (saying A1 beat 50% of testers, yet only 4/10 testers found LESS vulns than A1, and A1 had a nearly 50% false positive rate)

* AI agents were given 16 hours while human testers were given 10

* Their human testers gave up when a modern browser refused to open a webpage with weak TLS ciphers so....clearly not professional testers unless the bar is REALLY low these days

Exactly the points I wanted to make. The article is vague at best and no statistical evidence is presented anywhere.