Based on trying to buy anything retail that is scarce, bot detection systems are definitely not working. I wonder how their system would do against https://refractbot.com/ or other popular retail bots.
To be fair, most bot detection can't and isn't trying to catch 100% of bots. Generally the idea is to prevent repeated abuse, rather than stop every single instance of it.
In the case of the scalper bot you've linked, the goal of bot protection would be to make the bot's transactions so computationally expensive/slow that they miss out on most of their intended purchases.
Did they use the browsers against sites they owned or 3rd party ones? If the latter, then lots of sites have Cloudflare Bot Management on, without actual strict restrictions on bots (a lot of sites using CLoudflare as a CDN for instance)
In your bot detection systems I do not see open source system like anubis, safeline or sibuna (I am the author of sibuna). Sibuna is built as self-hosted alternative to cloudflare with console UI providing OWASP CRS, JA4 TLS signature from ingress, these are still necessary besides bot detection and cloudflare offer it as enterprise offering.
I have a smaller benchmark for them in my book at https://insanai.github.io/sibuna/. If you can provide your benchmarking code as open source will incorporate this report into sibuna book.
I feel like newer systems are even worse, probably in about 50% of my online experiences now I just get "banned" immediately. Opening a new account = sign of a bot = banned.
Hyper focusing on detecting bots without thinking about the humans you're blocking is making the Internet a crappier place.
conception | 19 hours ago
routerl | 15 hours ago
In the case of the scalper bot you've linked, the goal of bot protection would be to make the bot's transactions so computationally expensive/slow that they miss out on most of their intended purchases.
AznHisoka | 18 hours ago
vikrantrathore | 16 hours ago
I have a smaller benchmark for them in my book at https://insanai.github.io/sibuna/. If you can provide your benchmarking code as open source will incorporate this report into sibuna book.
mimimile | 14 hours ago
aurareturn | 14 hours ago
If blocking 1 bot request is +1 then making a human solve a captcha should be something like -100. That's how we should grade.
nickabe | 7 hours ago
Hyper focusing on detecting bots without thinking about the humans you're blocking is making the Internet a crappier place.
asksomeoneelse | 13 hours ago
Also, the methodology just compare the true-negative detection rate for known negative, it seems?
> "Proof of Human identified 99% of automated sessions"
> "We measured detection rates for automated sessions but did not evaluate false-positive rates among human respondents."
Trivially easy measure to game if so. "isHuman(x) = return false" gets you a 100%.