BotOrNot (2021)

https://incolumitas.com/pages/BotOrNot/

by peter_d_sherman • 3 years ago

53 12 3 years ago

incolumitas.com

thevania• 3 years ago

link to bot detection page - https://bot.incolumitas.com/ - is down

Forbo• 3 years ago

I just get a 403. I assumed it was because of my VPN but even without it I get the same result.

risingson• 3 years ago

It's been down for months, sadly. I was working on some dummy Puppeteer human-behavior script back then and the site helped a lot. Hope the guy's doing alright :/

1vuio0pswjnm72• 3 years ago

Not down for me.

   echo 167.99.241.135 bot.incolumitas.com|sed -i -e1r/dev/stdin -e1N /etc/hosts
   FTPUSERAGENT= tnftp -4vdo/dev/stdout https://bot.incolumitas.com
   links -no-connect https://bot.incolumitas.com
   sed -i 1d /etc/hosts

   curl --resolve *:443:167.99.241.135 https://bot.incolumitas.com

1vuio0pswjnm72• 3 years ago

False. It's up.

Alas, does not accept TLS1.3.

     printf 'HEAD / HTTP/1.0\r\nhost: bot.incolumitas.com\r\nconnection: close\r\n\r\n'|openssl s_client -connect 167.99.241.135:443 -ign_eof
     printf 'GET / HTTP/1.0\r\nhost: bot.incolumitas.com\r\nconnection: close\r\n\r\n'|openssl s_client -connect 167.99.241.135:443 -ign_eof
     echo|openssl s_client -connect 167.99.241.135:443  -showcerts|openssl x509 -text

batch12• 3 years ago

Ugh, just remembered hotornot circa 2004. Those neurons haven't been exercised in a while...

luth0r• 3 years ago

Kind reminder Google broke almost all TCP fingerprints (i.e. JA3) when introducing grease values and other stuff such as random order in the ciphers in Chromium. I work in a bot detection company and we put a lot effort on it, currently have an advanced implementation that ignores some extensions, order the fields, remove the grease values etc.

_lvbh• 3 years ago

I have an open source implementation of normalized JA3 fingerprinting at https://github.com/fidraC/canary (Includes CreepJS and other stuff to be added / WIP) it’s not very complicated but I did waste some time before realizing that grease was everywhere, not just extensions.

Note: That is my secondary account for school. I’m not stealing someone else’s work. I use a different name as a sort of mental compartmentalization between different spheres of my life.

mike_d• 3 years ago

Google's plan is to break every type of client fingerprinting and then funnel everyone into using their browser attestation.

iOS isn't any better and already sends a unique "Private Access Token" with requests to Cloudflare.

didntcheck• 3 years ago

Well I get a 403 when trying to load the detection page (in Firefox) from behind a VPN. Guess that's a fail then?

mateuszbuda• 3 years ago

Some parts are outdated now. Bot detection vs web scraping is an ever evolving arms race.

Anyway, incolumnitas blog is a great source of information about bot detection and techniques to avoid being detected. We have implemented many successful methods based on it in https://scrapingfish.com/.

dharma1• 3 years ago

What are the current SOTA open source projects for bot detection?

BotOrNot (2021)

Related Stories