Dark Visitors – A list of known AI agents on the internet

https://darkvisitors.com

by johneth • 3 years ago

134 66 3 years ago

darkvisitors.com

aeturnum• 3 years ago

This seems like a useful index, but I just wanted to say that I love the name. A good name is so much fun.

johneth4• 3 years ago

I just stumbled on it while searching for AI crawler user agents.

I'm not sure how often it's updated. It seems fairly complete from other sources I've been looking at (mainly robots.txt files for well-known publishers).

gavinhking10• 3 years ago

It's updated daily. Thanks for posting by the way.

gavinhking10• 3 years ago

Ha thanks, it was a shower thought

vntok• 3 years ago

Why even bother adding those to a purely declarative robots.txt, when you can just block them outright?

johneth4• 3 years ago

Defence-in-depth? Why not both?

Areibman• 3 years ago

I'm not sure I'd classify most of these as AI agents, it's mostly web crawlers. (Though the definition of agent is fuzzy enough that I suppose you could lump them in).

I'm also curious, will adding "Common Crawl" to the user-agent disallow list in your robots.txt actually do anything?

Added context: I run a tracker of vetted AI agents with verticalized use cases https://staf.ai

gavinhking10• 3 years ago

Yeah they respect robots.txt: https://commoncrawl.org/faq

blacksmith_tb• 3 years ago

Kind of a funny choice of names, "dark visitor" is a common English translation of 黑客, the title of a book[1] about hacking in China, for example.

1: https://taosecurity.blogspot.com/2008/02/review-of-dark-visi...

Macaroni8152• 3 years ago

Oh that is a good point! 黑客 could also be 駭客！

moritzwarhier• 3 years ago

Great idea & execution!

There is a typo in one of the classification texts:

  not currently classified as artificailly intelligent

gavinhking10• 3 years ago

Fixed, thanks!

vennom• 3 years ago

This idea is awesome! I'm definitely going to be referencing this for some of my upcoming projects.

Love the design and name, too.

scopeh• 3 years ago

it would be great to have a raw json file or something of this list so we can just start blocking the requests instead of using robots.txt. do they really respect it? probably not.

SushiHippie• 3 years ago

Most of these bots already publish their IP ranges, so it would be just a simple firewall rule.

For chatgpt user:

https://platform.openai.com/docs/plugins/bot

And for gtpbot:

https://platform.openai.com/docs/gptbot https://openai.com/gptbot.json

ano-ther• 3 years ago

Great list.

Would be nice if they had a fully filled robots.txt for download (I could only find the example file).

gavinhking10• 3 years ago

Right here: https://darkvisitors.com/robots-txt-builder

This example blocks everything, which is probably not what you want, but it's meant to be a starting point.

jahewson• 3 years ago

Don’t block Common Crawl! They’re a charity and do great work producing an open dataset that everyone can use.

alphabettsy• 3 years ago

Including all of the other crawlers. Interesting dilemma.

yellowpencil• 3 years ago

Them being a charity isn't an automatic license to use anything on the internet?

JohnFen2• 3 years ago

If you want to avoid having your site contents used to train AI and don't want to make your website unavailable to the open web, then blocking Common Crawl (among others) is absolutely mandatory.

We have very few tools to protect ourselves here, and need to make the most of the ones we have.

dzhiurgis2• 3 years ago

Why use a deliberately open standard then? Create an app like every other successful monopolistic megalomaniac?

flir2• 3 years ago

I want to publish it on the internet, but I want to keep control of it.

Honestly, it seems like we've been here before. I hope y'all have got your right-click-disable scripts in place too.

ethbr1• 3 years ago

Does Cloudflare have an equivalent?

I assume they'd have visibility into a huge amount of this

nyx• 3 years ago

Huh, didn't know about "ChatGPT-User"... I've always been a fan of a slightly blacker-hat approach to this problem: rather than block those user agents outright, wouldn't it be way more fun to serve them nonsense or irrelevant garbage-filled versions of your pages?

dzhiurgis2• 3 years ago

As a user of non-google search engine (which came long way) I'm pretty disappointed how many website owners explicitly granting monopoly powers to google.

Modern browsers already have LLMs built in so anti-scraping is a folly.

chupapimunyenyo• 3 years ago

I don't understand anything from this comment. Is there something wrong with me?

gavinhking10• 3 years ago

Hi all, I made this site. I wasn't expecting this post, but happy to answer questions and take feedback.

Hopefully it's pretty self-explanatory but I made this website as a simple resource for people who want to stay up to date with the ever-changing cast of AI user agents.

Feel free to sign up with the Google Form to get notified when this list is updated. And if you know of any agents I'm missing, please submit them. Thanks!

schleck82• 3 years ago

cohere-ai is an unconfirmed agent possibly dispatched by Cohere's AI chat products in response to user prompts when it needs to retrieve content on the internet.

This is almost certainly the coral web grounding found as an option on coral.cohere.ai

gavinhking10• 3 years ago

Thank you, will try to confirm this.

lancesells• 3 years ago

Great resource! I'm wondering why you put links to your site in the robots.txt example? It's like there's a clear utility you've made but then the links cheapen it a bit. Maybe I'm missing the reason?

gavinhking10• 3 years ago

You're not missing anything, they're just for any humans reading the file who want to know more about the agent. This is also just an example file.

lakomen• 3 years ago

Is there a "export all to robots.txt disallow" feature? If not, that would be great to have

gavinhking10• 3 years ago

Yup, here is an example with all listed agents, although you probably want to modify it to not fully block all of them: https://darkvisitors.com/robots-txt-builder

thatguysaguy4• 3 years ago

This is a bit confusing since "AI Agent" has come to mean a specific thing, whereas this is about values of the User-Agent field.

mnky9800n2• 3 years ago

Can you explain what you mean?

thatguysaguy4• 3 years ago

The term "AI Agent" is currently being used a lot to describe the setup where you have an LLM doing multiple generation rounds using tools etc so it can interact with an environment or other LLMs or whatever. Feel free to pass your own judgement on whether that's going to go anywhere, but that's what I think of when I hear the term, rather than "web crawler for company making LLM".

thatguysaguy4• 3 years ago

I'm kinda unclear on why I wouldn't want these to be able to access my site, what's the reasoning?

dirtyhippiefree• 3 years ago

YMMV

cdot2• 3 years ago

exfiltration of ideas

Isnt that generally the point of putting something on the internet?

schleck82• 3 years ago

If I can industrially espionage your company using publicly accessible data, I think you should ask chatgpt for some advice on how to maintain secrecy

urbandw311er• 3 years ago

Ask the New York Times

Tao3300• 3 years ago

Maybe if you're worried that your hypothetical My Little Pony erotic fanfic site might accidentally poison one of these AIs. Maybe. Hypothetically speaking. Of course, neither you nor I have such a site, and if we did, the writing would be of high enough quality, and the characters depicted faithfully enough to be good for AI training!

weregiraffe• 3 years ago

And that's the story of CelestAI

osigurdson• 3 years ago

I think most traffic on the internet will eventually come from "ChatGPT-User" or similar. Who wants to sift through things when ChatGPT can do this for you? What is needed is a way to monetize this information it seems.

jmarbach• 3 years ago

I was thinking the same. Wouldn’t it be better to let agents buy access to your site? It wasn’t feasible with the Google monopoly. Now with dozens of agents competing for info it seems feasible to put a tollbooth in front of your site.

mistrial92• 3 years ago

doesn't that depend on the purpose of the visit? Is it not the choice of a publisher?

sgt101• 3 years ago

Because the sift may be biased and unfai

WarOnPrivacy• 3 years ago

This is helpful.

An updating list of IPs for these agents wouldn't go amiss. I might have access to some site logs I could mine for that.

__loam3• 3 years ago

It would be cool if you could download this list as a robots.txt

gavinhking10• 3 years ago

Here it is: https://darkvisitors.com/robots-txt-builder

__loam3• 3 years ago

Oh sick

ErikAugust• 3 years ago

At least they are being “good net citizens” [0] and setting their User Agent honestly.

0: https://x.com/pm/status/1391462220861566977?s=46

olx_designer• 3 years ago

Looks like we might end up with the new web designed entirely for machine intelligence.

_a_a_a_• 3 years ago

What RDF was supposed to be for, though an extra to the web not an alternative.

pimlottc• 3 years ago

Is this for crawling requests used to train/update models, or for real-time requests made in response to user queries?

Dark Visitors – A list of known AI agents on the internet

Related Stories