ByteDance’s Bytespider is scraping at much higher rates than other platforms

https://fortune.com/2024/10/03/bytedance-tiktok-bytespider-scraper-bot/

by wmstack • 2 years ago

141 90 2 years ago

fortune.com

MaKey• 2 years ago

Somehow the headline made me think of a parent with a TikTok account.

skrebbel• 2 years ago

Maybe the people running the crawler are also parenting tiktokers

bilekas2• 2 years ago

Leave no bot behind!

beAbU• 2 years ago

All of a sudden the title makes so much more sense. Thanks. Now I might read the article actually.

smittywerben• 2 years ago

Title edited out the apostrophe from the headline, plus a few small differences shown in brackets.

TikTok[’s] parent launched a [web] scraper [that’s] gobbling up [the] world’s [online] data 25x[-times] faster than OpenAI

bilekas2• 2 years ago

does not respect robots.txt research shows.

It would be nice then for the investigators to help people with the identifying markers for such crawlers. Apart from a mention of darkvisitors, which it seems is a paid service to "Block agents who try to ignore your robots.txt"

I'm not sure how much that could be trusted given their business model also.

koolba• 2 years ago

The Bytespider bot, much like those of OpenAI and Anthropic, does not respect robots.txt, the research shows. Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data.

Does any of these scrapers uniquely and unambiguously identify themselves as a bot?

Or are those days long over?

spiderfarmer• 2 years ago

This one does and I blocked them categorically from all my domains.

diggan2• 2 years ago

Does any of these scrapers uniquely and unambiguously identify themselves as a bot?

It seems like all of them do, yeah: https://github.com/eob/isai/blob/b9060db7dc1a7789b322b8c2838...

Not sure if they're really "scrapers" though, if they're initiated by a user for a single webpage/website, more like "user-agents" in that case, unless it automatically fans out from there to get more content.

netdevnet• 2 years ago

Some of the scrapers used by big companies do identify themselves as bots by using unique user agents. Of course, it does not mean that they don't have other bots running around without the bot user agent name.

Whether those days are over or not will greatly depend on the outcome of the ongoing New York Times vs OpenAI lawsuit. If OpenAI wins, then it pretty much green lights all the other scrappers to feast upon the web

jeroenhd• 2 years ago

Most of the good ones will tag themselves in the user agent and follow robots.txt.

The ones that don't are the ones people are trying to block the most. Sometimes Google or Bing go crazy and start scraping the same resource over and over again, but most scraping tools causing load peaks are the badly written/badly configured/malicious ones.

stroupwaffle• 2 years ago

Im thinking a lot of those issues might be related to “smart” scraping which parses JavaScript. Could lean in to the bot and just make it easier for them to scrape by removing JavaScript from the websites.

I realize this is somewhat off-topic, but the big companies kind of destroyed the internet with all the JavaScript frameworks and whatnot.

rockwotj• 2 years ago

I worked for a short time on SearchGPT, and I can tell you OpenAI does respect robots.txt , at least when I was there and does now. They are also careful to shard per domain and only crawl each domain at a small rate (~1 qps) as to not ddos the site. OpenAI also uses User Agent strings to identify itself: https://platform.openai.com/docs/bots

They have dedicated user agents for search crawling, when a user directly asks about a site and for training data.

jsheard• 2 years ago

They are also careful to shard per domain and only crawl each domain at a small rate (~1 qps) as to not ddos the site.

Maybe that's their intent, but this was only a month ago: https://www.gamedeveloper.com/business/-this-was-essentially...

"The homepage was being reloaded 200 times a second, as the [OpenAI] bot was apparently struggling to find its way around the site and getting stuck in a continuous loop," added Coates. "This was essentially a two-week long DDoS attack in the form of a data heist."

benreesman4• 2 years ago

Indiscriminate scraping is a dick move.

But if you’re going to do it, do it properly. I would have hung it off the Like button with an ungodly ZooKeeper ensemble and trained a GBDT on which parts of which URLs I could just obliterate with Proxygen.

We’d have it all in about 4 days. Don’t ask me how I know.

The second worse thing about the AI megacorps after being evil is being staffed by people who use Cursor.

Edit: on the back of the valued feedback of a valued commenter I’d like to acknowledge that I made a sloppy mistake and have corrected in haste, making no excuses. It would be super great if the largest private institutions in history of the world took the care with give or take everything that I do with trolling on a forum.

viraptor• 2 years ago

is being staffed by people who use Cursor.

Any specific reason?

benreesman4• 2 years ago

Plenty of smart people use Cursor I shouldn’t have been dismissive.

I meant people who don’t work at Cursor.

trunch• 2 years ago

Not OP but other than what core functionality they can demo to investors, every AI company seems to have extremely lacking:

- web design (basic features take years to implement, and when done break the website on mobile)

- UI/UX patterns (cookie cutter component library elements forced into every interface without any tailoring to suit how the product is actually used, also makes a Series C venture indistinguishable from something setup in a weekend)

- backend design (turns out they've been hemorrhaging money on serverless Vercel function calling instead of using Lambda and spending a minute implementing caching for repeat requests)

- developer docs (even when crucial to business model, often seems AI generated, incomplete, incoherent)

And this usually comes from hiring much less developers than is needed, and those that are hired are 10x Cursor/GPT developers which trust it to have done a comprehensive job at what seems like a functional interface on the surface, and have little frame of reference or training for what constitutes good design in any of these aspects.

vasco2• 2 years ago

But if you’re going to do it, do it proerly

Top shelf unintentional irony.

Rinzler89• 2 years ago

Exactly. That line of reasoning just feels like established players kicking the ladder from under them in order to maintain their moat, when competitors start to catch up: "Hey, web scraping and data mining should only be allowed the right way, where the right way = our way."

"Free market" to them = the market where they get to write the rulebook.

sshine• 2 years ago

people who use Cursor

What's Cursor?

ro_bit• 2 years ago

AI based code editing tool (https://www.cursor.com/)

jl6• 2 years ago

I have observed this bot requesting URLs that haven’t been live for over a decade, and to which no reference can now be found in search engines. I imagine there must be a private trade in URL lists.

is_true• 2 years ago

maybe they are using commoncrawl, webarchive, yandex as indexes?

jefozabuss2• 2 years ago

In addition to those it's also possible they just found a website that published a scraped list back then and got de-indexed for obvious spammy content.

I would not be surprised if there are still some auto generated link directories left from the "golden ages" of blackhat.

Havoc• 2 years ago

Going to be hard to enforce anything against this if it’s happening across jurisdiction like this.

I don’t see how copyright survives long term in this sort of context

jfoster3• 2 years ago

It was originally a way to motivate creation of artistic works, since they used to involve a lot of effort.

diggan2• 2 years ago

It was originally a way to motivate creation of artistic works, since they used to involve a lot of effort.

So true, then abstract expressionism appeared and suddenly copyright wasn't a thing anymore.

neilv• 2 years ago

Copyright isn't required once any work can be created faster than you can snap your fingers.

Breathing isn't required if someone strangles everyone to death.

(Now we can all transcend breathing, in the new post-living higher plane of existence. Which surely is viable and great, and totally won't be abused to enrich the worst people, to the detriment of everyone else.)

blibble2• 2 years ago

it still requires a lot of effort, but by other people

not a lot of effort by the parasites

icehawk• 2 years ago

It was originally created because printers needed a constant stream of new works because once they published something their competitors could immediately copy it and republish it without the initial cost of making the work.

That's why its copyright and not artistright.

fny3• 2 years ago

Punitive measures.

Narrow tariffs, competitive subsidies, sanctions, divestment, export restrictions are all viable deterrents.

Aside from this behavior, China has been subsidizing industries and dumping products into economic rivals for years. Never mind all the IP theft. It’s absurd the US has responded so weakly for the last 30 years.

sofixa• 2 years ago

US-based companies do the same (scraping content and training models on it, regardless of copyright or licenses or attribution).

US subsidises and protects tons of industries (agriculture, chips, automobiles, aerospace).

Does that mean that other countries can impose tariffs and sanctions on the US to punish this obviously anticompetitive and anti-free market behaviour? Or is it just the normal stuff we'd expect a country to do?

VHRanger4• 2 years ago

The proper response to a foreign country subsidizing something is to just buy a lot of it. They're losing money on the unit economics, you can profit from it by buying.

We've known for decades that protectionist policies and subsidies make industries less competitive, not more. It is literally textbook stuff [1].

A typical result of protectionism is somethint like GM in the US, where they grow uncompetitive as they don't have to compete with foreign markets for domestic demand.

You get similar uncompetitive dependent behavior from subsidies - just look at Intel right now.

1. https://web.pdx.edu/~ito/Krugman-Obstfeld-Melitz/8e-text-PDF...

Note the author here is Paul Krugman, who literally won a Nobel Prize for his work on infant industry protection.

indymike2• 2 years ago

Narrow tariffs, competitive subsidies, sanctions, divestment, export restrictions are all viable deterrents.

The only thing that stops this is when a nation has more to lose than to gain... and that will happen soon as other emerging economies follow the grand tradition of cheating their way to prosperity. Then slowing down the comepetition will be the only play.

All of us born after 1970 have seen Japan, Taiwan, Hong Kong (before being re-absorbed), and China run the cheat your way to the top playbook and will see it at least a few more times.

mschuster91• 2 years ago

At the point and scale we're at, I'd classify China's actions as cyberwarfare and would add similar responses to your list.

sct202• 2 years ago

Bytedance has physical presences in most major markets now for ad sales/support so there are measures that can be taken and money is flowing that could be halted if needed.

lovethevoid• 2 years ago

You don't see how large companies with vast resources are going to protect their copyright? You really believe that ByteDance et al. are going to make their data freely and publicly available for everyone?

This is a way to permanently entrench their positions while maintaining ownership. Not an eradication of copyright.

wtk• 2 years ago

https://archive.ph/https://fortune.com/2024/10/03/bytedance-...

neonate• 2 years ago

https://archive.md/btyIo

shellac• 2 years ago

I'm pretty sure this bot has been operating for much longer than the article suggests (April this year), and truly is a pain. I work in academia and see a lot of ill considered web scraping by ML / AI researchers, but Bytespider is in a league of its own.

welder• 2 years ago

So what, who cares? Is this newsworthy? It's definitely not something to get upset about, web scraping is a normal part of the internet.

OutOfHere• 2 years ago

Just how would a scraper catch up with the internet if not by accelerating the rate? It is to be expected if the scraping is to succeed.

buro9• 2 years ago

Also the Facebook hit scraper.

Which does not respect robots.txt and definitely is just scraping.

AS blocks are the only really effective tool now, there are many scrapers that do not even respect user agent

wiredfool• 2 years ago

Facebook and someone who’s using a Firefox UA are the big hitters for me today, each at a sustained 6 req per second for the last 24 hours on one site.

Today is actually pretty good, there’s some real looking UA traffic in the top 10.

486sx33• 2 years ago

It’s unfortunate and kind of dystopian. We have an opportunity to properly archive all of the worlds online data and catalog it for very very low cost (historically), so that the future of our planet will have a much better reference point for the past.

Instead of that, companies are sucking up as much crap as possible, and tokenizing it and then scrubbing it, and adding “safety” to it.

Reality is always much stranger than fiction.

Spivak2• 2 years ago

We have billions of people, we can accomplish two maybe three things at a time. This is a valid use as any of that archived data. The part that sucks isn't that people are doing unusual things with it like training AI, but that copyright & capitalism make it so that everyone has to go get their own data themselves to the annoyance of web admins.

The biggest technical hurdle to sharing the work among interested parties is the web only authenticates the pipe, not the content.

jrochkind1• 2 years ago

CommonCrawl tries to archive the web and share it openly so everyone doesn't have to scrape it themselves.

"Our goal is to democratize the data so that everyone, not just big companies, can do high-quality research and analysis."

Because they share it openly including with those doing AI, they wind up on "AI crawler" lists, which are increasingly used by blocking tools that just "use the AI list", by people who don't like AI, or, quite ironically, people who are trying to prevent the excess traffic that poorly mannered AI crawlers cause. (Common Crawl's crawler is well mannered, uses good user-agent, respects robots.txt including crawl-delay, etc)

https://commoncrawl.org/

anileated• 2 years ago

Copyright & capitalism is the crucial part of how we have the technical foundation that got us ML and most of the material used for training it. Big tech companies that want to monetize it at scale would like us to not think about that (or any long term consequences that do not affect shareholder value beyond current management), of course. If anything, the problem with intellectual property law is that they feel it’s safe for them to ignore it when it comes to ordinary people’s work (good luck suing ClosedAI).

throw109202• 2 years ago

This is a valid use as any of that archived data.

No, it's really not, as most of the people who actually spend the time and effort to produce that content did not consent to it being used to train AI.

copyright & capitalism

That's a really disingenuous way to say "the creators of that data didn't consent to training or commercial use and I want to steal their effort".

Ironlikebike• 2 years ago

In my last job, we observed ByteDance scraping TBs of OS testing data using the restful API that our OSS community front-end was using to serve it's CI result to the OSS community. The scraping was so relentless it was causing performance problems. We were also worried they were going to cause us large network egress fees as well. We specifically locked down the API after that, and anyone who wanted to use results had to ask explicit permission and be granted access.

jgrahamc• 2 years ago

Stuff like this is why Cloudflare launched the AI Audit feature and the ability to block "AI bots". We're about to launch a feature that'll enforce your robots.txt.

notachatbot123• 2 years ago

It would be nice to share this tooling free and open-source so that anyone can protect themselves.

andrethegiant• 2 years ago

I’m working on a platform[1] (built on Cloudflare!) that lets devs deploy well-behaved crawlers by default, respecting robots.txt, 429s, etc. The hope is that we can introduce a centralized caching layer to alleviate network congestion from bot traffic.

[1] https://crawlspace.dev

zebomon• 2 years ago

I love the sentiment, but the real issue is one of incentives and not ability. The problem crawlers have more than enough technical ability to minimize their impact. They just don't have a reason to care right now.

kgen• 2 years ago

To be honest, it's probably not enough to just block these scrapers if they are acting maliciously, people should just start serving generated content back to it and see how long it takes for them to catch on and fix the problem

OuterVale• 2 years ago

This pops to mind: https://youtube.com/watch?v=Hi5sd3WEh0c

nubinetwork• 2 years ago

The China-based parent company of video app TikTok released its own web crawler or scraper bot, dubbed Bytespider, sometime in April

Uh, no... bytespider has been around for a long time...

ByteDance’s Bytespider is scraping at much higher rates than other platforms

Related Stories