Google cut a deal with Reddit for AI training data

https://www.theverge.com/2024/2/22/24080165/google-reddit-ai-training-data

by bx376 • 2 years ago

165 141 2 years ago

whoopdedo• 2 years ago

Google Search is currently expanding the test of a "forums" filter that lets you browse through results from sites with human discussion, like Reddit

Yeah, about that whole "human discussion" thing...

lcnPylGDnU4H9OF• 2 years ago

I think Reddit is accurately described as a site "with human discussion". It's another matter whether such discussion is organic and actually shows up in search results.

0ckpuppet• 2 years ago

Is that like the taco bell meat that "contains" some meat.

salt-thrower• 2 years ago

Fair point that a lot of Reddit is already astroturfed, but this is definitely the final nail in the coffin. Now SEO is coming for the forums, too. Wonderful.

jimbob45• 2 years ago

In fairness, Reddit's built-in search works well enough that I could still find stuff on Reddit if Google croaked tomorrow.

p1mrx• 2 years ago

Didn't Google have a "forums" filter like 20 years ago? I can't seem to find results pages on archive.org.

whycome• 2 years ago

You may be thinking of the Gmail default sorted inboxes

silisili3• 2 years ago

It was called discussions search, and showed up in the suggestion bar with images and whatnot. I was beyond mad when they removed it. Everything's a circle.

malfist• 2 years ago

If you want that now, kagi has a forums lens

A_D_E_P_T• 2 years ago

Hah. In fairness, PR firms, digital marketers, "reputation management" experts, and SEO gurus are "human" too.

These days at least 90% of product-related discussions on Reddit are astroturfed, though, and that's for sure.

The main problem, among many others, is that it's far too easy to game the upvote/downvote system with bots, and most pros have bot networks of their own, or know how to hire them. The second problem is that Reddit "karma" is site-wide, which means that most "high karma" accounts that are sold on secondary markets are very useful for digital marketing purposes generally. The third problem... well... I could go on all day. Reddit's system is horrendous if you're looking for human discussion and unbiased advice.

digging• 2 years ago

Kagi already has such a filter by default, although I've never tested it.

LorenDB• 2 years ago

Hm, now I have to wonder if it'd be possible to create a tool that poisons your Reddit posts so they look fine to a human but completely trash LLM output.

I'm not saying we should do it, but it's a fascinating thought.

berbec• 2 years ago

I'm saying we should do it for you.

barryrandall• 2 years ago

davidjl and several others did it.

duffyjp• 2 years ago

https://www.smbc-comics.com/comic/got-you

mlekoszek• 2 years ago

What about Markov chain babble? It would be interesting to see Bard learn to mimic people going far even as decided to use even go want to do look more like.

monkeydust• 2 years ago

This can't be exclusive for that amount or could it?

dmix2• 2 years ago

No way it’s exclusive. Reddit is making a push to find new $$ streams for their IPO and the API is a big part of that with AI being hot. They probably have lots of deals for their new API already and this one is just making headlines because it’s Google and it’s yearly/unlimited/relatively large.

purpleblue2• 2 years ago

Someone should create a bot that responds to highly upvoted comments and remind the redditor that Reddit is making money off the comment and the information is being used by AI and they aren't seeing a dime of it.

dageshi• 2 years ago

It was going to get scraped and used by AI anyway? Given that I'm not sure I care that much if reddit makes a bit of cash off it. Honestly, this is the future, if it's publically available on the internet it's going to be scraped and used by AI. The inevitable consequence of which is probably more conversation moves into private places like Discord.

But that's the future we're in, it's not gonna get rolled back.

belinder• 2 years ago

I don't see why Discord would not follow suit if it has not already

Everyone is privacy focused until they're not

Imnimo• 2 years ago

With only $60M a year to go around, I'm not sure my share of comments would even add up to a full dime anyway. They can have it.

hobs3• 2 years ago

That's arguably more true in advertising driven reddit than AI reddit - no one comment is going to be worth the most in a data scraper but eyeballs win in advertising.

I think the entire thing is scummy too, but its interesting to me how people think that advertisers were somehow less bad than this.

purpleblue2• 2 years ago

It's interesting how you're creating an argument out of thin air.

The article was about mining comments for AI, not about advertisement. Yet you brought in the topic of advertisement, and then created a fake argument about how people think advertisers are better than AI scraping. It's really weird that you did that.

vdaea• 2 years ago

They are using a service that allows them to make their opinion public and publicised and to know other people's opinions. Or does that have no value now?

asadotzler3• 2 years ago

It does have value, enough value that the owners can sell that content to third parties and so reasonably I should get my cut.

mediumsmart• 2 years ago

oh cool so the contributors of the data are all getting paid?

MakeThemMoney• 2 years ago

Or course! Same way we get paid by for the ideas we discuss here

brookst• 2 years ago

Exactly. If you don’t get value from HN, don’t use HN. If you don’t get value from Reddit, don’t use Reddit.

wombat-man• 2 years ago

Paid in glorious internet points.

Alupis3• 2 years ago

oh cool so the contributors of the data are all getting paid?

If people on reddit wanted to get paid for their opinions, they wouldn't be posting them for free on a publicly accessible forum that does not even require an account to view...

This same sentiment came up when StackExchange started to use/sell StackOverflow posts for AI training too - yet everyone arguing that position failed to recognize StackExchange has been making money off their free contributions since day one. Like, how else would StackExchange even exist? They can sell job postings now because of all the free contributions and community built up over the past decade - same with Reddit...

tayo42• 2 years ago

I think people would write blogs and stuff, but reddit communities are so hostile to anything off the platform.

Scoundreller7• 2 years ago

At least for the time-being, you can still do full downloads/extracts of stack for your needs, e.g.:

https://www.brentozar.com/archive/2015/10/how-to-download-th...

23B13• 2 years ago

If people on reddit wanted to get paid for their opinions, they wouldn't be posting them for free on a publicly accessible forum that does not even require an account to view...

People post on the internet 'for free' with the expectation that they're discussing ideas which have their own value, not providing training data. Yeah yeah ToS blah blah blah, but the goodwill that sites have built – even ones like HN – is on the premise that we're participating in the marketplace of ideas, not enriching shareholders.

This will ultimately be bad for the open internet.

swyx• 2 years ago

you guys are getting paid?

dmix2• 2 years ago

All platforms are moving to monetize content contributors. It’s worked amazing for YouTube, TikTok, and Twitch. X is trying it slowly.

Reddit could potentially do this but I don’t think they have a solid business model to support it yet. They are scrambling for money streams, not content. Although high end supported content drives ad views so who knows.

drexlspivey• 2 years ago

Paying people to shitpost, what could go wrong with that.

steveBK1232• 2 years ago

$60M/year for GOOG to access all their data when they purport to be targeting a $5B valuation at IPO is really cheap.

Arguably Reddit's value is it's data, and GOOG is renting it for 1.2%/year?

taion• 2 years ago

Google are already indexing Reddit, so this might reflect the incremental value of direct API access plus keeping everyone happy. I would be surprised if Reddit’s threats to block Google’s crawler were all that credible.

hobs3• 2 years ago

It's 100% this - Google is going out and shopping so they can get easy access and avoid any future lawsuits if things change in the AI training space.

I imagine this wont be the last announcement in this space.

leros• 2 years ago

It wouldn't surprise me either if something like half of Reddit's traffic was Google constantly crawling them.

declaredapple2• 2 years ago

Arguably Reddit's value is it's data

Reddit has a chicken and egg problem - nobody knows what the value of it's data even is. They've never sold it before. You can sell pet rocks for 1 billion dollars, but nobody will value them if there's been 0 sales. (ask vs bid vs last sale price).

I'm guessing reddit is rushing to sell their data so they can show that their data is worth anything at all.

Scoundreller7• 2 years ago

They've never sold it before.

Do we know this?

swatcoder• 2 years ago

Is this being announced as permanently exclusive?

Otherwise, Reddit is productizing its data, which was the obvious next step after closing public API access. This establishes a market price that at least one buyer was willing to accept, and shapes the terms for later deals with more customers.

Presumably, they think they can sell this kind of access many time over as one of their major revenue channels while also benefiting from the incoming traffic it should generate.

paxys2• 2 years ago

1. It is pointless to compare total company value with an annual payment.

2. It isn't an exclusive license. There are dozens of companies training language models, and if they are all forced to pay the same amount then that's some serious cash.

Scoundreller7• 2 years ago

It’s going to be a sad day when /u/cumbuster69 doesn’t get the attribution they deserve for their excellent financial advice

tyingq2• 2 years ago

1. It is pointless to compare total company value with an annual payment.

Mozilla being an exception, I suppose?

coffeemug• 2 years ago

I thought that's really cheap too. Tacking on "wiki" and "reddit" probably accounts for ~50+% of my google searches now. From that perspective $60M annually feels really low-- reddit search is probably a quarter of the value that google search offers me as a consumer.

steveBK1232• 2 years ago

Companies are generally valued based on cash flows, and some multiples based on profit/growth/etc.

Analemma_• 2 years ago

The flipside to this argument is that paying Reddit $60M to access data which was posted to the public internet for public viewing and consumption, by people who don't work for Reddit, is obscene and antithetical to how the Internet is supposed to work. Why on earth should Reddit get paid for this, especially if they're not giving the users who made the value a cut?

elektrontamer• 2 years ago

Because they developed and hosted the site and the users agreed to it.

paulddraper4• 2 years ago

A. Nothing said it was exclusive.

B. Reddit's value is in the community more than the data. A company with Reddit's data but not Reddit's users is worth relatively little.

newsclues• 2 years ago

How do you extract the value from the community?

vineyardmike• 2 years ago

No, arguably it’s worth some subset of $60M. That number includes future access, but it also includes a license to all that has already been written.

skhunted• 2 years ago

I think in the race for AI dominance that human generated data is extremely valuable. The community is valuable only to the extent that is contains lots of human generated responses on specific topics.

I’m not in AI research so I know I could be completely off base.

zzleeper• 2 years ago

If the value of Reddit was its data, then it should have encouraged more people to use it, instead of e.g. cutting off almost every Android and iPhone app.

My usage went completely downhill and my posts/comments as well

jiveturkey422• 2 years ago

AI is about to become really sarcastic, pedantic, and absurdly moralistic

Almondsetat• 2 years ago

That's already what googlers developing it are

mvdtnz2• 2 years ago

And even more confidently wrong.

Scoundreller7• 2 years ago

Can’t wait for heavily downvoted unfortunate truths to be eliminated from AI

Freedom2• 2 years ago

Confidently wrong? I thought it was getting access to Reddit, not HN.

HelloMcFly• 2 years ago

"Often wrong, never in doubt"

kromem2• 2 years ago

Every LLM almost certainly already is trained on Reddit from before 2015 when the entire site content was compiled for research.

This is just adding in more recent content.

jiveturkey422• 2 years ago

makes a banter
gets a pedantic reply

were you also trained on reddit?

Ajay-p2• 2 years ago

I don't understand why they chose Reddit, was 4Chan not willing? Seriously though, why not use the comment and discussion sections of Wikipedia and other sites that are not drowning in social insanity.

we_love_idf• 2 years ago

This is dystopian. A company like Google will be able to surveil more and the only outcome is more invasion of privacy and more targeted ads. Why doesn't FTC stand up for this?

paulddraper4• 2 years ago

For what exactly? Doing something you don't like?

7373737373• 2 years ago

Copying this right from the Reddit TOS for all those who consider posting anything of any value there, present or future:

You retain any ownership rights you have in Your Content, but you grant Reddit the following license to use that Content:
When Your Content is created with or submitted to the Services, you grant us a worldwide, royalty-free, perpetual, irrevocable, non-exclusive, transferable, and sublicensable license to use, copy, modify, adapt, prepare derivative works of, distribute, store, perform, and display Your Content and any name, username, voice, or likeness provided in connection with Your Content in all media formats and channels now known or later developed anywhere in the world. This license includes the right for us to make Your Content available for syndication, broadcast, distribution, or publication by other companies, organizations, or individuals who partner with Reddit. You also agree that we may remove metadata associated with Your Content, and you irrevocably waive any claims and assertions of moral rights or attribution with respect to Your Content.

(https://www.redditinc.com/policies/user-agreement-february-1...)

tl;dr: "We will be able to do anything we fucking want with anything you contribute to our site"

jpalawaga• 2 years ago

I know reddit has hired a lot. This sort of feel like a fundraising round to keep operations afloat.

This must be the real reason for destroying public API access.

SeanAnderson• 2 years ago

Source for Reddit hiring more than average tech company? I feel like every tech company scaled up in the bull market and then trimmed back down when hype subsided. For example, Google grew by 72,000 between 2019 and 2022 and has since shed about 20,000.

Reddit is IPOing next month. IMHO they cut public API access as a way to simplify and pump bookkeeping numbers to get a better listing price.

bbor2• 2 years ago

To be fair, “they were only as insane as everyone else” doesn’t necessarily mean that they didn’t vastly over-hire. I mean Jesus I got hired in Silicon Valley in 2022, it was crazy out there

shmatt2• 2 years ago

In the long term no company offers free unlimited APIs like that. It was a dumb decision to offer it in the last 5 years

Scoundreller7• 2 years ago

Reddit should be paying Google.

Some of Reddit’s success can be attributed to the difficulty of getting your personal blog indexed about your ‘87 Nissan Sentra anymore on Google.

CamelCaseName• 2 years ago

Maybe. 75% of their incremental users are logged out and likely coming from Google.

Still, maybe Google also derives a lot of value from Reddit.

Would either of the two risk their position to extort some money from the other?

silisili3• 2 years ago

I hope they're filtering it heavily and specifically. While Reddit does still have some valuable discussion in the more niche subs, I've noticed the main subs moving further left, hatefully and almost violently so.

I can't wait to search for who the Republican nominee is, and Google tell me to kill myself.

khazhoux2• 2 years ago

I've noticed the main subs moving further left, hatefully and almost violently so.

I’ve not seen this. What do you mean?

silisili3• 2 years ago

So, I'm only replying to give some color into that thought, not to start a flame war or debate, because it is a legitimate concern I have with using the site as data.

I'm not necessarily complaining that it's left leaning, it always has been in US parlance at least. My worry is that the tone as changed from disagreement to some form of disgust and hate, and yes even at times calls for murders. Maybe that's just the landscape now, broadly?

Just look at some of the 'mainstream' comments linked below. This is probably a 'both sides' thing, but I don't see the other side represented here and I would have equal concerns if it were in similar manner.

That said, I don't have a Reddit account, so just opened the site to see what it showed me. I'm not sure of its algo, as a lot of it is video games, but here are a few that showed up in the top 15 or so -

https://www.reddit.com/r/nottheonion/comments/1axe9rk/darryl...

https://www.reddit.com/r/LeopardsAteMyFace/comments/1axas1q/...

https://www.reddit.com/r/AnythingGoesNews/comments/1axad5x/t...

head over to /news, and you see -

https://www.reddit.com/r/news/comments/1ax8mnl/tax_evasion_b...

https://www.reddit.com/r/news/comments/1axe1ac/darryl_george...

/politics, you get

https://www.reddit.com/r/politics/comments/1axa6m9/donald_tr...

https://www.reddit.com/r/politics/comments/1ax49rq/alabamas_...

evandale• 2 years ago

here's a few examples

https://old.reddit.com/r/toronto/comments/100j6md/new_year_n...

https://old.reddit.com/r/toronto/comments/10t393y/final_upda...

https://old.reddit.com/r/toronto/comments/17i5x9u/posts_rela...

https://old.reddit.com/r/toronto/comments/17tszml/free_pales...

https://old.reddit.com/r/toronto/comments/18jctpk/propalesti...

https://old.reddit.com/r/toronto/comments/1apxrbc/torontos_m...

Not even allowing discussions of crime and city-wide threats is completely asinine and the mods reasoning is that everybody who visits r/toronto is racist and can't be trusted to discuss these issues in good faith. Yet .. it has NEVER been tried and this "experiment" has been going on 2 years now. What I think is really going on is that the Toronto mods are lazy and using that to implicitly pushing an agenda based on what they allow to be posted.

Fervicus2• 2 years ago

An AI company with an agenda getting their data from a social platform with an agenda. What could go wrong?

khazhoux2• 2 years ago

What would you say is the agenda of Reddit (the company, not the users)?

add-sub-mul-div• 2 years ago

Any ideas that make you uncomfortable are an "agenda" now.

lijok• 2 years ago

On one hand a measly 60 mill for Reddits data

On the other hand they bought a poison pill

Is whoever behind the decision making here blindsided by Reddit being a massive trove of data, and are not realizing most of that data is shit?

xyst• 2 years ago

Reddit is really pumping their P&L sheet in preparation for IPO.

The parasites at the top have $ in their eyes rn

ChrisArchitect• 2 years ago

[dupe]

Discussion on official post here: https://news.ycombinator.com/item?id=39471317

rebelde• 2 years ago

Google is paying Reddit instead of just taking it for free like they do from all other websites?

BadHumans• 2 years ago

Second line in the article: "In an update on Thursday, Reddit announced it will start providing Google “more efficient ways to train models.”

They will helping Google create a real-time pipeline to Reddit's data.

huitzitziltzin• 2 years ago

$60 M for absolutely cutting edge dick joke generation in next gen LLMs.

I question whether Reddit data is worth anything at all.

shmatt2• 2 years ago

Reddit and Facebook Groups (the latter being much better) are the only 2 places I can search for reviews or recommendations to pretty much anything these days. Google & co. are impossible to use for "best" anything

bpanon• 2 years ago

Most of my google searches include "reddit" now. From what I've read from others, this is pretty common.

crazysim• 2 years ago

Amazing. It used to be a free BigQuery dataset. Apparently it was worth $60M.

izyda• 2 years ago

The lack of revocability, marginal temporal value, and downstream governance I think makes the prospect of more such data deals happening slim -- or at least, slim without regret.

I wrote an essay on this here: https://magis.substack.com/p/llm-data-sales-a-market-for-lem...

PedroBatista• 2 years ago

Only $60M? Damn.. Reddit is for the streets.

And that's a sweet deal for Google, paying pennies to partially patch their deteriorating reputation about their search engine becoming useless.

Mo33• 2 years ago

Lol, it's essentially just advertisement and psyops bots, echo chambers and specific discussions in specific subreddits at this point.

lolpanda• 2 years ago

Reddit blocks all search engines from crawling their comments, essentially blocking all user generated contents.

See this directive in https://www.reddit.com/robots.txt `Disallow: /r//comments////*`

But I can still search Reddit on Google. How does Google manage to get the data?

gruez• 2 years ago

I think you're misinterpreting the rule.

The relevant robots.txt rules are:

    User-Agent: *
    Disallow: */comment/*
    Disallow: /r/*/comments/*/*/*/*

The url for the comment sections looks like: https://www.reddit.com/r/[subreddit]/comments/[id]/[slug]/. This doesn't match the above rules because there's only 2 parts after "comments", not 4 parts as specified by the rules.

hollerith• 2 years ago

Now it says "Disallow: /"

Ajay-p2• 2 years ago

Does Reddit really block all search engines, or do "all" search engines abide by No Spiders? If it's freely available it's ripe for scraping, no matter what Reddit may say.

Mo33• 2 years ago

Just a reminder this exists:

https://github.com/j0be/PowerDeleteSuite

https://github.com/andrewbanchich/shreddit

latentcall• 2 years ago

This worked amazingly for me when I stopped using Reddit.

klipklop2• 2 years ago

Who knows if the "deleted" data can still be sold off. Sure you can't view it on the website, but nobody knows what Reddit will hand to Google. Then again certain laws passed recently might make such a thing illegal...

klipklop2• 2 years ago

What incentive is there for posters on Reddit to continue to produce new content that is sold off for a profit? Sure they agreed to it in the TOS, but I suspect people might leave the platform.

Also how many of the posts are already generated by AI? Seems like a raw deal to pay for data that could be a large percentage of bot activity.

Why visit Reddit at all for information when you can just ask Gemini.

RecycledEle• 2 years ago

I hope they get all versions of edited and moderated comments. An AI might be able to determine when the moderation was valid and when it was playing games.

I would love to see a leak of that dataset, sorted my username.

iteratethis• 2 years ago

When I first learned that AI companies are vacuuming up all internet content without any regard for permission, attribution or compensation for the content creator, I found that deeply immoral.

I figured they should pay for it. But now that they do (in this instance), I'm realizing this might be even worse. They can just buy the entire market in the same way they buy Google search users by paying Apple billions a year.

And still the actual content creator, a Reddit user in this case, is not compensated.

It's truly wild how lax regulation is. This is probably the most important technology ever created and we just let 2 companies have it all: the data and the compute.

AH4oFVbPT4f8• 2 years ago

This was one of the reasons I no longer contribute to reddit. I don't post, comment, upvote nor downvote. I'll still consume it from time to time, something I try to stop doing. All I can do is vote with my money and my time and I try to spend it else where.

asadotzler3• 2 years ago

Section 230 lets them pretend they don't own the content, shielding them from liability, yet they can simultaneously claim ownership enough to sell it. That's effed up. Either it's not theirs, it's mine or ours, in which case I get to sell it or at least get a cut, or it's theirs in which case they're liable for any illicit content and should face those suits. Which is it, Silicon Valley? Is my post yours or is it mine? You shouldn't get to have it both ways.

jfghi• 2 years ago

I wonder how well they identify posts written via AI (and what proportion of posts that is). Also, there is a ton of misleading astroturfing for some types of businesses.

gremlinsinc• 2 years ago

Do people still use reddit? I haven't since RelayForReddit went subscription only thanks to the new Reddit API rules.

rldjbpin• 2 years ago

i like how one half of the discussion is about getting your share of pay from the deal, while the other disregards whatever is posted on the site.

Google cut a deal with Reddit for AI training data

Related Stories