Eliezer Yudkowsky on large language model economics

https://twitter.com/ESYudkowsky/status/1635577836525469697

by nabla9 • 3 years ago

66 115 3 years ago

twitter.com

coldtea2• 3 years ago

This is like "Malcolm Gladwell on horse racing"...

herculity275• 3 years ago

"Have you tried making your Yud bigger?"

thecleaner• 3 years ago

Could you explain this joke please ?

xdavidliu5• 3 years ago

my possibly incomplete guess: "articulate person opines on a topic that they have no expertise in, and uses articulation to hide their lack of expertise"

cs702• 3 years ago

As usual, Yudkowsky has valuable insights, but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be.

TL;DR: If your customers can get text input-output samples from your super-secret LLM, whether via web interface or API, then your customers can easily and cheaply train their own LLM to learn to act like your super-secret one.

For example, if you're OpenAI, your customers can spend ~$100 to get input-output samples from text-davinci-003, plus another ~$500 to fine-tune LLaMA on those samples, and -- voila! -- they get Alpaca.

PaulHoule4• 3 years ago

It’s not accidental at all that EY writes this way. He is writing out of L. Ron Hubbard’s style guide. He doesn’t want to communicate to outsiders, he is writing to members of his Cthulhu cult.

canadiantim• 3 years ago

Yeah I was liking his post and following along until he kept repeating "shoggoth", very weird. I stopped at that part

jerry1979• 3 years ago

What do you mean by "L. Ron Hubbard's style guide"? Would that be doing things like employing in-group metaphors?

bnralt• 3 years ago

I’ve noticed a similar writing style in a lot of others from the same group (AI Doomsdayers/“Rationalists”). Most of the time I stumble across something from that corner of the web it feels like the author was thinking “Why convey an idea in just one sentence when I can convey it in ten?”

gpderetta• 3 years ago

Similarly on how people have fine tuned SD on Midjourney.

substation134• 3 years ago

Is it a good insight though? The training data for many of these premier models is already available, and that's even better than some input-output pairs.

zarzavat• 3 years ago

He’s talking about RLHF - reinforcement learning with human feedback (the process that trained ChatGPT), the training data for which is not publicly available.

And the point is that you don’t need to RLHF as long as you have access to another model that has been trained with RLHF that you can blackbox.

sdwr• 3 years ago

That's only because these models are all pretty much the same (?) in terms of architecture, and basically do the same thing. New features, like memory, medium-term planning, and iterating towards a design goal, won't be imitatable in the same way.

yunohn• 3 years ago

but uses a lot of words, many of them fancy, to explain himself, making his writing less accessible than it could be

Why does he use 2 different niche analogies to explain the thing that is still quite densely written at the end? Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence…

dragonwriter5• 3 years ago

Basically I read his point 3 times, and felt it could’ve been distilled into 1 sentence…

“Making profit on AI is basically the same as making profit on other software, except that the sensitive bits include the direct inputs and outputs to the core model, not just the software itself.”

Yes, it could. But then, the attempt to sell the idea this was some kind of radical phase transition that the players in the AI industry (largely, very experienced players in the software industry) simply were fundamentally not prepared for would be a lot harder.

substation134• 3 years ago

I can't help but feel like there is an element of sour grapes here after Yudkowsky's attempts to build an AI failed (assuming that's what he was really trying to do, and not just to lure young and enthusiastic grad students into his lair...)

terafo3• 3 years ago

But did he try to build AI? As far as I know MIRI is pure-play safety research lab.

substation134• 3 years ago

He did, MIRI was a later rebrand.

Source: https://youtu.be/0A9pGhwQbS0?t=60

xdavidliu5• 3 years ago

kinda like the Chomksy op-ed, though you could argue that Chomsky never technically tried to build an AI

renewiltord2• 3 years ago

Considering Chomsky's work in linguistics, I was disappointed by that piece.

tlb• 3 years ago

While cloning kind of works, I wouldn't expect the results to be quite as good as the original.

Right now we're in this weird phase where we have no accurate metrics for how good an LM is, so people just try them with a handful of queries and are subjectively impressed or not. Maybe clones can pass that test, but as the market gets more sophisticated, people will prefer superior models.

boole1854• 3 years ago

It may not be exactly as good, but it's also a 7B parameter model vs 175B parameters for GPT-3.5 (i.e., text-davinci-003). People are running the new model on their phones, laptops, and Raspberry PIs.

People's mouths were watering over the commercial implications of the recent 90% drop in cost for the new ChatGPT model. Now imagine if you can get similar performance on a model that requires <5% of the parameters.

sharemywin3• 3 years ago

not to mention I think you want your knowledge in some kind of database and just let the AI just do the conversational part of it.

which would be a much easier and more reliable(explainable) way to do it. just find and parrot the facts.

pilooch• 3 years ago

This has been true of machine learning APIs from the early renaissance of neural networks. Computer vision APIs have been used for years to grow datasets and fine-tune models on the data.

As a matter of fact, the API/SaaS business for machine learning 'direct' model results have suffered from this since the early days. Now, what may make more sense and survive longer are full data-storage-oriented platforms, and niche products that encapsulate all underlying model calls, including to publicly available models. Sometimes glue code or even devops becomes the true solid business case of AI/ML apps ;)

optimalsolver• 3 years ago

It's really weird to see a blog-length Twitter post.

dragonwriter5• 3 years ago

Its a new paid-membership benefit, so it will probably become more common.

ipsum23• 3 years ago

Yudkowsky has a very shallow understanding of machine learning/AI, but loves to write in a conspiratorial way with in-group terminology to make people think that he's onto some dark secrets.

What he's describing is common knowledge. This technique is called knowledge distillation (there are some small differences, but the idea is the same). A large model (here, openai's davinci) is used as a teacher model to train a smaller student model (llama). This is used practically everywhere in industry to improve inference times.

polygamous_bat2• 3 years ago

Yudkowsky has a very shallow understanding of machine learning/AI, but loves to write in a conspiratorial way with in-group terminology to make people think that he's onto some dark secrets.

Perhaps I am a little behind the times, but what are reasons/achievements/credentials behind his huge, prophet like status within the cult of rationalism? Beyond writing a popular Harry Potter fanfic, I mean.

nwah1• 3 years ago

Charisma, humor, and religious-like ideological framing that can acquire true believers who believe they are saving the world.

agentwiggles2• 3 years ago

I think EY's big contribution to the LessWrong/Rationalist cult/movement/ethos is writing the "Sequences," which purport to be a guide to training yourself to think rationally. (https://www.lesswrong.com/tag/original-sequences)

The rationalists are an interesting group. I like a lot of the vibe of the whole thing, but there is also some very weird navel gazing that has cropped up. It can be hard to differentiate from the good and the bad.

I am overall a little surprised by the vitriol towards EY in this thread though. While I agree that the guy is somewhat shrill and his ideas are pretty out there, he really doesn't seem like a con artist from my standpoint. It seems to me that he is genuinely extremely concerned about AI alignment/risk, and his actions seem to make sense in that framework without needing to ascribe a bunch of weird and sinister motivations to him.

Not professing agreement or disagreement with him on any particular thing, just my two cents.

didericis6• 3 years ago

He’s an excellent writer and has spent a long time trying to come up with high level easily understandable ways to try to measure alignment abstractly through pure logic/outside of any given system.

Essentially him and his crowd have been trying to create a rationally grounded purely logical non contradictory moral framework for action from axiomatic first principles (he’s not the first or the last to do this), and they believe such a system is essential if AGI comes to be.

I personally think AGI is impossible, his purely logical moral system is impossible, and think what he’s working on is not applicable to current AI (I think he knows this, but his conclusion seems to be that the path to AGI is out of the bag/was made incredibly foolishly and should have been constructed differently with more ability to add guardrails)

But I do think the high level abstract way he thinks is fruitful/very valuable, and a good moderating counterbalance to the frenzied “harder better faster stronger” ML stampede that’s not really thinking about the ramifications of what they’re doing with any sort of discipline.

traverseda• 3 years ago

If scientific knowledge were hidden in ancient vaults (rather than hidden in inconvenient pay-for-access journals), at least then people would try to get into the vaults. They'd be desperate to learn science. Especially when they saw the power that Eighth Level Physicists could wield, and were told that they weren't allowed to know the explanation.
[...]
If the Great Secret of Natural Selection, passed down from Darwin Who Is Not Forgotten, was only ever imparted to you after you paid $2000 and went through a ceremony involving torches and robes and masks and sacrificing an ox, then when you were shown the fossils, and shown the optic cable going through the retina under a microscope, and finally told the Truth, you would say "That's the most brilliant thing ever!" and be satisfied. After that, if some other cult tried to tell you it was actually a bearded man in the sky 6000 years ago, you'd laugh like hell.
And you know, it might actually be more fun to do things that way. Especially if the initiation required you to put together some of the evidence for yourself—together, or with classmates—before you could tell your Science Sensei you were ready to advance to the next level. It wouldn't be efficient, sure, but it would be fun.
[...]
And no, I'm not seriously proposing that we try to reverse the last five hundred years of openness and classify all the science secret.

~ Eliezer Yudkowsky, https://www.lesswrong.com/s/6BFkmEgre7uwhDxDR/p/3diLhMELXxM8...

That's really all there is to Yudkowsky's persona of occultism at the end of the day. It's an intentional affectation.

uoaei• 3 years ago

He spends a lot of time writing on LessWrong, noted online agora for the Dunning-Krugers among us.

z3c0• 3 years ago

Thanks for clearing this up. This seemed like much ado about nothing, but I didn't want to assume that this guy was trying to appear more knowledgeable than he may be.

I think the most impressive feat of GPT/LLMs thus far has been all the ML experts its caused to materialize from thin air.

James_Henry• 3 years ago

Good rule of thumb: Yudkowsky is always trying to appear more knowledgeable than he is. He's also been larping an AI expert for a long time.

Kinrany2• 3 years ago

The common knowledge you described is that the technique exists, not the economic implications of its existence.

ipsum23• 3 years ago

I thought the implications were pretty obvious. OpenAI prohibits this in their TOS for explicitly this purpose and is called out in the Alpaca post[0]

You may not ... use the Services to develop foundation models or other large scale models that compete with OpenAI

[0] https://crfm.stanford.edu/2023/03/13/alpaca.html

Eliezer2• 3 years ago

(1) This is not knowledge distillation, which requires knowing the logits and not just the output text.

(2) "Training a smaller model off davinci using the logits" would have been vastly more expensive than what they did, which was to fine-tune another pretrained foundation model, which happened to be smaller, using output of another foundation model, which happened to be larger. The relative size isn't the key idea, it's that they were able to extract the fine-tuning for instruction-following (which is relatively computationally cheap to train, but requires a custom expensive small dataset) and port it from one foundation model to another. This is what has implications for the companies that planned to have this custom fine-tuning as part of their competitive moat.

(3) I understand that ad hominem is against the culture that Hacker News tries to cultivate here. Consider reading or rereading the local guidelines on discussion, as well as reading or rereading an article on knowledge distillation, the Stanford publication of Alpaca, and my tweet.

shenberg• 3 years ago

It's referred to as distillation in the literature in separate sub-fields (e.g. LRspeech, a paper about low-resource speech-to-text & text-to-speech from Microsoft Research, 2020, uses datasets generated by a large model to train a small one and refers to this as knowledge distillation: https://arxiv.org/abs/2008.03687 )

Imnimo6• 3 years ago

Matching logits is merely a special case of knowledge distillation. See section 2.1 https://arxiv.org/abs/1503.02531

ipsum23• 3 years ago

1) It is a general version of knowledge distillation. For example, this paper from 2016 describes the same technique: Sequence-Level Knowledge Distillation [0]

This sequence-level approximation leads to a simple training procedure wherein the student network is trained on a newly generated dataset that is the result of running beam search with the teacher network

2) Fine-tuning is a step in the training process. Language models are first pre-trained, then fine-tuned. This is a pedantic quibble.

3) It is unsurprising that you don't understand ad hominem. Giving background information and pointing out the style of writing is relevant to arguments made.

[0] https://arxiv.org/abs/1606.07947

YeGoblynQueenne2• 3 years ago

> the mask of the shoggoth

I threw a null cultural reference exception on that one. Can you please clarify?

Edit: I know the "Masks of Nyarlathotep" and what a shoggoth is; I just don't know what is the "mask of the shoggoth".

Edit 2: Really, the one thing I wanted to be able to do with a language generator has always been to get it to GM a CoC campaign for me. It's still the case that there's no chance of that. Just as an aside.

karaterobot• 3 years ago

To be generous, I did not know this was a thing, and if you'd told me it was a thing, I would have said "gosh, you don't say" and not realized the implications. Not everybody is embedded in this world, so we need explainers.

PoignardAzur2• 3 years ago

What he's describing is common knowledge.

The implications aren't quite common knowledge yet: a lot of companies are still operating with the "we'll beat the competition by building a better AI than them" mindset.

I don't think the broader market has caught up with the idea that investing in AI gives you no long-term competitive advantage. (eg StabilityAI hasn't gone bankrupt yet)

seydor• 3 years ago

Another way to put it is : current crop "AI companies" don't really have a moat. Transformers have existed for many years, and a bit of RL on top of it is not enough to get ahead. It's all about who has the most money to spend training larger and larger, but what happens when they all run out of money for that? LLMs will become cheaper than web hosting

webmaven3• 3 years ago

That's overstating things.

Web hosting will always be less demanding, unless something weird happens and GPU/APU/TPU type hardware (along with HBM and related ancillaries) somehow becomes less expensive than (or subsumes) general purpose CPUs and RAM.

5cott0• 3 years ago

not reading yuds unless it takes the form of of harry potter fanfic

zadler2• 3 years ago

I don’t believe the idea that they would lose their moat. That’s like saying, any product you can copy has no moat, it’s not true at all.

sharemywin3• 3 years ago

but what ties you to the product?

chat history but is that really that important?

It's one api call from an integration stand point.

It's not a marketplace so you don't have network effects much.

the interface isn't all that much.

it's kind of a boring bot. it doesn't even tell dirty jokes.

I really do feel like this is a race to the bottom.

zadler2• 3 years ago

We have a very simple view of the product right now, and it's also new, so the different available options don't have much history to inform preference. That will change, and the stakes are very high, since this technology is capable of removing alot of jobs, so the company that f*cks up the least is going to become preferential. This effect will be magnified as regulation / beaurocracy come in more. Also, the post seems to assume that a single model will be published and then not change for a long time, enough time that a competetor can copy it and sell their copy; but this is also not true, models will be constantly updated and as soon as regulation catches up it'll be a legal requirement that they are constantly updated. And due to the power of the models, the state will quickly want to pick favorites, and that will also impact the difficulty of displacing a model provider. So, forget it, there's alot more involved than just having a copy of the result of some algorithms.

rafaelero• 3 years ago

There will be behaviors smaller models won't be able to mimic, though. So sure, fine-tuning advantages will be rapidly copied, but huge models that require large data centers won't. Economic incentives are still aligned with companies going through the path of larger and larger models.

blueyes• 3 years ago

Cutting-edge LLMs are commoditized anyway. There are very few tech moats here, so his point doesn't change the economics much. Data will sometimes be a moat, but companies with proprietary data will also expose their models through GUIs and therefore have other means of preserving that moat. OpenAI is moving toward economies of scale with their API pricing, which is smart.

ozten• 3 years ago

Subscribers get 4000 character limit? Another way in which Twitter is not twitter any more.

kybernetyk• 3 years ago

But Twitter is still shit.

Imnimo6• 3 years ago

Completely missed the point, as usual. Consider the amount of world knowledge present in a 175B model vs a 7B model. You can elicit instruction following behavior with a relatively small number of samples , but you're never going to get that coverage of memorized facts. The Alpaca blog post gives the example that Alpaca doesn't know the capital of Tanzania, for example. The most is that depth of knowledge and understanding, not the surface level instruction following behavior.

radu_floricica• 3 years ago

The point is that you can split the work in two: the one you mentioned, creating and training the model, and a separate step of shaping it with reinforcement learning. Which is quite expensive, because it uses human operators in one of its steps. But apparently you can duplicate easily enough this with an API of an already educated engine, which makes the second step not be a moat anymore.

Imnimo6• 3 years ago

How expensive do you imagine collecting human feedback data is compared to the cost of doing the original language modelling training?

cubefox• 3 years ago

But that's not a new point?! The second, expensive step (RLHF) was already automated by Anthropic with their Constitutional AI approach several months ago.

terafo3• 3 years ago

You don't need to store it in weights. It would be too expensive to constantly update model with new data. It wouldn't be able to contain everything anyways. LLaMa 30B with access to 100TB of storage(you can store whole libgen plus arxiv and much much more in there) and trained to use it should be ahead of LLaMa 65B with no such access in terms of world knowledge. And it would cost less.

Imnimo6• 3 years ago

You can retrieve small amounts of specific information this way. But suppose the model was asked to write code in a new programming language that it doesn't have existing knowledge of. Could it do that given unguided access to the docs?

cm2012• 3 years ago

EY is a rare thing. Right or wrong, he is a modern day philosopher. He has ideas he makes the case for strongly and eloquently.

williamtrask• 3 years ago

EY is not rare, but he is famous by proximity to powerful/wealthy actors. There are many philosophers, likely more than ever in history. Aside from professional philosophers, there’s a large pipeline of PhDs “doctor of philosophy” who work at a high enough level of abstraction in their own field to be considered a philosopher.

originalcopying2• 3 years ago

all I see are more arguments for my opinion that digital assets + capitalist marketplace = human stupidity.

you're giving away your business crown jewels to competitors that can then nearly-clone your model without all the hard work you did to build up your own fine-tuning dataset.

you're loosing the capacity to continue to earn money for work that is already done. This just means they're not gonna be able to charge rent on the done work for too long.

the work is the training of the model. I chose to reject this to way to frame the issue; i.e. I reject that it is problem that they're giving away that hard work. It's only a problem due to social (market) ideologies of ownership which are necessary only by the logic of trade (or commerce).

As I see it, Eliezer is pointing out hat people who use the "product API"/ "chaptGPT service" can use this in such a way that they can clone it so to stop being a customer.

I see some kind of divine comedy in this (due to my own ideological beliefs).

OR can successfully sue any competitor that engages in shoggoth mask cloning.

this is just the legal, modern day equivalent of old school "break their knees unless they pay you"

PoignardAzur2• 3 years ago

this is just the legal, modern day equivalent of old school "break their knees unless they pay you"

I don't EY is advocating for this, so much as saying "unless AI companies can find a way to establish such a racket, they won't be profitable". That's a factual claim, not a value judgment.

originalcopying2• 3 years ago

I've recently started to believe that known current legitimate governments are (and have always been) the legacy of people in history doing rackets like these in the now historical past.

dragonwriter5• 3 years ago

this is just the legal, modern day equivalent of old school “break their knees unless they pay you”

In the same way that proprietary software, or property rights in commercially useful property more generally, are that, yes.

BryantD• 3 years ago

Simon Willison wrote about this on Monday, so “I don’t think people realize what a big deal it is…” seems false.

renewiltord2• 3 years ago

The idea was fairly natural. I suppose having done it successfully is what is interesting.

https://news.ycombinator.com/item?id=35113781

Perhaps we will provide feedback to open source Llama using ChatGPT. The cost to adjust the model is presumably what's hard?

What I am so impressed by is that the state of the art is so democratic now. All the top AI posts on HN are things that I was able to get to myself days before the were posted here. This must mean either that the true SOTA is way ahead of what we see or that anyone can hit the current SOTA. Both are very exciting developments!

dragonwriter5• 3 years ago

The conditions for AI profitability he suggests as a “new idiom” for AI have fairly exact parallels in the world of software, but where the secret sauce has expanded to include inputs and outputs, not just the software itself:

The AI companies that make profits will be ones that either have a competitive moat not based on the capabilities of their model,

This is the “open source” case, where profitable firm provide ancillary services and support, rather than directly selling the capabilities of the software, because the software is not at all exclusive.

OR those which don’t expose the underlying inputs and outputs of their model to customers

This is the “internal software” case, where the profitable firm does not provide access to the software at all, but uses it internally in a way which is opaque from the outside to support other products or services it provides, maintaining exclusivity because no one outside actually gets access to the software itself. (Arguably, non-source-available closed source software might fit in here, though its often possible to reverse engineer software that you have access to without the source, from the executable, so I'd argue that even without source available its more the next category.)

OR can successfully sue any competitor that engages in shoggoth mask cloning.

This is the “proprietary software” case, where the profitable firm protects its exclusivity through legal constraints on people who have access to the software, while giving customers access to it.

Given that the big firms in AI largely either are or are closely associated with big firms in software that have successfully used all three models for different parts of their portfolios, I don’t think that them figuring out AI profitability is particularly challenged by this “new idiom”.

Eliezer Yudkowsky on large language model economics

Related Stories