What does Alan Kay think about LLMs?

https://www.quora.com/What-does-Alan-Kay-think-about-programming-and-teaching-programming-with-copilots-and-LLMs-of-today

by agomez314 • 2 years ago

184 149 2 years ago

quora.com

skadamat• 2 years ago

This answer is amazing and classically Alan Kay! There's so much here to unpack because of all the different areas Alan draws from in his work (he's like a computing philosopher).

All I will say is that for people who want to understand his perspective, there's a large epistemological load to overcome. Sampling his talks is a good starting point though: https://tinlizzie.org/IA/index.php/Talks_by_Alan_Kay

agomez314• 2 years ago

wow this list is super useful, thanks for sharing!

layer86• 2 years ago

He always criticizes how everyone got computing wrong (fair enough), but never offers anything but the most vague suggestions of how to do things differently.

detourdog2• 2 years ago

I think trustability is pretty specific and without a foundation of trust the rest of the effort”s value is unknowable.

theGnuMe2• 2 years ago

True message passing is one of the key ideas behind smalltalk which would improve computer security considerably. This is analogous to the design of the internet that he references. The internet scales well. So imagine building entire computing systems this way.

sebastianconcpt4• 2 years ago

He did tho. He mentioned more than once that OOP was more about message passing than behavior + state.

mempko4• 2 years ago

What? He literally built systems that did things differently. Look at smalltalk.

carapace2• 2 years ago

You are impressively wrong. He just wrapped up an incredible research program that demonstrated many concrete implementations of great computing ideas.

After 16 years of continuous research and important contributions toward its mission - "Improve 'powerful ideas education' for the world's children and to advance the state of systems research and personal computing" - Viewpoints Research Institute concluded its operations at the beginning of 2018.

http://www.vpri.org/

joshmarinacci2• 2 years ago

His entire 60 year career is a suggestion of how to do things differently. ;)

https://en.wikipedia.org/wiki/Alan_Kay

agumonkey2• 2 years ago

one real experiment he contributed to was VPRI ometa and the os built on top of that (every part of the OS built on top of an eDSL)

fmap• 2 years ago

Others have already mentioned this, but Alan Kay spent his entire career experimenting with approaches to radically simplify computing. The progress reports from vpri give a good overview of a recent project that has since wrapped up, e.g.: https://tinlizzie.org/VPRIPapers/tr2012001_steps.pdf

Osmose• 2 years ago

He has generally great insights about principles to aim for, which is part of the answer, but no, he does not have the entire answer. His insights are inspiration for others who contribute their own parts to potential answers—that's how open collaboration works.

KorematsuFredt• 2 years ago

I wish someone could provide a tl;dr of that answer to us simpletons. His writing style is something I am unable to comprehend.

dpflan• 2 years ago

There is a lot in here, various paths to venture off, but the bottom line seems to be trust is important when running commands on a machine, and LLMs are not trustable. What else?

altruios• 2 years ago

what would be required for trusting an LLM?

1: 100% transparency. Open Source code, fully (and correctly) attributed training data.

2: A predictable model of what these models are actually encoding (so that hypothetical new models (or modifications) can be reasoned about).

thesz3• 2 years ago

I think I have to know what chain of reasoning is behind this or that fact and/or deduction. i would like to be able to verify that.

For example, the proof of the absence of solution in SAT should be accompanied with the easily verifiable chain of reasoning. This shows the absence of incorrect deductions and missing assignments. Another example is autovectorization in contemporary compilers, they can show you why parts of your loops are not eligible for vectoriztion.

All LM's can do is to show me that these parts of those inputs are important for that output, but nothing else. Thus, they cannot be trusted even for minimally critical tasks.

utopcell• 2 years ago

Open sourcing would only guard against human programmers. The fact of the matter is, we don't _really_ know why LLMs work.

falsaberN12• 2 years ago

Transparency won't help a lot from a technical standpoint (seems more like a solution to a legal issue than a technical one). I can't trust LLMs because they just...recombine text by probabilities and aren't deterministic. I get incorrect information every time I ask them a thing, and it's incorrect in different ways every time. The only things they seem to get consistently correct are very widespread facts that are much faster and trivial to google.

I still find it funny we managed to get image generation working so much better than text.

markrages• 2 years ago

Even with these in place, the output of the LLM is not trustworthy. In the sense that it doesn't care what it true, only what is plausible.

tjr• 2 years ago

I would add, repeatable / reproducible results. Given the same input, you get the same output every time. If the input includes some sort of random seed for the express purpose of getting different output intentionally, then so be it, but I can't trust a program that I never know what it will do in response to what I tell it.

fzzzy6• 2 years ago

Isn't the gist that you should be able to trust your teacher?

sebastianconcpt4• 2 years ago

Questioning trustfulness to ontological hallucinatory automatons. Nailed it!

sebastianconcpt4• 2 years ago

It is by definition that the silicon automatons cannot escape from their ontologically stochastic imitative hallucinations.

All they can do is be observed by us and with that expose us to get induced by association, ontologically human hallucinations and deal with its outcome of real consequences.

devjab2• 2 years ago

I use LLMs quite a lot to help me in my work, but they are wrong so often that it’s ridiculous. This isn’t a major issue when you’re an expert using the tools to be more efficient, because you’ll spot and laugh at the errors it makes. Sometimes it’ll be things that anyone would notice, like how a LLM will simply “invent” a library function that has never existed. Even if you’re not an expert, you’re not going to get that PNP-whatever function to work in Powershell if it never existed in the module to begin with.

Where it becomes more dangerous, at least in my opinion, is when the LLM only gets it sort of wrong. Maybe the answer it gives you is old, maybe it’s inefficient, maybe it’s insecure or a range of other things, and if you’re new to programming, you’re probably not going to notice. Hell, I’ve reviewed code from senior programmers that pulled in deprecated things with massive security vulnerabilities and never noticed because they were too focused on fast delivery and “it worked”. I can’t imagine how that would work out for people trying to actually learn things.

I’m not sure what we can really do about it though. I work a side gig as an external examiner for CS students. A lot of the curriculum being taught (at least here in Denmark) are things I’ve seen the industry move “beyond” in the previous 20 years. Some of it is so dated that it really makes no sense at all. Which isn’t exactly a great alternative to the LLMs, and it’s only natural that a lot of people simply turn to these powerful tools.

I tend to tell people to ask their favorite LLM to help them solve a crossword. When you ask it to give you words ending on “ing” it’ll give you words that don’t end on “ing” because of how the tokens used work. This tends to be an eye opener for people in regards to how much they trust their LLM. At least until they get refined enough that they can also do these things.

Anyway, it’s a good answer.

bongodongobob2• 2 years ago

I'm sorry, but it sounds like you're using ChatGPT3.5. I get very very few invented libraries unless it's a very specific version of an old or rare language. I tried your "ING" example and GPT4 got 100/100.

Idk, ymmv, but GPT4 beats the pants off of 3.5.

samatman• 2 years ago

I use GPT4 and it confabulates constantly. Try it on a language in the 10th-20th range of popularity (let alone lower) and I expect you'll see it as well. I don't see that as often with Python, the only top-5 language I currently work with.

sumtechguy• 2 years ago

Yesterday I asked it the date. Then asked it if it was possible to give me the date. It gave a very different answer that contradicted what it just did. It is kind of hard to trust when it can not even make itself consistent between 2 prompts. For creative prompts this thing is very cool. For logical prompts it is frustrating to no end how much detail it gets wrong.

dimal• 2 years ago

Off topic, but I didn't realize Alan Kay was regularly answering questions on Quora. I can ask "What does Alan Kay think about X?" and get an answer from Alan Kay?!?

YeGoblynQueenne3• 2 years ago

Well, you can try. And if he doesn't answer you can always ask an LLM :|

stana• 2 years ago

Can we trust it though that is what Alan Kay would say?

swyx• 2 years ago

when you're alan kay you can literally just title things in the third person and it'd get read haha

mempko4• 2 years ago

I see a lot of people here are missing his "big deal" which he talks about in the end where he references the "Spaceship Earth" problem.

What I believe he is getting at is people are going to use LLMs to build systems at scale to further strip mine society.

The "Spaceship Earth" problem is a reference to Limits to Growth. For those who haven't read "Limits to Growth", and the more recent Re-calibration of Limits to Growth, I implore you to do so.

https://onlinelibrary.wiley.com/doi/full/10.1111/jiec.13442

lucianbr3• 2 years ago

Most tech seems to be used, at least amongst other things, to further strip mine society. It's the extreme of wealth accumulation, and there's always some people wanting to do that. Microsoft uses the idea of personal computers to get a monopoly on OSes, and get as much rent with that. Google and Amazon use the internet. Apple the iPhone with the walled garden. OPEC uses oil to extract as much wealth as possible. Toyota and VW would also get a monopoly and then raise prices it if they could. Why do you think VW bought so many other car makers? Boeing now that it is too big to fail, focuses on making more money, and less safety and excellence in engineering. Everyone does it. Well, every big business.

It's a bad thing for sure, but doesn't seem specific to LLMs, and a solution would not be specific either.

mempko4• 2 years ago

It's not specific to LLMs, but the point is LLMs will make things more efficient and therefore make 'a bigger straw' to drink the milkshake.

LightFog• 2 years ago

This stood out to me too - the technology helping us move toward a ‘greed singularity’.

ozten• 2 years ago

That humans also do this all the time is “interesting”, “dangerous” etc., but it is also why trying to move from superstition (this is actually what “reasoning by correlation” amounts to) to more scientific methods is critical for anything like civilization to be created.

"reasoning by correlation" as superstition is a brutal insight.

kurthr• 2 years ago

It seems like the other part of this is the ability of LLMs to rationalize even answers that are not rational (he says piling BS on BS). That has previously been a very human (and quite distracting/confusing) activity and it's very effective on most people most of the time.

kergonath2• 2 years ago

"reasoning by correlation" as superstition is a brutal insight.

I don’t think he is right on that one, though. Reasoning by correlation is a kind of empiricism, and it can be tested. “These things happen together” or “if I do this, then that happens” can be disproven with statistical analysis even without any understanding of the underlying mechanisms at play their causes.

Superstition is beyond that; it is a belief that cannot be proven and that is not based on logic. The problem with superstition is not that there is no causal link between the supposed cause and the effect, it’s that there is not even a correlation.

halayli• 2 years ago

Superstition is assuming a causation when you identify a correlation.

Phiwise_9• 2 years ago

A road trip by random walk is also a kind of empiricism that can be tested. This does not imply it is worth the energy to actually do.

n4r9• 2 years ago

There are plenty of superstitions that are perfectly falsifiable. People continue to believe in them for comfort. For example, wearing odd socks to a sports match helps your team win. Or, leaving a fan on overnight might kill you.

alexvoda3• 2 years ago

I think the author and you are using different meanings for "reasoning".

We do know that many people even in professional fields confuse correlation with causation. And even when that doesn't happen, when only doing measurments, correlation is often considered good enough without giving any thought to the underlying mechanism. This may be the root of Goodhart's law stating that when a measure becomes a target it stops being a good measure. The measure was only correlated with the behaviour the person doing the measuring thought was measuring.

Superstition is similar, but drops any shred of statistical rigour, relying on mere anecdotal evidence.

atoav• 2 years ago

The problem here is that correlation is not a hard term for the people doing drawing the lines. People who take homeopathics for example will assure you they always get healthy when they took the sugar pills. What they fail to see is that they would have gotten healthy without it as well.

If you always rub your lucky penny before playing a sports game a superstitious person would account wins for the penny and excuse losses by not having rubbed the penny enough, the right way, not concentrated enough. This also feeds into correlation.

We are not talking about things actually being correlated, we are talking about people believing they are. Those are two wildly different things one is physical reality and doesn't change when you ignore it, the other is bound to individuals and tbeir world view and can take on absolutely ridiculous forms.

codethatwerks• 2 years ago

Causation is just correlation, and a good story. :-)

ein0p2• 2 years ago

It’s great that he recognizes that most people aren’t to be trusted either. Most people outside the field are talking about “safety” as though humans never lie or make up bullshit, whereas in reality I trust GPT4 much more than I’d trust an average human, just not 100%. Literally people keep hyperventilating about “fake news” generation even though their entire news consumption already consists solely of pre approved narratives, many of which are fake, by omission or otherwise.

I can’t wait for the day when ultra long context LLMs are able to point out omissions and factual contradictions in human produced fake news for any given event

ribosometronome• 2 years ago

Are context windows not long enough to do this for most news articles and forum posts nowadays? I'd've thought the issue would be finding the relevant and reliable sources of truth for comparison would be the difficult part.

eska• 2 years ago

The comparison is not between an LLM and an average human but an LLM and an expert on a subject.

discreteevent2• 2 years ago

Here's an excellent recent comment along the same lines (in response to whether our intelligence is statistically based like LLMs):

mjburgess 9 days ago

Suppose you touch a fireplace once, do you touch it again? No.

OK, here's something much stranger. Suppose you see your friend touch the fireplace, he recoils in pain. Do you touch it? No.

Hmm... whence statistics? There is no frequency association here, in either case. And in the second, even no experience of the fireplace.

The entire history of science is supposed to be about the failure of statistics to produce explanations. It is a great sin that we have allowed pseudosciences to flourish in which this lesson isnt even understood; and worse, to allow statistical showmen with their magic lanterns to preach on the scientific method. To a point where it seems, almost, science as an ideal has been completely lost.

The entire point was to throw away entirely our reliance on frequency and association -- this is ancient superstition. And instead, to explain the world by necessary mechanisms born of causal properties which interact in complex ways that can never uniquely reveal themselves by direct measurement.

https://news.ycombinator.com/item?id=39661963

better_sh• 2 years ago

but what does Ja Rule think?

amelius• 2 years ago

LLMs are horribly broken.

But we somehow like the idea of putting everything behind an API and then call it a solved problem.

ianbicking2• 2 years ago

This feels like a limited and perhaps naive perspective on LLMs. If you looked at computers as adding machines in the 60s/70s then you'd be missing most of what was interesting about computers. And if you look at LLMs as a question answering service now, you are also missing a lot.

It's hard to compare trust of LLMs to other computing, because many of the things that LLMs get wrong and right were previously intractable. You could ask a search engine, but it's certainly no more trustworthy than an LLM, gameable in its own way. The closest might be a knowledge graph or database, which can formally represent some portion of what an LLM represents.

To be fair the relational systems can and will give "no answer" when an LLM (like a search engine) always gives some answer. Certainly an issue!

But this is all in the realm of coming up with answers in a closed system, hardly the only way LLMs can be used. LLMs can also come up with questions, for instance creating queries for a database. Are these trustworthy? Not entirely, but the closest alternative supportive tool is perhaps some query builder...? I have seen expert humans come up with untrustworthy queries as well... misinterpretation of data is easy and common.

That's just one example of how an LLM can be used. If you use an LLM for something that you can directly compare to a non-LLM system, such as speech recognition or intent parsing, it's clear that the LLM is more trustworthy. It can and does do real error correction! That is, you can get higher quality data out of an LLM than you put in. This is not unheard of in computing, but it is uncommon. Internet networking, which Kay refers to, might be an analog... creating reliable connections on top of unreliable connections.

What we don't have right now is systematic approaches to computing with LLMs.

r00fus3• 2 years ago

If you use an LLM for something that you can directly compare to a non-LLM system, such as speech recognition or intent parsing, it's clear that the LLM is more trustworthy

Except for the hallucination problem, sure. How do you ensure the answer you get from an LLM is not a hallucination? It sure doesn't.

pixl974• 2 years ago

How do you ensure any bits of data? You going to fist fight your uncle at the next family reunion when he promises you pigs can actually fly?

There are plenty of bits of data that can be hard references to datasets or actual physics to assign their probability value. And there are other things that can't.

ianbicking2• 2 years ago

"How do you ensure the answer you get from an LLM is not a hallucination?"

Intent parsing and speech recognition both frequently return inaccurate responses. It is true that with an LLM it can return something that is sensible but invented, where other systems typically return less inventive wrong answers.

With any of these systems you want to balance the probability of an incorrect interpretation against the impact of the action being taken, and get confirmation based on that. That's pretty normal engineering and UX.

pixl974• 2 years ago

“All models are wrong, some are useful.”

What's kind of funny is almost all the people that complain about accuracy of LLMs would gladly answer the question of "What's the chance for rain today" without giving giving you a 5 minute lecture on what forecasts actually mean.

soperj• 2 years ago

"I'm unable to provide real-time weather updates as my data is not current. To find out the chances of rain today, you can check your local weather forecast through a reliable weather website or app, or tune in to a local news station for the most up-to-date information." - chatgtp

andrewflnr• 2 years ago

Yeah because most everyone you talk to knows how to treat weather forecasts, but LLMs are still widely misunderstood. The right idea seems to be getting out, but in no small part due to the "complaining" you mock.

alexvoda3• 2 years ago

There was a time when a search engine would return no results if it didn't find anything instead of returning increasingly irrelevant results.

It was a conscious decision by corporations to implement the dropping of constraints and search terms when too few results would have been returned. Today the search operators are a joke.

LAC-Tech• 2 years ago

Typical Alan Kay. Very long, very thought provoking, full of historical references, and barely answered the question at all. :)

r00fus3• 2 years ago

He did give a TL;DR about halfway through his answer: "Finally, to try to answer the question … (Summary: I don’t think it would be a good idea at all)"

keybored• 2 years ago

What I like about programming (real programming) is that it is dumb and obtuse. You can see why things happen. Because the instruction languages are painfully literal. They will throw their hands up if you omit a step. Or crash.

You can understand it.

That’s why I dread the AI future.

aaroninsf• 2 years ago

Always a welcome and thoughtful opinion, but not correct about the potential for LLM. Waving away their output as "BS" is awfully flip, and IMO is hence one more example of Ximm's Law (that every critique of AI assumes to some degree that contemporary implementations will not, or cannot, be improved upon.)

One might find ambiguity in his criticism though, that LLM alone are insufficient... but, that's what Ximm's Law is saying. It's not very interesting to (as I would say, he does here) take on straw, rather than steel.

(A steely defense of LLM is to say that no one is particularly interested in scaling LLM without other improvements, though scaling alone provides improvements; multi-modal, multi-language, long-context, and most of all augmented systems which integrate LLM into systems rather than making them "systems on a chop", are where things look and IMO will be interesting.)

svieira• 2 years ago

By “help” I mean that — especially when changes in epistemological points of view from one’s own common sense are required , it can make a huge difference to be near a “special human” whose personality is strong enough to make us rethink what we think we know.

This is how I measure Ed-Tech companies. Do they have an awareness that you cannot replace the connection with other human beings that is an essential part of teaching with "facts" or not? If "yes, they have that awareness" how do they mitigate the problem?

slaymaker1907• 2 years ago

If we look at human anthropology, we see a species that treats solitary confinement and banishment from society as punishments — we are a society because we can cooperate and trust a little — but when are safely back in society, we start competing like mad (and cheating like mad) as though the society is there to be strip-mined.

I really like this quote. We simultaneously value trust and community, yet so many people also treat it as just another resource to turn into money and power. Alan Kay is a real gem.

PCMPSTR• 2 years ago

This part of his answer:

A key part of their design was to not allow direct sending of commands — only bits could be sent. This means that (other) software inside each physical computer has the responsibility to interpret the bits, and the power to do (or not do) some action

seems, at least on a basic reading, to contradict this famous little argument (or maybe trolling?) he had on HN with Rich Hickey where he seems to be suggesting that one shouldn't just send raw bits, but also a little interpreter along with the data: https://news.ycombinator.com/item?id=11945722

Maybe this is an inevitable consequence of always speaking so abstractly/vaguely, but it also makes it difficult to know what exactly he's suggesting the industry, that he is so routinely critical of, should concretely do next.

discreteevent2• 2 years ago

The main point he was making in that argument is that data is nothing without an interpreter. Sending an interpreter with the days is secondary. Rich hickey was saying that data is meaningful on its own. But that makes no sense.

What does Alan Kay think about LLMs?

Related Stories