Systems design 2: What we hope we know

https://apenwarr.ca/log/20230415

by Amorymeltzer • 3 years ago

181 29 3 years ago

"I've never seen anyone say ChatGPT feels like magic"

This is a curious line - I have, and I personally have that reaction when I use it. It might even be the most magical technology I've ever encountered.

jt2190• 3 years ago

This is a curious line…

Literally in the next sentence the author also remarks that he finds it odd:

“That makes me curious.”

He finds it curious because he also finds the technology is magical.

Eventually he concludes:

“Let's say you have a Large Language Model AI, trained on all the text in the world. We understand the mechanisms (engineering) but the outcomes are surprising and emergent (magic). Its job is to repeatedly predict the next word that would come if a person wrote it – a form of negative latency, or non-causality. From this, for reasons we don't really understand, it can say an impressive variety of unexpectedly meaningful things.”
“When you have all the text in the world memorized, that means you have access to all the insights that have ever been written. You need only recognize them, and have a good idea of what the reader knows already, and you can produce insights – things the reader has never heard before – on demand. You can get an A+ from my English teacher. Can our AI do that? I don't know. Certainly sometimes, maybe not reliably yet. Any time it does, it will feel like magic.”

MH15• 3 years ago

While I personally don't think ChatGPT feels magical, I know many people who feel this way. It's a really common opinion, especially among non-programmers.

amadeuspagel• 3 years ago

I guess it doesn't feel like magic because it feels human?

carapace• 3 years ago

FWIW, apenwarr is one of the most intelligent humans I've ever met. When people bring up 10x programmers I think of him. I don't really have much to add, I just wanted to mention that for context.

tudorw• 3 years ago

thanks, a good read, I'm always curious to go back and look at early knowledge system innovators who were looking at the 'causes' of 'breakthrough hypothesis' and looking to then 'engineer' a surrounding system to support more frequent breakthroughs.

rektide• 3 years ago

Placing emerged complexity/magic in a bucket together & saying science despises them is a novel & revelatory point of view. Avery is such a treasure!

Lovely idea towards the bottom too,

Or maybe we need to draw a distinction between producing insight and recognizing insight.

I personally identify with this & see it as super core a distinction. I'm reasonably bright & can find some clever ways of doing things even if I haven't any learnings here. But like 90% of what I love & appreciate about self is a resourcefulness & ability to recognize & integrate insight.

kqr• 3 years ago

On the quote opening the article:

Any sufficiently advanced technology is indistinguishable from magic.

I had a lecturer who used to counter "if something ever seems like magic it means you're not asking enough questions."

That stuck with me, although the article goes on to counter it fairly well.

Sniffnoy2• 3 years ago

Since we now know what magic means, we know what magical thinking means: a tendency to interpret something as magic instead of trying to understand it.

No, "magical thinking" has a particular meaning, and it's not that: https://en.wikipedia.org/wiki/Magical_thinking

Mind you, people often use the phrase in a pretty loose way as a way of dismissing an argument, but it does seem to be based on the technical use, and not the meaning Apenwarr puts forth.

epgui6• 3 years ago

You’re allowed to define different meanings for words if it makes your argument go forward and if you make it clear what it is.

However I still find the argument very unpersuasive, and the usual meaning of “magical thinking” makes me feel uncomfortable too.

Sniffnoy2• 3 years ago

Sure, but in this case he's explicitly discussing other people's use:

On the other hand, people who believe AIs and specifically LLMs possess "intelligence" are often accused of "magical thinking."

singpolyma3• 3 years ago

That definition seems pretty closely related to his.

epgui6• 3 years ago

I appreciate that the author is using very specific meaning for the term “magical thinking”, but even with this, I am uncomfortable with the assertion that somehow understanding emergence has anything to do with magic or magical thinking, as (re)defined in the article.

Anyone who spends enough time studying biochem will have an intuitive understanding / acceptance that biology emerges from chemistry, which emerges from physics… But there’s not really any “magical thinking” (as defined in the article) involved.

Emergence of cognitive processes and experiences in the brain is not really any different in this respect. If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.

analog312• 3 years ago

My thought about emergence is that it is not a phenomenon in its own right, but a category of phenomena that are identified subjectively. A lot has been written describing the wonders of emergence. Synthetic model systems have been developed with seemingly emergent behaviors, without leading to novel, empirically testable predictions. So far the only empirically testable models leading to emergent behaviors such as we observe in nature, are in fact the laws of physics and chemistry, more or less as we know them today.

To me, "magic" means that a phenomenon is inherently inaccessible to naturalistic inquiry, not merely that such inquiry is temporarily stalled due to inattention, lack of the right insight, or insufficient technology.

epgui6• 3 years ago

To that short list I would add some purely mathematical topics such as cellular automata (which show great promise towards helping formalize the concept of emergence IMO), but also biology, ecology, neuroscience, psychology, sociology, economics... There's layers upon layers of emergence all around us.

Even just within "physics" there are probably multiple layers of emergence (a striking and popular example is particle-like behaviour arising from wave-like fields). There is "reasonable speculation" as to whether space, time, gravity might all be emergent rather than fundamental...

geocar2• 3 years ago

Anyone who spends enough time studying biochem...

...knows about how LSD was used to discover DNA. Agreeing it possible is one thing, but to actually expect it? That is magical thinking.

I don't think putting random chemicals in your brain is all that different than putting a bunch of random coin-flips in a training function, so I don't think it's a terrible analogy either: The fact that something amazing comes out of that process is magic.

I am uncomfortable with the assertion that somehow understanding emergence has anything to do with magic or magical thinking

That's common amongst scientists. Words matter, and replacing magic with "emergent" keeps it at arms length, and detachment is important to do science.

But I suggest you play along anyway, instead of pretending they meant "emergent" every time they say "magic" because words matter...

If anything, it’s people who insist that human experiences are “special” and “just different, obviously” that are doing the magical thinking.

I hear you, but it is hard to operate on the assumption you're not special, after all, you're probably more successful than a lot of people, and I bet you think your hard work had something to do with it.

The science says this isn't true, of course, that the colour of your skin, the postcode you were born in, that the wealth and other genetics of your parents, are all greater predictors of success, and you probably know this, but if you're a human being you can't help but think that you have willpower and ethics, even if you don't believe in a soul.

Or another example: if you've ever looked at someone who behaved badly, and said you wouldn't have done that? Of course, if you really did have their life, their parents, their postcode, their skin, their abuse and traumas over the years, their joys and friends, stories of parties and life, and so on, well then you definitely would have, or you've got to believe you're special.

This kind of magical thinking has kept you alive- it's provided impulse to the bag of biology and chemistry to twitch and react its way through your life- something you've got to admit sounds a lot less fun when I use science-words to put it that way.

epgui6• 3 years ago

I really tried to read this comment, but I can't get past the word salad. I'm sorry, I don't know what you're trying to say.

Is there some sort of allusion to the meritocracy trap or fundamental attribution bias? How does any of that relate to my comment? I'm just confused.

In any case I believe the connection you make between LSD and the discovery of DNA is apocryphal.

nyanpasu64• 3 years ago

In short, in software engineering, we acknowledge that failures happen and we measure them, characterize them, and compensate for them. We don't aim for perfection.

If software engineering revolves around writing software complex enough to have bugs not understood, or accepting memory safety errors as a natural byproduct of programming (rather than programming mistakes to be exterminated on sight, and made impossible by design unless it prevents writing useful programs), I'd be more effective and happier practicing applied math than engineering.

skybrian• 3 years ago

It's also accepting that performance is variable, that the network is not reliable, that machines can fail at any time, and that memory corruption happens.

When you build systems at scale you need to worry about these things.

Programming language API boundaries don't make real-time performance guarantees, not even in Rust. We hope to get typical performance. In a mathematical context (for example, proving programs correct), you often ignore these things and worry only about whether the program will give you the right answer, eventually, when the hardware works.

akkartik• 3 years ago

Minor quibble.

"Summarization can be done with little to no insight at all. Perhaps to do it better requires some insight."

By this standard caching requires little to no insight at all. Perhaps some predictors require insight.

apenwarr• 3 years ago

It’s possible to build a cache without any insight, but to make a cache that works well in a given domain requires knowledge from outside the world of caching itself. The same is true of summarization and compression.

ChatGTP• 3 years ago

Many people keep saying the brain isn’t magic so blah blah sentience etc, but if that’s the case, we cannot claim LLMs are magic either, or any “emergent properties”, else you do believe in magic. Which is it, magic or no magic? We can not have it both ways.

Maybe the idea of magic shouldn’t keep being referenced, we should refrain from comparing and just apply equal respect to all beings and phenomena and just admit some things are, and always will be mysterious and move on with life ?

tuatoru• 3 years ago

I love the meta-joke about the "levels of indirection" quote.

ignoramous2• 3 years ago

Can you point me to it? ;)

Systems design 2: What we hope we know

Related Stories