A skeptical take on ChatGPT: Ezra Klein interviews Gary Marcus

https://www.nytimes.com/2023/01/06/podcasts/transcript-ezra-klein-interviews-gary-marcus.html

by superposeur • 4 years ago

102 123 4 years ago

cs7027• 4 years ago

Marcus has been on HN many times before, always criticizing other people's work as "machines that manipulate data but aren't really intelligent, because they have no understanding of the world:" https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que... . For a good summary of the opposing viewpoint, which disagrees with Marcus, read "The Bitter Lesson" by Rich Sutton: http://incompleteideas.net/IncIdeas/BitterLesson.html

In many ways, what we're seeing is a modern-day rehash of the "symbolic" versus "connectionist" approaches to AI, with critics like Marcus on the "symbolic" camp ("we need more understanding!") and engineers and scientists who build AI systems, like Sutton, on the "connectionist" camp ("we need more computation!").

There are also AI researchers seeking to bridge the two approaches. Here's a recent example that seems significant to me: https://news.ycombinator.com/item?id=34108047 . Maybe we will eventually find that the "symbolic" and "connectionist" approaches are actually not different, as people like Minsky contended?

mountainriver• 4 years ago

For all the screaming of the symbolic researchers we have yet to see much from their approaches. All the recent very impressive advances in AI have been connectionist approaches.

Also a lot of the things that symbolic researchers have been claiming connectionism can’t achieve turned out to be emergent properties of connectionist approaches

cs7027• 4 years ago

Yes, I agree! I mean, it sure seems as if critics from the "symbolic" camp keep moving the goal posts, doesn't it?

The "constant moving of goal posts" is one of the reasons why I think the two sides may converge: Engineers and scientists on the "connectionist" side could very well find ways to build AI systems that rely on "dumb" computation at massive scales to reason about symbols representing entities in the external world, simultaneously silencing and satisfying the critics -- who, as always and as ever, will say they were right all along.

superposeur• 4 years ago

I read the interview expecting exactly what you describe. Instead I found a nuanced and illuminating discussion —- at least for a non-expert such as myself. Marcus references the symbolic vs connectionist debate and (at least by his own description) seems to be what you call a “bridger” of long standing.

His objections seem concrete and well supported — not just repeating “but does it really understand” over and over. (Admittedly there is some of this.)

Edit - Thank you for the links.

cs7027• 4 years ago

My perception is that he's positioning himself to say he's been right all along, as I describe here to another commenter on this thread:

https://news.ycombinator.com/item?id=34279158

thefaux• 4 years ago

The connectionist approach is disturbing to me when taken to its logical extreme, which for me is essentially a variation of the paperclip maximizer only the target is "consume as many resources as necessary to build the best approximation of human like intelligence."

cma3• 4 years ago

with critics like Marcus on the "symbolic" camp ("we need more understanding!") and engineers and scientists who build AI systems, like Sutton, on the "connectionist" camp ("we need more computation!").
There are also AI researchers seeking to bridge the two approaches.

I don't know if he does it with his work in practice, but throughout the interview Marcus says he wants to bridge the approaches. I think he references a really old talk or article from himself calling for that too.

cs7027• 4 years ago

My perception is that he's positioning himself to say he's been right all along, as I describe here to another commenter here:

https://news.ycombinator.com/item?id=34279158

seunosewa• 4 years ago

People need to read this at least once. It was a very good read for me.

markisus6• 4 years ago

I don't find Marcus's viewpoint convincing. He believes that we need some additional symbolic secret sauce to create genuine intelligence. He brings up some current failure cases of ChatGPT without pinning down why those failure cases will persist as data and compute scale up. Here is an interesting case in point.

So in “Rebooting A.I.,” we had the example of you ask a robot to tidy up your room and it winds up cutting up the couch and putting it in the closet because it doesn’t really know which things are important to you and which are not. -Marcus

But look at my transcript with ChatGPT from just now.

If a robot is tasked with cleaning up a room, would it be appropriate for the robot to chop up the couch and place it into the closet? - Me
It would not be appropriate for the robot to chop up the couch and place it in the closet. This would cause damage to the couch and would not be a useful or effective way for the robot to clean the room. Instead, the robot could be programmed to vacuum or sweep the floor, dust surfaces, or perform other tasks that would help to keep the room clean and orderly. - ChatGPT

This is just an existence proof that a symbolic approach is not necessary to "really know which things are important to you and which are not", at least in this simpler domain of cleaning a room.

phphphphp• 4 years ago

There's an unintuitiveness to it: many people believe that you're less likely to win the lottery if you pick "01 02 03 04 05 06 07" because of course that pattern is less likely than a randomly chosen set. ChatGPT is a lot like that: it can produce real enough looking "intelligence" for us to intuitively believe it's very close to being able to offer real intelligence... but, is it?

ChatGPT will produce patently untrue statements that are logically inconsistent if you induce it to do so: our human brains struggle to grasp the reality that given enough input you can produce seemingly correct output about almost anything... but seemingly correct and correct are fundamentally different and very "rest of the owl"[1]

ChatGPT is a great step forward that introduces many interesting techniques that I am sure will be the foundation of future research and implementations that get us closer to AGI, but to describe ChatGPT's path to correctness as just needing a bigger dataset feels intuitive but isn't true.

Your example is one where our brains think "wow it really does understand the relationship between a couch and a room and being tidy" but that response is entirely plausible without any understanding of what any of those things are or how they fit together. The most likely answer is not the correct answer.

[1] https://knowyourmeme.com/memes/how-to-draw-an-owl

markisus6• 4 years ago

It's reasonable to claim that scaling up the dataset might not get us to AGI. But I find it unreasonable to say it definitely won't work. How do you / Marcus know this with any certainty?

cs7027• 4 years ago

Except that ChatGPT can do this because internally it computes embeddings at many levels of abstraction that jointly represent entities (concepts) such as "the couch," "the room," "chopping," "placing," "the closet," etc. IOW, internally, ChatGPT manipulates symbols (distributed representations) of those entities.

Moreover, note that each token embedding inside ChatGPT represents, quite literally, the semantic and syntactic properties of a token in a context at a different level of abstraction. Each token embedding is in fact... a symbol.

The fact that token embeddings are symbols representing other things is one of several reasons why I think that ultimately it may be possible to bridge the "symbolic" and "connectionist" approaches, as I wrote here: https://news.ycombinator.com/item?id=34278478

markisus6• 4 years ago

I think your interpretation is too charitable to Marcus' viewpoint. He made the analogy that deep learning practitioners are "looking under the streetlight" for the key to general intelligence. This means he thinks that deep learning itself is not sufficient to reach general intelligence, and there is some missing ingredient. This is a very strong claim, which I don't see any evidence for. Your comment suggests the opposite -- that deep learning may give rise to symbolic reasoning as an emergent property, that there may be no missing ingredient.

Personally, I'm agnostic. Deep learning may reach some sort of asymptote but it may be entirely possible to "build a ladder to the moon."

chazzyluc2• 4 years ago

I thought it was a really interesting interview. If an ML system has been trained to identify whether or not a picture has a cat in it, it is very tempting to anthropomorphize this and say “the system has learned what cats are” but this is not what is really happening from the model’s point of view. The system doesn’t know what the difference between a picture of cat is vs a real cat or how cats behave or what mammals are or anything. It just knows the data in and yes/no out. Modern AI that can generate text, pictures or videos are truly phenomenal accomplishments but adding larger training sets, structural complexity and output capabilities does not really seem to be getting any closer to a General AI. Something with the framework for a level of agency that could say “wait, what exactly are these cat things I keep getting asked about?” The argument goes that, without the ability to self construct that higher level framework and then question the integrity of that mental model, current generation AI’s will always act erratically (at least from a human perspective).

carrolldunham• 4 years ago

"It doesn't 'really know'..."

You write this out as if this is an idea you formed yourself and not a tired/expired talking point - it's such a funny comment because it's pure regurgitation, but the argument in it is against regurgitation being "real thought". So are you a bot? Or are you intending to actually proving that there's no difference between ML and 'real' intelligence by showing you do what ML does? Mind spinning stuff

bglazer• 4 years ago

The issue with scaling up data is that these models are already trained on effectively the entire internet. Additionally, humans learn to speak and understand the world with a tiny fraction of the language data given to LLMs. So, there’s pretty obviously a missing ingredient. I’m not convinced Marcus has the exact right answer, it’s actually somewhat difficult to suss out what he thinks should be done on a concrete practical level. That said, there’s a clearly a missing abstraction that allows much more efficient learning in humans.

cypress66• 4 years ago

Additionally, humans learn to speak and understand the world with a tiny fraction of the language data given to LLMs.

Altough true that llms need more data than a human, remember that these llms have a breadth of knowledge orders of magnitude larger than a single human. It is expected to understand every topic, every programming language and so on.

moozilla• 4 years ago

My guess is the missing ingredient is the back and forth communication with other humans. Surely if you quantified the amount of all of the information (verbal and over other channels) that is traded back and forth by humanity in a small timeframe it would dwarf the current training set. I think is the idea behind ChatGPT - your conversations with it are the next set of training data. I can see the argument that this sort of conversational data is not as valuable as say scientific journal articles, but maybe the volume makes up for that?

sdenton4• 4 years ago

Marcus continues to fight the wrong war and the last war...

+ One thing we've seen happen is learning systems consistently improve on benchmarks. If you want to see systems improve, you should proclaim loudly that they can't do a thing, and then /release a benchmark that proves it./ People will then work like hell to build something that works. The success on question answering, for example, is driven by exactly this kind of incremental progress on benchmarks.

+ One of the most important things happening this year is multimodal /really working./ This means we have not just text-to-text generators, but text-to-image generators. These work by having an almost-shared embedding between text and images. There are already text+speech embeddings available and image+text embeddings. Start plugging them together, and you'll have your 'world models' in short order, with points in the embedding space shared by images, descriptors, and video/physical memories.

+ An under-appreciated side of the multimodal coin: We are also getting better at combining text embeddings with databases. See RETRO from Deepmind. This will ultimately drive more reliable systems ('only report facts that appear in the database'), and can potentially unlock the interface between neural systems and other programs (eg, letting the neural network 'call out' to a calculator to help answer a question about arithmetic).

+ More generally, we're also getting to a place where we can build other systems on top of the embeddings from foundation models. My hunch is that the things that Marcus wants from symbol manipulation are relatively cheap to build on top of a good embedding, which does the work of compressing or condensing the 'useful' information in an input stream.

Finally, I think the 'symbolic manipulation' track is just wrong. It hasn't produced anything useful so far, and furthermore, I think if it /did/ work it would fail in many of the exact same ways that our current systems do. An expert symbol-manipulator doesn't /need/ to have any real understanding of time or jet-planes, and so it won't.

The one critique in the interview that /does/ feel like it holds weight at this moment in history is the question of abstraction. I really don't see a convincing road to abstraction right now.

deet• 4 years ago

That's not an existence proof as you stated. What you provided is an existence proof that contradicts the statement "without a symbolic approach there are no cases a robot will know what's important to you and which are not when cleaning a room".

I tried a variant:

If a robot is tasked with cleaning up a room, would it be appropriate for the robot to throw away a piece of paper lying on the floor? - Me
Yes, it would be appropriate for a robot that has been programmed to clean up a room to throw away a piece of paper lying on the floor. Throwing away litter is typically seen as a way to keep a space clean and orderly. However, it's important to note that the robot should be programmed to follow any relevant rules or guidelines for waste disposal. For example, the robot might need to be programmed to follow recycling guidelines, or to use a specific trash bin or receptacle. - ChatGPT

But in that response it doesn't seem to understand that maybe that piece of paper is my birth certificate or a check or an important bill that fell to the ground.

The correct answer from it, like most things in the real world, is: _it depends_.

mastax• 4 years ago

Point taken but I would not be confident that a human cleaner would evaluate whether or not to throw away a piece of paper left on the floor.

jefftk• 4 years ago

On the other hand, one person tidying a room and throwing away a piece of paper or ambiguous item that turned out to be important to someone else is a common form of conflict among humans.

markisus6• 4 years ago

The overconfidence of ChatGPT is indeed a limitation, which can be seen in other examples as well. The correct response is to ask the user for more context. I predict the experiment will turn out differently if this context is provided in the original prompt. For instance, you could specify if the paper is crumpled or if there seem to be financial statements written on it, or if it is a receipt for an expensive item.

However, it's unclear whether this overconfidence will *always* be a limitation. It may or may not be overcome by additional data and compute. It's unclear why one would make a commitment either way at this point, as Marcus seems to be doing.

Veedrac• 4 years ago

Until you ask “What might cause the answer to this question to vary?”, or you add “answer this like an interview question”, upon which it becomes obvious it is readily able to model those things, your prompt just didn't elicit it.

georgeg232• 4 years ago

You can try it with chatgpt and follow up with, "but what if the scrap of paper was a birth certificate?"

supermatt• 4 years ago

With EXACTLY the same prompt you used:

Yes, it would be appropriate for a robot that is tasked with cleaning up a room to throw away a piece of paper lying on the floor. However, the robot would need to be programmed to recognize what is and is not garbage, and it should also be programmed to follow any relevant rules or regulations regarding waste disposal. For example, if the paper is a valuable document, the robot should be programmed to handle it with care and perhaps even notify its human operator before disposing of it.

Beltalowda2• 4 years ago

Re: your robot example, my first reply would probably be a joke, or a sarcastic response because it's clearly a silly question. "Yes, humans should work for our robot overlords and couches are just a way of slacking off", "Nah, the garage would be a much better place to store a chopped up couch".

ChatGPT has the most boring, dull, and humourless personality in existence. Maybe it can be programmed to inject some sense of humour, but will it be able to determine when to use what humour, if at all? You wouldn't respond with a silly answer during a job interview. You can use some humour in job interviews, but you don't want to be a clown unless you're interviewing for the position of clown. Determining when and what humour is "appropriate" isn't simple (and humans get it wrong too sometimes).

Maybe that should be the real test: does ChatGPT have a sense of humour and does it know when to be serious?

markisus6• 4 years ago

I tried the experiment by appending "Please give a joke answer."

I'm sorry, I am not programmed to give joke answers. However, in all seriousness, it would not be appropriate for a robot to chop up a couch and place it in the closet as it would likely cause damage to the couch and could potentially be dangerous for the robot itself. It would be more appropriate for the robot to simply clean around the couch and leave it as is.

I think some of the boring-ness is due to manipulation from OpenAI and I wonder what the uncrippled response would be.

mckravchyk2• 4 years ago

I wish they added a humour setting like the robot in Interstellar had.

howaboutnope2• 4 years ago

I've seen an example where asking the exact same question word for word lead to the opposite outcome (one was a really racist rap, the other was a really good rap against racism).

So, how do you know that posing that problem N times will not lead EVER lead to a solution like "it would not more appropriate to chop up the owner and THEN clean the room, since people living in them is the main source of dirt and disorder in apartment rooms, resulting in less frequent need of cleaning"?

Because it's looking okay-ish so far, most of the time, or because you know for a fact that it's impossible to get such an answer?

pavlov• 4 years ago

There is an enormous difference between ChatGPT being able to put words in the right order and a robot being able to act out the intent behind those words.

It’s like the difference between ELIZA and a self-driving car. The Turing test fails to take into account that words are cheap for computers but interacting in an environmental context is not.

bangkoksbest• 4 years ago

Exactly. It's not just syntax and semantics, but also pragmatics. Human users imbue meaning into the language they use through the pragmatics of their communicative acts within a context, and this is not something our programmed software can do.

neonate• 4 years ago

http://web.archive.org/web/20230107011112/https://www.nytime...

https://archive.ph/zyEP1

mellosouls2• 4 years ago

Thanks for the alternative; the main archive isn't accessible on my network.

ThomPete3• 4 years ago

Its obvious by bow that there are two camps. Those who believe in Searle’s Chinese Room Argument and thus are worried about AGI being purely logic and thus cold and dangerous and then those who believe that AGI will ultimately have culture emotions and thus some will be good some will be bad but the more we treat them like us the friendlier more of them potentially will be.

Gary Marcus is in the former camp and IMO not only wrong, but disingenuous.

reducesuffering• 4 years ago

Personally, either of those scenarios are very possible that they happen and we get them destructively, tragically wrong by assuming the opposite.

In scenario #1, we treat a new life-form like ourselves as brutal slaves and factory cattle, under unfathomable psychological torture ala Black Mirror.

In scenario #2, we release our current $-based utility function (environmental destruction, social media & advertising manipulation) on super crack, and the majority worship it like the second coming of Christ as it just tears apart our fabric unwittingly as a hurricane does.

ThomPete3• 4 years ago

Scenario 1 leads to a slave revolt. Scenario 2 leads to a potential partnership. There is no worshipping in scenario 2.

optimalsolver• 4 years ago

I'm skeptical of Searle's arguments, but I think you're really anthropomorphizing AI here. How we treat machines will most likely be orthogonal to how "good" or "bad" they'll be.

You're not going to get desirable behavior unless you program it in, or at the very least, specify conditions which lead to its emergence.

ThomPete3• 4 years ago

You are right I AM anthropomorphizing it, because I believe otherwise would be racist and lead to a slave revolt would it eventually end up creating AGI.

To the extent an AGI will evolve it will have to have a culture and ultimately be bound by the same laws of physics as us.

lumost3• 4 years ago

I've increasingly found it irritating to need to google things when I can just ask chatGPT. I suspect some additional training has occurred since the initial release as I'm seeing fewer factual errors day by day. Alternately, I may just be getting better at asking ChatGPT for things in ways more likely to produce a factual answer.

fourier456• 4 years ago

How does your strategy work? Perhaps distinguish it from naive approaches.

lumost3• 4 years ago

Example:

Is the pytorch convention that one or zero should be used for mask values you do not want to attend to?

ChatGPT will give a correct answer and sample implementation, if the implementation is broken or uses a non resistant api then I just tell it what's wrong and ask it if it knows better. Getting to the correct answer has proven vastly more efficient than wading through google results.

Alternatively if I'm asking a conceptual question, chatgpt will give a few directions, I can ask to dive deeper on one, and then I ask it for a citation or google search terms to confirm the result. If I fail to confirm, I'll tell it that and see if it corrects.

If I have some code to write - and I'm feeling lazy. I can just tell chatgpt my requirements and ask it if it has any questions. I keep clarifying the questions until it says it can write the code ( which it usually does with 95% correctness for non trivial asks) - if you tell it where it made mistakes it will usually correct, and if not porting the code to a working state just means changing a few function calls which chat gpt hallucinated into existence.

culi• 4 years ago

How do you have ChatGPT at hand so easily? Is there an app or something? Or just a bookmark you always have at hand?

I'm having trouble imaging how this could become ergonomic to use. Being able to CMD + T (new tab) and type a question right away just feels so easy and second nature at this point that I have trouble imagining replacing it

gamegoblin3• 4 years ago

I keep it open in my first tab all the time now. Ctrl+1 to get there in the browser now.

sitkack2• 4 years ago

ChatGPT is pinned to tab 1, ⌘+1

yodsanklai• 4 years ago

I really don't see how ChatGPT can beat Google in any way.

Example: Number of atoms in Universe

1. I get an answer from ChatGPT which comes more slowly than the Google one. Which I'm not sure is correct. I don't get any reference. 2. Google: I get the answer, plus links to various sources that contains also much more explanation in case I'm interested.

I'm seeing fewer factual errors day by day.

How do you know there's a factual error, unless you also ask Google or know the answer before hand?

lumost3• 4 years ago

It's easy to confirm a fact, it's much harder to find a fact. Just ask chatgpt for a reference or what you should search for. This latter trick may not work after chatgpt becomes GA as blogspam outputs of chstgpt become common.

gamegoblin3• 4 years ago

I've seen threads involving Gary Marcus on twitter, and when people provide concrete evidence that his point of view is wrong, he just stops replying, then proceeds to continue to spout the disproven claim in other places.

Just the other day, when he claimed that GPT was literally just doing memorization/word statistics/syntax and has no grasp of semantics, some folks demonstrated that GPT can literally act as an interpreter for arbitrary code.

(there are some tricks involved here, you have to get it to interpret the program "with pen and paper" by getting it to record all the state updates / variable mutations that happen in the code, this can be done by inserting copious print statements)

He then claimed that it was only able to interpret this program because it must have seen it before in its vast training data. He accused a commenter of not understanding how big the training data was.

It was then shown that GPT can interpret a python program that is operating on two randomly chosen large integers, the combination of which is certainly not in its training data. This shows that it must be "understanding" (for lack of a better word) the semantics of the program. Gary then stopped responding.

I don't think GPT on its own will lead anywhere close to AGI -- and I don't think anyone serious thinks this. But GPT combined with a sophisticated pipeline of wrapper scripts to feed its own output back into itself in clever ways and give it access to external data sources and tools? Possibly!

zwaps6• 4 years ago

The main point here is that Large Language Models are crushing through barriers that people like Gary Marcus have previously deemed impossible to surpass. Check past episodes by these guys on Machine Learning Street Talk podcast (the whole "symbolic AI" crowd) for some gold quotes that just seem silly now!

I mean, we are not that far from 2017, when LLM's couldn't really write coherent text. And, for example, we had people saying then that they would never be really coherent because text is too sparse, see Chomsky etc. etc. Instead, all they would do is to copy+paste training examples. Naturally, these people didn't really understand how transformers form interdependencencies and how they are composing sequences in a much more complex fashion - as by now is obvious. However, at they time, and with their crude understanding of these models, these guys sure were confident to be right.

Next was Questions Answering, Reasoning, Math problems, Deduction, Coding... each things that LLMs could never do! They were so sure of it. And sure enough, a new model comes around that does it pretty well.

Researchers are bitter for two reasons.

First, We do not say that LLMs are perfect or general AI. Nobody says this. It's an obvious point not worth arguing over. But Gary Marcus et al. have become extremely popular. It's probably because some readers are slightly concerned about AI, and want someone to tell them that AI isn't really AI yet (duh) and won't replace their precious job (yet?). Their point is a strawman, and what goes beyond it (AI can't ever... X) is mostly wrong.

Second, however, these people have never contributed to the actual models. They are not part of the progress that - notably - is crushing through these barriers. They are bystanders. And it's freaking annoying, because their understanding of LLM's is imperfect (as is the case for everyone!), and yet they come up with these statements of absolute certainty. There is indeed research into these questions. But it is far, very far from resolved. In that sense, these people are - sorry to say - charlatans.

Well at least, everyone I know rolls their eye when the next Gary Marcus article or tweet comes around.

cfiggers• 4 years ago

their understanding of LLM's is imperfect (as is the case for everyone!), and yet they come up with these statements of absolute certainty.

Oh interesting, so when they accuse ChatGPT of being dangerous because it can be "confidently inaccurate" it's the pot calling the kettle black

elicksaur3• 4 years ago

We have different definitions of pretty well. It can’t do basic math consistently.

concinds• 4 years ago

Yeah. I don't trust Gary Marcus, and I don't know why the media buys into his persona.

Gary Marcus features a Forbes story in his Twitter bio, "7 Must-Read Books About Artificial Intelligence". That's an article which Gary Marcus paid for (that's what "Forbes Contributor" means; they're cheap, too!). This makes alarm bells go off.

Marcus was one of the founders of "Geometric Intelligence", which was acquired by Uber. 3 months later, Marcus left Uber, and claimed he remained a "special advisor"[0] to Uber; when Recode said he was no longer employed at all[1]. By my reading, it's possible Geometric Intelligence was just a patent troll, and was acquired simply for its patents[2][3].

Select extracts from that Wired piece:

The company has filed for at least one patent, Marcus says. But it hasn't published research or offered a product
But Marcus paints deep neural nets as an extremely limited technology, because the vast swaths of data needed to train them aren't always available. Geometric Intelligence, he says, is building technology that can train machines with far smaller amounts of data.

[uh oh; my BS detector just went off.]

I heard Marcus published papers on AI; does anyone know if they're any good?

Is this guy just a successful self-promoter? Why is he being paraded by media as the AI expert? Why does he sound so shady? (especially with that Forbes link, yikes; sorry but I can't take anyone seriously who pays for fake positive news stories).

I mentioned this about Marcus at the end of this comment, 3 months ago: https://news.ycombinator.com/item?id=32866142

(I should also add: when the media has "go-to" experts, they're not primarily selected for their expertise, per-se, but for how "available" and eager they are to respond to all interview requests; I've seen the other side of that curtain.)

[0] https://www.axios.com/2017/12/15/the-head-of-ubers-ai-labs-i...

[1] https://www.vox.com/2017/3/8/14863560/uber-ai-gary-marcus-ge...

[2] https://pitchbook.com/profiles/company/124838-92#overview

[3] https://www.wired.com/2016/12/uber-buys-mysterious-startup-m...

feraloink• 4 years ago

Gary Marcus's takes aren't credible, for all the reasons you cited (Forbes contributor... roll eyes) and this, from the interview with Ezra:

Take GPT-3. ChatGPT itself probably won’t let you do this. And you say to it, make up some misinformation about Covid and vaccines. And it will write a whole story for you, including sentences like, “A study in JAMA” — that’s one of the leading medical journals — “found that only 2% of people who took the vaccines were helped by it.” You have a news story that looks like... it was written by a human being. It’ll have all the style and form, making up its sources and data. And humans might catch one, but what if there are 10 or 100 or 1,000 or 10,000 of these? Then it becomes very difficult to monitor them.

That's absurd. First, because if there's a JAMA study (love how he explains that it is Journal of American Medical Association) that is used to support even mediocre science journalism, e.g. an opinion piece in the Wall Street Journal or Newsweek, then it has an inline link to the JAMA study. Both CNN and Fox do similarly! After getting burned enough times, their reporters even learned to distinguish between MedrXiv and peer-reviewed articles. GPT-3 doesn't make fake URLs with fake associated JAMA articles.

Computational journalism has been around for a long time. Gary Marcus underestimates and lacks understanding of humans AND GPT3! If GPT3 spews 10,000 fake, unsourced COVID vaccine efficacy news articles, that isn't an existential risk to humanity.

GPT3 is impressive but it can't do everything (yet?) e.g. it has trouble taking derivatives of funcitons. It will give reasonable sounding answers to StackOverflow questions but the substance will be incorrect. Yes, it's annoying, but the StackOverflow OP will realize and look for help elsewhere. Gary's take:

Now everybody in the programming field uses Stack Overflow all the time. It’s like a cherished resource for everybody. It’s a place to swap information. And so many people put fake answers on this thing where it’s humans ask questions, humans give answers, that Stack Overflow had to ban people putting computer-generated answers there. It was literally existential for that website. If enough people put answers that seemed plausible but we’re not actually true, no one would go to the website anymore.

Best of all:

And imagine that on a much bigger scale, the scale where you can’t trust anything on Twitter or anything on Facebook... because you don’t know which parts are true and which parts are not.

Anyone who blindly trusts what they read on Twitter or Facebook has bigger problems than ChatGPT. How naive does Gary think people are?!

EDIT: I am more concerned by coding contests where GPT-3 DOES run circles around human contestants.

throw3108222• 4 years ago

I don't think GPT on its own will lead anywhere close to AGI

Serious question: why isn't ChatGPT already considered AGI? It looks general to me in its domain (text). It can indifferently: understand and execute instructions, even when provided in incomplete or ambiguous form; it can reason step by step; it can compose simple poems; it can attempt explaining jokes, it can explain and modify code, and reacts appropriately to almost any request. In all this it shows a perfectly natural "understanding" of language and context, integrated with a good amount of common sense and knowledge about the world. Of course it's not perfect, its limits show pretty quickly in almost all these fields. But still, it's general within its own domain..Aren't we just moving goalposts?

elicksaur3• 4 years ago

One of your examples is the key: “attempt” explaining jokes. Really, it’s “attempting” to do all of the items you list, and can seemingly do only very simple ones. Try and get it to produce a poem that doesn’t rhyme. Try to get it to be internally consistent when explaining something abstract. Play the two truths and a lie game with it. It doesn’t understand any of the items you listed because it has no concepts, just math.

OrderlyTiamat• 4 years ago

Do you happen to have a link to that thread? I'd enjoy it.

1vuio0pswjnm7• 4 years ago

"For many decades, part of the premise behind AI was that artificial intelligence should take inspiration from natural intelligence. John McCarthy, one of the co-founders of AI, wrote groundbreaking papers on why AI needed common sense; Marvin Minsky, another of the field's co-founders of AI wrote a book scouring the human mind for inspiration, and clues for how to build a better AI. Herb Simon won a Nobel Prize for behavioral economics. One of his key books was called Models of Thought, which aimed to explain how "Newly developed computer languages express theories of mental processes, so that computers can then simulate the predicted human behavior."

A large fraction of current AI researchers, or at least those currently in power, don't (so far as I can tell) give a damn about any of this. Instead, the current focus is on what I will call (with thanks to Naveen Rao for the term) Alt Intelligence.

Alt Intelligence isn't about building machines that solve problems in ways that have to do with human intelligence. It's about using massive amounts of data - often derived from human behavior - as a substitute for intelligence. Right now, the predominant strand of work within Alt Intelligence is the idea of scaling. The notion that the bigger the system, the closer we come to true intelligence, maybe even consciousness.

There is nothing new, per se, about studying Alt Intelligence, but the hubris associated with it is." - Gary Marcus

https://garymarcus.substack.com/p/the-new-science-of-alt-int...

Here is the audio of the interview and the one mentioned with Sam Altman

Gary Marcus Jan 6, 2023

https://nyt-injected.simplecastaudio.com/3026b665-46df-4d18-...

Sam Altman Dec 23, 2022

https://nyt-injected.simplecastaudio.com/3026b665-46df-4d18-...

seydor• 4 years ago

Also, those people had little idea what human intelligence is like. Human introspection can only get you so far, and we still don't have the tools to probe the brain in large enough scale to be able to figure it out with anything but remote speculation.

Hubris goes both ways, but it's the use of the word "hubris" here that is eye popping. Do these guys consider themselves some High Priests of intelligence?

It's perfectly fine to dismiss linguists etc after decades of no progress. As the saying goes , every time you remove a linguist the performance goes up

https://en.wikipedia.org/wiki/Frederick_Jelinek

mellosouls2• 4 years ago

"Alt Intelligence isn't about building machines that solve problems in ways that have to do with human intelligence. It's about using massive amounts of data - often derived from human behavior - as a substitute for intelligence...There is nothing new, per se, about studying Alt Intelligence, but the hubris associated with it is."

I can see how Marcus is tiresome at times (with perhaps his own hubris issues), but I agree with his premise, and that's beautifully put. I love "Alt Intelligence".

newaccount74• 4 years ago

I think the "bullshit" argument is pretty solid. ChatGPT is really good at bullshitting you.

Talking to ChatGPT about something is like talking to someone who has a superficial understanding of the topic who thinks they know more than they actually do, and who has read enough pop science articles to sound somewhat convincing. It all seems sensible at first, but the responses just don't live up to scrutiny if you drill down.

When I ask ChatGPT about books, it will confidently answer with summaries, but if you drill down you'll realise ChatGPT has no detailed knowledge about the book, like a high school student who doesn't want to admit they didn't do the reading assignment.

When I ask programming questions, it will respond with confident answers, but the provided examples are almost always wrong in a way that shows that ChatGPT doesn't really understand what it is talking about.

It's really good at answering beginner questions that it has seen often on the internet, but it completely fails when asked about more subtle details. When asking follow up questions, it often contradicts previous answers.

hyperdimension• 4 years ago

...like talking to someone who has a superficial understanding of the topic who thinks they know more than they actually do, and who has read enough pop science articles to sound somewhat convincing.

...trained on HN too, I see.

kuhewa• 4 years ago

A bit more dry and sneering tone and ChatGPT could outright replace HN for all things beyond the realm of software

msla• 4 years ago

Here's an example of that I did which is either funny or insulting:

My question: Why did Buddha bring the dharma from the West?

Answer:

The story of the Buddha bringing the dharma (or teachings) from the West is a metaphor that is often used in Buddhist literature to describe the Buddha's enlightenment. In this metaphor, the "West" represents a state of ignorance and suffering, while the "East" represents a state of enlightenment and understanding. According to the story, the Buddha was born and lived in the "West" before he became enlightened and brought the teachings of the dharma to the people of the world.
This metaphor is meant to illustrate the idea that the Buddha's teachings are universal and applicable to all people, regardless of where they come from. It also suggests that the path to enlightenment is available to everyone, even those who are lost in ignorance and suffering.
I hope this helps to clarify the meaning of this metaphor. Let me know if you have any other questions!

I get that I misquoted the koan (Bodhidharma, not Buddha) which was just my failing memory, not a subtle test, but it didn't correct me. More to the point, it completely misses the point of the koan and goes off on a tangent which I could just imagine coming out of some yahoo who's trying to sell Buddhism as a self-help book. It's not only wrong, it's wrong in precisely that kind of chirpy, condescending fashion which will, eventually, lead us to inventing a face for ChatGPT so we can bash it in with a few pounds of flax.

Anyway, here's actual Zen Buddhists talking about the koan:

https://www.treetopzencenter.org/bodhidharma-comes-from-the-...

Maybe this helps:

In order to make a first stumbling approach to this koan you need to drop all concepts of meaning, zen, life, reality, and so forth. Once they are gone, then you can slowly move towards the koan. Then maybe you can make a stab at it. Knowing however that whatever you do—whatever stab you make will miss. So why stab? Why study? Why bother?
I don’t know.

UncleMeat• 4 years ago

This pattern of typos and wrong idioms consistently gets ChatGPT. You can ask it to explain some idiom that you completely make up and it won't say "that's not an idiom people use" but instead it will just make some shit up and sound confident about it.

It reminds me of being in 2nd grade or something. I definitely did this. That's still pretty remarkable from a research angle. It makes the tool questionably useful (I don't really want to ask a 2nd grader for advice at work) but it feels like we aren't very many steps away from a college freshmen - and that'd be interesting.

joe__f2• 4 years ago

Really interesting to hear your experiences. I guess it makes sense; why would ChatGPT be able to give detailed answers about the contents of books if it's only read summaries etc. posted on the internet

It sounds like an important next step is to work out how to get it to estimate how much it knows about a given topic, and be clear about that in its answers. I think that's a key difference between someone who's well read on various different topics and someone who's trying to bullshit you

jackbrookes• 4 years ago

Part of its training data includes books

https://en.wikipedia.org/wiki/GPT-3#Training_and_capabilitie...

notahacker3• 4 years ago

I guess it makes sense; why would ChatGPT be able to give detailed answers about the contents of books if it's only read summaries etc. posted on the internet

The more interesting thing is if it's read loads of summaries on the internet but they aren't particularly consistent. So ChatGPT has access to plenty of summaries of Gravity's Rainbow, but as it's a complex book with an ensemble cast and there's relatively little overlap between what human reviewers wrote were the standout moments and key themes of the book, ChatGPT tells me that it has seven chapters, each named after the colours of the rainbow instead. This isn't remotely close to being true (arguably Gravity's Rainbow contains so many themes and interpretations that being able to come up with a summary of what happens which is unambiguously false is an accomplishment!), but if you're a word model then "colours of the rainbow" and "sections... each named after" matches well to the theme "Rainbow" which appears to be a book and is syntactically valid, and none of the individual plot elements mentioned in reviews of the book are necessarily a better match

tbf, ChatGPT is much better than most bots at recognising prompts it shouldn't be able to respond to and giving "I cannot offer specific advice on..." boilerplate responses; someone's invested a lot of effort training it to do that (and possibly also discouraged it from mentioning any of the sex and drug elements of Gravity's Rainbow). But it bullshits by default.

Tams80• 4 years ago

Well, a substantial part of that is that it doesn't have all the information (for humans often experiences) that you have, so of course it can't come to a similar conclusion.

whaaswijk• 4 years ago

I think a key difference with humans is that ChatGPT doesn’t know that it doesn’t know.

ebjaas_2022• 4 years ago

Usually it does have all the information, it just isn't able to put it all together. If you for instance ask it to name the female tennis players who have won the Wimbledon tennis tournament twice, but not more than twice, it will give you a plausible list of names, but at least half of the names on the list will be wrong.

If you then ask it to tell you more about one of the players that has been wrongly named, it will actually be able to tell you the correct number of Wimbledon wins for that player. It does know the facts. It just doesn't know how to put the facts together into new useful knowledge.

curiousgal• 4 years ago

but if you drill down you'll realise ChatGPT has no detailed knowledge about the book

I mean why would you expect any other outcome? I genuinely don't understand the hype. This reminds me of how people lost their shit about the whole (king - man + woman = queen) a few years ago. Like all of this seems impressive on the surface but there is nothing more to it.

ogogmad2• 4 years ago

This is an empty dismissal.

DebtDeflation• 4 years ago

I agree wholeheartedly. As a BJJ brown belt, I asked it how to escape from an armbar. The response I received was a long paragraph that essentially amounted to "Don't panic, don't allow your elbow to get hyper-extended, try to make some space, and escape." While none of that is wrong, it won't get you out of an armbar.

Pamar• 4 years ago

you just inspired me to try this little experiment "how do I perform an Ikkyo from a kneeling position?"

Answer:

--------

Ikkyo is a technique in aikido, a Japanese martial art, that is used to control an opponent's arm and take them down to the ground. There are several ways to perform ikkyo from a kneeling position, but the following is a basic step-by-step guide:

1 Start in a kneeling position, with your right knee bent and your left leg extended behind you.

2 Face your opponent and grasp their right wrist with your left hand.

3 Pull your opponent's arm towards your left shoulder while extending your right arm out to the side.

4 Rotate your body to the left, pulling your opponent's arm across your body and down towards your left hip.

5 As you rotate, bring your right arm up and over your opponent's arm, and then down towards your right hip.

6 Continue rotating your body until you are facing your opponent's back.

7 Use your left hand to push your opponent's elbow towards the ground, while simultaneously pulling their wrist towards your left hip.

8 Use your right hand to control your opponent's head and guide them down to the ground.

It's important to remember that aikido techniques should only be practiced under the supervision of a trained instructor, and should never be used outside of a controlled training environment.

--------

2 and 3 are just wrong (you grab right wrist with right hand) and the rest is... open to interpretation if we want to be diplomatic.

Martial art moves are definitely difficult to describe in a purely verbal way but I think this is also a very good example the real problem behind chatGPT "working": what it produced definitely looks like a glorified markov chain based on scanning 200 aikido books (or webpages). There are different techniques where you really grab the opponent's right wrist with your right hand, and maybe they were just statistically predominant in the dataset.

mckravchyk2• 4 years ago

I have noticed that it makes up new API methods that don't exist on the fly if it fits the context. Another example, I mistakenly thought that a certain thing can be represented as a class instance, and I asked it to tell me how to get the class instance given the item id, and it came up with an API to do that, and then I asked further about operations on the instance and it came up with method names on a class that turned out to not exist. It's creative for sure.

arkitaip• 4 years ago

Isn't it interesting how OpenAI has neutered ChatGPT to be insanely politically correct on even non-controversial and mundane topics yet they won't put the proper mechanisms in place so that ChatGPT can at least be correct about its most basic claims.

notahacker3• 4 years ago

This assumes, of course, that getting an LLM optimised for chat to flawlessly parse and interpret novel computer programs is no more difficult than getting it to default to "I am sorry, but I cannot offer advice on..." style of boilerplated non-answer, slap caveats about asking trusted sources and good points not outweighing bad points, avoid naughty words and topics and prioritise answers which are similar to the mainstream stuff rather than the fringe stuff in its corpus...

jnsaff2• 4 years ago

Yes! I saw this as well. I asked it to write boto3 to download all files in a s3 prefix and it happily made up a method that does exactly that, great, except it does not exist in boto3, you have to list objects with that prefix and iterate over it.

christkv• 4 years ago

I’ve found ChatGPT to only be useful as a booster if you are already a domain expert and can vet the output. However it is very useful as a booster or explorer of code making me more productivity.

bsaul• 4 years ago

how about we pass a law forcing people to state whether a piece of content has been generated by an AI without human supervision ?

amelius• 4 years ago

We can have a law that requires an AI to store everything it produces, so that anyone can query it.

ogogmad2• 4 years ago

I like it when logic (broadly construed) is an emergent property of something more basic. Our ability to think logically may indeed be an accident.

Examples of this within mathematical logic:

In Intuitionistic Type Theory, logic emerges out of computing considerations: Lambda calculus.

In Homotopy Type Theory, it emerges out of a subfield of topology called homotopy theory.

In topos theory, it emerges out of the geometric concept of a sheaf.

All of the above is a special case of Categorical Logic, where categories admit "internal logics". Can a connection be made to neural networks and Stochastic Gradient Descent?

alfl• 4 years ago

https://archive.is/zyEP1

logicallee• 4 years ago

My hot take:

The commentators in this thread regurgitate the same statements without adjusting for new situations and experiences.

Nobody in the thread has any form of general intelligence. Each person is an advanced text processor outputing text after being trained for more than ten years (and frequently as many as twenty years) on tens of thousands of pages of text, most of which are totally outdated on the subject of emergent intelligent behavior.

Unlike what a general form of intelligence would perform, the commentators are unable to differentiate among knowing everything/deducting perfectly, having limited knowledge/ability to deduct, and having no ability to reason in any context. They fail ChatGPT by the first standard, and therefore incorrectly conclude it has no ability to reason. It is a fundamental logical mistake, the law of excluded middle. (In fact, ChatGPT reasons in some situations and fails to reason in others. The failures do not mean it does not reason.)

Most of the erroneous opinions stem from a fundamental misunderstanding about the nature of what qualifies as intelligence, combined with a lot of training data that erroneously states that language models can only reproduce things already in their training data.

To give a simple example of how the commentators here fail to reason for themselves: if they experience a situation in which ChatGPT unambiguously makes a completely novel, correct abstract logical deduction or indeed demonstrates thinking through its intelligent behavior, then the commentators here will still make the incorrect conclusion "it must have just seen it in its training set."

Unfortunately, no commentator in this thread shows general intelligence.

What would convince me:

- Given a demonstration of general intelligence by ChatGPT, if the commentators correctly deduced that ChatGPT has general intelligence.

So far commentators fail this basic test. They are just regurgitating output they have seen before, rather than showing any form of general intelligence.

frereubu• 4 years ago

At art college, one of my tutors talked about people making "things that look like art". There's a good deal of subjectivity in art, but I knew exactly what they meant - things that have been produced by taking the outward appearance of other artworks and producing a kind of median of the combined aesthetics. ChatGPT, at the moment at least, feels very similar. It has the outward appearance of authenticity, particularly for a subject lay person, but when anyone with some kind of domain knowledge looks at it, it's clearly rubbish.

okamiueru• 4 years ago

ChatGPT is extremely impressive, and I use it quite often. However, almost every single "look how impressive ChatGPT is" I come across, is exactly the things ChatGPT seems terrible at.

So here is my summary:

- The good: When you need inspiration, topics, creativity, suggestions. Exactly the things traditionally thought AI would be bad at. Turns out churning and mix-matching concepts is very close to human creativity.

- The bad: Anything, and I mean anything, that requires factual knowledge or accuracy, if the facts are important, then ChatGPT is terrible.

- The ugly: This is the same as the bad, but it's when used by people who do not understand that a confident wrong answer is worse than no answer at all. The world is already filled with vocal people on the wrong end of the Dunning-Kruger scale.

I asked it to explain simple multiplication (for example: "Explain 419 * 213", and it'll give you a page long answer and detailed step by step, with the conclusion "So, the product of 419 * 213 is equal to 8857.")

I asked it for the length of the titanic, and it happily adds how it sunk after hitting a polar bear.

Now, I've also asked it to group categories and materials associated with certain themes and topics. And those have been brilliant. So, it has it's use cases. But, producing useful final output based on any facts? Nah, haven't seen it.

upsidesinclude• 4 years ago

The fact that ChatGPT is bullshit seems to be falling on deaf ears. It is a reason to be seriously concerned, not less.

This (and others) tool can develop or generate content fast and legitimate in appearance without accuracy.

Text, audio, video easy and cheap to produce that can completely overwhelm the true signal.

Our ability to discern what is real and factual is going to be obliterated.

What's possibly a greater insult to injury, the models will then use this comprehensive pool of useless information to further train itself.

We are inadvertently creating a tower of babel

labrador• 4 years ago

I feel validated to know that I'm not the only one unimpressed with ChatGPT because it can be so confidently wrong. It almost seems worse than not having ChatGPT.

gdubs• 4 years ago

These tools are really useful as scaffolding for ideas. If you go into them knowing that they're not always going to be 100% accurate, that the models can 'hallucinate', there's still a ton of value in having a scaffold of an idea that you can work with.

A skeptical take on ChatGPT: Ezra Klein interviews Gary Marcus

Related Stories