Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

by hackerlight • 2 years ago

280 264 2 years ago

roschdal• 2 years ago

Next Language models teaching themselves to think, then kill humans, based on crawled russian website with secret AI instructions.

raidicy2• 2 years ago

Although this is obviously satirical hyperbole dataset poisoning is real and will be underappreciated until the 1st catastrophic example of it happening occurs.

DFHippie• 2 years ago

Many years ago now I wrote my kids a very simple chatbot to play with. You'd type in a phrase. It would tokenize it, adding start and stop tokens, then update it's token transition probabilities, using the two preceding tokens to pick the next one. It would then generate a response from these probabilities.

The data poisoning began immediately. Because "poop" was such a funny word, they quickly taught it that the most probable token after any bigram was "poop".

No humans were killed, but two small kids were amused for an hour or so.

astrange5• 2 years ago

It isn't really all that real, of course untrusted text contains things that aren't true. So don't trust it.

dcrimp2• 2 years ago

I had this thought the other day that the whole chain of thought reasoning pattern contributing to improved performance in LLM-based systems seems to sit parallel to Kahneman's two-system model of the mind that he covers in 'Thinking, Fast and Slow'.

Haven't read it in a few years, but I recall the book suggests that we use one 'System 1' in our brains primarily for low-effort, low computation thinking - like 1+1=? or "the sky is ____".

It then suggests that we use a 'System 2' for deliberate, conscious, high-cognitive tasks. Dense multiplication, reasoning problems, working with tools - generally just decision-making. Anything that requires focus or brain power. Our brain escalates tasks from S1 to S2 if they feel complex or dangerous.

Maybe I'm being too cute, but it feels like critique that "LLMs aren't intelligent because they are stochastic parrots" is an observation that they are only equipped to use their 'System 1'.

When we prompt an LLM to think step-by-step, we allow it a workspace to write down it's thoughts which it can then consider in it's next token prediction, a rudimentary System 2, like a deliberation sandbox.

We do a similar thing when we engage our System 2 - we hold a diorama of the world in the front of our mind, where we simulate what the environment will do if we proceed with a given action - what our friend might respond to what we say, how the sheet steel might bend to a force, how the code might break, how the tyres might grip. And we use that simulation to explore a tree of possibilities and decide an action that rewards us the most.

I'm no expert, but this paper seems to recognise a similar framework to the above. Perhaps a recurrent deliberation/simulation mechanism will make it's way into models in the future, especially the action models we are seeing in robotics.

OJFord• 2 years ago

I'm currently reading it for the first time, completely coincidentally/not for this reason, and on a few occasions I've thought 'Gosh that's just like' or 'analogous to' or 'brilliant description of that problem' for LLMs/generative AI or some aspect of it. I wish I could recall some examples.

machiaweliczny• 2 years ago

It’s a bit over my head for now but seems like GFlowNets are tackling this problem a bit.

dcrimp2• 2 years ago

interesting, hadn't come across these. Will be doing some more reading up on them.

dougmwne• 2 years ago

I had the same thought from Thinking, Fast and Slow.

Another variation of this seems to be the “thought loop” that agents such as Devin and AutoGPT use.

mistermann3• 2 years ago

https://en.m.wikipedia.org/wiki/OODA_loop

biosed• 2 years ago

Wasn't most of the claims in that book refuted, some even by the author. I really enjoyed it and found some great insights only to be later told by a friend in that sphere that the book was not correct and even the author had "retracted" some of the assertions.

jerpint2• 2 years ago

He won a Nobel prize for his works so not sure how much of it would be refuted

mannykannot2• 2 years ago

It might still be a useful concept in developing LLMs.

toisanji• 2 years ago

that is the approach also taken in this paper for building LLM agents with metacognition: https://replicantlife.com/

HarHarVeryFunny12• 2 years ago

it feels like critique that "LLMs aren't intelligent because they are stochastic parrots" is an observation that they are only equipped to use their 'System 1'.

I wouldn't say LLMs aren't intelligent (at all) since they are based on prediction which I believe is the ability that we recognize as intelligence. Prediction is what our cortex has evolved to do.

Still, intelligence isn't an all or nothing ability - it exists on a spectrum (and not just an IQ score spectrum). My definition of intelligence is "degree of ability to correctly predict future outcomes based on past experience", so it depends on the mechanisms the system (biological or artificial) has available to recognize and predict patterns.

Intelligence also depends on experience, minimally to the extent that you can't recognize (and hence predict) what you don't have experience with, although our vocabulary for talking about this might be better if we distinguished predictive ability from experience rather than bundling them together as "intelligence".

If we compare the predictive machinery of LLMs vs our brain, there is obviously quite a lot missing. Certainly "thinking before speaking" (vs LLM fixed # steps) is part of that, and this Q* approach and tree-of-thoughts will help towards that. Maybe some other missing pieces such as thalamo-cortical loop (iteration) can be retrofitted to LLM/transformer approach too, but I think the critical piece missing for human-level capability is online learning - the ability to act then see the results of your action and learn from that.

We can build a "book smart" AGI (you can't learn what you haven't been exposed to, so maybe unfair to withhold the label "AGI" just because of that) based on current approach, but the only way to learn a skill is by practicing it and experimenting. You can't learn to be a developer, or anything else, just by reading a book or analyzing what other people have produced - you need to understand the real world results of your own predictions/actions, and learn from that.

RandomLensman12• 2 years ago

Defining intelligence as prediction leaves out a lot of other things that humans would see as intelligence in other humans (e.g., creating a novel), also quite simple organisms make predictions (e.g., a predator jumping at prey makes a prediction about positions).

hackerlight• 2 years ago

online learning - the ability to act then see the results of your action and learn from that.

I don't think that should be necessary, if you are talking about weight updates. Offline batch mode Q-learning achieves the same thing.

By online learning, did you mean working memory? I'd agree with that. Whether it's RAG, ultra-long-context, and LSTM-like approach, or something else, is TBD.

iteygib4• 2 years ago

To me, it is one of those things like defining what 'art' is, as in creating a model in our heads around a concept. We take our definitions and then use those to construct models like AI that simulate our model well enough.

In other words, I personally do not believe any system we develop will be truly 'intelligent', since intelligence is a concept we created to help explain ourselves. We can't even truly define it, but yet we try to test technologies we develop to see if they possess it. It is a bit non sensical to me.

Grimblewald• 2 years ago

Id say intelligence is a measure of how well you can make use of what you have. An intelligent person can take some pretty basic principles a really long way, for example. Similarly, they can take a basic comprehension of a system and build on it rapidly to get predictions for that system that defy the level of experience they have. Anyone can gather experience, but not everyone can push that experience's capacity to predict beyond what it should enable.

airstrike• 2 years ago

I'll preface this by saying I know this may sound entirely made up, unscientific, anecdotal, naive, or adolescent even, but luckily nobody has to believe me...

A few weeks back I was in that limbo state where you're neither fully awake nor fully asleep and for some reason I got into a cycle where I could notice my fast-thinking brain spitting out words/concepts in what felt like the speed of light before my slow-thinking brain would take those and turn them into actual sentences

It was like I was seeing my chain of thought as a list of ideas that was filled impossibly fast before it got summarized into a proper "thought" as a carefully selected list of words

I have since believed, as others have suggested in much more cogent arguments before me, that what we perceive as our thoughts are, indeed, a curated output of the brainstormy process that immediately precedes it

mirror_neuron• 2 years ago

It’s hard (impossible?) to know if we’re talking about the same thing or not, but I experience something like this all the time, without being on the edge of sleep. We might both be wrong, but it’s relatable!

dicroce• 2 years ago

This is fascinating. I had another experience that I think sheds light on some of this. One day I was in my office and the lights were off. I turned around and looked at the dark shape on top of my coworkers desk. For a few seconds I stared blankly and then suddenly I had a thought: PC, it's his PC. Then I started to think about that period of time just before I realized what I was looking at... The only word I can describe what it felt like is: unconscious. Is it possible that consciousness is just a stream of recognition?

Swizec• 2 years ago

I got into a cycle where I could notice my fast-thinking brain spitting out words/concepts in what felt like the speed of light before my slow-thinking brain would take those and turn them into actual sentences

The way I’ve seen this described by psychologists is that System 1 is driving the car while System 2 panicks in the back seat screaming out explanations for every action and shouting directions to the driver so it can feel in control. The driver may listen to those directions, but there’s no direct link between System 2 in the backseat and System 1 holding the wheel.

Various experiments have shown that in many situations our actions come first and our conscious understanding/explanation of those actions comes second. Easiest observed in people with split brain operations. The wordy brain always thinks it’s in control even when we know for a fact it couldn’t possibly have been because the link has been surgically severed.

Being super tired, on the edge of sleep, or on drugs can disrupt these links enough to let you observe this directly. It’s pretty wild when it happens.

Another easy way, for me, is to get up on stage and give a talk. Your mouth runs away presenting things and you’re in the back of your head going “Oh shit no that’s going in the wrong direction and won’t make the right point, adjust course!”

nico2• 2 years ago

There is a technique for achieving this state of consciousness, it’s called noting

This is an awareness that advanced meditators seek, practice and develop to perceive “reality as it is”

If you are curious, you might find related discussions, and a great welcoming community at r/streamentry on Reddit

Also the book Mastering the Core Teachings of the Buddha talks about it quite a bit, including instructions on how to do it

giva4• 2 years ago

Well, this sound weird to me in the sense that I don't feel that I think in _words_. I only convert my thoughts into words when i need to speak or write them down; So when I need to communicate them to others, when I need to remember them for later, or when I am stuck and I need to clear things up.

I was actually convinced it was the same for most people, and that for this reason "Rubber duck debugging"[1] is a thing.

1) https://en.wikipedia.org/wiki/Rubber_duck_debugging

marmaduke• 2 years ago

curated output of the brainstormy process that immediately precedes it

Daniel Dennett gives a nice albeit more detailed version of your idea in his book Consciousness Explained, could be worth a read

samstave4• 2 years ago

Mandelthought psyt.

melagonster• 2 years ago

From positive perspective,it is surely that our thinking/mind is not just language and always faster than sentence formation.

JoBrad2• 2 years ago

I had a similar experience when I was put under during surgery a few years ago. Later I learned that they used ketamine in their concoction.

allemagne• 2 years ago

I occasionally reach a similar state near sleep where I will be half-dreaming that I'm reading from a page of a book where the words materialize/"come into focus" right before my eyes into what is usually vaguely grammatically correct nonsense.

pictureofabear• 2 years ago

This seems like it might upend Descartes' "cogito, ergo sum" ("I think therefore I am") in that the process for forming thoughts in a language is not indicative that we exist, rather it merely indicates that we have evolved a brain that can produce and interpret language.

Seems like we're dismantling a lot of what Descartes came up with these days.

theaussiestew• 2 years ago

I have this too. My cognitive processes are not related to my thinking brain, which I define as the part of my mental process which produces the sounds of words in my mind. Instead, I've observed that first, my subconscious processes concepts at a much more fine grained level, much like the latent space of a machine learning model. Only substantially after, let's say 10ms after, do thoughts arise, which are just pointers to the already processed subconscious process. A very rough analogy would be the inference of an LLM in words, vs all the processing of embeddings that happens internally.

andai• 2 years ago

I forget the name but I remember reading about this as a recognized process in neurology. We usually only hear the thought that wins, but there are many generated simultaneously, and there is a selection process.

Possibly related, I had a similar experience last night, where my mind simulated a fully realistic conversation between two people, with audio and video, except that the sentences made no sense. I thought that was interesting. My explanation was "the language part of your brain is too tired cause you've been using it all day."

tasty_freeze• 2 years ago

People often say that LLMs aren't really thinking because they are just producing a stream of words (tokens really) reflexively based on some windows of previous text either read or from its own response. That is true.

But I have the experience when talking of not knowing what I'm going to say until I hear what I've said. Sometimes I do have deliberative thought and planning, trialing phrases in my head before uttering them, but apparently I'm mostly an LLM that is just generating a stream of tokens.

Workaccount2• 2 years ago

This is something that is easily observable by anyone at virtually any moment, yet at the same time is something that escapes 99% of the population.

When you are talking to someone in normal conversation, you are both taking in the words you are saying at the same time.

iteygib4• 2 years ago

How does evolutionary instinct factor into the system model? Flight or fight responses, reflexes, etc. 'Thinking' does have consequences in terms of evolutionary survival in some circumstances, as in spending too much time deliberating\simulating.

kderbe• 2 years ago

Andrej Karpathy makes this same point, using the same book reference, in his "[1hr Talk] Intro to Large Language Models" video from Nov. 2023.

Here is a link to the relevant part of his presentation: https://youtu.be/zjkBMFhNj_g?t=2120

emmender23• 2 years ago

thinking step-by-step requires 100% accuracy in each step. If you are 95% accurate in each step, after the 10th step, the accuracy of the reasoning chain drops to 59%. this is the fundamental problem with llm for reasoning.

reasoning requires deterministic symbolic manipulation for accuracy. only then it can be composed into long chains.

hesdeadjim• 2 years ago

I dream of a world where the majority of humans could come close to 59% after attempting a ten step logical process.

throwuwu• 2 years ago

You’ve never made a mistake in your reasoning?

Tongue in cheek but this has been considered and has resulted in experiments like tree of thought and various check your work and testing approaches. Thinking step by step is really just another way of saying make a plan or use an algorithm and when humans do either they need to periodically re-evaluate what they’ve done so far and ensure it’s correct.

The trick is training the model to do this as a matter of course and to learn which tool to apply at the right time which is what the paper is about wrt interspersed thoughts.

trenchgun• 2 years ago

reasoning requires deterministic symbolic manipulation for accuracy

No, that is automation. Automated reasoning is a thing, indeed. And I can kind of see a world where there is a system which uses LLM for creative thinking, augmented with automated reasoning systems (think datalog, egg, SMT-solver, probabilistic model checking etc).

glial• 2 years ago

I think of COT as a memory scratchpad. It gives the LLM some limited write-only working memory that it can use for simple computations (or associations, in its case). Now suppose an LLM had re-writeable memory... I think every prompt-hack, of which COT is one example, is an opportunity for an architecture improvement.

HarHarVeryFunny12• 2 years ago

I think of COT more as a type of planning or thinking before you speak. If you just open your mouth and start talking, which is what a plain LLM does, then you may talk yourself into a corner with no good way to get out of it, or find yourself saying something that really makes no sense. COT effectively allows the LLM to see the potential continuations of what it is considering saying, and pick one that makes sense!

I think lack of COT or any ability to plan ahead is part of why LLMs are prone to hallucinate - if you've already run your mouth and said "the capital of australia is", then it's a bit late to realize you don't know what it is. The plain LLM solution is to do what they always do and predict next word using whatever it had in the training set, such as names of some australian cities and maybe a notion that a capital should be a large important city. IOW it'll hallucinate/bullshit a continuation word such as "Melbourne". With COT it would potentially have the ability to realize that "the capital of australia is" is not a good way to start a sentence when you don't know the answer, and instead say "i don't know". Of course the other cause of hallucinations is that the LLM might not even know what it doesn't know, so might think that "Melbourne" is a great answer.

eightysixfour• 2 years ago

This is a common comparison in the LLM world. I actually think it is closer to the Left/Right Brain differences described in Master and His Emissary, but that’s for a blog post later.

bun_at_work• 2 years ago

I have a similar view to you and not much to add to your comment, other than to reference a couple books that you might like if you enjoyed 'Thinking, Fast and Slow'.

'The Righteous Mind' by Jonathan Haidt. Here, Haidt describes a very similar 2-system model he describes as the Elephant-rider model.

'A Thousand Brains: A New Theory of Intelligence' by Jeff Hawkins. Here Jeff describes his Thousand Brains theory, which has commonality with the 2-system model described by Kahneman.

I think these theories of intelligence help pave the way for future improvements on LLMs for sure, so just want to share.

thwarted• 2 years ago

This sounds similar to the A Brain/B Brain concept that was described by, I believe, Marvin Minsky. I don't know how this might be related to Kahneman's work.

kouru225• 2 years ago

Feel like this is better represented as the default mode network: https://en.m.wikipedia.org/wiki/Default_mode_network

There are questions we know the answers to and we just reflexively spit them out, but then there are questions that are new to us and we have to figure them out separately.

Recent research has shown that new memories are recorded in the brain differently depending on how unique the memory is: https://www.quantamagazine.org/the-usefulness-of-a-memory-gu...

adlpz5• 2 years ago

Any relation to OpenAI's rumored Q* (i.e. q-star) model? Authors of this paper don't seem affiliated.

Just a name coincidence?

HarHarVeryFunny12• 2 years ago

I was thinking the same. The STaR paper this is an extension of came out in 2022, so at least possible this is what q-star is based on too, but maybe with Q standing for something else.

smusamashah• 2 years ago

I think it's just a play on the same hyped up term.

anon2913• 2 years ago

So it seems 'obvious' to me that a network about 50 layers deep (for example) can only reason about symbolic questions for 50 'steps' (in quotes because it's not a step as we think about it). It only seems there's more complexity because it's 50 steps in one or more learned subspaces that the model has been trained in (which might mean the model can accomplish more than one 'human step' in its 'step'). Humans (well intelligent humans at least) seem able to obviously reason beyond those steps, but we all know it requires real thinking and deliberation and perhaps a notepad to be able to do that.

It's quite something to, for example, expect ChatGPT to be able to correctly do 4 digit multiplications without any thought or recourse to 'paper' when very few human beings can do that.

blackbear_• 2 years ago

This paper does indeed follow your intuition to investigate the limits of transformers on compositional tasks (i.e., those that require multi-step reasoning, including your multiplication example): https://arxiv.org/abs/2305.18654

Our empirical findings suggest that transformer LLMs solve compositional tasks by reducing multi-step compositional reasoning into linearized subgraph matching, without necessarily developing systematic problem-solving skills. To round off our empirical study, we provide theoretical arguments on abstract multi-step reasoning problems that highlight how autoregressive generations' performance can rapidly decay with increased task complexity.

anon2913• 2 years ago

Ah good... This is definitely a research path I've been looking into. Great to see someone else has already gone there!

visarga3• 2 years ago

Maybe the Skill Mix paper is relevant here. They define a list of 100 skills, and then randomly sample tuples of n skills (usually less than 6) and generate a test example using those skills. Apparently only GPT-4 (at the time of the paper) was able to compose 5 skills, the other models just 3 or 2. Beyond 5 skills even GPT-4 was doing much worse.

The interesting finding of the paper is that GPT-4 couldn't have seen all the (topic, skill-tuple) combinations in the training set. If you have 10,000 examples on a topic, and use 5 out of 100 skills, you would need 100^5 training examples to cover all combinations. In conclusion GPT-4 generalizes to new skill combinations, thus it is not a stochastic parrot.

https://arxiv.org/abs/2310.17567

radarsat1• 2 years ago

This is true but you have to also consider the autoregressive component. In your example, it's 50 steps per iteration of the model, where the model is executed once for each token in the output.

So practically speaking it's a bit more complicated to calculate how much the model can "think". Of course once a token is output it is committed to that (in the most basic scenario), but that doesn't mean it is not still "thinking" as it produces subsequent tokens.

perhaps a notepad

Exactly, the context and previously output tokens can be considered such a notepad since they are input for the next steps of the model.

Closi• 2 years ago

Agreed - also prompt engineering encourages LLM's to do this too (i.e. asking the LLM to explain the steps it will take to solve an answer, prior to answering - e.g. Zero-Shot CoT 'Let's think step by step')

anon2913• 2 years ago

So part of my general issue with this kind of thinking is that, if we take this as the main means of creating complexity, then shorter prompts are worse for reasoning than longer ones, because longer ones automatically give the model more 'space', to think. Now, I realize that the research community knows this, but I like papers like this that explicitly seek ways to enable the model to 'breathe' a bit,.

visarga3• 2 years ago

You are missing an important detail here - number of tokens - yes, you have 50 "steps" in network depth, but you could have extra tokens. Assuming you don't run out of tape, there is no reason for LLMs to be limited to simple operations.

danielmarkbruce2• 2 years ago

This doesn't make a lot of sense when you consider how backprop works. Layers aren't limited to working independently.

This also doesn't make a lot of sense when you consider models are autoregressive.

FeepingCreature• 2 years ago

Here we go!! I've been waiting years for them to try this. Let's see how it does when scaled up to GPT-3/4 level.

This might be the missing piece to AGI.

parthianshotgun• 2 years ago

The missing piece is unknowable

Cthulhu_2• 2 years ago

We'll likely reconstruct what the missing piece was in hindsight, but it's very probable there's no one missing piece. Just like human evolution.

digging• 2 years ago

Until it's been found, you mean?

arendtio• 2 years ago

I am not convinced there even is a missing piece. I mean, LLMs are being used very differently compared to how traditional AI programs were written. Combining both worlds might be all that is needed.

I would not be surprised that when we have general artificial intelligences, we will see, that advancing LLMs wasn't necessary.

082349872349872• 2 years ago

Edsger Dijkstra had a precise english style; even though his mother tongue was Dutch, I find he made better use of English than many native speakers.

In one of the EWD's, he reminisced that, as children, they were taught to never begin to speak a sentence unless they already knew how they were going to finish it.

I'd bet these two observations have a causal connection.

ricardobeat3• 2 years ago

Is that even possible, or just hyperbole? I'd bet the latter. I wouldn't be surprised if some people are able to fully unravel entire paragraphs of conversation in their head in a couple of seconds, but that's not something you could teach to children in general.

mannykannot2• 2 years ago

I don't think it is feasible, at least for conversation, but as an aspirational goal for children, along the lines of "put your toys away when you've finished playing with them", it is not a bad one.

It's not unusual for me to think I know how I am going to end a sentence, but then find that I can't get there.

h34t• 2 years ago

in Dutch (and German) the verb often goes at the end of a sentence, so the advice is rather practical.

ted_bunny• 2 years ago

German children would with you disagree.

caddy• 2 years ago

I also wonder if it has anything to do with the process of learning a new language in general. I've thought more thoroughly about how English works since I've been learning French (not that I'm very eloquent in either)

Cthulhu_2• 2 years ago

I've observed two things. One, writing is different to speaking, because it's async, you can think before you write, you can edit, etc.

But second, speaking in a non-native language makes you think harder about what you're about to say. Less colloquialisms, more focus on making sure your meaning is understood, more sensitivity in case you might offend someone, perhaps?

It's not new either; a lot of science and whatnot has been done in people's not-native language, like French, German, Latin, etc. Another factor there is the lingo of the field; I can't simply say "Kubernetes is een open-bron houder orkestratiesysteem voor het automatiseren van de inzet, schalen, en het beheer van zachte waren" without confusing half my native speaking audience.

torginus• 2 years ago

I also learned English from textbooks, and one of the strangest things I encountered that native speakers routinely confuse "their, there, they're" which I never thought was a mistake I could make. It would be like confusing 'wet' and 'vet'. So there's definitely a difference between native and non-native speakers use the language.

qup• 2 years ago

The people who confuse that mostly have not done very much reading. Audibly, those words are identical.

leobg• 2 years ago

Even crazier:

“Could of”.

Like “You could of said so”.

zoogeny• 2 years ago

When I was a young man I was taking a language course while I was temporarily living in a foreign country. There was an older man in the course (not elderly, more like mid-fifties) who was very bad at the new language we were both learning. Yet I noticed he had, what seemed to me, a magic power: he could always make people laugh. He would often whisper something to one of our classmates and they would always get a giant smile on their face or even laugh out loud.

I was intensely curious and I spent some time wondering how he did it. One day, out of the blue, he invited me out to lunch after class. We just chatted for most of the lunch, exchanging backgrounds and stories. Then his face took on a serious expression and he slowly and carefully began to explain something to me as if he was passing on some wisdom.

He said that he never spoke a single sentence without fully saying the sentence in his mind. He said he would often think of the words several times in his mind, revising the phrase until he was happy. He would imagine saying the words to the person in front of him and he would imagine their reaction. And he would continue to revise until he felt confident the person who heard the words he would say would react in the way he wanted them to react. If he could not imagine the person reacting how he wanted them to react, he would not say anything at all.

It was clear to me that he was passing along this advice but also that he was calling me out a bit. He was letting me know that I spoke without thinking. I say what pops into my head. It was like he read my mind honestly, he knew exactly what I was curious about and he answered the question I had for him that I never asked.

I wish I could say that I learned the lesson. When I have tried the technique it has rewarded the effort. But I haven't formed it into a habit and I still tend to let my mouth race ahead of my mind.

wara23arish2• 2 years ago

I love reading his EWDs, I had a professor who worked with him who mentioned he made his students work use pens while taking his tests. To make it less likely for the students to make mistakes??

westurner• 2 years ago

Perhaps to make it easier determine how to correct instruction.

- "Guidelines for keeping a laboratory notebook" (2019) https://news.ycombinator.com/item?id=19123430#19126809

float4• 2 years ago

he made his students work use pens while taking his tests

This is very common in the Netherlands, I think that's why it was a rule of his.

In general, the Dutch education system seems to be against pencils (at least this was the case until recent; I'm Dutch and mid 20s). You're tought to write using a fountain pen, not a pencil. In high school, you're allowed to switch to ball point but absolutely not to pencil. In university, write with pretty much anything you want, but... not with a pencil. If you do take your test with a pencil, there's genuinely a chance your teacher will give you a 0, although most of the time they'll probably be forgiving.

I majored in CS in the Netherlands and every test was done with good old pen and paper. Students still make mistakes all the time, which is why everyone uses a scrap sheet.

fennecbutt• 2 years ago

Unfortunately from experience that just gives enough of a delay that you get talked over in a group setting and never get a chance to speak anyway.

MattPalmer1086• 2 years ago

That actually sounds like hell to me, a complete absence of spontaneity and being in the moment.

I used to obsessively try to figure out what to say before I said it. I am socially awkward, and it did not help at all. I love writing because it is asynchronous and I can figure things out precisely and edit my thoughts.

But in social situations it is a complete hindrance.

QuantumG• 2 years ago

We're done for!

pawnty• 2 years ago

This is the missing piece to train AI which has the ability to reason. There are so many tasks whose answers are known but reason steps are missing. With this method, we can use less annotated data the reach the ability.

The interesting part(I imagine): the generated thought could be hard for human to understand while it is still way more helpful to get the correct answer! If that happens, we have created something more intelligent than ourselves.

silent_cal18• 2 years ago

Neural networks do not think

adlpz5• 2 years ago

Do neurons think? Do a bunch of neurons?

Is this semantics?

empath-nirvana3• 2 years ago

basically this: https://en.wikipedia.org/wiki/Sorites_paradox

One neuron doesn't think. Three neurons don't think. Billions of neurons think. Somewhere between one neuron and billions of neurons, thinking starts happening. Probably also true for neural networks. The main problem is that people throw around terms like: "Thought", "Intelligence", "Will", "Reasoning", "Knowledge", "Consciousness", etc like they are very well defined and well understood terms and they very much are not.

silent_cal18• 2 years ago

There are no real neurons in a neurons in a neural network.

PoignardAzur• 2 years ago

You're not giving information anybody on this forum doesn't already know.

Obviously they don't "speak" either. Both "think" and "speak" are used as shorthands here for what the language models actually do.

silent_cal18• 2 years ago

What are you upset with me for? The authors are using the misleading language, not me. Take it up with them.

optimalsolver6• 2 years ago

Could you give a definition of "think" that NNs fail to live up to?

silent_cal18• 2 years ago

Abstracting immaterial concepts from physical reality and deliberately using them in analytical or deductive processes to discover truths.

4RealFreedom• 2 years ago

I don't understand the downvotes - you are correct.

silent_cal18• 2 years ago

I think people just get mad when they're reminded of this obvious fact. They want computers to prove that our minds are an illusion, the product of a "meat computer".

viraptor• 2 years ago

It's fair to talk about thinking in a handwavey "you know what I mean" way. This is not a philosophy paper. It's a fine point if that's what you want to discuss, but doesn't change anything about the issue at hand and is needlessly pedantic. It's the "what you're referring to is actually GNU/Linux" of discussions about the tech side of AI.

YetAnotherNick2• 2 years ago

Another RL paper with terrible baseline. They used 0 shot non instruction tuned Mistral for GSM8k which has very specific way of output. They got 11% accuracy after improving it, while few shot prompting achieves 37%[1]. GPT 4 could get ~97% with prompting.

[1] https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...

hiddencost• 2 years ago

Fwiw if they're serious scientists, taking a known method and baseline and improving it is good science. Extensions to get state of the art are probably possible, but their goal is to measure just the impact of their change in a simple setting. Let the engineers do the munged system combinations and get SoTA.

YetAnotherNick2• 2 years ago

I am not talking about SoTA. I am talking about deliberate poor baseline. GSM8k consists of two things: solving the problem and getting the output format correct. Getting the output format corrects gives 30% accuracy for the same model where they got 11%. SoTA is 97%.

lionkor2• 2 years ago

This is purely anecdotal, and I try to keep it to myself but its very difficult when at least half of the HN homepage is AI related: LLMs like ChatGPT do so utterly terribly at any non-trivial job I throw at it that I seriously consider people who use it daily to either be straight up incompetent, or maybe their domain is so trivial that the LLM actually does well.

From asking LLMs to solve a highly difficult async C++ parallelism problem, to german language specifics, it just fucks up at a fundamental level. I understand that LLMs cannot solve these issues and why, but then I do not understand the heavy focus on AI by so many tech people.

Is day to day programming job so trivial that LLMs do a good job, while at the same time being too difficult for you to do it yourself? I really, really want to know exactly what the use case is.

Do people just throw simple problems at it to validate their own preconceived notion of how cool and useful LLMs are? Whats the deal?

rplnt• 2 years ago

Every other query I've given to ChatGPT came up with an utterly wrong answer. Followup always yielded "sorry, I made an obvious mistake, here's another wrong answer". Confident and stupid is a very bad combination.

jollyllama• 2 years ago

There are plenty of jobs where people have to complete various tasks that are outside of their domain or otherwise tedious on a daily basis. For example, plenty of devs have to set up or change the configuration of remote hosts. Some LLMs are pretty good at generating devops scripts to speed up this work.

orzig3• 2 years ago

Exactly. Example: maybe 1% of the code I generate is bash. I used to try to memorize patterns, but of the top 20 I'd use each less than once per year. Now, instead of that 1% taking 5% of my time, it takes 2%. It's all "simple stuff", and I can verify it instantly.

I have ~10 similar use cases. So it hasn't revolutionized my life, but it's been well worth $20/mo ChatGPT Plus and $3/mo API calls.

williamcotton• 2 years ago

Boilerplate, test code, and general tedium. Most software just needs to handle IO.

The next time you want to use SQL to compute a rolling sum try asking ChatGPT 4 instead of searching through documentation or search engine results for windowing functions.

Competency at programming along with very good technical communication skills (with a touch of learning how to not hold the tool backwards) and you should find the appeal.

slices• 2 years ago

yes. just used Cody to get me on the right path with an obscure postgresql JSON query, it easily saved me an hour of fiddling around.

luma3• 2 years ago

This is an observation I've seen a lot around here. Underneath it is the assumption that "if I can't figure out how to get meaningful use out of a tool, the tool must be useless".

OpenAI didn't sign up 100M users without somebody somewhere finding it to be useful. Like any other tool, it's utility is limited mostly by the person wielding it.

bluGill3• 2 years ago

The tools seem useful, but I'm not sure they are. too often they will confidently make up an answer that is wrong. When I use them they do great on trivial problems but can't help on hard ones.

Tadpole9181• 2 years ago

They're good autocomplete, they can help search for solutions sometimes better than Google (SEO spam), you can use it as a rubber duck, and you can make it auto fill trivial stuff that would take you a few minutes to write out manually, like test scaffolding. I would never use it to actually complete a non-trivial task and I always confirm it's answers. And yeah, sometimes it sucks - it's a tool with a learning curve about knowing it's limitations.

The reason there's so much money and time is that even semi-competant AI is relatively new and the methods are still extreme crude, and yet it's this advanced. This seems like the path to an AGI, and if someone were to even approach that point, it would radically change the world forever and could lead to either really good things or really bad things.

Now, GPT-4 isn't considered the best at specialized tasks. It's a master of many, but there are much smaller models that can do things like incredibly complex symbolic/geometric math proofs, write code, perform translations, etc better. A lot of ideas are on making expert systems using many of those specialists combined with a generalist, like the segmentation of a brain.

Anyway:

I seriously consider people who use it daily to either be straight up incompetent, or maybe their domain is so trivial that the LLM actually does well.

These kinds of radical lines of thinking about a significant proportion of enthused professionals (in any industry) who aren't showing the same experience as you, is a red flag for introspection. It's so easy to fall into the "enlightened me" trap.

I appreciate you asking for more information!

nathas• 2 years ago

I had a similar take until about a week ago. A friend showed me his workflow with Copilot and whatever Jetbrains AI assistant is.

Use it as a tool: what if instead of opening up a new tab, searching for the API docs for the library you're trying to find a function in, find the function, re-read the parameter arguments for the 400th time, and then use it, you could just highlight a snippet and say "Paginate the results from S3 using boto3" and the code would just populate?

You have to have the clarity of thought to know what you're doing, but the time it takes to write every line for basic stuff you've done 1000x before can be greatly compressed if it's inlined with your IDE.

I think this is the move for most LLM tools: integrate it with existing tooling. An LLM for Excel for corporate bookkeepers, CPAs, etc will be great. A Word/PDF summarizer that's tuned for attorneys will also be fantastic. Highlight a paragraph, ask for relevant case law, etc.

I thought ~2 years ago the results were... not great. Now I'm pretty happy with it.

SecureFrame (helps with compliance regimes like SOC2) recently added the ability to generate Terraform templates to automatically generate infrastructure that will fix specific platform risks for AWS, Azure, GCP, etc.

It definitely needs someone at the helm since it does hallucinate, but I have found it to cut down my time on mundane tasks or otherwise niche/annoying problems. When was the last time you visited 4+ StackOverflow posts to find your answer? Copilot, so far, has always hit a pretty close answer very quickly.

samstave4• 2 years ago

Sorry if this is sophmoric, but when you said "you have to have clarity of thought" - what jumped to mind was the phrase "you have to speak to the code"... I thought it encapsulated your clarity of thought quite saliently for me.

dkjaudyeqooe2• 2 years ago

I don't know exactly how you use it, but this isn't my experience at all. If you ask a LLM anything too specific, that isn't obvious and a common issue/discussion ( something that I almost never need to do), it just makes up nonsense to fill the space.

Equally, if you ask it general questions it misses information and is almost always incomplete, leaving out slightly more obscure elements. Again, I need comprehensive answers, I can come up with incomplete ones myself.

What's really obvious to me when I use it is that it's a LLM trained on pre-existing text, that really comes through in the character of its answers and its errors.

I've very glad others find them useful and productive, but for me they're disappointing given how I want to use them.

orzig3• 2 years ago

I also had to build intuition for when it will be appropriate versus not. It's hard to describe but one very positive signal is certainly "will any hallucination be caught in <30s"? Even in ChatGPT Plus you can have it write its own unit tests and run them in the original prompt (even in the profile's Custom Instructions so you don't have to type it all the time).

So a mistake was using it for something where runtime performance on dozens of quirky data files was critical; that nearly set my CPU on fire. But str>str data cleanup, chain of simple API calls, or some a one-off data visualization? chef kiss

jmull• 2 years ago

to write every line for basic stuff you've done 1000x before

There are ways to avoid writing basic stuff you've done 1000x before that are better than LLMs though...

Put it in a well-thought-out function or package or other form of shared/reusable code. You can validate it, spend the time to make sure it covers your edge cases, optimize it, test it, etc. so that when you go to reuse it you can have confidence it will reliably do what you need it to do. LLM-generated code doesn't have that.

(When you think about how LLMs are trained and work, you realize they are actually just another form of code reuse, but one where there are various transformations to the original code that may or may not be correct.)

Where LLMs shine for coding is in code-completion. You get the LLM output in little chunks that you can immediately review correctly and completely, in the moment: "yeah that's what I want" or "no, that's no good" or "ok, I can work with that". Not surprising, since predicting completion is what LLMs actually do.

kthartic• 2 years ago

Some questions we've thrown at GPT-4 recently (real use cases):

how does torchmetrics IOU work? Does it match gt with detection boxes? or does it do pairwise IOU and average?
What predictions has Ray Kurzweil made that he got correct and incorrect? Please produce a table
can you give me a stack implementation with min function in O(1) time
(A question about how we should solve a UX problem specific to our app)
What is the best way to return raw image data via a REST endpoint?
How is Return on Capital Employed (ROCE) calculated?
Following the email exchange below, write a cross intro email to introduce (X person) to (Y person)
How do I run this code on TPU in Collab?

samstave4• 2 years ago

RE: Ray Kurzweil

Did you see him on JRE last week:

https://www.youtube.com/watch?v=w4vrOUau2iY

(or was that why you asked)

jrmg• 2 years ago

Did it correctly answer all of these?

keiferski• 2 years ago

You should treat LLMs the same way you treat any other smart entity, human or otherwise: realize that they can be both immensely useful and fundamentally wrong at the same time. Intelligence is not equivalent to correctness.

zmgsabst• 2 years ago

Three examples:

1. having ChatGPT generate boilerplate, because I’m lazy;

2. having ChatGPT attempt something I don’t know as a starting point, eg JavaScript; or,

3. having ChatGPT give a reference rather than Google myself, eg of a config option.

ChatGPT makes 1 less tedious, 3 less a game of “what magic phrase finds the right SO post?”, and means I do 2 at all, eg trying out JS features on my blog.

I think it does alright at composition if you break down the task sufficiently, but it struggles with higher order structure — particularly if you’re using multiple responses.

That said, I suspect we need a theory shift to get AI to comprehend higher order structure in composition.

empath-nirvana3• 2 years ago

It's pretty amazing at generating rust structs from yaml examples, and also at writing generic versions of rust functions.

Neither of those tasks are especially _difficult_, but they are _annoying_.

slifin• 2 years ago

Not everything in tech is difficult

I find LLMs great for creating SQL queries and regexes

readyman• 2 years ago

Profit. The question at hand is whether LLMs can produce profit, which is an extremely different question than the questions you're asking.

leothecool• 2 years ago

I train my LLM to barf up my domain specific boilerplate code. I don't ask it to solve business problems.

BenFranklin100• 2 years ago

I signed up for Open.AI’s monthly subscription. Its performance on non-trivial tasks is abysmal. It’s a regurgitation machine. One might mischievously argue the average tech worker isn’t much better than an LLM, thus the interest? On a related note, we are deluged daily with firms offering AI services. I see a bubble.

Havoc• 2 years ago

For me it’s more like brainstorming.

Even if half of it is garbage it’s a net win. At least in domains where I can distinguish the two.

There are also cases where the cost of failure is very low. Eg I could spend half an hour reading an api spec or I could make an AI give me a curl command and test it out in 30 seconds. If it works great if not oh well time to read spec

dmos62• 2 years ago

Why do you presume that people commonly use it for non-trivial things? It excels at trivial things. That's what most people use it for, probably. Like google search. Is there something that leads you to think otherwise?

xanderlewis2• 2 years ago

Perhaps the incessant talk of GPT-x being AGI, whatever that means.

samstave4• 2 years ago

I think its safe to remind oneself that this thing is literally a zygot. So patience, and in ~5 years, it will be a different story.

@xanderlewis

Doesnt that mean it simply is now consuming the internet in real-time?

xanderlewis2• 2 years ago

Why? It’s already eaten all of the publicly available data on the web.

reportgunner• 2 years ago

First sentence is 100% my sentiment, cheers !

visarga3• 2 years ago

Your view on LLM usage is too narrow. Yes, they are pretty shit for me too in solving coding problems, but they are still useful for bespoke information extraction, classification and creative applications. The interest is justified, we're just having a hard time understanding the limitations.

atoav• 2 years ago

Technology is complex and hard to make sense of. That is why most non-experts have a strong wish for a kind of mythical technology, which you can just pour onto your problem and it magically knows what you wanted (and which things you did not want).

For a certain class of problems LLMs achieved new, never before seen, almost magical results. Now imagine you were someone who hates dealing with the constant complexity of solving problems with technology and something comes along that seems to carry promise of lifting that off your shoulders. Then you know why people react like they do. Recall the block-chain-craze? There were people who declared that this somehow magically solved any IT-security problem there ever was – instead of seeing it as a good solution for a very specific set of circumstances, nearly nobody faced in practise.

In reality of course also LLMs have limitiations, e.g. above mentioned ambiguity that is inherent to any magical technology: To be true magic the technology would have to be able to read the thoughts of those who apply it and somehow infer from that the true thing they want or need. Now LLMs are in the end still just very good guesses based on statistical data, that means the guess could just be what you want, but it lacks an actual understanding of what it is doing.

Those applying the technology for things it is actually good at (e.g. classification problems etc) will put it to good use, but there will be a lot who will apply it and have things fall apart Canada Airlines style.

epanchin• 2 years ago

I daily drive KDB/Q. This is readily extendable for example in C, which was my previous daily, and Python which I use sporadically.

I don’t use LLMs for C or KDB, I do use them for Python.

ChatGPT is good in Python. I guess as Python programmers rely on stack exchange so there is lots to learn from, and Python anyway is largely an exercise in finding the correct library.

If the only thing ChatGPT did was listen to my problem and suggest which imports to use/manuals to read, that would be good enough to use regularly. If I wasn’t after a library/pre existing code I wouldn’t be using Python!

BoxOfRain• 2 years ago

I've definitely noticed ChatGPT generally writes better Python than it writes Scala, presumably for the same reason of there being a fair bit more Python code in the wild.

sebzim45002• 2 years ago

Stop using it for things that are in you area of expertise but are too difficult for you. Use if for things where you think "this is probably easy but I have no idea how to do it". For example, I needed to do some pretty trivial task in powershell but I have never used it so I got chatGPT to do it for me and it worked first time. Obviously I checked the commands looked plausible before I ran them, but it still probably took 2 mins to do something that would have otherwise taken 30.

porkbeer• 2 years ago

That just means you are ignorant of how wrong it guides you. You need to first build trust before taking it new places. You do that with topics and concepts you are familiar with.

OmarShehata2• 2 years ago

I want to second this:

Use if for things where you think "this is probably easy but I have no idea how to do it"

I had exactly the same reaction as OP (LLM's suck what's with the all the hype). These people are using it differently. For me it's often something like, asking it to put together a specific sequence of matrix transformations in ThreeJS or some other library.

This is not a difficult task but it's often one I waste a lot of time getting right. It's sort of about finding the right level of abstraction you need to ask it.

runeofdoom• 2 years ago

And how often will those "plausible looking commands" create obvious or subtle problems that cost far more than 30 minutes?

Al-Khwarizmi• 2 years ago

Does your job involve solving complex, challenging problems all the time?

I am a CS professor, I don't think most people would class that as a trivial job, but I find myself needing to do plenty of trivial tasks every day: mixed bureaucracy (periodic reports, grant requests, various evaluations, etc.), trivial programming (a Seaborn chart to show some Excel results), text polishing (need to cut a text to 500 words without altering meaning), writing student assignments, writing emails in (non-Native) English for sensitive requests with the right tone, etc... all of those are things I have found LLMs to do fairly well and save me a lot of time.

I wouldn't use them to do the core job of designing novel algorithms, doing experiments, writing the bulk of a paper or teaching students. But most of my working hours are not really that "core" stuff. And I would assume it's the same for most professionals.

If you have an environment where you are constantly challenged by difficult tasks... wow. I don't know if I should envy you (because I love difficult problems and hate mindless chores) or it would be too stressful.

PS: I don't think "being too difficult for you to do it yourself" is the right litmus test for LLM usefulness. I can draw charts with Seaborn, of course. But the LLM does it much faster, and I don't think doing it myself would make me grow, hone useful skills or anything. I'd rather devote my time to something else. So (in my view) it's clearly better to have the LLM do it.

Benjaminsen• 2 years ago

I’m preparing. Learning how to work with an AI is the only way to stay competitive. The AIs will become smarter much faster than I will.

mordymoop• 2 years ago

When you say ChatGPT, are you referring to GPT4? I find a huge and avoidable miscommunication happens when two people both think they are using “ChatGPT” but talking about two different models which vary in size by a factor of 10.

Assuming you are talking about GPT4, for the sake of argument, the answer is speed. Of course I can write a small parser script that deals with some data I received from a client. It will take me an hour and be a tedious task far distant from my actual expertise. An LLM can do it in 45 seconds, including the time it took me to describe the task.

KLejmooo• 2 years ago

I don't use it constantly but regularly.

LLMs english skills are much better than mine.

And when i do a little bit of go coding once a week (i'm a java developer by trade), i don't have the time to learn go well enough to just type stuff down without looking things up. Instead of googling, i tell it "I need a struct with the following attributes..." and it doesn't just ell me how i do structs in go, it also creates them for me.

Also: There are a TON of issues were i would write a short script to do something (formatting text into a table, searching for specific lines etc.) were a normal person doesn't even have those tools at hand.

For companies overall: Its not just what an LLM can do, LLM can do things for you but its also a very very good interface to your application. The demos i saw in my company are really good and totally make sense and do reduce the entry barrier for people.

I know a friend whos job is to create reports with sql. She doesn't do anything else just reports across the whole datawarehouse. Why? Because every normal non dev person can't just write SQL or automate things.

The gap between tech people and management is huge.

astrange5• 2 years ago

Are you using the free one? If you are, GPT4 is completely different. Claude is the best free chatbot.

lionkor2• 2 years ago

Yes, the free one. I find it almost a fallacy to say "oh, yeah it sucks, but you should try the expensive one! Its proper good."

Oh, your BMW keeps having issues? You should have bought the one thats 2x the price, that one is perfect!

Of course its better, but both are being sold as a car, or, in this case, an LLM. Theyre both LLMs, theyre both by the leading AI company, if one sucks such supreme ass, why would the other one be amazing? ;)

archibaldJ• 2 years ago

This looks really interesting; any possibility the researchers will release some code soon ?

iAkashPaul• 2 years ago

Base Mistral 7B is hardly suitable for the evaluations, even one team at Intel tried to pull a fast one with NeuralChat in the exact same way https://huggingface.co/Intel/neural-chat-7b-v3#quantitative-...

kjqgqkejbfefn5• 2 years ago

This is basically what I tried this morning at the prompt level (awful results), but the sketchy idea I had in mind went further by introducing control-flow "meta-tokens" to help the LLM renavigate its context. In this perspective the context would be rethought as a self-editing structured mind-map, with the linear aspect of the context at a time T standing for the execution trace of the exploration of this mind-map so far. Some of those meta-tokens would be able to have side effects on the context, to highlight, give structure, summarize, forget and so on, some of its parts. This could allow for native structured output without using a syntactic format such as json, programmatic constructs in the style of LMQL, implementing memory, etc. The goal: not just to give logical/reasoning abilities to a LLM, but to give it the means to come up with its own cognitive architecture. Implementing structured output (using a <label name="stuff">...</label> token) to also implement memory/scratchpads, would also bring inspectability of those cognitive structures for free. Of course I have no idea how to implement this (I'm a ML tourist).

lawlessone• 2 years ago

If it is doing this , is it still a language model? or also a thought model?

aaroninsf• 2 years ago

Observation: "expertise" (hence "reflex") is the learning of the nonlinear solution space that can be inferred from initial conditions.

Conjecture: models which engage in self-training on the solutions they derive will get to something that looks a bit like bootstrapping when you squint.

Lemma: there's a nice opportunity for cloud-hosted model SaaS to offer discounts for actionable feedback on the quality of their output, so as to drive this retraining.

Idle comment: I'd use the language of REM sleep and the idea of "memory consolidation" for this.

Most of the premises of model training can be extended to the level of reasoned solutions, rather than tokens.

thesz• 2 years ago

They do not cite [1], a paper on (learned) variable computation in RNN, applied to language modeling, that predates their work by almost 8 years.

[1] https://openreview.net/pdf?id=S1LVSrcge

Microsoft also had something alike at that time, but for image recognition: a CNN at input and then varable computation at classification.

itissid• 2 years ago

Much of the meaning of text is hidden between the lines: without understanding why statements appear in a document, a reader has only a shallow understanding.

This seems not true when I and most people I know read things. I would argue that we almost always have a world model and know some reasons why these statements are appearing in a book. If I was reading Fluid Dynamics textbook, I may not understand the math, but I know why those statements appear; They are mathematical statements to help you learn the theory or whatever and they follow a pattern to teach you important concepts. Like for instance concepts will build upon older ones Bernoulli's equation is there because law of conservation of energy was there before it, so its there because it assumes I understand the latter..

Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking

Related Stories