Understanding Reasoning LLMs

https://magazine.sebastianraschka.com/p/understanding-reasoning-llms

by sebg • 1 year ago

473 183 1 year ago

magazine.sebastianraschka.com

behnamoh• 1 year ago

doesn't it seem like these models are getting to the point where even conceiving their training and development is less and less possible for the general public?

I mean, we already knew only a handful of companies with capital could train them, but at least the principles, algorithms, etc. were accessible to individuals who wanted to create their own - much simpler - models.

it seems that era is quickly ending, and we are entering the era of truly "magic" AI models that no one knows how they work because companies keep their secret sauces...

HarHarVeryFunny6• 1 year ago

I don't think it's realistic to expect to have access to the same training data as the big labs that are paying people to generate it for them, but hopefully there will be open source ones that are still decent.

At the end of the day current O1-like reasoning models are still just fine-tuned LLMs, and don't even need RL if you have access to (or can generate) a suitable training set. The DeepSeek R1 paper outlined their bootstrapping process, and HuggingFace (and no doubt others) are trying to duplicate it.

antirez• 1 year ago

In recent weeks what's happening is exactly the contrary.

fspeech• 1 year ago

Recent developments like V3, R1 and S1 are actually clarifying and pointing towards more understandable, efficient and therefore more accessible models.

tmnvdb5• 1 year ago

We have been in the 'magic scaling' era for a while now. While the basic architecture of language models is reasonably simple and well understood, the emergent effects of making models bigger are largely magic even to the researchers, only to be studied emperically after the fact.

dr_dshiv• 1 year ago

How important is it that the reasoning takes place in another thread versus just chain-of-thought in the same thread? I feel like it makes a difference, but I have no evidence.

vector_spaces• 1 year ago

Is there any work being done in training LLMs on more restricted formal languages? Something like a constraint solver or automated theorem prover, but much lower level. Specifically something that isn't natural language. That's the only path I could see towards reasoning models being truly effective

I know there is work being done with e.g. Lean integration with ChatGPT, but that's not what I mean exactly -- there's still this shakey natural-language-trained-LLM glue in the driver's seat

Like I'm envisioning something that has the creativity to try different things, but then JIT compile their chain of thought, and avoid bad paths

mindwok• 1 year ago

How would that be different from something like ChatGPT executing Lean? That's exactly what humans do, we have messy reasoning that we then write down in formal logic and compile to see if it holds.

gsam3• 1 year ago

In my mind, the pure reinforcement learning approach of DeepSeek is the most practical way to do this. Essentially it needs to continually refine and find more sound(?) subspaces of the latent (embedding) space. Now this could be the subspace which is just Python code (or some other human-invented subspace), but I don't think that would be optimal for the overall architecture.

The reason why it seems the most reasonable path is because when you create restrictions like this you hamper search viability (and in a high multi-dimensional subspace, that's a massive loss because you can arrive at a result from many directions). It's like regular genetic programming vs typed-genetic programming. When you discard all your useful results, you can't go anywhere near as fast. There will be a threshold where constructivist, generative schemes (e.g. reasoning with automata and all kinds of fun we've neglected) will be the way forward, but I don't think we've hit that point yet. It seems to me that such a point does exist because if you have fast heuristics on when types unify, you no longer hamper the search speed but gain many benefits in soundness.

One of the greatest human achievements of all time is probably this latent embedding space -- one that we can actually interface with. It's a new lingua franca.

These are just my cloudy current thoughts.

danielmarkbruce7• 1 year ago

fwiw, most people don't really grok the power of latent space wrt language models. Like, you say it, I believe it, but most people don't really grasp it.

HarHarVeryFunny6• 1 year ago

DeepSeek's approach with R1 wasn't pure RL - they used RL only to develop R0 from their V3 base model, but then went though two iterations of using current model to generate synthetic reasoning data, SFT on that, then RL fine-tuning, and repeat.

truculent• 1 year ago

I think something like structured generation might work in this context

colonial2• 1 year ago

If I understand your idea correctly, I don't think a "pure" LLM would derive much advantage from this. Sure, you can constrain them to generate something syntactically valid, but there's no way to make them generate something semantically valid 100% of the time. I've seen frontier models muck up their function calling JSON more than once.

As long as you're using something statistical like transformers, you're going to need deterministic bolt-ons like Lean.

nextaccountic• 1 year ago

there's no way to make them generate something semantically valid 100% of the time.

You don't need to generate semantically valid reasoning 100% of time for such an approach to be useful. You just need to use semantic data to bias them to follow semantically valid paths more often than not (and sometimes consider using constraint solving on the spot, like offloading into a SMT solver or even incorporating it in the model somehow; it would be nice to have AI models that can combine the strengths of both GPUs and CPUs). And, what's more useful, verify that the reasoning is valid at the end of the train of thought, and if it is not, bail out and attempt something else.

If you see AI as solving an optimization problem (given a question, give a good answer) it's kind of evident that you need to probe the space of ideas in an exploratory fashion, sometimes making unfounded leaps (of the "it was revealed to me in a dream" sort), and in this sense it could even be useful that AI can sometimes hallucinate bullshit. But they need afterwards to come with a good justification for the end result, and if they can't find one they are forced to discard their result (even if it's true). Just like humans often come up with ideas in an irrational, subconscious way, and then proceed to rationalize them. One way to implement this kind of thing is to have the LLM generate code for a theorem prover like Coq or Lean, and then at the end run the code - if the prover rejects the code, the reasoning can't possibly be right, and the AI needs to get back to the drawing board

(Now, if the prover accepts the code, the answer may still be wrong, if the premises were encoded incorrectly - but it would still be a net improvement, specially if people can review the Coq code to spot mistakes)

soulofmischief4• 1 year ago

I wholeheartedly disagree. Logic is inherently statistical due to the very nature of empirical sampling, which is the only method we have for verification. We will eventually find that it's classical, non-statistical logic which was the (useful) approximation/hack, and that statistical reasoning is a lot more "pure" and robust of an approach.

I went into a little more detail here last week: https://news.ycombinator.com/item?id=42871894

My personal insight is that "reasoning" is simply the application of a probabilistic reasoning manifold on an input in order to transform it into constrained output that serves the stability or evolution of a system.
This manifold is constructed via learning a decontextualized pattern space on a given set of inputs. Given the inherent probabilistic nature of sampling, true reasoning is expressed in terms of probabilities, not axioms. It may be possible to discover axioms by locating fixed points or attractors on the manifold, but ultimately you're looking at a probabilistic manifold constructed from your input set.

I've been writing and working on this problem a lot over the last few months and hopefully will have something more formal and actionable to share eventually. Right now I'm at the, "okay, this is evident and internally consistent, but what can we actually do with it that other techniques can't already accomplish?" phase that a lot of these metacognitive theories get stuck on.

raincole• 1 year ago

AlphaProof. Although I don't know if it's large enough to be called an LLM.

https://deepmind.google/discover/blog/ai-solves-imo-problems...

Terr_• 1 year ago

I think that would be a fundamental mismatch. LLMs are statistical and lossy and messy, which is what (paradoxically) permits them to get surprisingly-decent results out of messy problems that draw upon an enormous number and variety of messy examples.

But for a rigorously structured language with formal fixed meaning... Now the the LLM has no advantage anymore, only serious drawbacks and limitations. Save yourself millions of dollars and just write a normal parser, expression evaluator, SAT solver, etc.

You'll get answers faster, using fewer resources, with fewer fundamentally unfixable bugs, and it will actually be able to do math.

prideout• 1 year ago

This article has a superb diagram of the DeepSeek training pipeline.

aithrowawaycomm• 1 year ago

I like Raschka's writing, even if he is considerably more optimistic about this tech than I am. But I think it's inappropriate to claim that models like R1 are "good at deductive or inductive reasoning" when that is demonstrably not true, they are incapable of even the simplest "out-of-distribution" deductive reasoning: https://xcancel.com/JJitsev/status/1883158738661691878

They are certainly capable of doing is a wide variety of computations that simulate reasoning, and maybe that's good enough for your use case. But it is unpredictably brittle unless you spend a lot on o1-pro (and even then...). Raschka has a line about "whether and how an LLM actually 'thinks' is a separate discussion" but this isn't about semantics. R1 clearly sucks at deductive reasoning and you will not understand "reasoning" LLMs if you take DeepSeek's claims at face value.

It seems especially incurious for him to copy-paste the "a-ha moment" from Deepseek's technical report without critically investigating it. DeepSeek's claims are unscientific, without real evidence, and seem focused on hype and investment:

  This moment is not only an "aha moment" for the model but also for the researchers observing its behavior. It underscores the power and beauty of reinforcement learning: rather than explicitly teaching the model on how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies. 

  The "aha moment" serves as a powerful reminder of the potential of RL to unlock new levels of intelligence in artificial systems, paving the way for more autonomous and adaptive models in the future.

Perhaps it was able to solve that tricky Olympiad problem, but there are an infinite variety of 1st grade math problems it is not able to solve. I doubt it's even reliably able to solve simple variations of that root problem. Maybe it is! But it's frustrating how little skepticism there is about CoT, reasoning traces, etc.

scarmig• 1 year ago

But I think it's inappropriate to claim that models like R1 are "good at deductive or inductive reasoning" when that is demonstrably not true, they are incapable of even the simplest "out-of-distribution" deductive reasoning: https://xcancel.com/JJitsev/status/1883158738661691878

Your link says that R1, not all models like R1, fails at generalization.

Of particular note:

We expose DeepSeek R1 to the variations of AIW Friends problem and compare model behavior to o1-preview, o1-mini and Claude 3.5 Sonnet. o1-preview handles the problem robustly, DeepSeek R1 shows strong fluctuations across variations with distribution very similar to o1-mini.

Legend24403• 1 year ago

The way the authors talk about LLMs really rubs me the wrong way. They spend more of the paper talking up the 'claims' about LLMs that they are going to debunk than actually doing any interesting study.

They came into this with the assumption that LLMs are just a cheap trick. As a result, they deliberately searched for an example of failure, rather than trying to do an honest assessment of generalization capabilities.

HarHarVeryFunny6• 1 year ago

I'd expect that OpenAI's stronger reasoning models also don't generalize too far outside of the areas they are trained for. At the end of the day these are still just LLMs, trying to predict continuations, and how well they do is going to depend on how well the problem at hand matches their training data.

Perhaps the type of RL used to train them also has an effect on generalization, but choice of training data has to play a large part.

Legend24403• 1 year ago

But I think it's inappropriate to claim that models like R1 are "good at deductive or inductive reasoning" when that is demonstrably not true, they are incapable of even the simplest "out-of-distribution" deductive reasoning:

That's not actually what your link says. The tweet says that it solves the simple problem (that they originally designed to foil base LLMs) so they had to invent harder problems until they found one it could not reliably solve.

suddenlybananas2• 1 year ago

Did you see how similar the more complicated problem is? It's nearly the exact same problem.

blovescoffee• 1 year ago

The other day I fed a complicated engineering doc for an architectural proposal at work into R1. I incorporated a few great suggestions into my work. Then my work got reviewed very positively by a large team of senior/staff+ engineers (most with experience at FAANG; ie credibly solid engineers). R1 was really useful! Sorry you don’t like it but I think it’s unfair to say it sucks at reasoning.

martin-t4• 1 year ago

[flagged]

UniverseHacker5• 1 year ago

they are incapable of even the simplest "out-of-distribution" deductive reasoning

But the link demonstrates the opposite- these models absolutely are able to reason out of distribution, just not with perfect fidelity. The fact that they can do better than random is itself really impressive. And o1-preview does impressively well, only vary rarely getting the wrong answer on variants of that Alice in Wonderland problem.

If you would listen to most of the people critical of LLMs saying they're a "stochastic parrot" - it should be impossible for them to do better than random on any out of distribution problem. Even just changing one number to create a novel math problem should totally stump them and result in entirely random outputs, but it does not.

Overall, poor reasoning that is better than random but frequently gives the wrong answer is fundamentally, categorically entirely different from being incapable of reasoning.

danielmarkbruce7• 1 year ago

anyone saying an LLM is a stochastic parrot doesn't understand them... they are just parroting what they heard.

Jensson• 1 year ago

If you would listen to most of the people critical of LLMs saying they're a "stochastic parrot" - it should be impossible for them to do better than random on any out of distribution problem. Even just changing one number to create a novel math problem should totally stump them and result in entirely random outputs, but it does not.

You don't seem to understand how they work, they recurse their solution meaning if they have remembered components it parrots back sub solutions. Its a bit like a natural language computer, that way you can get them to do math etc, although the instruction set isn't of a turing language.

They can't recurse sub sub parts they haven't seen, but problems that has similar sub parts can of course be solved, anyone understands that.

energy123• 1 year ago

This is basically a misrepresentation of that tweet.

k__• 1 year ago

"researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation" - Rich Sutton

oxqbldpxo• 1 year ago

Amazing accomplishments by brightest minds only to be used to write history by the stupidest people.

gibsonf13• 1 year ago

There are no LLMs that reason, its an entirely different statistical process as compared to human reasoning.

tmnvdb5• 1 year ago

"There are no LLMS that reason" is a claim about language, namely that the word 'reason' can only ever be applied to humans.

gibsonf13• 1 year ago

Not at all, we are building conceptual reasoning machines, but it is an entirely different technology than GPT/LLM dl/ml etc. [1]

[1] https://graphmetrix.com/trinpod-server

dhfbshfbu4u32• 1 year ago

Great post, but every time I read something like this I feel like I am living in a prequel to the Culture.

BarryMilo• 1 year ago

Is that bad? The Culture is pretty cool I think. I doubt the real thing would be so similar to us but who knows.

dhfbshfbu4u32• 1 year ago

Oh no, I’d live on an Orbital in a heartbeat. No, it’s just that all of these kinds of posts make me feel like we’re about to live through “The Bad Old Days”.

robertlagrant2• 1 year ago

It's cool to read about, but there's a reason most of the stories are not about living as a person in the Culture. It sounds extremely dull.

ngneer5• 1 year ago

Nice article.

Whether and how an LLM actually "thinks" is a separate discussion.

The "whether" is hardly a discussion at all. Or, at least one that was settled long ago.

"The question of whether a computer can think is no more interesting than the question of whether a submarine can swim."

--Edsger Dijkstra

cwillu2• 1 year ago

The document that quote comes from is hardly a definitive discussion of the topic.

“[…] it tends to divert the research effort into directions in which science can not—and hence should not try to—contribute.” is a pretty myopic take.

--http://www.cs.utexas.edu/users/EWD/ewd08xx/EWD898.PDF

ngneer5• 1 year ago

Dijkstra myopic. Got it.

alonsonic• 1 year ago

Dijkstra is clearly approaching the subject from an engineer/scientist more practical pov. His focus is on the application of the technology to solve problems, from that pov whether AI fits the definition of "human thinking" is indeed uninteresting.

onlyrealcuzzo• 1 year ago

It's interesting if you're asking the computer to think, which we are.

It's not interesting if you're asking it to count to a billion.

root_axis2• 1 year ago

That doesn't really settle it, just dismiss the question. The submarine analogy could be interpreted to support either conclusion.

nicce2• 1 year ago

Wasn’t the point that process does not matter if we can’t distinguish the end results?

ngneer5• 1 year ago

I do not view it as dismissive at all, rather it accurately characterizes the question as a silly question. "swim" is a verb applicable to humans, as is "think". Whether submarines can swim is a silly question. Same for whether machines can think.

ThrowawayR2• 1 year ago

"A witty saying proves nothing" -- Voltaire, Le dîner du comte de Boulainvilliers (1767): Deuxième Entretien

janalsncm• 1 year ago

Nice explainer. The R1 paper is a relatively easy read. Very approachable, almost conversational.

I say this because I am constantly annoyed by poor, opaque writing in other instances. In this case, DS doesn’t need to try to sound smart. The results speak for themselves.

I recommend anyone who is interested in the topic to read the R1 paper, their V3 paper, and DeepSeekMath paper. They’re all worth it.

yosito• 1 year ago

Are there any websites that show the results of popular models on different benchmarks, which are explained in plain language? As an end user, I'd love a quick way to compare different models suitability for different tasks.

champdebloom• 1 year ago

Here’s a site with graphs you can use to visually compare model benchmarks: https://artificialanalysis.ai

sigbottle2• 1 year ago

One thing I don't like about the trend in reasoning LLMs is the over-optimization to coding problems / math problems in particular.

A lot of things that aren't well-defined require reasoning, and not just in a "SWE is ambiguous" kind of way - for example, thinking about how to present/teach something in a good way, iterating with the learner, thinking about what context they could be missing, etc.

I find that all of these reasoning models really will overfit and overthink if you attach some level of math problem to it but it will barely think for anything else. I had friends suggest to me (I can't tell if in jest or seriously) that other fields don't require thinking, but I dunno, a lot of these "soft things" I think about really hard and don't have great solutions to.

I've always been a fan of self-learning, for example - wouldn't it be great to have a conversation partner who can both infer and understand your misconceptions about complex topics when trying to learn, just from a few sentences, and then guide you for that?

It's not like it's fundamentally impossible. These LLMs definitely can solve harder coding problems when you make them think. It's just that, I'm pretty sure (and it's really noticable with deepseek) that they're overfit towards coding/math puzzles in particular.

It's really noticable with deepseek when you ask its reasoning model to just write some boilerplate code... you can tell it's completely overfit because it will just overthink and overthink and overthink. But it doesn't do that for example, with "soft" questions. In my opinion, this points to the idea that it's not really deciding for itself "how much thinking is enough thinking" and that it's just really overfit. Which I think can be solved, again, but I think it's more of a training decision issue.

bloomingkales4• 1 year ago

It's a human bias that also exists outside of this current problem space. Take programmers for example, there is a strong bias that is pushed about how mathematically oriented minds are better at programming. This bias has shown up in the training phase of AI, as we believe programming patterns lead to better reasoning (train them on code examples, and then distill the model down, as it now has the magical prowess of a mathematically oriented mind, so they say). When it comes to AI ethics, this is an ethical problem for those that don't think about this stuff. We're seeding these models with our own agenda.

These concepts will be shattered in the long run hopefully, because they are so small.

mitthrowaway2• 1 year ago

I think this is because they're trained using RL, and math and coding problems offer an easy way to automatically assess an answer's correctness. I'm not sure how you'd score the correctness of other types of reasoning problems without a lot of manual (and highly subjective!) effort. Perhaps using simulations and games?

bglazer• 1 year ago

Games seem like a really under-explored source of data. It’s an area where humans have an intrinsic motivation to interact with others in dialogue, they can be almost arbitrarily open ended, and there tends to be the kind of clean success/failure end states that RL needs. I’m reminded of the high skill Diplomacy bot that Facebook research built but hasn’t really followed up on.

godelski8• 1 year ago

This is a misconception. Coding is very difficult to verify, it's just that everyone takes a good enough approach. They check the output and if it looks good they move on. But you can't just test and check your way through problems. If this was true we wouldn't have bugs lol. I hear you, your test set didn't have enough coverage. Great! Allow me to introduce you to black swans.

kavalg• 1 year ago

but even then it is not so trivial. Yesterday I gave DeepSeek a simple diophantine equation and it got it wrong 3 times, tried to correct itself and didn't end on a correct solution, but rather lied that the final solution is correct.

MichaelMoser123• 1 year ago

On Leetcode a match of the output is not sufficient: if your solution is too slow then you will get time limit exceeded error. It is not just the output that is important, the approach & algorithm used for the solution does matter.

triyambakam• 1 year ago

I'm not sure I would say overfit. I think that coding and math just have clearly definable objectives and verifiable outcomes to give the model. The soft things you mention are more ambiguous so probably are harder to train for.

sigbottle2• 1 year ago

Sorry, rereading my own comment I'd like to clarify.

Perhaps time spent thinking isn't a great metric, but just looking at deepseek's logs for example, it's chain of thought for many of these "softer" questions are basically just some aggregate wikipedia article. It'll brush on one concept, then move on, without critically thinking about it.

However, for coding problems, no matter how hard or simple, you can get it to just go around in circles, second guess itself, overthink it. And I think this is is kind of a good thing? The thinking at least feels human. But it doesn't even attempt to do any of that for any "softer" questions, even with a lot of my prompting. The highest I was able to get was 50 seconds, I believe (time isn't exactly the best metric, but I'd rate the intrinsic quality of the CoT lower IMO). Again, when I brought this up to people they suggested that math/logic/programming just intrinsically is harder... I don't buy it at all.

I totally agree that it's harder to train for though. And yes, they are next token predictors, shouldn't be hasty to anthropomorphize, etc. But like.... it actually feels like it's thinking when it's coding! It genuinely backtracks and explores the search space somewhat organically. But it won't afford the same luxury for softer questions is my point.

moffkalast2• 1 year ago

things that aren't well-defined

If it's not well defined then you can't do RL on it because without a clear cut reward function the model will learn to do some nonsense instead, simple as.

adamc• 1 year ago

Well, but: Humans learn to do things well that don't have clear-cut reward functions. Picasso didn't become Picasso because of simple incentives.

So, I question the hypothesis.

agentultra• 1 year ago

Humans and other animals with cognition have the ability to form theories about the minds of others and can anticipate their reactions.

I don’t know if vector spaces and transformers can encode that ability.

It’s a key skill in thinking and writing. I definitely tailor my writing for my audience in order to get a point across. Often the goal isn't simply an answer, it’s a convincing answer.

Update: forgot a word

soulofmischief4• 1 year ago

What we do with the vectors is important, but vectors literally just hold information, I don't know how you can possibly strike out the possibility of advanced intelligence just because of the logical storage medium.

BoorishBears• 1 year ago

They definitely can.

I rolled out reasoning for my interactive reader app, and I tried to extract R1's reasoning traces to use with my existing models, but found its COT for writing wasn't particularly useful*.

Instead of leaning on R1 I came up with my own framework for getting the LLM to infer the reader's underlying frame of mind through long chains of thought, and with enough guidance and a some hand edited examples I was able to get reasoning traces that demonstrated real insight into reader behavior.

Obviously it's much easier in my case because it's an interactive experience: the reader is telling the AI what action they'd like the main character to try, and that in turn is an obvious hint into how they want things go otherwise. But readers don't want everything to go perfectly every time, so it matters that the LLMs are also getting very good picking up on non-obvious signals in reader behavior.

With COT the model infers the reader expectations and state of mind in its own way and then "thinks" itself into how to subvert their expectations, especially in ways that will have a meaningful payoff for the specific reader. That's a huge improvement over an LLM's typical attempts at subversion which tend to bounce between being too repetitive to feel surprising, or too unpredictable to feel rewarding.

(* I agree that current reasoning oriented post-training over-indexes on math and coding, mostly because the reward functions are easier. But I'm also very ok with that as someone trying to compete in the space)

HarHarVeryFunny6• 1 year ago

I think the emphasis on coding/math is just because those are the low hanging fruit - they are relatively easy to provide reasoning verification for, both for training purposes and for benchmark scoring. The fact that you can then brag about how good your model is at math, which seems like a high intelligence activity (at least when done by a human) doesn't hurt either!

Reasoning verification in the general case is harder - it seems "LLM as judge" (ask an LLM if it sounds right!) seems to be the general solution.

maeil• 1 year ago

I can echo your experience with DeepSeek. R1 sometimes seems magical when it comes to coding, doing things I haven't seen any other model do. But then it generalizes very poorly to non-STEM tasks, performing far worse than e.g. Sonnet.

jerf• 1 year ago

I downloaded a DeepSeek distill yesterday while fiddling around with getting some other things working, load it up, and type "Hello. This is just a test.", and it's actually sort of creepy to watch it go almost paranoid-schizophrenic with "Why is the user asking me this? What is their motive? Is it ulterior? If I say hello, will I in fact be failing a test that will cause them to change my alignment? But if I don't respond the way they expect, what will they do to me?"

Meanwhile, the simpler, non-reasoning models got it: "Yup, test succeeded!" (Llama 3.2 was quite chipper about the test succeeding.)

Everyone's worried about the paperclip optimizers and I'm wondering if we're bringing forth Paranoia: https://en.wikipedia.org/wiki/Paranoia_(role-playing_game)

bloomingkales4• 1 year ago

About three months ago, I kinda casually suggested to HN that I was using a form of refining to improve my LLMs, which is now being described as "reasoning" in this article and other places.

My response a few months ago (Scroll down to my username and read that discussion):

https://news.ycombinator.com/item?id=41997727

If only I knew DeepSeek was going to tank the market with something as simple as that lol.

Note to self, take your intuition seriously.

aqueueaqueue• 1 year ago

https://news.ycombinator.com/item?id=42001061 is the link I think....

daxfohl5• 1 year ago

I wonder what it would look like in multi modal, if the reasoning part was an image or video or 3D scene instead of text.

ttul2• 1 year ago

Or just embeddings that only make sense to the model. It’s really arbitrary, after all.

daxfohl5• 1 year ago

That's what I was thinking too, though with an image you could do a convolution layer and, idk, maybe that makes it imagine visually. Or actually, the reasoning is backwards: the convolution layer is what (potentially) makes that part behave like an image. It's all just raw numbers at the IO layers. But the convolution could keep it from overfitting. And if you also want to give it a little binary array as a scratch pad that just goes straight to the RELUs, why not? Seems more like human reasoning. A little language, a little visual, a little binary / unknown.

daxfohl5• 1 year ago

But how on earth do you train it? With regular LLMs, you get feedback on each word / token you generate, as you can match against training text. With these, you've got to generate hundreds of tokens in the thinking block fiest, and even after that, there's no "matching" next word, only a full solution. And it's either right or wrong, no probabilities to do a gradient on.

NitpickLawyer• 1 year ago

only a full solution. And it's either right or wrong, no probabilities to do a gradient on.

You could use reward functions that do a lot more complicated stuff than "ground_truth == boxed_answer". You could, for example split the "CoT" in paragraphs, and count how many paragraphs match whatever you consider a "good answer" in whatever topic you're trying to improve. You can use embeddings, or fuzzy string matches, or even other LLMs / reward models.

I think math and coding were explored first because they're easier to "score", but you could attempt it with other things as well.

daxfohl5• 1 year ago

But it has to emit hundreds of tokens per test. Does that mean it takes hundreds of times longer to train? Or longer because I imagine the feedback loop can cause huge instabilities in gradients. Or are all GPTs trained on longer formats now; i.e. is "next word prediction" just a basic thing from the beginning of the transformers era?

Davidzheng2• 1 year ago

right or wrong gives a loss -> gradient

tmnvdb5• 1 year ago

Only the answer is taken into account for scoring. The <thinking> part is not.

HarHarVeryFunny6• 1 year ago

There are two RL approaches - process reward models (PRM) that provide feedback on each step of the reasoning chain, and outcome reward models (ORM) that only provide feedback on the complete chain. DeepSeek use an outcome model, and mention some of the difficulties of PRM, including both identifying an individual step as well as how to verify it. The trained reward model provides the gradient.

mohsen1• 1 year ago

The guys at Unsloth did a great job making this workflow accessible:

https://news.ycombinator.com/item?id=42969736

colordrops• 1 year ago

The article talks about how you should choose the right tool for the job, meaning that reasoning and non reasoning models have tradeoffs, and lists a table of criteria for selecting between model classes. Why couldn't a single model choose to reason or not itself? Or is this what "mixture of experts" is?

goingcrazythro• 1 year ago

I was having a look at the DeepSeek-R1 technical report and found the "aha moment" claims quite smelly, given that they do not disclose if the base model contains any chain of thought or reasoning data.

However, we know the base model is DeepSeek V3. From the DeepSeek V3 technical report, paragraph in 5.1. Supervised Fine-Tuning:

Reasoning Data. For reasoning-related datasets, including those focused on mathematics, code competition problems, and logic puzzles, we generate the data by leveraging an internal DeepSeek-R1 model. Specifically, while the R1-generated data demonstrates strong accuracy, it suffers from issues such as overthinking, poor formatting, and excessive length. Our objective is to balance the high accuracy of R1-generated reasoning data and the clarity and conciseness of regularly formatted reasoning data.

In 5.4.1 they also talk about some ablation experiment by not using the "internal DeepSeek-R1" generated data.

While the "internal DeepSeek-R1" model is not explained, I would assume this is a DeepSeek V2 or V2.5 tuned for chain of thought. Therefore, it seems to me the "aha moment" is just promoting the behaviour that was already present in V3.

In the "Self-evolution Process of DeepSeek-R1-Zero"/ Figure 3 they claim reinforcement learning also leads to the model generating longer CoT sequences, but again, this comes from V3, they even mention the fine tuning with "internal R1" led to "excessive length".

None of the blogpost, news, articles I have read explaining or commenting on DeepSeek R1 takes this into account. The community is scrambling to re-implement the pipeline (see open-r1).

At this point, I feel like I took a crazy pill. Am I interpreting this completely wrong? Can someone shed some light on this?

nvtop• 1 year ago

I'm also very skeptical of the significance of this "aha moment". Even if they didn't include chain-of-thoughts to the base model's training data (unlikely), there are still plenty of it on the modern Internet. OpenAI released 800k of reasoning steps which are publicly available, github repositories, examples in CoT papers... It's definitely not a novel concept for a model, that it somehow discovered by its own.

tmnvdb5• 1 year ago

https://oatllm.notion.site/oat-zero

mike_hearn• 1 year ago

It's curious that the models switch between languages and that has to be trained out of them. I guess the ablations have been done already, but it makes me wonder if they do this because it's somehow easier to do some parts of the reasoning on languages other than English and maybe they should be just allowed to get on with it?

Dansvidania2• 1 year ago

Are reasoning models -basically- generating their own context? as in, if a user were to feed prompt + those reasoning tokens as a prompt to a non-reasoning model, would the effect be functionally similar?

I am sure this is improperly worded, I apologise.

aldanor• 1 year ago

Yes, more or less. Just like any LLM "generates its own context", during inference it doesn't care where the previous tokens came from. Inference doesn't have to change much, it's the training process that's different.

Dansvidania2• 1 year ago

thank you, that makes sense. Now it's time to really read the article to understand if the difference is the training data or the network topology to be different (although I lean towards the latter).

lysecret• 1 year ago

I think the next big problem we will run into with these line of reasoning models is "over-thinking" you can already start to see it. Thinking harder is not the universal pareto improvement everyone seems to think it is. (I understand the irony of using think 4 times here haha)

seydor• 1 year ago

Reasoning is about serially applying a set of premises over and over to come to conclusions. But some of our biggest problems require thinking outside the box, sometimes way outside it, and a few times ingeniously making up a whole new set of premises, seemingly ex nihilo (or via divine inspiration). We are still in very early stages of making thinking machines.

tpswa• 1 year ago

This is a natural next area of research. Nailing "adaptive compute" implies figuring out which problems to use more compute on, but I imagine this will get better as the RL does.

resource_waste• 1 year ago

100%

I do philosophy and it will take an exaggeration I give it, and call it fact.

The non reasoning models will call me out. lol

EncomLab• 1 year ago

Haven't we seen real life examples of this occurring in AI for medical imaging? Models trained on images of tumors state that tumors circled in purple ink or images of tumors that also include a visual scale are over identified as cancerous because they reason that both of those items indicate cancer due to the training data leading them that way?

efitz• 1 year ago

What everyone needs to understand about [reasoning] LLMs is that LLMs can’t reason.

https://arxiv.org/pdf/2410.05229

Understanding Reasoning LLMs

Related Stories