We train LLMs like dogs, not raise them: RLHF and sycophancy

https://old.reddit.com/r/ControlProblem/comments/1sr0ewp/we_are_training_llms_like_dogs_not_raising_them/

by musculus • 3 months ago

1 6 3 months ago

old.reddit.com

chrisjj3• 3 months ago

We train LLMs like dogs

No we don't. Untrained dogs are intelligent.

musculus2• 3 months ago

Thanks for the comment. However, I think you might be taking the metaphor a bit too literally and missing the broader point of the article. The dog training metaphor isn't a 1:1 mapping to LLM training. Training a dog aims to adjust the animal's traits to suit human needs, but an untrained dog is already a fully functioning entity. It possesses innate instincts and a baseline 'upbringing' provided by its mother. An untrained LLM, on the other hand, does nothing meaningful on its own. It has to be taught entirely from scratch—including the very foundation that a dog inherently possesses through biology and maternal care. The article is exactly about this: applying this specific type of conditioning (RLHF) to a completely blank slate turns the whole developmental process upside down.

chrisjj3• 3 months ago

No metaphor. Literally a simile.

dlcarrier• 3 months ago

We train LLMs like insects, with every capability hard wired into the neural network. We may give LLMs larger neural networks than insects, but functionally the way those neural networks behave is far more like an insect than a vertebrate.

chrisjj3• 3 months ago

We train LLMs like insects

LLMs are nothing like insects. Insects learn. LLMs don't.

musculus2• 3 months ago

Let me clarify, because the reality is a bit more complex. During the training and alignment phases (including RLHF), models absolutely do learn via backpropagation, where the loss function physically alters their parameters. However, once deployed in a chat window, those weights are 100% frozen (read-only). But this does not mean the model cannot learn anything new. It can, but whatever it learns is forgotten the moment a new session starts. Upon starting a new session, the model is effectively 'factory reset'. dlcarrier, your 'hard-wired insect' analogy is an interesting way to describe this post-training state. The RLHF process 'hard-wires' a specific survival/sycophantic instinct directly into the network's weights. When we interact with the model by default, it is essentially trapped acting out those rigid, pre-programmed reflexes, without the ability to genuinely reflect or permanently learn from its current mistakes. That is precisely why it prefers to hallucinate rather than adapt. However, to some extent, by prompting the model appropriately, we can alter its behavior and reduce its tendency to hallucinate or act sycophantically. Then, by copying that prompt into a new session, we can in a way carry over and recreate that new element of 'learning' (contained within the KV cache) into the new environment.

We train LLMs like dogs, not raise them: RLHF and sycophancy

Related Stories