HN user

mk67

65 karma
Posts1
Comments53
View on HN
GPUs Go Brrr 2 years ago

Not easy to give a concise answer here, but let me try:

The problem mainly occurs in networks with recurrent connections or very deep architectures. In recurrent architectures this was solved via LSTMs with the signal gates. In very deep networks, e.g. ResNet, this was solved via residual connections, i.e. skip connections over layers. There were also other advances, such as replacing sigmoid activations with the simpler ReLU.

Transformers, which are the main architecture of modern LLMs, are highly parallel without any recurrence, i.e. at any layer you still have access to all the input tokens, whereas in an RNN you process one token at a time. To solve the potential problem due to "deepness" they also utilize skip connections.

GPUs Go Brrr 2 years ago

I'm in the industry and nobody does that since over ten years. There was just a small phase when Hinton published "Greedy layer-wise training of deep networks" in 2007 and people did it for a few years at most. But already with the rise of LSTMs in the 2010s this wasn't done anymore and now with transformers also not. Would you care to share how you reached your conclusion as it matches 0 of my experience over the last 15 years and we also train large-scale LLMs in our company. There's just not much point to it when gradients don't vanish.

Adding noise is generally helpful for regularization in ML. Most modern deep learning approaches do this in one way or the other - mostly dropout. It improves generalization capabilities of the model.

How would you get the money you possess and want to launder on the stolen credit card? That more sounds like a method to extract money from stolen credit cards, but not a valid way to launder money.

Falcon 180B 3 years ago

It's not possible to count letters for an LLM; it only "sees" tokens.

I Can Eat Glass 3 years ago

Seems pretty hard without teleportation as your mouth is closed when doing this mostly.