Ask HN: How training of LLM dedicated to code is different from LLM of “text”

https://news.ycombinator.com/item?id=37737149

by transformi • 3 years ago

2 3 3 years ago

news.ycombinator.com

Does it go beyond exposing to the LLM only "code"?, or there are extra steps in the training? (like giving the compiler/interpreter rules)? Since programming are more structured , I think that using grammar that are dedicated to those language might be useful.

sp3322• 3 years ago

The tokenizer might need tweaking. Base Llama models, for example, are trained on text that has had consecutive spaces reduced to a single space. This is unhelpful for coding where specific amounts of whitespace is at least very nice to have and can even be meaningful.

When you talk about a grammar, that's in the decoder, right? You don't need to retrain a model to use one of those.

transformi• 3 years ago

Regarding tokenizer- You are right in the context of python (but other language are more like pure "English" No?)

I'm talking about the "coding grammar" (like for python/CPP..), in order that when he write the code- it should gain benefit from using some syntax checking

sp3322• 3 years ago

Yeah, the LLM outputs a distribution of likely next tokens. It is up to the decoder to select one, and it can use a grammar to enforce certain rules on the output. https://github.com/Hellisotherpeople/Constrained-Text-Genera... or https://github.com/ggerganov/llama.cpp/blob/master/grammars/... for example.

Ask HN: How training of LLM dedicated to code is different from LLM of “text”

Related Stories