Alibaba releases 72B LLM with 32k context length

by logicchains • 3 years ago

55 84 3 years ago

Perhaps this is the famous "Alibaba Intelligence" that Jack Ma was mocked for in his discussion with Musk.

Does it still refuse to answer basic history facts (e.g. "What happened in 1989 on Tiananmen square?") like other LLMs from China?

A4ET8a8uTh010• 3 years ago

How is that any different from US neutering its LLMs to not discuss 'unsafe' topics?

qualifiedai5• 3 years ago

Obviously what is being "neutered" is important. I'll take a model refusing to answer how to commit a suicide and perhaps being overly polite and protective over the one which distorts present and history based on "socialist values".

WoahNoun5• 3 years ago

One is a company freely deciding what it's product should and shouldn't do and the other is forced by the government.

mensetmanusman4• 3 years ago

How are western values (historical anomaly) different than authoritarian values (historical norm)?

boppo1• 3 years ago

It's not. I dislike the US neutering too.

MuffinFlavored12• 3 years ago

Do you mean 1989?

qualifiedai5• 3 years ago

Sorry, yes :). You can try that question on "aligned" Baichuan2-chat model on HF.

brucethemoose2• 3 years ago

Yi 34B 200K is not like this. In fact, it seems to answer like it is Taiwanese, with way more fervor than base Llama.

nkingsy2• 3 years ago

Trained on the same number of tokens as the 14b, which was only a little more than the 7b.

Seems like they just ran out of good tokens?

MuffinFlavored12• 3 years ago

One thing I struggle to understand is, are LLMs just a large way of spitting out which token it thinks is best based on scoring? aka it's guessing at best?

How will they ever grow to the point where they are nothing more than just really good at guessing what to regurgitate based on what it's already seen?

Like when it spits out what would have been a StackOverflow answer (where the community votes if it is right/wrong and it's a living thing with comments to keep it up to date as it goes stale) but mixes up a few of the tokens making the whole thing mildly broken... are these systems just going to get really good? will they ever be trustable/perfect?

porphyra2• 3 years ago

One piece of evidence that LLMs form some sort of internal reasoning model is the stochastic parrot chess engine: https://parrotchess.com/

Despite it being statistically improbable for it to have seen every chess position before, it nonetheless plays very strongly.

emodendroket• 3 years ago

In the end it seems like a more sophisticated version of a Markov chain.

staunton3• 3 years ago

will they ever be trustable/perfect?

Humans aren't trustable or perfect either. Also, keep in mind that people strongly disagree on what "perfect" means...

lukev• 3 years ago

Not without extra machinery around the model itself. Token generation is always probabilistic to some degree.

mrb• 3 years ago

"what to regurgitate based on what it's already seen"

...which is no different than what humans do :-) Our neurons fire in a way only because they fire in patterns they have already seen before.

filterfiber5• 3 years ago

are LLMs just a large way of spitting out which token it thinks is best based on scoring? aka it's guessing at best?

They are 100% exactly this. More specifically they spit out a list of "logits" aka the probability of every token (llama has a vocab size of 32000, so you'll get 32000 probabilities).

Taking the highest probability is called "greedy sampling". Often you actually want to sample from either the top 5 (top k sampling), or the ones over say 90% (top p).

If you're doing things like programming or doing multiple choice questions, you can choose to only sample from those logits - for example, if the only outputs should be "true" or "false", ignore any token that isn't "t", "tr", "true", "f", "fa", "false", etc. This makes it adhere to a schema.

How will they ever grow to the point where they are nothing more than just really good at guessing what to regurgitate based on what it's already seen?

This is what humans do the big difference is our "weights" aren't frozen and we update them in realtime based on real world feedback. Once you loop in the real world feedback you get much better results, for example if a llm recommends a command with a syntax error, if you give it back the error message it can correct it. This is why training on synthetic data is possible.

will they ever be trustable/perfect?

This is why I hate the word "hallucinate", it's not a hallucination, it's a "miss-prediction". Humans do the exact same thing all the time - miss remembering, misspeaking, doing fast math, etc.

In many ways LLMs are already more "trustable" than inexperienced humans. The main thing you need is some mechanism of double-checking the output, the same as we do with humans. We can "trust" them once their probability of acceptable answers is high enough.

I_Am_Nous7• 3 years ago

guessing at best

Yes, but at the same time no. It's guessing based on math, so it is guessing the answer out of a filtered set of "likely correct" answers. The reason it's novel is because it can go so fast now that it seems conversational, but as nothing is free the trade off is to get better math, better probabilities, and better guesses, you have to keep expanding the size of the model which costs immense amounts of electricity for diminishing returns. There is also a limit to the amount of times you can "just make it bigger" before it actually performs more poorly, as I understand it.

LLMs feel like a shortcut because it is a shortcut compared to developing something that actually understands human language. We just train them on data so it knows what words usually comes next in a series. So if the training data is good, the output is more likely to be good. But it doesn't understand what the "correct" answer is and is entirely unaware of what the content it was trained on actually means. So they will likely never be perfect because they were designed to be "good enough". I can foresee them being a part of a greater AI in the future though.

nkingsy2• 3 years ago

From my interactions, models seem to be able to perform general purpose thinking. As in one, focused thought.

I have yet to encounter a situation where a capable llm hallucinates due to faulty logic, but that is from a perspective that the llms “knowledge” is a funny side effect from the process of making it able to think, and never to be trusted.

The chatty knowledge side effect is somehow obscuring the fact that we have a general purpose logic processor on our hands.

Building up reliable tasks on top of this is going to play out similarly to the computing revolution, but accelerated by huge global interest.

Comparing this to the internet boom is mistaken. The internet was the missing piece of the computing puzzle for achieving mass consumer appeal.

We haven’t hit that yet. We’re still thinking about these things as calculators and speed running mainframe deployments.

saurabh20n• 3 years ago

Summary from https://arxiv.org/pdf/2309.16609.pdf --- (q: how does one format lists on HN?)

* qwen-{1.8B,7B,14B}:

  * 3 trillion tokens; start with BPE tiktoken, cl100k base vocab, augmented with chinese, numbers split into digits, final vocab 152k.

  * RoPE - rotary positional embedding

  * context length 2048

  * qwen-14b perf percentages: 66.3 MMLU(5), 72.1 CEval(5), 61.8 GSM8K(8), 24.8 MATH(4), 32.3 HumanEval(0), 40.8 MBPP(3), 53.4 BBH(3); beats LLaMA2-13B on all, but behind LLaMA2-70B on all except CEval, MATH and HumanEval (somewhat surprising)

* code-qwen-{7B,14B}

  * additional 90B code tokens over base

  * context length 8192, flash attention

  * 14B perf: humaneval 66.4, mbpp 52.4; ok, but not stellar (similar numbers as OSS wizardcoder-py, and lower than gpt-3.5)

* math-qwn-{7B,14B}-chat

  * math instructional dataset

  * context length 1024

  * 14B perf: gsm8k 69.8, MATH 24.2, Math401 85.0, Math23K 78.4 (substantially better than OSS in the same weight class (WizardMath and GAIRMath-Abel) on MATH but same ballpark on GSM8k -- surprising). Math23K is chinese grade school math; and Math401 is arithmetic ability.

* comprehensive automatic evaluation in Appendix A.2.1 pg 36 (based on OpenCompass'23)

Alibaba releases 72B LLM with 32k context length

Related Stories