How long can open-source LLMs truly promise on context length?

https://lmsys.org/blog/2023-06-29-longchat/

by dacheng2 • 3 years ago

189 62 3 years ago

When it rains, it pours.

We have the SuperHOT LORA and the associated paper, the Salesforce model, MPT, and this. Any other long context news I am missing?

ianbutler5• 3 years ago

Yup, someone supposedly improved superhot scaling already.

Here you go, https://www.reddit.com/r/LocalLLaMA/comments/14lz7j5/ntkawar...

brucethemoose27• 3 years ago

Ah yeah I confused rope with SuperHOT.

ijk3• 3 years ago

Turns out that dynamic NTK-Aware scaling further improves on that, with perplexity performance equal to or better than non-scaled at all context lengths: https://www.reddit.com/r/LocalLLaMA/comments/14mrgpr/dynamic...

ignoramous• 3 years ago

SuperHOT: https://kaiokendev.github.io/til

LoRA: https://news.ycombinator.com/item?id=35288015

The Salesforce Model (XGen): https://news.ycombinator.com/item?id=36514936

MPT (Mosaic): https://www.mosaicml.com/blog/mpt-30b

jrm4• 3 years ago

On one hand, I confess that I'm not familiar with exactly how the tech works, but this absolutely sounds like cope. I've literally never ever ever ever seen open source get beat by "proprietary" in this regard. We shall see.

anamexis• 3 years ago

In what regard?

1024core• 3 years ago

Why don't they evaluate GPT-4, like they evaluated GPT-3.5? And why not try longer context windows?

xrd• 3 years ago

Is the tl;dr: without sufficient context length, the conversation gets confusing, because the AI can't keep track of the full context of the conversation?

It's like being in conversation with an Alzheimer's patient, who keeps forgetting the last sentences?

So, open source models don't do well, while closed source models do?

And, this post documents a way to better measure that important metric?

brucethemoose27• 3 years ago

LLM's have no short term memory. Their internal state never changes.

When you have a conversation with one, you are actually feeding it the entire conversation every time you reply, usually with specific formatting the model is trained to "recognize" as a ongoing conversation, like its continuing a long story. There are some other formatting details, like the initial "instruction" always being preserved at the top as the old conversation gets truncated, and there is some caching done so it doesn't have to encode the entire conversation every time.

The context size is how much conversation you can feed them at once.

The popular open models were essentially limited to ~2K tokens until recently.

ftxbro• 3 years ago

Yes the entire conversation is fed every time. But also the web interfaces for example on ChatGPT or Bing do this automatically as part of their implementation detail, so the one above you is exactly right.

Der_Einzige2• 3 years ago

We already know how to do "short term memory". We can mathematically condense/average embedding representations of text and use these as input (i.e. combine a page of text into the representation space of one or several tokens, textual inversion style). Once embeddings get good enough, this kind of memory will be ubiquitous. Not sure why we are collectively having amnesia about sentence transformers and related ideas

It's worth noting that "soft-prompts" were a thing that the coomer crowd was already using with openassistant. Not sure why the mainstream NLP crowd is being so damn slow in implementing stuff like this

NLP is shocking undertooled, and I wrote a whole gist complaining about this that got to the front page of HN awhile ago: https://gist.github.com/Hellisotherpeople/45c619ee22aac6865c...

skissane4• 3 years ago

LLM's have no short term memory. Their internal state never changes.

Many have interpreted recent advances in LLM as a sign that strong AI is just around the corner. Yet, their static nature is an obvious blocker to ever achieving that – humans, even animals, learn continuously, unlike LLMs where training and runtime are two completely separate processes.

Which makes me wonder – how difficult would it be to have something like an LLM which continuously learnt just like humans do? Could it be done by modifying existing LLM architectures such as Transformers, or would it require a completely different architecture?

magicalhippo• 3 years ago

Why truncate the conversation? Why not have the LLM generate a summary, and use that instead of a truncated conversation?

That seems closer to what we do, we recall the important bits without every single word said.

regularfry2• 3 years ago

There's a hack here though, right? Run the LLM once to say "summarise this conversation in 500 words", then tack the new exchange on the end and run it again?

VMG• 3 years ago

Wow, somehow I have never heard about this before. In one sense, this makes LLMs even more impressive - the time it takes to consume a large amount of input seems very small.

ianbutler5• 3 years ago

I think this is more of an issue of smaller models with larger context aren't a magic bullet and I'm not sure anyone really paying attention thought they would be.

Falcon 40BN, after extensive use, is the only model that has crossed the "I'd build a product on this" threshold in open source. I expect my tune won't change there until we see larger Open Source models trained on high quality data. These are the models that will benefit from the extended context lengths.

But longer context without a model capable of utilizing the context to actually do things, is yeah, unsurprisingly, not useful.

There's also not really an incentive to currently make that happen in Open Source. If you can train a big model, with a unique high quality dataset you've assembled, using some novel techniques on some performant serving infrastructure, you basically have a blank check right now.

Edit: On my last point, I'm going to say I was being very jaded. I think the community is doing a lot of awesome things and I know of a few efforts personally to train these larger models for open source consumption. I don't mean to be so overly negative here.

simonw2• 3 years ago

It's pretty amazing to me that there are any open source models that have crossed that threshold - especially given that prior to the Llama release in March there were hardly any open source models that felt like they could catch up with GPT-3 at all.

If you're looking for an incentive to make this happen in open source, how about the team behind MPT-30B just getting acquired for $1.3 billion.

There are a LOT of well-funded groups trying to make an impact in open source models right now.

ianbutler5• 3 years ago

I'm highly skeptical that people are just going to leave the secret sauce out in the open for anyone to directly compete with them, feels a bit too idealistic to me.

If you look at Mosaic, their value, their secret sauce, is the infrastructure. The model, to say it cynically, was marketing. Now it's true it can be marketing and help push open source forward at the same time.

More of a separate point, but I also think models are going to be fairly quickly commoditized on a couple years timescale, so the infrastructure to train a model or serve a model, or the thing built on the model is the value, but right now models are still the value because they're so new and we haven't finished (in fact we're really only starting) the race to the bottom.

brucethemoose27• 3 years ago

Falcon was trained by a research think tank, and this is going to happen more and more as AI compute gets cheaper and the frameworks get better. We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long.

High quality "open" data is an tricky problem, but the community has already come up with some impressive datasets for finetunes, like the roleplaying collection used for Chronos and Pygmalion. I think this is going to get better too, epecially if cheap models can "distill" lower quality human datasets.

ianbutler5• 3 years ago

We aren't going to be stuck with overpriced A100s running early PyTorch eager mode code for long.

I agree and am eagerly awaiting this on the hardware side and trying to participate in this where I can on the software side.

KRAKRISMOTT2• 3 years ago

Falcon was trained by a research think tank

A research think tank subsidized by UAE’s oil wealth. Somebody get Masayoshi on the line, the Saudis must be hungry for deals for their vision fund.

PeterStuer• 3 years ago

The phi-1 paper suggests small models trained on higher quality data hold promise as well. If confirmed that would be great news for the potential accesibility of training open source models.

https://arxiv.org/abs/2306.11644

cypress66• 3 years ago

Really? I found falcon 40b underwhelming compared to llama 65b

19h• 3 years ago

Try falcon-40b-instruct, that’s optimized for better interaction with instructions and performs A LOT better

regularfry2• 3 years ago

Doesn't this imply some kind of scaling law? An LLM of complexity Y will only benefit from a context length of at most Z? That feels truthy to me.

brucethemoose27• 3 years ago

It dependends on the context (hah).

For, say, continuing a story, even a small LLM will benefit from a mega context even if it struggles to retrieve specific information... As long as perplexity isnt hit too much, which depends on the method.

But this is more of an issue for summarization or information retrieval.

ReactForAll• 3 years ago

LOL it was released like a few days ago, lets give it sometime maybe, but yes it's great to raise that these things should be tested better.

gtirloni• 3 years ago

Why was the title editorialized?

d4rkp4ttern• 3 years ago

I’m not sure I even understand what the title is saying.

shubhamgrg04• 3 years ago

Isn't condensing rotary embeddings just a clever hack to circumvent the original training limitations? Is it really a sustainable solution for extending context length in language models?

kaiokendev• 3 years ago

I am not sure how to parse this. Can you clarify?

bioemerl2• 3 years ago

Yes and no.

Yes, it actually works. That's what matters at the end of the day.

But no, eventually you're going to have to fine tune on something that has a larger context or training a brand new model with the position embeddings semi randomized so that it learns to generalize in the first place instead of needing hacks like this to function.

But training a model costs millions and millions of dollars, so we're going to have to wait until some very generous group decides to do all that training and then release that model open source.

Or releases the model file for a paid charge fee, I'd pay like $200 for a good one.

jpdus• 3 years ago

There is already another breakthrough on top of RoPE that promises even better scaling to larger contexts (based on NTK-awareness) - see my submission here:

https://news.ycombinator.com/item?id=36533816

The lmsys team is already looking into it, expect further progress soon.

It's really amazing to follow this progress in realtime, even if the pace can sometimes feel exhausting!

How long can open-source LLMs truly promise on context length?

Related Stories