InternLM2

by milliondreams • 2 years ago

136 24 2 years ago

TLDR; 1. InternLM2 is an open-source Large Language Model that has shown improvements over previous models, particularly in long-context modeling. 2. The model uses a unique approach, combining traditional training with Supervised Fine-Tuning and Conditional Online Reinforcement Learning from Human Feedback. 3. It offers a variety of model sizes and training stages to the community, demonstrating significant advancements in AI research and application.

jerpint• 2 years ago

Excited to see how it will perform on the lmsys leaderboard

ilaksh• 2 years ago

Does anyone know how the free commercial license works? Do they usually grant it? https://wj.qq.com/s2/12727483/5dba/ looks like a form there.

Apache 2 code, free commercial license with application form for weights.

esha_manideep• 2 years ago

Pretty amazing to see training data being discussed more openly

WiSaGaN• 2 years ago

Indeed. I think part of the reason when they are not discussed openly may be that much of the data used is copyrighted, which introduces some legal ambiguities.

YetAnotherNick• 2 years ago

IANAL but hiding something doesn't make someone legally immune. Any company could sue LLM companies and they can't hide it during the case. e.g. there is already a similar case on OpenAI.

dannyw• 2 years ago

How good is the base (non-instruction-tuned) model? Everyone is trying to make chat bots, but for my use cases, I find base models more suitable.

fragmede• 2 years ago

Interesting. What are some of those use cases?

viraptor• 2 years ago

The repo is here: https://github.com/InternLM/InternLM

zone4112• 2 years ago

We really need better long context benchmarks than needle-in-a-haystack. There is LV-Eval (https://arxiv.org/abs/2402.05136) with multi-hop QA that's better but still pretty basic.

andersa• 2 years ago

Yes, I don't understand why they are using a search benchmark for these... it would be much better to have something like giving it a story up to the context length (from a book? how to find one that's not in the training data?) and have it write a new chapter/ending that is consistent with all prior text and introduces zero inconsistencies.

But how can you automatically evaluate whether it did this?

ricw• 2 years ago

Because that’s how people use llms. You go to ChatGPT to ask a question and get an answer, rather than searching on Google, revising your search because you didn’t know a term, and then look at 3-5 different links to find the answer to what you were searching for

s-macke• 2 years ago

We got RULER 3 days ago.

https://arxiv.org/abs/2404.06654

https://github.com/hsiehjackson/RULER

barsonme• 2 years ago

Is it normal for papers to have that many authors?

esafak• 2 years ago

Only in fields like foundation models and high-energy physics, where immense resources are required. Look at GPT-4's credits: https://openai.com/contributions/gpt-4

barkingcat• 2 years ago

It's not abnormal in many fields. A lot of biology or physics papers have more than that number.

In academic labs, so you're a postgrad working the overnight shift watching some petri dish to make sure the bacteria doesn't die, etc. It's super boring grunt work but you do it so you get on the paper's author list.

exe34• 2 years ago

Particle physics papers usually have more pages for the names than for the work.

m3kw9• 2 years ago

Anyone that even sniffed it can get in on the action

pilotneko• 2 years ago

I experimented with this model and vLLM around a month ago. The long context length is attractive, but it was incredibly slow on a g5.12xlarge (4 NVIDIA A10G GPUs). I actually could not get it to respond for single examples longer than 50K tokens.

Kwpolska• 2 years ago

The name suggests this is interns posing as a chatbot, especially considering today’s date.

InternLM2

Related Stories