Paving the way to efficient architectures: StripedHyena-7B

https://www.together.ai/blog/stripedhyena-7b

by minimaxir • 3 years ago

221 65 3 years ago

together.ai

mmaunder• 3 years ago

Is the model available or is this just an API/app?

lelag• 3 years ago

Weights seem available at https://huggingface.co/togethercomputer/StripedHyena-Nous-7B....

SparkyMcUnicorn• 3 years ago

And here's the base model: https://huggingface.co/togethercomputer/StripedHyena-Hessian...

And GH repo: https://github.com/togethercomputer/stripedhyena

anon3738392• 3 years ago

This is a seriously impressive model.

kcorbitt• 3 years ago

For short context tasks looks like it's slightly stronger than Llama 7B and slightly weaker than Mistral 7B. Really impressive showing for a completely new architecture. I've also heard that it was trained on far fewer tokens than Mistral, so likely still room to grow.

Overall incredibly impressive work from the team at Together!

tempusalaria• 3 years ago

Did they disclose the training compute/token count?

stellaathena3• 3 years ago

Nope :(

bratao• 3 years ago

And this uses Hyena, that can be considered a "previous generation" of Mamba. I think that this anwsers the question about the scalability of SSM and the transformer finally found an opponent.

algo_trader• 3 years ago

H 7B is always faster than optimized Transformers (>10%, >20% and >50% end-to-end faster than FlashAttention

50% improvement on very large sequences is great. But it is not yet transformative (pun intended..)

Presumably FlashAttention already does lots of the less-than-quadratic improvement

ttul2• 3 years ago

Well, since Together did hire the author of Mamba…

skerit• 3 years ago

7B models are so exciting. So much is happening with those smaller models.

minimaxir2• 3 years ago

I wish they were a smidge smaller since 7B LLMs just barely run on a 16GB VRAM GPU (like a T4 server GPU) without quantization shenanigans.

Fortunately the learnings from finding better 7B models will trickle down, or more will be done with distillation (e.g. Gemini Nano)

3abiton• 3 years ago

What's wrong with quantization?

triyambakam2• 3 years ago

Can you explain for a noob why?

ttul2• 3 years ago

Quantization means reducing the number of bits used to encode each floating point number constituting a parameter in the model So instead of having billions of possible values per weight, you might have just 255. The model has to have its weights crammed into a much smaller number of possible values, which reduces its ability to produce good outputs.

orbital-decay• 3 years ago

Easier to train, easier to experiment with. Most research and prototyping happens on the scale that is just barely out of the "toy" category.

Metricon2• 3 years ago

BTW, for anyone who might not be aware of it, this model trained by Intel based on the Mistral architecture is probably the single best general 7B model available currently:

https://huggingface.co/Intel/neural-chat-7b-v3-2 (also see https://huggingface.co/Intel/neural-chat-7b-v3-1 from the previous version for more details)

It's licensed Apache 2.0 and unaligned (uncensored).

gardnr• 3 years ago

How is it better than the model from the team that made the dataset? https://huggingface.co/Open-Orca/Mistral-7B-SlimOrca

firejake308• 3 years ago

Darn, I was hoping the RWKV people had finally obtained reportable results. This is still interesting, though. Maybe we will see more alternatives to transformers soon

refulgentis3• 3 years ago

There are RWKV numbers in there, so you sort of got your wish sideways. :)

Has RWKV not released anything until now? I thought it was an open project that was in use, if sort of by a hipster 1%

stellaathena3• 3 years ago

Paper: https://arxiv.org/abs/2305.13048

Models (v4 are the ones from the paper): https://huggingface.co/RWKV

GitHub: https://github.com/BlinkDL/RWKV-LM

senseiV2• 3 years ago

V5 7b is out, close to hyena, gets 1400 t/s on a 3090, while an h100 llama 7b 8bit is 1200 t/s

stellaathena3• 3 years ago

RWKV had a paper accepted at EMNLP and released models which match the performance of equivalent transformers.

What else are you looking for?

senseiV2• 3 years ago

They Do, the latest rwkv v5, matches mamba at 3b scale, and from the benchmarks I see, its similar to hyena

goalonetwo2• 3 years ago

There seems to be a new model every single day. How do people have time to keep track with everything going on in AI?

hinkley• 3 years ago

From decades of observing at a distance and observing observers at a distance, I think it's safe to say that, like fusion, there are walls that AI run into, not unlike the risers on a staircase, and when we collectively hit one, there's a lot of scuttling back forth. A lot of movement, but no real progress. If that plateau goes on too long, excitement (and funding) dry up and things die down.

Then someone figures out how to get past the current plateau, and the whole process repeats. That could be new tech, a new architecture, or it could be old tech that was infeasible and had to wait for Moore's Law.

Right now we are on the vertical part of the sawtooth pattern. Everyone hopes this will be the time that takes us to infinity, but the old people are just waiting for people to crash into the new wall.

goalonetwo2• 3 years ago

Thanks for putting this so eloquently. That's exactly how I feel as well.

fvv• 3 years ago

Why things should dry up when contrary to fusion ai is already usable by millions daily ? Even if prpgress should stall a bit the product or fine-tunes or normal progress will still be super supeful , the "too soon" point has been surpassed

YetAnotherNick• 3 years ago

We know for a fact that there is no wall till GPT 4 and open source still has long way to reach there.

holoduke• 3 years ago

I know. The new reddit look sucks big. But the sub reddits still give you good insights in the latest developments. I am playing arround a lot with image related stuff arround stable diffusion. Comfyui subreddit gives me daily. Now after a few weeks i think i have a fair good understanding in what is hot. Checkpoints, ipadapters, facemodels etc. Just playing arround and you will get a grasp of it. I guess its similar with text generation.

tavavex• 3 years ago

FYI: if you want a faster experience, https://old.reddit.com/ still works perfectly

refulgentis3• 3 years ago

I honestly don't think I could if I still had a job not in it

benreesman• 3 years ago

I sorta keep huggingface open next to HN now. Follow like “TheBloke” and a few others and you’ll know what’s up.

ekianjo• 3 years ago

Thebloke just quantizises existing models what is the point of following him?

m3kw9• 3 years ago

You look at the comments here to see if it’s any different or getting rave reviews. Most model are crap, the new mistral one from yesterday seem to be pretty good. But most models are not very practical or useful for anything other than amusement right now. I imagine in a year we’d get close to gpt4 models locally and with low spec requirements where a 2 series NVidia card can run.

taneq• 3 years ago

Is almost like everything’s accelerating exponentially.

icyfox• 3 years ago

Common wisdom in most industries is to release bad PR announcements on a Friday and good ones towards the start of the week. It's interesting how the advent of twitter communication has shifted the ML ecosystem to publishing work whenever it's ready versus trying to find an optimal weekday. Or maybe they're optimizing for the weekend hackers that will take improvements released late in the week and put them into practice on Saturdays and Sundays.

refulgentis3• 3 years ago

It is interesting, been thinking about that myself a bit: some random associated thoughts: NeurIPS is next week, & there was OpenAI drama news that went under the radar due to Friday rule.

alsodumb• 3 years ago

NeurIPS is also starting this Sunday, I'm sure some of these releases are meant to get the word going during the conference.

mkesper• 3 years ago

Please stop using non-zero-based y axes. Every comparison chart in this post uses it, overemphasizing the difference between models.

Paving the way to efficient architectures: StripedHyena-7B

Related Stories