MiniMax-M1 open-weight, large-scale hybrid-attention reasoning model

https://github.com/MiniMax-AI/MiniMax-M1

by danboarder • 1 year ago

349 70 1 year ago

swyx• 1 year ago

1. this is apparently MiniMax's "launch week" - they did M1 on Monday and Hailuo 2 on Tuesday (https://news.smol.ai/issues/25-06-16-chinese-models). remains to be seen if they can keep up the pace of model releases for the rest of this week - these 2 were big ones, they aren't yet known for much else beyond llm and video models. just watch https://x.com/MiniMax__AI for announcements.

2. minimax m1's tech report is worthwhile: https://github.com/MiniMax-AI/MiniMax-M1/blob/main/MiniMax_M... while they may not be the SOTA open weights model, they do make some very big/notable claims on lightning attention and their GRPO variant (CISPO).

(im unaffiliated, just sharing what ive learned so far since no comments have been made here yet

behnamoh• 1 year ago

they did M1 on Monday and Hailuo 2 on Tuesday

It would've been fun to see them name their models like Apple chips: M1, M1 Pro, M1 Ultra.

plufz• 1 year ago

Yeah MiniMax M1 certainly directed my thoughts to Mac mini M1. :)

drewbitt• 1 year ago

They are also known for audio models, having the best TTS on some leaderboards (and my personal favorite) https://artificialanalysis.ai/text-to-speech/arena?tab=leade...

noelwelsh7• 1 year ago

A few thoughts:

* A Singapore based company, according to LinkedIn. There doesn't seem to be much of a barrier to entry to building a very good LLM.

* Open weight models + the development of Strix Halo / Ryzen AI Max makes me optimistic that running great LLMs locally will be relatively cheap in a few years.

rfoo• 1 year ago

A Singapore based company, according to LinkedIn

Nah, this is a Shanghai-based company.

manc_lad• 1 year ago

It seems more and more like an inevitability we will run models locally. Exciting and concerning implications.

If anyone has any suggestions of people thinking about this space they respect, I'd love to listen to more ideas and thoughts on the developments.

noelwelsh7• 1 year ago

I think the main limitation, right now, is hardware. For GPUs the main limit is the VRAM available on consumer models. CPUs have plenty of memory but don't have the bandwidth or vector compute power for LLMs. This is why I think the Strix Halo is so exciting: it has bandwidth + compute power plus a lot of memory. It's not quite where it needs to be to replace a dedicated GPU, but in a few iterations it could be.

I'm interested in other opinions. I'm no expert on this stuff.

pantulis• 1 year ago

Honest question: what is the concerning aspect to it?

psychoslave• 1 year ago

I don't know, what's worst with people running LLM locally compared to running any software locally?

There is nothing fundamentally new in having freedom in edge of societies. Yes it can lead to horrible situation, like someone kill neighbors, using the single handable bright new tool available to all. But that's far less of a concern than having the powerful new tool staying in full concentrated control of the greediest humans out there, who will gladly escalate any hindrance to genocide whenever something doesn't fit their perspective.

ulfw• 1 year ago

They are a Chinese company based out of the city of Shanghai, not Singapore.

They're also planning to IPO at HKEX in Hong Kong soon

https://www.scmp.com/tech/tech-trends/article/3314819/deepse...

7moritz73• 1 year ago

I'll keep an eye out on that ipo

thedudeabides5• 1 year ago

ask it what country taiwan is part of...

freeqaz• 1 year ago

In a linked Twitter post[0], they trained this for $500k-ish. I wonder how?

RL at unmatched efficiency: trained with just $534,700

0: https://x.com/MiniMax__AI/status/1934637031193514237

vintermann• 1 year ago

"We publicly release MiniMax-M1 at this https url" in the arxiv paper, and it isn't a link to an empty repo!

I like these people already.

npteljes5• 1 year ago

This is stated nowhere on the official pages, but it's a Chinese company.

https://en.wikipedia.org/wiki/MiniMax_(company)

iLoveOncall2• 1 year ago

Why would you expect them to mention that on their project's page?

noelwelsh7• 1 year ago

1. It's conventional to do so.

2. It's a legal requirement in some jurisdictions (e.g. https://www.gov.uk/running-a-limited-company/signs-stationer...)

3. It's useful for people who may be interested in applying for jobs

npteljes5• 1 year ago

Forget the project page, I couldn't find definitive information on any of the official pages.

They state HQ in Singapore on LinkedIn, and San Francisco elsewhere. Compared to this, it's outright disingenuous that they don't mention that they are a Chinese company.

As a layman, I'm mostly indifferent to this information.

If I were a project manager, this would be vital information. And the people running projects know this. So it begs the question: why not disclose, and why obscure it?

7moritz73• 1 year ago

Many people know that Minimax is Chinese because their video generator has a super obviously Chinese name (Hailuo), and that's what they've been known for so far

npteljes5• 1 year ago

It's best when they state it themselves, and we can also verify by third parties. Not stating it, or outright obscuring it is also information. The Hailuo name is definitely an indicator, but it could be Taiwanese or Singaporean as well, or just foreign branding, like Häagen-Dazs.

htrp• 1 year ago

They apparently building buzz for an IPO

https://www.bloomberg.com/news/articles/2025-06-18/alibaba-b...

markkitti• 1 year ago

Please come up with better names for these models. This sounds like the processor in my Mac Studio.

chvid• 1 year ago

https://en.wikipedia.org/wiki/Minimax

They named themselves after a classic ai algorithm.

JoeDaDude• 1 year ago

As best I can tell from a gloss-over read, it doesn't use anything like the Minimax algorithm. Astute readers are aware that one of the first applications of Minimax was in an AI chess program designed by Claude Shannon.

https://en.wikipedia.org/wiki/Claude_Shannon#Shannon's_compu...

badc0ffee• 1 year ago

But then there's the "M1" part.

diggan4• 1 year ago

Also sounds like my long lost dog whose name was Max but he was tiny. Absolutely horrible name, borderline criminal I say.

seydor• 1 year ago

Your Mac is made by 'Apple' and literally named after an apple cultivar

kridsdale12• 1 year ago

Is that like a pineapple that doesn’t grow on pine trees?

quesera• 1 year ago

... but the cultivar is named for a person. :)

reedlaw• 1 year ago

In case you're wondering what it takes to run it, the answer is 8x H200 141GB [1] which costs $250k [2].

1. https://github.com/MiniMax-AI/MiniMax-M1/issues/2#issuecomme...

2. https://www.ebay.com/itm/335830302628

incomingpain3• 1 year ago

That's full quantization. If you run Q4 or Q8 you can run this on <$10,000 equipment.

cma• 1 year ago

And if you add in heavy sparsification it should fit and run on a raspberry pi.

deadbabe• 1 year ago

No point in running anything but full quantization.

tgtweak• 1 year ago

My experience with heavily quantized models is they do better than a similar sized unquantized model but don't really perform anywhere near the pre-quantized model.

GTP• 1 year ago

How many parameters does this model have?

7moritz73• 1 year ago

456 bn, about 46 bn active at a time (it's moe)

kridsdale12• 1 year ago

Can’t you run it on a Mac Studio with 512GB? That’s about $8,500.

hu3• 1 year ago

It's also 1/20 of the speed. So not very useable.

b0a04gl• 1 year ago

if they trained this scale without western cloud infra, i'd want to know what their token throughput setup looks like

econ• 1 year ago

Sneakernet

jaggs• 1 year ago

They trained on 512 H800 GPUs for three weeks, equivalent to around half a million dollars. https://xcancel.com/MiniMax__AI

yorwba2• 1 year ago

That is for the reinforcement learning part. The base model was likely trained on more GPUs for significantly longer.

killerstorm• 1 year ago

"In our attention design, a transformer block with softmax attention follows every seven transnormer blocks (Qin et al., 2022a) with lightning attention."

Alright, so it's 87.5% linear attention + 12.5% full attention.

TBH I find the terminology around "linear attention" rather confusing.

"Softmax attention" is an information routing mechanism: when token `k` is being computed, it can receive information from tokens 1..k, but it has to be crammed through a channel of a fixed size.

"Linear attention", on the other hand, is just a 'register bank' of a fixed size available to each layer. It's not real attention, it's attention only in the sense it's compatible with layer-at-once computation.

MiniMax-M1 open-weight, large-scale hybrid-attention reasoning model

Related Stories