Mamba-3: Improved Sequence Modeling Using State Space Principles

by anentropic • 4 months ago

4 1 4 months ago

anentropic• 4 months ago

"Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks. At the 1.5B scale, Mamba-3 improves average downstream accuracy by 0.6 percentage points compared to the next best model (Gated DeltaNet), with Mamba-3's MIMO variant further improving accuracy by another 1.2 points for a total 1.8 point gain. Across state-size experiments, Mamba-3 achieves comparable perplexity to Mamba-2 despite using half of its predecessor's state size. Our evaluations demonstrate Mamba-3's ability to advance the performance-efficiency Pareto frontier."

It's apparently optimised for inference efficiency

Additionally: https://www.together.ai/blog/mamba-3

"Mamba-3 SISO beats Mamba-2, Gated DeltaNet, and even Llama-3.2-1B (Transformer) on prefill+decode latency across all sequence lengths at the 1.5B scale."

Mamba-3: Improved Sequence Modeling Using State Space Principles

Related Stories