"MiniMax" Releases Trillion-Parameter MoE Model abab 6.5 | Oasis Capital Vitality

On April 17, 2024, MiniMax officially launched the abab 6.5 series of models. Back when MoE had yet to become an industry consensus, MiniMax bet more than 80% of its efforts on the architecture — and in January of this year, released abab 6, China's first MoE-based model. Over the past three months, we've pushed further to unlock the potential of the MoE architecture, developing the more powerful abab 6.5.

Along the way, we've found more and more ways to accelerate the realization of Scaling Laws, including improvements to model architecture, a rebuilt data pipeline, and optimizations to training algorithms and parallel training strategies. The abab 6.5 and abab 6.5s we're releasing today represent the fruits of this accelerated Scaling Laws process.

The abab 6.5 series comprises two models: abab 6.5 and abab 6.5s. abab 6.5 contains trillions of parameters and supports a context length of 200k tokens; abab 6.5s uses the same training techniques and data as abab 6.5, but is more efficient — it supports a 200k-token context length and can process nearly 30,000 Chinese characters in one second.

Across core capability benchmarks, abab 6.5 is beginning to approach the world's most advanced large language models, including GPT-4, Claude-3, and Gemini-1.5.

Core Capability Benchmarks

We tested both models using industry-standard open-source benchmark suites, comparing them against leading language models across knowledge, reasoning, math, coding, and instruction-following dimensions.

Scores marked with an asterisk were obtained through our own API testing; all other scores are from the corresponding technical reports.

We also conducted the industry-standard "needle in a haystack" test within a 200k-token context — inserting a single unrelated sentence (the "needle") into a very long text, then querying the model in natural language to see if it could accurately retrieve that needle. Across 891 test runs, abab 6.5 answered correctly every time.

abab 6.5 and abab 6.5s will be rolled out across MiniMax's product suite, including the productivity tool Hailuo AI and the MiniMax Open Platform. We welcome everyone to try them out.