Yunqi Capital | Angel Portfolio Company MiniMax Releases Trillion-Parameter MoE Model, Closing In on World's Most Advanced LLMs
Betting on MoE models before the industry consensus caught up.
Compute has always been one of the biggest challenges facing large AI models.
In our annual review, we observed the industry experimenting with multiple approaches to solve this, including MoE (Mixture of Experts) architectures, vector databases, and dedicated AI chips.
Long before MoE became an industry consensus, MiniMax — a leading Chinese foundation model company and Yunqi Capital angel-round portfolio company — had already committed over 80% of its compute and R&D resources to MoE. It has now achieved阶段性领先成果.
In January this year, MiniMax released China's first MoE large model, abab 6, and continued iterating to release the more powerful abab 6.5 series, with core capabilities beginning to approach the world's most advanced large language models.
This edition of "Yunqi Partners" brings you MiniMax's latest progress on MoE models. Enjoy.
➤➤➤ Today, MiniMax officially launches the abab 6.5 series. Building on the foundation of abab 6, the company has further tapped the potential of the MoE architecture to develop the more powerful abab 6.5.
Along the way, MiniMax has discovered increasingly more pathways to accelerate the realization of Scaling Laws, including improvements to model architecture, reconstruction of data pipelines, training algorithm optimizations, and parallel training strategy refinements. The abab 6.5 and abab 6.5s released today represent the company's 阶段性成果 in accelerating the Scaling Laws process.
The abab 6.5 series comprises two models: abab 6.5 and abab 6.5s. abab 6.5 contains trillions of parameters and supports a context length of 200k tokens; abab 6.5s uses the same training techniques and data as abab 6.5 but is more efficient, supporting a 200k-token context length and capable of processing nearly 30,000 Chinese characters in one second.
Across various core capability benchmarks, abab 6.5 is beginning to approach the world's most advanced large language models such as GPT-4, Claude-3, and Gemini-1.5.
Both abab 6.5 and abab 6.5s will be rolled out progressively across MiniMax's product suite, including the productivity app Hailuo AI and the MiniMax Open Platform. Welcome to try them out 👏
Currently, the MiniMax Open Platform serves over 20,000 enterprise and individual developers, spanning more than ten industry scenarios including office collaboration, interactive entertainment, customer service, search, and education. It has established partnerships with Tencent, Kingsoft Office, China Literature, Xiaohongshu, Gaoji Health, DiDi, Meituan, and Xiaomi.
At the same time, MiniMax operates on a "dual-wheel drive" strategy and is the earliest, most prolific, and most invested Chinese large-model startup when it comes to building products. Its first product, Glow, launched in October 2022, followed by at least four more products including STARFIELD and Hailuo AI — spanning both companion-style social entertainment apps and productivity tools like Q&A. Multiple apps have surpassed 1 million DAU.
Core Capability Benchmarks
The two models were tested using industry-standard open-source benchmark datasets, comparing them against leading language models across dimensions including knowledge, reasoning, mathematics, coding, and instruction following.
Results marked with asterisks were obtained by MiniMax through API testing; all other scores are from the respective technical reports.
Within the 200k token range, the company conducted the industry-standard "needle in a haystack" test — inserting a single unrelated sentence (the "needle") into a very long text, then querying the model in natural language to see if it could accurately retrieve that needle. Across 891 tests, abab 6.5 answered correctly every time.
💡 P.S. The company is recruiting more believers in AGI to co-create intelligence with users. Click for details.
