MaHui | Sand.ai Releases World's First 100B+ Open-Source MoE Video Model

Today, MaHui member Sand.ai officially released and open-sourced MAGI-2 Preview — the world's first open-source, hundred-billion-parameter MoE video generation model, with approximately 114B total parameters and only about 6B activated per forward pass.

Open-source links:

https://github.com/SandAI-org/MAGI-2-preview

https://huggingface.co/sand-ai/MAGI-2-preview

With just 6B activated parameters, MAGI-2 Preview ranks sixth on the Artificial Analysis image-to-video generation leaderboard.

Based on current monthly rental rates for 8-card H100 servers, the inference cost to generate a 10-second 1080P video using the distilled model is approximately 0.5 RMB — roughly one-tenth the per-second cost of mainstream industry models.

MAGI-2 Preview sets out to answer two questions: how can video models scale stably to push capability ceilings higher, and how can training and inference costs be controlled as models grow larger?

Parameters, leaderboard rankings, and costs all ultimately come down to picture and sound. First, here are some videos generated by MAGI-2 Preview.

[Cinematic Video Scenes]

[Complex Motion and Camera Language]

[Product and Brand Advertising Scenes]

[Anime Scenes]

1

Unified Audio-Video: Letting the Shot Grow from Inside the Model

In the videos above, sound and image are generated in a single pass.

When a character speaks, the model simultaneously generates corresponding lip movements, facial expressions, and body gestures. When an impact sound occurs in the environment, objects in the frame must react at the same moment. When musical rhythm shifts, character performances and camera movements must adapt accordingly.

These fine-grained synchronizations determine whether a shot becomes a complete performance.

Traditional multi-stream approaches typically assign separate backbone networks to image and sound, connecting them through cross-attention or post-processing. Audio and video are modeled separately in the early stages of generation, making audio-visual alignment dependent on subsequent connections — the longer the chain, the more latency and error accumulate.

To address this, MAGI-2 Preview extends the single-stream architecture proposed in our daVinci-MagiHuman paper, Speed by Simplicity: A Single-Stream Architecture for Fast Audio-Video Generative Foundation Model. Text, video, and audio enter the same context and continuously exchange information through self-attention at every layer within the same Transformer.

The model also employs shared experts alongside modality-specific experts. Shared experts handle commonalities across the three modalities, while dedicated experts process information unique to text, video, and audio respectively. Capabilities for language understanding, character appearance, motion timing, speech, and ambient sound can all operate together within a single shot.

A unified backbone also shortens the generation pipeline, reducing interface overhead, waiting time, and maintenance costs from chaining multiple models. For advertising, short dramas, animation, and game content, a single generation can more closely approximate a complete shot containing image, sound, and performance simultaneously.

Bringing audio and video into one model also raises computational pressure. A video is split into numerous spatial patches and continuous frames, plus audio and text, causing sequence length and information volume to increase rapidly. How to make a 114B unified audio-video model actually run became the next challenge.

2

10-Second 1080P Video for Just 0.5 RMB

Based on current monthly rental rates for 8-card H100 servers, MAGI-2 Preview's distilled model can generate a 10-second 1080P video for approximately 0.5 RMB in inference cost.

The difference between a few mao and a few kuai may seem like merely a decimal point, but in real-world production, it changes cost structures and content creation workflows.

Generated videos rarely come out perfect on the first try. A character's gesture, a camera turn, even a pause in dialogue can affect the final result. Lower per-generation costs mean the same budget can test more prompts, adjust motion, composition, and style, and select usable shots from multiple versions.

Creative workflows will gradually shift toward multi-version generation, automatic filtering, and human curation. For content tools, advertising systems, game asset generation, and virtual human interactions, low costs also mean models can be called continuously, giving video generation the conditions to enter high-frequency applications.

Behind these cost figures lies MAGI-2 Preview's MoE architecture: approximately 114B total parameters, with only about 6B activated per forward pass.

3

114B Total Parameters, Only 6B Activated

As video generation evolves from short, silent clips toward longer, more complex unified audio-video content, models must simultaneously handle character and scene consistency, motion and camera changes, dialogue, ambient sound, and rhythm. Expanding capability boundaries requires increasing model capacity. When scaling dense models, every token passes through all parameters — the larger the model, the more computation needed for training and inference.

Video amplifies this problem further. Text models typically process hundreds or thousands of tokens; video must additionally accommodate continuous frames, numerous spatial patches, and audio information. Sequence length and modality complexity rise together, quickly pushing up computation and cross-device communication overhead.

MoE addresses this by maintaining a large pool of experts within the model, where each token selects only a small subset relevant to the current task. The model can thus continue expanding capacity while controlling the number of parameters actually invoked.

MAGI-2 Preview uses approximately 114B total parameters to encompass richer information about characters, motion, camera work, materials, semantics, speech, and ambient sound, while using approximately 6B activated parameters to control computation per generation.

To enable finer expert specialization, MAGI-2 Preview splits 3072-dimensional hidden representations into 12 heads of 256 dimensions each, employing Multi-Head MoE across 36 layers of the main network.

Each head has 256 experts, selecting 6 per operation. Each layer thus forms 3072 head-local expert units, with a token activating 72 small experts across 12 heads. We call this structure Ultra-fine-grained MoE.

Traditional MoE typically routes based on a token's complete hidden representation, requiring a few experts to handle multiple feature types simultaneously. Multi-Head MoE first distributes representations into multiple low-dimensional subspaces, then lets each head independently select experts.

Different heads can thus develop finer specializations — handling character appearance, motion timing, language semantics, speech, or ambient sound separately. The 12 heads' results are later recombined in subsequent layers, allowing the model to access richer expert combinations while keeping each expert's task scope limited.

As expert counts increase, a new problem emerges: these experts are distributed across different devices. How should tokens be routed to them?

4

The MoE Challenge for Video Generation Models Hides in Communication

Traditional Expert Parallel first selects Top-K experts for each token, then sends tokens to the devices where those experts reside.

Routing results vary dynamically with input. One GPU may suddenly receive a flood of tokens while another sits idle; the more experts activated, the more data typically needs to transfer between devices. At video-scale sequence lengths, communication and load fluctuations quickly become bottlenecks.

MAGI-2 Preview, inspired by Multi-Head LatentMoE and Head Parallel, introduces Head Parallel to change the communication order.

The model first splits hidden representations into multiple heads, distributing fixed-shape head representations to different devices before dynamic routing occurs. Once data reaches its device, experts are selected and executed locally.

Traditional Expert Parallel's communication volume depends on Top-K results from each routing decision. Head Parallel's primary communication is determined by input representations, avoiding linear growth in communication with the number of activated experts. This reordering creates more room for fine-grained experts — the model can accommodate more experts and improve routing precision while keeping cross-device traffic relatively stable.

The architecture is only half complete at this point. The other half depends on systems that can actually run it.

5

Behind Hundred-Billion Parameters Lies a Self-Developed Infrastructure

Multi-Head MoE imposes new requirements on infrastructure. Thousands of fine-grained experts bring frequent routing, data reshuffling, and small-matrix computation. Using general-purpose MoE systems directly, the added model capacity can easily be offset by communication and scheduling overhead.

To address this, we developed MagiMoE, Head Parallel, and the distributed optimizer MagiMuon around MAGI-2 Preview, covering expert computation, cross-node communication, and stable training for hundred-billion-parameter models.

MagiMoE is our self-developed high-performance MoE kernel library. It fuses expert routing, sorting, and computation steps to reduce intermediate results and memory movement, and optimizes forward and backward passes for the numerous small-matrix computations in Multi-Head MoE. It also supports different kernel backends and low-precision paths, leaving room for future hardware adaptation and model iteration.

Head Parallel handles cross-node communication. Tensor parallelism, context parallelism, expert parallelism, and parameter sharding jointly scale the main network and hundred-billion parameters. Multiple parallel strategies must cooperate within one system — any prolonged wait in one link leaves large numbers of GPUs idle.

For training optimization, distributed MagiMuon uses Muon for main matrix parameters and AdamW for parameters better suited to conventional updates, re-adapting parameter organization and cross-device computation for massive fine-grained experts. This hybrid design lets different parameter types use appropriate update methods, and allows optimization approaches to scale stably from small-to-medium experiments to hundred-billion-parameter MoE training.

MagiMoE, Head Parallel, and MagiMuon together form MAGI-2 Preview's infrastructure. Scaling video models has never been purely about parameter increases — what's harder to scale is communication efficiency, computation efficiency, numerical stability, and training monitoring.

MAGI-2 Preview's value lies in validating both model design and systems capability within the same scaling training run.

From "Clean-First" to "Coverage-First": Scaling Requires Rethinking Data

As model scale grows, data pipelines must evolve too. The industry has often understood data quality as layered filtering: removing low-quality footage, poorly composed shots, or atypical motion, leaving a cleaner dataset.

Layered filtering reduces noise and improves training stability, but may also eliminate complex motion, unusual camera work, rare subjects, atypical sounds, and long-tail combinations. Real-world diversity often hides in this "non-standard" data.

Therefore, in MAGI-2 Preview's training, we emphasize data scale, diversity, and distribution coverage more strongly. The focus of data work shifts from deletion to organization: through more accurate, finer-grained annotations, helping the model understand subjects, motion, scenes, camera work, timing, and relationships between sound, image, and text.

Pre-training covers the data distribution as completely as possible; post-training focuses on preference, safety, controllability, and product adaptation. The more thoroughly the base model learns, the less pressure on product-side modules to patch motion consistency, image detail, and audio-visual relationships.

For video models, scaling is the joint expansion of model, data, and systems. Any bottleneck prevents parameter scale from stably translating into generation capability.

MAGI-2 Preview validates a clear technical path: using a unified single-stream architecture to jointly generate image and sound, using Ultra-fine-grained MoE to expand model capacity, then controlling communication and computation costs through Head Parallel and self-developed infrastructure.

Going forward, we will continue expanding model and data scale, exploring longer-duration video generation, further improving generation quality, audio-visual synchronization, and long-term consistency, while continuously reducing per-unit generation costs.