CodeMark | Uni-3DAR Unifies Micro and Macro 3D Worlds Through Autoregression, Outperforming Diffusion Models by 256% with 21.8x Faster Inference

• This article is republished from Synced.

From molecular and material structures in the microscopic world to geometry and spatial intelligence in the macroscopic world, creating and understanding 3D structures is a cornerstone of advancing scientific research. 3D structures carry rich physical and chemical information, providing scientists with essential tools for deconstructing complex systems, running simulations, and driving cross-disciplinary innovation. How to accurately and efficiently build 3D models and comprehend and generate 3D worlds is becoming a shared focus across three of AI's hottest fields: AGI, AI for Science, and Embodied Artificial Intelligence. As AI technology evolves, the powerful next-token prediction capabilities of large language models (LLMs) and large multimodal models (LMMs) are already being applied to create and understand 3D structures. This opens up new possibilities for AI for Science.

Recently, a groundbreaking model of this kind has emerged!

Called Uni-3DAR, from DeepWise, the Beijing Institute of AI for Science, and Peking University, it is a framework that unifies 3D structure generation and understanding through autoregressive next-token prediction. According to available information, Uni-3DAR is the world's first scientific model of its kind. Its author list is notably strong, including DeepWise AI Algorithm Lead Guolin Ke, Chinese Academy of Sciences Academician Weinan E, and DeepWise Founder & Chief Scientist and Beijing Institute of AI for Science President Linfeng Zhang, among others.

Guolin Ke shared on 𝕏 that the core of Uni-3DAR is a general-purpose coarse-to-fine tokenization method that converts 3D structures into one-dimensional token sequences.

• Paper title: Uni-3DAR: Unified 3D Generation and Understanding via Autoregression on Compressed Spatial Tokens
• Paper: https://arxiv.org/abs/2503.16278
• Project page: https://uni-3dar.github.io
• Code repository: https://github.com/dptech-corp/Uni-3DAR

Built on this general tokenization method, Uni-3DAR uses an autoregressive approach to unify generation and understanding tasks for 3D structures. Extensive experiments show that Uni-3DAR achieves state-of-the-art performance across multiple tasks including molecular generation, crystal structure generation and prediction, protein binding site prediction, molecular docking, and molecular pretraining. Particularly in generation tasks, compared to existing diffusion models, it delivers up to 256% relative performance improvement and 21.8× faster inference, fully validating the framework's effectiveness and efficiency. Moreover, the model applies not only to microscopic 3D molecules but also to macroscopic 3D tasks, demonstrating cross-scale capability.

Specifically, Uni-3DAR addresses two pain points in 3D structure modeling:

First, non-unified data representations. Current 3D structures employ multiple representation formats, with significant differences across scales. Macroscopic structures commonly use point clouds, meshes, and similar formats, while microscopic structures typically rely on atomic coordinates or graph structures. These representational differences lead to fundamentally different modeling approaches. Even at the same scale, different structure types (crystals, proteins, molecules) often use specialized representations and models due to varying data characteristics, making compatibility difficult. This fragmentation severely limits model generality and obstructs the possibility of building general-purpose foundation models that can leverage large-scale training data.

Second, non-unified modeling tasks. 3D structure-related tasks fall into generation and understanding categories, but they have developed independently. Generation tasks mostly rely on diffusion models, synthesizing stable structures from random noise step by step, while understanding tasks primarily use unsupervised pretraining methods. By contrast, LLMs have successfully unified generation and understanding through autoregression, yet this unified paradigm has seen little exploration in 3D structure modeling. If autoregressive methods can unify 3D task modeling, they could bridge the gap between understanding and generation, potentially incorporating 3D structures as a new modality into multimodal LLM paradigms — following images and video — and laying groundwork for building general-purpose multimodal scientific models oriented toward the physical world.

Uni-3DAR overall architecture

Below, we examine the two core technical contributions of this paper in detail.

Compressed Spatial Tokens: Unifying Microscopic and Macroscopic 3D Structures

3D structures exhibit significant sparsity at both microscopic (atoms, molecules, proteins) and macroscopic (object wholes, mechanical structures) levels: most space is empty, with important information concentrated in local regions. Traditional full voxel-grid representations consume enormous computational resources and fail to exploit this sparsity.

To address this, Uni-3DAR proposes a hierarchical, coarse-to-fine tokenization method that enables efficient compression and unified representation, applicable to both microscopic and macroscopic 3D structure modeling, providing a solid foundation for subsequent autoregressive generation and understanding tasks.

1. Hierarchical Octree Compression This method first uses octrees for lossless compression of the entire 3D space. Specifically, it starts with a large grid containing the whole structure, then recursively divides non-empty grids (regions containing atoms or other structural information) into eight equally-sized sub-units. After multiple levels of subdivision, a coarse-to-fine hierarchical structure forms, where tokens at each level record not only whether a region is empty but also preserve its spatial position information (determined by its level and grid center coordinates), providing explicit spatial priors for subsequent autoregressive generation.

2. Fine-Grained Structure Tokenization While octrees effectively compress empty regions, they only provide coarse spatial partitioning and cannot capture important information such as atom types, precise coordinates (in microscopic structures), or object surface details (in macroscopic structures). To address this, the team introduces the concept of "3D patches" in the final-level non-empty regions — analogous to 2D patches in image processing. By discretizing local structural details (using techniques such as vector quantization), continuous spatial information is converted into discrete tokens. This allows both microscopic-scale individual atom information and macroscopic-scale object surface details to be represented in the same format.

3. Secondary Subtree Compression Since even with octree structures the token count may remain large, the method further proposes secondary subtree compression. Specifically, it merges a parent node and its eight child nodes into a single token (using the parent node's fixed state and child nodes' binary features, combining into 256 possible states), reducing the total token count by roughly 8×. This significantly improves computational efficiency and enables efficient modeling of large-scale 3D structures.

In summary, this method fully exploits the inherent sparsity of 3D structures through octree decomposition, fine-grained tokenization, and secondary subtree compression. It substantially reduces data representation complexity while achieving unified representation from microscopic to macroscopic 3D structures, providing an efficient, general-purpose data foundation for subsequent autoregressive generation and understanding tasks.

Masked Next-Token Prediction: A Unified Autoregressive Framework for Generation and Understanding

In traditional autoregressive models, token positions are fixed — in text generation, for example, the i-th token is always followed by the (i+1)-th token, so the next token's position can be directly inferred without explicit modeling.

However, in the coarse-to-fine 3D tokenization proposed in this paper, tokens unfold dynamically with substantial position variation across samples; without explicit position information, autoregressive prediction becomes considerably more difficult. To address this, the paper proposes the Masked Next-Token Prediction strategy.

Specifically, the method duplicates each token to create two copies with identical position information, then replaces one copy with a [MASK] token. During autoregressive prediction, because the masked token and target token share exactly the same position information, the model can directly leverage this explicit positional information to predict the next token's content, more precisely capturing the next token's positional features and improving prediction quality. Although duplicating tokens doubles sequence length, experimental results show this strategy significantly boosts performance while only reducing inference speed by 15% to 30%.

Based on Masked Next-Token Prediction, the paper constructs a unified autoregressive framework enabling 3D structure generation and understanding tasks to proceed simultaneously within a single model.

Specifically, generation tasks (including single-frame and multi-frame generation) execute on masked tokens, using the autoregressive mechanism to progressively build structures; token-level understanding tasks (such as atom-level property prediction) rely on fine-grained structure tokens; and structure-level understanding tasks introduce a special [EoS] (End of Structure) token to capture global information about the overall structure.

This design keeps tokens corresponding to different tasks independent and non-interfering within the model, supporting joint training. Meanwhile, the autoregressive nature facilitates unifying other modality data (such as natural language text, protein sequences, instrument signals, etc.) into a single model, further enhancing generalization capability and practical utility.

Experimental Results

The paper designs a series of tasks in the microscopic 3D structure domain, including molecular generation, crystal structure generation and prediction, protein binding site prediction, protein-small molecule docking, and pretrained molecular property prediction.

Results show that in generation tasks, Uni-3DAR substantially outperforms diffusion model approaches; in unsupervised pretraining-based understanding tasks, its performance is roughly on par with bidirectional attention-based models. These results fully demonstrate that Uni-3DAR not only unifies different types of 3D structure data and tasks, but also achieves significant improvements in both effectiveness and speed.

3D small molecule generation task performance

Crystal structure prediction, and crystal structure resolution performance based on multimodal information (powder X-ray diffraction spectra)

Protein binding site prediction results

Protein-small molecule docking results

Pretrained small molecule property prediction results, where Uni-Mol and SpaceFormer are also specialized models proposed by DeepWise; Uni-3DAR surpasses Uni-Mol and is roughly on par with SpaceFormer

Pretrained polymer property prediction, where Uni-Mol and MMPolymer are also specialized models proposed by DeepWise; Uni-3DAR surpasses Uni-Mol and is roughly on par with MMPolymer

Future Outlook

Currently, Uni-3DAR's experiments focus primarily on microscopic structures, so further validation of its generality and scalability in macroscopic 3D structure tasks is urgently needed.

Additionally, to ensure fair comparison with prior work, Uni-3DAR currently uses independent training for each task. An important future direction is to integrate multiple data types and tasks, building and jointly training a larger-scale Uni-3DAR foundation model to further improve performance and generalization.

Meanwhile, Uni-3DAR has natural multimodal extension potential. Going forward, it can incorporate more modalities of information — such as protein sequences, amino acid composition, and even integration with large language models and scientific literature knowledge — to jointly train a multimodal scientific language model with physical world understanding capabilities, laying groundwork for building general-purpose scientific agents.