Code View | The Sora Moment for 3D AIGC May Happen in China
From text-to-text, represented by ChatGPT, to text-to-image, represented by Midjourney, to text-to-video, represented by Sora, and text-to-music, represented by Suno — the "aesthetics of brute force" keeps pushing the ceiling of content production, and multimodality has become the consensus trajectory. What's next? Within human line of sight, there aren't many answers to choose from, and more people are willing to bet on 3D.
It represents the sustained anticipation from the metaverse, XR devices, and Apple Vision Pro — about our relentless pursuit of content consumption experiences, about our imagination of humanity's future life.
Yet the 3D track also seems to be a domain not yet covered by tech giants, with no foundation model in place. Everything remains chaotic, still developing rapidly, but a horse race for the future has already begun: Ship products or refine technology? Iterate continuously or deliver industrial standards? Start from images or go 3D-native? Driven by these questions, 3D technology has iterated at breakneck speed over the past year, and a shakeout has already begun.
This article attempts to map the technical development path of the 3D field, and through different players' products and market judgments, tries to answer the questions above.
What's interesting is that the most valuable work in 3D currently seems to be done mostly by Chinese researchers. One team told us bluntly: "The Sora moment for image-to-3D will happen in China." That's an exhilarating prediction.
1
The Dimension-Ascending Path Starting from Google's DreamFusion:
Simulating 3D Models from 2D Image Generation
Unlike images and video, people rarely encounter true 3D information through pure visual perception in daily life.
Because of binocular imaging and stereoscopic vision mechanisms, the human brain is remarkably easy to fool. We can turn 2D information into 3D sensations through various means — the videos brought by Sora recently, panoramic images, dome films, 3D movies, and glasses-free 3D all work this way.
Moreover, 3D assets are far more complex than 2D. For 2D AIGC, there's basically only one choice: generate pixels. But models, textures, rigs, (keyframe) animations, and more can all count as parts of 3D assets. And even for the most mainstream 3D asset — the 3D model — its representations are divided into Mesh, Voxel, point cloud, SDF, NeRF, and so on.
Once you consider actual deployment into rendering pipelines, due to foundational constraints built up over years of industry development, there's basically only one mainstream representation to choose: Mesh. A basic Mesh looks roughly like the image below. Hundreds or thousands of quadrilateral or triangular faces together construct every angle of a three-dimensional object.
Image from the Polygon mesh Wikipedia page
In non-AI-generated cases, such 3D models can be created through scanning or manually by modelers. Later, external material textures are applied, producing a dolphin that can be used in games — either in a blue cartoon style or a gray realistic style.
Although 3D models belong to three-dimensional models, when we observe them, we can still view them as 2D images from any face — this was also the route people considered when first attempting AI generation of 3D models. This approach is called the "2D dimension-ascending school."
On September 29, 2022, Google released the text-to-3D technology DreamFusion, using a pretrained 2D text-to-image diffusion model to complete open-domain text-to-3D synthesis without 3D data for the first time.
In between, it borrowed a representation called NeRF, short for Neural Radiance Field.
A radiance field is the term describing how light emits or scatters from surfaces in a 3D scene. NeRF achieves error feedback by analyzing the radiance field residual between model outputs and target data (such as 3D scan results) during model training, allowing the model to iteratively optimize predictions and thereby improve its ability to generate precise 3D representations.
In Google's 3D generation method, the model never saw 3D data — it saw enough 2D data.
In fact, the human eye can only see 2D images too. But when a person observes 2D images of an object from multiple angles, they can become aware of the object's actual 3D state.
Image source: Mildenhall, Ben, et al. "Nerf: Representing scenes as neural radiance fields for view synthesis." Communications of the ACM 65.1 (2021): 99-106. From a certain perspective, NeRF can be understood as an intermediate state from 2D ascending to 3D representation. Once a person has seen enough angles of an object, they can take a lump of clay and sculpt out a 3D fitted state of that object — that clay is NeRF. After Google introduced this approach, 3D generation for a long time afterward followed similar routes, distilling 3D information from 2D pretrained models.
Among the more influential works were NVIDIA's AI tool Magic3D, launched shortly after Google. Though similar in technical approach, it made further improvements that enhanced quality.
The Best Paper at SIGGRAPH 2023 proposed a new representation that could replace NeRF — 3DGS (3D Gaussian Splatting). It surpassed NeRF in rendering speed, generation quality, and other aspects, and showed significant value in 3D reconstruction, though it couldn't convert to high-quality mesh. And since the entire 3D industry has developed with mesh as its foundational unit, with software, hardware, and engines only able to edit mesh, this created certain constraints.
2
How to Solve the Multi-Head Problem?
Foundation Models Based on 3D-Native Data
Ascending from 2D to 3D, large models, like humans, observe enough flat planes to become aware of their 3D state.
From every angle — that is, on every 2D plane — we can see a complete cross-section of the object. But when this lump of clay is sculpted into a 3D state this way, the biggest problem with this method is that AI-generated 3D objects have multiple heads or multiple faces. The multi-head problem is also known as the Janus Problem.
The Janus Problem arising from ProlificDreamer's lack of 3D shape consistency. On the left is a seemingly normal front view of a blue bird. On the right is a confusing image depicting a bird with two faces. To be practically applicable to content production, an ideal 3D generation model should meet the following requirements: (a) capable of generating shapes with geometric detail and arbitrary topology. (b) output textured meshes, the primary representation used by standard graphics software like Blender and Maya.
3D models must have consistency in object shape from different viewing angles to avoid situations where every front and side view looks normal but the whole becomes abnormal.
Is it possible to directly train 3D foundation models with 3D-native data?
Before OpenAI's Sora emerged, people generally borrowed the more mature text-to-image approach, hoping to use the method of generating 2D images plus some dimension-ascending work to complete video generation — and indeed achieved some results.
Sora's arrival almost overturned all previous work — as long as sufficient complete videos were fed into the Transformer architecture for training, the intelligence that emerged from large models far exceeded what humans could patch together for it. The large model learned much from the original training data on its own, achieving massive improvements in generated video length and coherent multi-angle shooting.
As Sora's technical revolution demonstrated, although image generation is a more mature technical domain and people are accustomed to deriving other forms from it, directly training on the asset to be generated actually allows models to gain greater benefits.
For Sora, directly training on video greatly improved inter-frame consistency within the video. Video generation based on image generation, because training lacked sufficient temporal information constraints, often produced flickering in generated videos, couldn't ensure consistency with basic physical laws, and couldn't guarantee video length. Directly training with video as the unit, though requiring greater computational resources, directly circumvented this problem.
The "3D-native school" trains directly on 3D datasets like ShapeNet, with both training and inference based on 3D data.
At NeurIPS 2016, 3D-GAN was proposed — the 3D version of GAN, one of the dominant models in computer vision — generating 3D models in voxel units.
3D GAN principle and generation results In 2022, NVIDIA released the GET3D architecture: its core is through differentiable surface modeling, differentiable rendering, and 2D generative adversarial networks, training models from collections of 2D images to directly generate explicit textured 3D meshes with complex topology, rich geometric detail, and high-fidelity textures.
GET3D generation models (Source: GET3D paper official site https://nv-tlabs.github.io/GET3D/) In May 2023, OpenAI launched the 3D-native data trained text-to-3D model Shap·E, but its results still fell short of the production-ready standard pursued by the 3D field.
Training large models with 3D-native data, the problem is obvious: 3D datasets (ShapeNet with 51K models, Objaverse with 800+K, commercial model site SketchFab with 5M) differ from 2D LAION's 5B scale by at least three orders of magnitude, and due to the natural scarcity of 3D data and collection difficulty, achieving data diversity is also hard. Data limitations have led researchers to generally believe it's difficult to train sufficiently good large models on this basis — effective in specific domains but with limited generalization.
Is there a way to enhance model generalization? Current team practice involves attempting to encode the 3D forms of all things into a single representation, i.e., tokenization, then feeding into model training. According to reports, combining this with the same Diffusion Transformer architecture as Sora, native 3D large models without any 2D priors have already shown good generalization performance.
Large model CLAY
3 Does the Data Problem Have to Be Solved?
New Progress in 2D Dimension-Ascending
In July 2023, researchers from UCSD and other institutions released One-2-3-45, breaking away from the per-object optimization generation paradigm, capable of transforming any single 2D image into 3D in 45 seconds.
First, they use a special model (a view-conditioned 2D diffusion model) to generate multiple 2D images from different angles from the original 2D image. This is like looking at the same object from different perspectives. Then, they use an SDF (signed distance function)-based neural network model to build a 3D model from these multi-view images. This is like "stacking" these 2D images from different angles to form a complete 3D shape.
Large amounts of 3D AIGC work emerging over the past year
Compared to other existing 2D dimension-ascending methods, this approach has several advantages:
- Fast speed: it can generate 3D models in a very short time, much faster than other methods.
- Good geometry: the shape of generated 3D models is closer to real objects' shapes.
- Good 3D consistency: the appearance of generated 3D models is more consistent from different viewing angles.
- Closely follows input image: the generated 3D model more accurately reflects the original 2D image's content.
Compared to methods relying purely on 3D-native data, this approach leverages 2D models' prior knowledge for stronger generalization, capable of generating more types and more diverse 3D models without being constrained by limited 3D training data.
Inspired by this, large amounts of academic work began following this new generation paradigm. Recently, much research has focused on using 2D diffusion models to generate consistent multi-view images, or training 3D networks to generate 3D models from multi-view images.
The One-2-3-45 team also released new work One-2-3-45++ in November 2023, achieving higher-precision geometry and texture generation.
Image source: https://sudo-ai-3d.github.io/One2345plus_page/
In November 2023, Adobe Research and the Australian National University (ANU) jointly announced a new algorithm for their AI model, using a highly scalable neural network called LRM (Large Reconstruction Model), containing one million datasets and five hundred million parameters, covering various data including images, 3D shapes, and video.
LRM achieves fast and accurate reconstruction from single images to 3D models through the combination of large-scale data and high-capacity models.
Project page: https://yiconghong.me/LRM/
These two studies drove technical upgrades for the 2D dimension-ascending school. Joint training models on 2D and 3D circumvented the 3D-native data problem.
4 The Distance to Production-Ready
Large model capabilities have advanced rapidly over the past year or two, and one frequently criticized point is that while model capabilities have developed at astonishing speed, demonstrating remarkable potential, the commercialization of AI applications hasn't happened as fast as people imagined.
Many applications, like text-to-image, though with outstanding results and lively C-end user engagement, haven't penetrated professional production pipelines to generate value very quickly.
If this scenario were replicated in the 3D generation field, it could be fatal for startups.
Currently, B-end 3D model usage is mainly divided into two major categories: one is industrial design. 3D models directly serve design and production, and in such industrial workflows, requirements for 3D models in various aspects need to be very precise. Because once produced, they really need to fit together seamlessly.
All AI generation technologies currently struggle with precise generation control. With current 3D generation technology, reaching usability in this aspect is still quite difficult, and commercialization remains relatively distant.
The other major category is artistic creation, including modeling of game characters, backgrounds, etc. In this field, precision requirements are lower relative to production, making it the more likely route for commercialization currently.
However, with current 3D model generation precision, directly generating finished products to replace stable manual generation is still quite difficult. Whether model surface polygon design is smooth and symmetrical, whether models can have drivable capabilities, etc. are all obstacles preventing current 3D models from directly providing finished products.
And can 3D model generation services enter game design pipelines?
It seems possible currently, but still faces difficulties.
In normal game modeling, human hair, heads, clothes, and shoes are all modeled separately, but current algorithms still struggle with this. Moreover, the quality of AI-generated 3D models actually still has a considerable gap from professional modeler standards. How much value does AI introduction to production pipelines really bring?
In game production, one currently feasible approach is providing assets for game and animation background filling. Multiple teams have already begun relevant cooperation with film, television, and game companies.
In an animation scene, besides main character modeling, there is also extensive background object modeling. For example, a protagonist might stand before a table covered with various bottles and jars. Each of these bottles and jars actually needs modeling, and if all done manually, it's extremely time-consuming.
In specific cooperation, 3D modeling teams directly photographed many animation-needed materials from museums, generated corresponding 3D models with models, and charged for API calls to their models.
But whether such business models can scale still depends on technology maturity. The 3D generation technical route is still far from converging to a single technical path.
The overall foundation model level, analogized to image generation, may still be at the Midjourney V3-V4 state, so model competition will continue for at least another year. After that may come competition in application-side packaging and personalized content generation.
How to tokenize 3D data, the size of model training data, and post-generation material generation are all issues that 3D model generation needs to compete on.
However, precisely because of this, the technology gap between domestic and international AI 3D models is not large.
5 What Is 3D Actually Good For?
Human content consumption has evolved through text, images, music, short video, film, and many other content forms. Yet today, if you have 30 minutes, would you choose to open Douyin, or open Honor of Kings, or open a book?
Regardless of right or wrong, good or bad, mainstream human content consumption has always moved toward higher information density per unit time. In the same amount of time, short video or games have far greater information density than a book.
If we're chasing a content form with even higher unit information density, that might be 3D — this is why the metaverse and XR once so excited the market.
How do ordinary people perceive 3D content visually? Perhaps we need interaction with 3D information beyond vision, and this depends on terminal device transformation. With mobile phones as the primary terminal device, current content forms have reached their end. To the extent XR devices can replace phones, 3D can replace current content forms. In this sense, Apple Vision Pro is Apple's hardware device to open and define the future.
We currently write diaries in text, record precious moments with photos, document life with short video — in the future, perhaps we'll enter an era of recording scenes with 3D information.
How big is this market? A 3D industry professional calculated for us: current UE engine users + Unity users + Blender users ≈ 3D market users. Excluding people using multiple software simultaneously, such 3D model users number roughly 10 to 20 million.
But in the long term, if there really comes a day when 3D becomes humanity's main content consumption form, current 2D creators may all become 3D creators. What's the incremental increase between these? About 1.5 billion. That seems like a distant future, when our life scenarios may have already changed dramatically. But let's hold onto that anticipation.
