The Data Path Toward Embodied AI's GPT-3 Moment | Yang Gao x Beipo Initiative

Signal and Noise, Beipo Initiative Episode 1: We sat down with Yang Gao, co-founder of Spirit AI and assistant professor at Tsinghua University's Institute for Interdisciplinary Information Sciences.

He offered a clear prediction: by mid-2027, embodied intelligence foundation models will reach their GPT-3 moment — the equivalent inflection point that large language models hit. However, he argued that the binding constraint on reaching this stage isn't the model itself, but data.

Building on our podcast conversation, we've compiled an overview of Gao's academic research. Thoughts and feedback are welcome — feel free to leave us a comment.

Enjoy.

---

In 2017, the Transformer architecture was introduced.

Over the nearly decade-long evolution of AI that followed, the Scaling Law has been regarded as a fundamental logic of intelligence development — even a "first principle."

Jared Kaplan's team, in Scaling Laws for Neural Language Models, reached a critical conclusion: stable power-law relationships exist between model performance and parameter scale, data scale, and compute volume. When all three expand in tandem, capability gains are continuous and predictable, with phase transitions occurring once certain thresholds are crossed. This judgment was reflected in the engineering leap from GPT-2 to GPT-3: training data expanded from roughly 10B tokens to 300B, parameters grew from 1.5B to 175B, and data distribution shifted from single-source WebText to a multi-source blend covering Common Crawl, Books, Wikipedia, and more. For the first time, the model crossed the "generalization threshold," demonstrating stable few-shot capability.

A consensus was thus established: "massive data produces intelligence." When data and model scale reach sufficient levels, capabilities emerge automatically from distributional learning rather than relying on manually engineered structures.

But this law holds only because of the statistical structure of language modeling: internet text provides an approximately infinite, distributionally stable, and repeatedly sampleable data source, allowing models to continually approach the true distribution through scale expansion. When model capacity is large enough and data coverage broad enough, the learning process shifts from "fitting specific tasks" to "approximating a universal distribution" — and only then do cross-task transfer and few-shot generalization emerge.

Applied to embodied intelligence models, this logic does not fully hold.

At least not at the time.

Before 2020, in the same window when GPT-3 emerged, embodied systems remained stuck in modular, hierarchical paradigms.

As illustrated by the classic Hybrid Deliberative/Reactive Architecture framework, task planning, path decision-making, and low-level control were handled by separate modules, with information passing and execution coordinated through hierarchical structure. Final performance depended on local optima within each module, and because modules differed in data modality, temporal scale, and optimization objective, the system could not achieve unified capabilities through simple scaling.

In other words, while language models had already entered an evolution stage driven by data scaling, embodied intelligence still relied on modular decomposition for training and execution. The two were methodologically misaligned.

The unified modeling paradigm represented by Transformer made this misalignment addressable for the first time.

Yet even then, this was not the mainstream line of thinking.

In 2017, during his doctoral studies at Berkeley, Gao published End-to-end Learning of Driving Models from Large-scale Video Datasets, an end-to-end driving model trained on large-scale video data that attempted to map visual input directly to control output using a single model.

Against a backdrop where modularity still dominated, this work demonstrated the foundational value of video as pretraining data, and implied a deeper conjecture: translated to the embodied domain, models need not depend on the structured decomposition of "perception—planning—control," but could instead advance learning through data optimization within a unified model.

This judgment did not form industry consensus at the time. Six years later, VLA (Vision-Language-Action) models were systematically developed and gradually became one of the important directions in embodied intelligence.

Following through on Gao's early insight: if vision can map directly to action, then can "action" itself be defined as a modelable data modality? If action can be serialized, predicted, and combined like language, then the problem of embodied intelligence ceases to be one of controller design and becomes one of data generation under a unified model.

This is the core judgment of the VLA path. It incorporates action into the same modeling framework as language, enabling the model to process vision, language, and behavior within a single sequence space, thereby obtaining cross-task generalization capability.

But once definitions are rewritten, underlying constraints change as well.

Unlike language models, "action" has no natural data supply. Real-world interaction is costly, collection efficiency is low, and scaling is difficult — quickly exposing the data constraints in VLA model training: insufficient per-unit data efficiency, limited data sources, and inability to scale overall. It was in the process of breaking through these constraints that Gao's research gradually crystallized into the later "data pyramid" training strategy, allowing embodied intelligence models to more effectively leverage diverse data for continuous iteration.

Of course, these conclusions are our retrospective synthesis.

Returning to the "dark forest" of that period, Gao's exploratory path was not achieved overnight, but rather involved nearly a decade of progressively rewriting three layers of constraints.

The first constraint: when data scale cannot be expanded, how to increase the value per unit of data.

Gao's 2021 EfficientZero algorithm dramatically improved sample utilization efficiency in reinforcement learning. Traditional methods reuse interaction data through experience replay but struggle to extract long-term decision information from limited trajectories; EfficientZero introduces a latent dynamics model and cross-temporal value consistency constraints, enabling the model to perform multi-step rollouts and repeated re-evaluation of existing trajectories in latent space.

On the Atari 100K benchmark, this method surpassed average human performance using only about 100,000 steps of interaction data. Traditional DQN-based methods typically require tens of millions to hundreds of millions of frames to achieve comparable results — a data efficiency improvement of over two orders of magnitude.

This result shows that insufficient data is not first and foremost a scale problem, but a utilization problem.

When the learning process is redesigned, the effective information density of data can be significantly amplified. This step, in effect, deferred the system's dependency on scaling under the premise that "data cannot be expanded." Beyond EfficientZero, Gao also proposed EfficientImitate and other high-performance imitation learning algorithms, all targeting the same core objective of improving sample efficiency.

However, the boundary of this strategy is clear. As task complexity rises, efficiency improvements alone cannot sustain continuous capability growth, inevitably bringing the question back to: "where does data come from?"

The second constraint emerges.

Starting in 2023, Gao's team shifted its focus: no longer assuming "data must come from real interaction," they attempted to convert passively observed data into training signals usable for decision learning — redefining the supervision signal itself.

For example, the team published Become a Proficient Player with Limited Data through Watching Pure Videos (ICLR 2023), which recovers latent dynamics information from video without action annotations, enabling models to learn from pure observation that "if a certain action is taken, the state will likely change in this way"; the 2024 ATM (Any-point Trajectory Modeling) work went further, making explicit the physical constraints implicitly contained in pixel changes by predicting future trajectories of arbitrary points in video, providing approximate action supervision for policy learning. This work was subsequently accepted at RSS 2024, the top international robotics conference, receiving full marks from all reviewers.

The common feature of these works is the transformation of "action" from annotation to structural information implicit in state changes. In other words, supervision signals no longer depend on manual collection but can be recovered from environmental changes themselves, migrating data supply from "low-frequency, high-cost interaction data" to "high-frequency, scalable observation data."

This is a redistribution of information density: decision information originally concentrated in limited annotations is diffused into large-scale unannotated data. When models gain the ability to parse this structure, internet video becomes a reliable source of embodied intelligence data.

This step, in essence, opened the ceiling on data scale.

Additionally, Gao's two most recent research works from 2024–2025 — General Flow and EgoMono4D — further complete the transformation from "usable data" to "computable structure." The former learns 3D flow from large-scale RGBD video as a cross-task general manipulation prior; the latter reconstructs first-person video into 4D representations containing depth, pose, and scene flow through multi-view geometry and temporal consistency. In Gao's sustained research, video data has transformed from pixel sequences into structured representations with explicit physical meaning, enabling models to reason and generalize at higher levels.

Only when both data sources and data representations were simultaneously reconstructed did embodied intelligence first gain the conditions to validate the Scaling Law.

If the previous phase solved "where does data come from," the subsequent phase saw Gao focus on "how does the model understand data," achieving the transition from end-to-end reinforcement learning (RL) to VLA, from "control" to "understanding + control."

In 2024, Gao's team published Data Scaling Laws in Imitation Learning for Robotic Manipulation (ICLR 2025), using extensive robot demonstrations and real-machine validation to systematically prove that the Scaling Law holds in embodied intelligence: across over 40,000 demonstration trajectories and 15,000 real robot tests, model performance improved as a power law with environment count and object count. Meanwhile, experiments showed that under fixed collection costs, increasing environment and task diversity yielded more significant performance gains than repeated sampling of the same scene.

Transforming "data scale" from an empirical variable into a modelable variable, model performance fluctuates with data increase and can be predicted and planned through distribution design. This is consistent with the role of the Scaling Law in language models, providing systems with a clearer growth path.

Placing these three steps together reveals a clear progression: EfficientZero improves data efficiency, giving systems learning capability under small-scale data; video and multimodal modeling expand data sources, making scale expansion possible; validation of the embodied Scaling Law transforms this expansion into a predictable capability growth mechanism.

Importantly, the entire path does not revolve around model training architecture in isolation, but consistently reconstructs around "data" — the underlying constraint blocking VLA work. When the industry still devotes primary energy to model design and algorithmic improvement, this choice to sink the problem to the data layer along "first principles" is itself a prescient judgment.

If we contrast this process with the GPT-2 to GPT-3 evolution, we see structural consistency: when data supply is opened and scale can be continuously expanded, system capabilities begin shifting from "dependency on structural design" to "dependency on data distribution." Capability gains no longer come from local optimization, but from leaps in overall scale.

The foresight of Gao and his team's research has also begun receiving feedback in engineering results.

In March 2025, Spirit AI's Spirit v1 achieved breakthroughs in deformable object manipulation — a task long considered difficult to generalize; in January 2026, version v1.5 surpassed the long-leading overseas model π 0.5 in the RoboChallenge Table30 real-machine benchmark, becoming the first Chinese model to reach the top; simultaneously, through self-developed wearable collection devices, Spirit AI reduced data collection costs to one-tenth of traditional solutions, enabling more efficient continuous scaling of real interaction data.

Looking back at Gao's nearly decade-long research, a pragmatic idealism runs throughout.

His "data pyramid" strategy is a pragmatic solution to embodied model training; while the vision of "enabling 10% of the world's population to have their own personal robots in 10 years" represents an idealistic pursuit of science benefiting humanity. In this global technology race, Gao's convergence on key problems before stable paradigms formed and before consensus crystallized — in a field where neither yet existed — undoubtedly contributes important Chinese wisdom to Chinese teams defining the next generation of intelligence paradigms.

Search for Signal and Noise on Xiaoyuzhou to listen to the full podcast.

Also follow us on WeChat Channels to experience the conversation live.