Is the $1 Billion "World Model" Bet by a Turing Award Winner AI's Next Decade? (Part 2)
True intelligence doesn't begin with language, but with the world.
True intelligence doesn't begin with language — it begins with the world.
👦🏻 Author: Shirley
🥷 Editor: Koji
🧑🎨 Layout: NCon
At the end of Part 1, we left off with a question: LeCun's "world model" sounds convincing, but can it actually outperform today's large language models? Can it control a real robot?
In this episode, his world model steps out of theory and faces reality for the first time. Again guesting on Welch Labs, the conversation shifts from "why" to "whether it works": the most impressive robots today — the ones that can peel zucchini and take out the trash — use precisely the VLA architecture that LeCun dismisses. What makes him so sure "VLA is doomed"? And is his own world model actually better?
We're publishing this because it offers a rare look at both sides of a high-stakes technical bet — JEPA's genuine achievements alongside LeCun's frank admissions of its weaknesses.
By the end, you'll have your own read on how far this contrarian path has actually progressed.
Guest Background
Yann LeCun | Turing Award winner, father of convolutional neural networks (CNNs), and former Chief AI Scientist at Meta. Widely recognized as one of the founding figures of deep learning, he pioneered convolutional networks in the 1980s that laid the groundwork for modern computer vision, and his 2015 "cake analogy" accurately foretold the era of self-supervised learning dominance. Having left Meta and raised roughly $1 billion to found AMI Labs, he is now betting on an entirely new, non-generative path centered on JEPA and world models — making him the most heavyweight, senior voice questioning the mainstream LLM approach today.
Roadmap: VLA Is Doomed → Layer 1: Visual Encoders → Layer 2: VL-JEPA → Layer 3: Robotics → Where Things Stand Now
Opening Salvo: VLA Is Doomed
Host: What's your expectation? Do you think JEPA will eventually surpass VLA?
Yann LeCun: Oh, absolutely. Yes. VLA is doomed — they basically don't work very well.
Startup Physical Intelligence has built one of the most impressive robot "brains" to date. This is their 𝝅0.7 model, capable of peeling zucchini, folding pinwheels, and taking out the trash. And it is a VLA (Vision-Language-Action) model.
Like VLA models, the JEPA approach can also control robots — but JEPA's demonstrated capabilities lag noticeably behind. Here's JEPA taking 60 seconds just to move a cup off a platform:
On one side, VLA robots fluidly peeling zucchini and taking out the trash; on the other, JEPA spending a full 60 seconds to nudge a cup aside. Yet LeCun — the man who has bet everything — declares that the stronger-looking path is the dead end.
What gives him such confidence? Are these currently dazzling VLAs truly destined to fail?
In many respects, VLA models represent the pinnacle of today's mainstream "generative, language-driven" AI approach.
VLA models are built on top of VLMs (Vision-Language Models), which in turn are constructed from visual encoders and large language models.
At every layer of the VLA stack, there exists a JEPA-based alternative. So in this episode, we'll explore the alternative VLA stack layer by layer:
- A video foundation model called V-JEPA 2 — how does it compare to the "language-supervised encoders" common in AI systems?
- Vision-language models, including AI assistants like ChatGPT and Claude, and how JEPA would reframe their training.
- How JEPA actually learns a world model usable for robot planning and control, and examining the potential advantages this method holds over VLA.
Let's start with Layer 1 — the foundation of the entire stack: the visual encoder.
Layer 1: The Visual Encoder Battle — CLIP vs. V-JEPA 2
Modern AI systems are remarkably adept at integrating vision and language: chatbots can describe images in extreme detail, and we've even learned to map text descriptions back to realistic images and videos.
Much of this progress traces back to a 2021 paper and model from OpenAI: CLIP.
As mentioned in Part 1, contrastive learning can train joint embedding architectures: encoders are trained to output similar vectors for "corrupted" and "uncorrupted" versions of the same image, and dissimilar vectors for different images.
CLIP works similarly, but instead of corrupted and uncorrupted versions of the same image, it uses image-text pairs: images are fed into a visual encoder, captions into a separate text encoder, and CLIP is trained to maximize embedding similarity for paired image-text combinations while minimizing it for unpaired ones.
Once trained, CLIP's visual and text encoders can be repurposed across a vast range of AI systems.
One common application is making large language models multimodal: when you give an AI assistant an image, it's typically fed into an image encoder trained with a CLIP-like method, which extracts meaningful information for the LLM to use. This "visual encoder + LLM" combination is called a Vision-Language Model (VLM).
CLIP relies on image-text pairs — fundamentally, it still depends on language to guide vision. So is there a visual encoder that learns entirely without language, from video alone? This is JEPA's alternative answer: V-JEPA 2.
Two Opposing Training Philosophies
V-JEPA 2 was trained by the Meta team in 2025 on 1 million hours of video with up to 1 billion parameters, making it one of the most ambitious JEPA models to date.
As described in Part 1, JEPA feeds input X and output Y through encoders to obtain embedding vectors, then uses a separate predictor to predict Y's embedding from X's embedding.
The V-JEPA 2 team employed self-supervised training: they corrupted video clips by removing patches, fed both corrupted and uncorrupted versions through the encoder, and trained the predictor to reconstruct the missing regions' embeddings.
The core idea is that by learning to fill in missing video patches, JEPA learns to understand video — and through that, to understand how the world depicted in the video works.
Though both CLIP and V-JEPA produce visual encoders that take images/video and return embedding vectors, their training objectives are fundamentally different:
| Dimension | CLIP (2021, OpenAI) | V-JEPA 2 (2025, Meta) |
|---|---|---|
| Training signal | Image-text pairs (image ↔ caption) | Pure video, no language exposure |
| Learning approach | Make paired image-text embeddings similar, unpaired ones dissimilar | Remove video patches, predict missing patches' embeddings |
| Relationship to language | Guided/constrained by language descriptions | Completely ignorant of language; free-form representations |
So as long as V-JEPA's learned representations help it fill in missing parts of cat videos, the model can represent "cat" however it likes — unconstrained by language.
The Critical Question: Can a Language Model Actually Use a Video-Only Encoder?
But this freedom raises a crucial question: Are V-JEPA 2's representations actually usable by our language models? Can a model trained only through vision interact with one trained only through language? What happens if we swap out CLIP for V-JEPA 2 — equipping a vision-language model with a V-JEPA 2 encoder?
The V-JEPA 2 authors prove in their paper [1] that not only is this possible, but replacing the encoder with V-JEPA 2 achieves SOTA (state-of-the-art) on a suite of video understanding benchmarks:
We demonstrate a video encoder pretrained without language supervision that can be aligned with a language model and achieves state-of-the-art performance, contrary to conventional wisdom.
These video understanding benchmarks span diverse skills. Here's an example from the TempCompass benchmark:
The model watches a video of "a person picking up a pineapple" and selects what happened from multiple options. Notably, in a variant of this question, the video is reversed — and the correct answer changes accordingly.
For reference, in our testing, ChatGPT 5.5 got both forward and reversed versions wrong. Only certain versions of Claude and Gemini answered correctly.
Thus V-JEPA 2 demonstrates: When used to train the vision component of vision-language models, a JEPA-based approach can achieve competitive results — even reaching state-of-the-art on certain benchmarks.
Layer 2: VL-JEPA — Rebuilding the Entire VLM as JEPA
Still, this remains a hybrid approach: JEPA is only used in the model's vision component, while the full vision-language model (VLM) still employs standard generative next-token prediction for language. So can we apply the JEPA architecture to the entire VLM?
In the most common VLM architecture, images/video are fed into a vision encoder; the resulting embedding vectors, together with tokenized prompts, go into an LLM, which outputs text one token at a time.
Now let's map this to the JEPA architecture. Following JEPA's approach, instead of directly generating output text, we feed the target output text into an encoder and train a predictor to predict the output text's embedding vectors.
Beyond this, the rest of the standard VLM architecture actually maps quite cleanly to the JEPA architecture:
In standard VLM architecture, visual embeddings and prompts are fed into the LLM; in JEPA architecture, the predictor receives embedded images/video, with the input prompt enabling it to process both visual and textual inputs.
So architecturally, the language model in a VLM and the predictor in JEPA perform similar tasks and receive identical inputs.
The key difference: the JEPA predictor's target is the embedding vector of the output text, not the output text itself.
So how does this JEPA-flavored vision-language model actually perform?
From the "dashcam" example in Part 1, we learned that a key advantage of the JEPA architecture is not needing to reconstruct the complete output. Theoretically, then, the encoder extracts salient features of the output while ignoring irrelevant details (like the random fluttering of roadside leaves).
For example, if we ask a VLM "Can I eat this mushroom in the picture?", there are many valid phrasings for the correct answer, but the training data might only contain one. If the training data's correct answer is "Don't eat this mushroom" but the model returns "This mushroom is unsafe, you can't eat it," the model gets penalized during training for an essentially correct response.
With the JEPA architecture, these phrasings get mapped to very similar embedding vectors, abstracting away irrelevant semantic differences in the prediction target.
In late 2025, a Meta research team demonstrated the striking efficiency gains from this vision-language JEPA architecture (which they call VL-JEPA).
In a controlled experiment, when VLM and VL-JEPA were trained with the same vision encoder, same data, and same training configuration, VL-JEPA learned significantly faster: after 5 million training samples, it achieved 35% video classification accuracy versus just 20% for the traditional VLM architecture.
In other words, by abstracting away irrelevant semantic details in the target training text (predicting the embedding of target text Y, rather than Y itself), VL-JEPA achieves higher learning efficiency.
This efficiency advantage translates into concrete results. GQA is a benchmark testing compositional visual reasoning with deliberately tricky questions — e.g., given an image, asking whether "the tray that holds the cup, does it have fruit on its left side?"
And it's on exactly this benchmark that a 1.6-billion-parameter VL-JEPA directly beat a 7-billion-parameter model.
However, since VL-JEPA isn't generative, it doesn't directly output answers by default. The team therefore employed several workarounds:
- Feed the image and question into the model to obtain a predicted embedding vector, then feed all candidate answers from a benchmark through the Y-encoder, selecting whichever is closest to the predicted embedding — effectively giving VL-JEPA multiple-choice questions;
- Train a text decoder to map VL-JEPA's predicted embeddings into text, allowing it to function like a generative model during inference.
Layer 3: Robotics — VLA vs. JEPA's Ultimate Showdown
The JEPA framework has intriguing overlap with the vision-language models powering AI chat assistants, offering paths toward stronger vision encoders (like V-JEPA 2) and, through architectures like VL-JEPA, the benefits of "embedding-space training objectives for more efficient learning."
JEPA won decisively in the first two layers. But in layer three, the story starts to shift.
So what about those vision-language-action (VLA) models from the opening?
These models essentially turn an LLM into a "robot brain": using a pretrained VLM, they're trained to output robot control signals based on instruction prompts and input data from robot cameras/sensors.
Early VLA models had LLMs output control signals directly. More recent approaches like 𝝅0.7 use a separate model called an action expert to interface with the LLM and produce the final control signals.
What's interesting is that VLA models represent the sharpest contrast to LeCun's JEPA philosophy. LeCun's critique of VLAs boils down to two points:
The scalability problem of behavioral cloning, and the lack of explicit planning.
First, here's LeCun on behavioral cloning:
Oh, VLAs are dead, they basically don't work very well. The only way to make them work is essentially to collect a massive amount of demonstration data and do behavioral cloning. And this only works in very few scenarios with limited variability — once the system encounters even slightly novel situations, it becomes helpless.
Human demonstrations are a key training data source for many VLA implementations, including Physical Intelligence's 𝝅 model. The training dataset is collected using precision controllers where the robot mimics the operator's hand position.
But LeCun argues this approach fundamentally doesn't scale: you can't possibly collect human demonstrations for every variation of every task a robot might perform.
It's worth noting that VLA models have been shown to generalize to new tasks beyond their training demonstrations.
In fact, the 2023 breakthrough moment for VLAs — when Google's RT-2 moved a can of Coke next to a photo of Taylor Swift — demonstrated exactly this.
To complete the task, RT-2 had to combine its internal vision-language model's pre-trained concept of "Taylor Swift" with the "move object" action it had learned from human demonstrations.
Of course, generalization itself is a spectrum: these specific tasks weren't in the human demonstration data, but similar tasks were; ask it to do something too far removed from the demonstrations and it will likely fail.
But the bigger question here is whether VLAs can achieve sufficient generalization beyond demonstration data.
LeCun's second major criticism of VLAs is their lack of explicit planning.
VLAs are trained and deployed end-to-end: at each timestep, a new set of camera images and robot joint positions is fed in, and the model is trained to directly output the next set of joint positions. The robot moves to these new positions, captures new images, and the cycle repeats.
In one Physical Intelligence demo, a robot had to pass a key back and forth between two grippers to adjust it to exactly the right orientation for unlocking; the internal LLM reasoned about how to hold the key and broke this goal down into the sequence of handoffs.
The challenge is that we have extremely limited control and visibility into the entire planning process. We're essentially dealing with a black box that receives text instructions and camera frames, then spits out actions.
And this is precisely what LeCun questions: a system that cannot predict the consequences of its own actions, he argues, cannot serve as a foundation for reliable agents. A reliable agent must be able to anticipate the outcomes of a sequence of actions before planning them.
In this framing, reasoning shifts from "autoregressive prediction" to "search" — and that's the whole point of world models.
push-T: Letting Robots Rehearse in Their Minds
Unlike VLAs, LeCun's approach to building world models with JEPA architecture isn't end-to-end learning, nor does it imitate humans through behavioral cloning. Instead, it uses the JEPA architecture to learn an action-conditioned world model, then uses it for explicit action planning.
This is a task called push-T: a robot must move a T-shaped object to a marked target position on a table.
It's trickier than it looks because it's difficult to predict how the T will translate and rotate with each push. The robot's actions are essentially limited to 2D joystick control — moving the end effector up, down, left, and right.
So how does LeCun's world model work when simulating the push-T task?
First, use JEPA to learn a world model by recording images and actions from the pusher. At each step, we train the predictor to predict the next frame's environmental image embedding based on the current image's embedding vector, plus some action indicated by the arrow keys.
The "brown T" is the target position, the "blue T" is the object we're pushing, and the control inputs move the yellow actuator.
Now things get really interesting: given some initial configuration, we feed this image into the encoder to get the starting position embedding; then we feed any action into the predictor, and the predictor returns its prediction of the world's next state based on that action.
This prediction is still an embedding vector — hard to interpret directly. But for simple environments like push-T, we can train a separate decoder (vision decoder) to map the predicted embedding back to an environmental image.
Input an "up" action, and the actuator (yellow dot) in the decoded image moves up. Same for left, right, down. Chain actions together, feeding each predicted new state (embedding vector) back into the predictor along with the next action, and so on... This becomes a learned sketch of the push-T world's dynamics.
This push-T demo comes from an open-source JEPA implementation called LeWorldModel (LeWM for short). Proposed with LeCun's own involvement, it's trained end-to-end from raw pixels and actions, from scratch, without any pre-trained encoder.
From this data alone, the world model learns the physics of the environment — including the fact that the "blue T is a rigid body that can be moved" — and the complex interactions between the actuator and the T.
With this in place, we can predict sequences of actions and, through optimization, find the optimal action sequence to achieve a particular outcome. This is classic optimal control theory.
To plan a sequence of actions, the LeWorldModel team used a very general planning method — the Cross-Entropy Method (CEM).
Given a starting image and a target image, CEM begins with a set of completely random actions (say, 500 random trajectories for the actuator). The world model then picks out the most promising trajectories:
For each path, use the world model to predict what would happen, and compute the Euclidean Distance between the final predicted embedding vector and the target embedding vector as a score; color-code paths by distance, select the top 30 trajectories to form an Elite Set, then sample a new batch of trajectories using the mean and standard deviation of this elite set. Repeat until converging on a final planned path.
Finally, following the planned path, the actuator pushes the T cleanly to its target.
What's truly remarkable is that the entire planning process unfolds entirely within the embedding space learned by the model. The score assigned to each path — the distance between the predicted embedding vector and the target embedding vector — guides the entire planning process.
Thus, this system offers a clean rebuttal to LeCun's critique of VLAs:
It does not learn by imitating humans, so it doesn't need to observe how humans solve the task; instead, it can autonomously find solutions through its world model and explicit planning process.
Shortcomings and Solutions: Hierarchical World Models
However, this elegant approach has one shortcoming that must be honestly confronted: LeWorldModel's demonstrated performance to date lags far behind VLAs. It can only reliably plan forward about 5 steps, restricting it to relatively simple operations.
This was precisely the sharp follow-up in the interview: what good is a JEPA robot if it can't even plan a long task like "clean the kitchen for 10 minutes"?
LeCun's answer is a hierarchical model:
Lower levels handle high-detail, short-term prediction; higher levels handle coarse-grained, long-term prediction.
The reasoning is that the more detail you preserve in prediction, the faster it tends to diverge from reality; so the further out you look, the more you must discard detail and grasp only the broad direction.
He illustrated with a scenario everyone knows:
If I'm sitting in my office in New York and want to get to Paris tomorrow, I can't plan the entire journey with "millisecond-level muscle control" — most of the time, I simply don't have that information.
Real planning unfolds hierarchically: the top level is "get to airport, catch flight," which breaks down into subgoals like "get to the airport first," then "go downstairs, hail a cab," until at some level you're facing something as practiced and thoughtless as "stand up from chair, walk to elevator."
Applying this hierarchical world model back to the push-T task, a mere two-layer structure extended the planning horizon from 5 steps to 15 steps. What's interesting is that the higher-level predictions happened to serve as subgoals for the lower-level planner.
What excites LeCun even more is a potential free lunch:
If the architecture is designed properly, these hierarchies might emerge spontaneously — much like how CNNs automatically learn feature hierarchies from simple to complex — with the system developing appropriate levels of abstraction through self-supervised training where "lower levels learn short-term, higher levels learn long-term."
But he added a caveat: this only works with "semi-expert" trajectory data, meaning the successful emergence of these hierarchies depends heavily on training data quality.
Feed it completely random, causally disconnected training data, and the system learns no high-level patterns at all.
AMI Labs' Industrial Blueprint: Controlling Systems That Defy Equations
V-JEPA 2 and VL-JEPA have shown us the potential of this framework, and proven that the JEPA path does not conflict with the current mainstream language-driven approach.
But LeCun didn't shy away from reality. Once you zoom out to true agent and robotics problems, JEPA-driven world models remain quite limited.
For this he offered a telling analogy: 30 years ago, his early deep learning system for recognizing handwritten digits was also quite limited in its time — as unimpressive as today's push-T demo — yet those core ideas were eventually scaled into the powerful AI we have today.
Will JEPA follow the same trajectory?
When asked about AMI Labs' next steps, LeCun's direction wasn't humanoid robots, but rather controlling complex systems whose behavior "cannot be reduced to a small set of equations":
If you can write down the equations — for a simple robotic arm, or even a humanoid robot — you just write down the dynamics equations; when NASA launches a rocket, it has complete dynamics models that can plan the entire flight trajectory. The real difficulty lies in places where no equations can be written: a chemical plant, a jet engine, or a diabetic patient. What treatment regimen should you use to control his blood sugar? How do you guide a stem cell to become a beta cell that secretes insulin in the pancreas?
These systems cannot be described by a few equations, but LeCun believes that perhaps a phenomenological model can be learned from data, and then used for control.
There are already many such attempts in materials science and chemistry — for instance, training a model of complex collective phenomena, then using it to design new materials, new catalysts, new batteries, and so on.
And he frankly admitted that at present this may not be about generating revenue, but rather accumulating experience in pushing this methodology toward practical applications.
As for the endgame, his ambition is hardly modest:
In a few years, he hopes AMI Labs can become a major supplier of intelligent systems, whatever the application scenario.
Supporting this ambition is a credo written on AMI Labs' homepage, and also the entire premise of his grand bet:
Real intelligence does not start in language. It starts in the world.
Closing: A Bet Without an Answer Yet
Looking at both parts together, LeCun's $1 billion bet has its bright spots and its sobering realities.
JEPA has proven that the "no generation, only understanding" path not only works, but can actually be more efficient: V-JEPA 2, which doesn't rely on language, reached SOTA in video understanding, while 1.6-billion-parameter VL-JEPA directly outperformed 7-billion-parameter rivals.
But in robot control, JEPA still lags far behind VLAs: it can only plan a measly 5 steps to push a T-shaped block.
LeCun has a complete vision, and a few impressive individual wins, but the capability he values most — getting robots to rehearse in their minds like humans before acting — remains a long way from being truly usable.
So this is not a victory dispatch declaring "JEPA wins." But don't forget: 30 years ago, his little network for recognizing handwritten digits looked just as unremarkable in its time.
Will world models stage the same comeback this time?
Within a few years, we may have our answer.
Crossing is looking for freelance writers to produce AI product and model reviews.
If you've written articles like: Hands-on with PixVerse C1[1], Hands-on with LibTV[2], please contact zeo0811@gmail.com. Your email should include: ① personal introduction, ② AI review articles you've written.
We offer competitive rates. Looking forward to observing and documenting the AI era together 🎪
References
[1] Paper: https://arxiv.org/pdf/2506.09985