Physical Intelligence: Building a General-Purpose Embodied Foundation Model | Vital Views

As the embodied intelligence industry evolves, while robotic hardware is iterating rapidly, the embodied model serving as the "decision-making center" is gradually becoming the critical variable in technical progress.

Spirit AI, an Oasis Capital seed portfolio company, has secured additional capital within 30 days after completing nearly 2 billion RMB in funding this February, bringing its total raised to 3 billion RMB. Spirit AI adheres to the principle of "defining generality through delivery," anchoring itself in enterprise partnerships and real-world scenarios, and advancing model scaling through diverse data. To date, it has accumulated over 200,000 hours of real interaction data.

Expanding the lens overseas, Physical Intelligence* (π, or PI)* is also among the earlier companies to articulate the core thesis of a "general-purpose physical intelligence model." In a recent in-depth interview, co-founder Sergey Levine offered a systematic exposition of the technical convictions, key challenges, and path to deployment for this direction.

We have compiled and edited the interview content below. The full text runs approximately 20,000 Chinese characters, with an estimated reading time of 30 minutes.

Enjoy

Left: Patrick O'Shaughnessy, CEO of Positive Sum. Right: Sergey Levine, Co-founder of Physical Intelligence

A General-Purpose Embodied Model Makes More Sense Than Specialized Ones

Patrick: Could you start by sharing what Physical Intelligence is working on?

Sergey: Essentially, the goal of Physical Intelligence is to build robotic foundation models capable of executing any task on virtually any embodied system. You can draw an analogy to the trajectory of language models, which are evolving into systems that can complete any task expressible in language. Our core conviction is that solving at the general level may be easier to achieve than specialized training for every individual scenario. This aligns with the experience from language models: rather than separately optimizing for narrow tasks like machine translation or sentiment analysis, modeling at the level of natural language as a whole proved more effective.

Patrick: This isn't a common view. Why not just build a robot that does something like washing dishes? What led you to this decision, and what are the key trade-offs?

Sergey: I'll answer in two parts.

First, the analogy from language models. Second, what this specifically means in robotics.

In natural language, there was once extensive work on specialized solutions — for instance, people would study the differences between English and French in depth to build machine translation systems. The reason large language models could eventually replace these niche applications is fundamentally that they can leverage much broader data sources. It's not simply merging data from different tasks; it goes further. When you can utilize weakly labeled data — for example, massive text scraped directly from the internet — the model is actually learning a holistic understanding of the world. Building specific applications on top of that world model works much better.

Conditions in robotics aren't identical, because we don't have internet-scale datasets comparable to what language models use, but "understanding the world" is actually even more important here. If you're facing multiple tasks and even multiple different physical systems, rather than training separate models for washing dishes, folding laundry, and so on, it's better to train a model that truly understands physical interaction.

Humans can quickly acquire new skills precisely because we have intuitive understanding of the physical world — we can rapidly judge what will happen in unfamiliar situations and act accordingly. Similarly, if a model can learn from multi-source, multi-task, multi-robot systems, it has the opportunity to build this kind of physical understanding, and extending to new applications on that foundation becomes easier and more efficient.

Patrick: If we go back to the most fundamental model layer to understand Physical Intelligence's path, could you simplify it further? I'd like to contrast it with other approaches you've considered but not adopted.

Sergey: In my view, the most critical thing is to give the system "generality," particularly in terms of how it can be improved.

For example, hand-designed robot controllers are very ungeneral in how they can be improved, because every optimization requires engineer intervention. Learning-based systems are more general — take perception systems, where you just need people to label more data for continuous improvement. If a system can automatically acquire data and learn from its own experience, that's another step forward, because even human labeling is no longer needed. Many of our decisions are fundamentally oriented around this.

As for specific implementation forms — whether a robot should have three cameras, whether it needs tactile sensors — these are relatively neutral questions, and we experiment with different configurations. Even in the long run, whether language models are definitively necessary, I'm not sure; perhaps there will be other model forms trained on diverse data.

But the unchanging core is the pursuit of this generality.

Patrick: What other paths have you paid attention to? Which directions seem interesting to you?

Sergey: A very critical question in this field — one that neither the research community nor industry has truly solved — is the divergence between different data sources, especially the trade-off between "real data" and "simulated data." This is a highly contentious topic, and I have fairly clear views myself. But objectively, there are indeed two very different paths here.

Take humanoid robots, for instance. When you see videos of complex gymnastics maneuvers, you'll typically find they rely on highly simulation-based methods, with very little — even close to zero — real-world data. In robotic manipulation, by contrast, the dominant approaches tend to be the opposite: little use of simulation data, heavy reliance on large amounts of real-world data and large-scale foundation models.

What's interesting is that in these two subfields, the dominant methodologies are so different. There are several possibilities for the future: one path ultimately wins and becomes the unified solution; or some kind of fusion emerges between the two, forming a new paradigm.

I'm not certain of the final answer, but I'm confident in our current path. Still, understanding why these two methods diverge so dramatically is itself a valuable question.

Patrick: Compared to paths that are more intuitive and easier for ordinary people to grasp — like a robot dedicated to a specific task with visible, clear behavior — what's the most difficult part of the choice you've made?

Sergey: Actually, the difficulty has run through my entire career.

When you decide to work on robot learning, the more you emphasize generality, the more obvious the difficulty becomes. What's truly effective, what real generalization looks like, is not the easiest path to "impressive demos." If the goal is to make a flashy demo, the most direct way is to pick a cool task, control the environment completely, make it extremely clean and controllable, and just nail that specific scenario.

But generalization doesn't work that way — it can't be demonstrated through a single scene. The essence of generalization is that it does things that look ordinary to humans, but completes them reliably across different environments.

For example, we released a set of demos last April showing a robot cleaning a kitchen. Viewed individually, the videos weren't that impressive — just picking up plates and such, something anyone could do. The key was that the robot was simply placed there for the demo; the model had never been trained in that specific scene. Without knowing the setup, it's hard to realize why this represented a breakthrough in capability boundaries.

Patrick: How do you think about the importance of this? What would count as success, rather than vaguely saying "crossing the chasm of general physical intelligence"?

Sergey: If a general embodied foundation model truly succeeds, a very exciting outcome would be that it massively unleashes people's imagination in building robots and all kinds of embodied systems.

Personal computers mattered because they let people build all sorts of cool things, and there was subsequently something like a "Cambrian explosion" of applications. This wave started in the 1990s and was further accelerated by the internet. I think something similar could happen in robotics. The reason it hasn't yet is that if you want to create a new robotic application or realize a new idea, you often need to build an entire massive, complex technology stack — essentially, you have to solve "intelligence" yourself. This makes most attempts unaffordable.

If in the future there's a foundation model available as infrastructure that people can call upon, obtaining basic capabilities through prompts and doing some fine-tuning or adaptation to their own application scenarios, then large numbers of companies, individuals, and teams will start trying all kinds of directions.

We sometimes imagine robots as a single form — "now there are humans, next we'll build metal humans" — but technology rarely evolves this way. What's more likely is an entire toolkit that people can freely combine and create various forms and applications from, even very different things — like a robot with five arms hanging from the ceiling to solve a specific problem, or all kinds of experiments at the software level.

And I believe embodied foundation models are precisely the right platform for that.

Humanoid Is Not the Only Solution

Patrick: What do you see as the advantages and limitations of humanoid robots on this path?

Sergey: The obvious advantage is that it's "cool" — the demonstration is more intuitive and easier to resonate with. For example, the Tesla Optimus hand design that everyone discusses now is inherently appealing. That intuitiveness helps spark imagination and makes it easier for people to think about the future.

But humanoid robots are just one of many possible forms, and the more fundamental issue is the challenge of intelligence itself. Different robot forms are actually similar to each other. I don't think we should solve intelligence under any specific body structure — it should be handled in a more general-purpose way, otherwise it's hard to achieve systematic breakthroughs. At the same time, a general approach is also better for data accumulation and utilization. One important advantage of robots is that they don't need to be constrained to human form; you can design the most suitable structure for the specific task. For example, imagine using a swarm of 1,000 quadcopters to build a house, rather than relying on a single-form robot.

The more likely long-term path is to first build out a robot foundation model, then adapt it to different forms and scenarios, covering everything from bulldozers to humanoid robots to robotic arms. For specific deployments, you might need fine-tuning, or to let the model understand how a particular "body" works through context.

But the underlying principles are shared — how to interact with objects, how objects move in the world, how causality works — these remain consistent across different systems.

Patrick: Do you have a particularly compelling example of what true general intelligence could enable, that would be difficult to achieve relying solely on humanoid robots?

Sergey: There are a few directions worth considering. One is the scale of the machine form itself, which opens up many possibilities, especially in medicine and surgery.

Future robots don't necessarily need to look human, or even be directly controlled by humans. Take current surgical robots — they're essentially still teleoperation, requiring real-time control by a doctor with sufficient precision. This limitation exists in current learning-based systems too. But in the long run, this constraint could potentially be broken.

Patrick: Michael Crichton wrote a fascinating novel called Prey, which raises speculations about form — for any given problem, there seems to exist an optimal, or a set of optimal, robot forms to complete the task. Following this logic, what you should really do is analyze the problem first, then let the system "morph" into a form suitable for that task. How do you think about this? Compared to the data and model side, how do you view innovation in robot morphology?

Sergey: Morphological innovation in robotics has always been heavily constrained, and the core reason lies in the challenges of AI itself. If you're on the traditional AI path — say, motion planning-based methods — it's very hard to arbitrarily assemble new robots. Because once you change the form, you need to re-characterize the system dynamics, do system identification, build the entire control framework — these costs are high.

But if you could build a robot in your garage, load a foundation model, and have it execute various tasks directly, the situation changes. Maybe it doesn't work perfectly at first and needs more data to optimize, but at least you can get the system "moving." That capability itself is a very strong motivator, allowing more people to participate and experiment.

I don't consider myself the kind of person who designs "optimal robot forms" — there are more specialized people for that. But looking at the bigger picture, this is more like the development path of personal computing: the key is to let more people tinker and try things, dramatically lowering the barrier. Once that happens, more creativity will emerge. Early personal computers had very limited forms, but now computing devices exist in phones, cars, even refrigerators — the forms are incredibly diverse. And what supports all of this is generality, plus the software foundation on which applications can be built. The same holds true for robotics.

Patrick: Your co-founder Lachy Groom once described "physical intelligence" to me as something like learning to ride a bicycle — at a certain moment, it goes from not knowing to knowing, and that feeling of "suddenly getting it" is physical intelligence itself.

Sergey: Actually, this phenomenon has a physiological explanation. There was research done on monkeys using tools, observing which neurons activate to judge hand position. What's interesting is that when monkeys use tools, these neurons don't activate based on hand position, but based on the position of the tool's tip. In other words, "the tool is an extension of the body" isn't just a metaphor — it's a real physiological mechanism, the brain actually works this way.

Patrick: How does understanding this affect your research approach?

Sergey: It means physical intelligence should be somewhat form-agnostic. A good foundation model should be able to learn to control whatever "body" it corresponds to, and whatever tool is at hand. Fundamentally, there is no such thing as a "humanoid robot problem," a "car problem," a "bulldozer problem," or a "robotic arm fixed to a desktop problem" — there is only one unified problem. If you can solve it at the general level, the capability becomes extremely powerful.

The Historical Thread of Robot Learning

Patrick: Before discussing the present and future, it's useful to establish a clear historical thread. Looking back at the development of robotics research, what were the key inflection points that led to today's situation? Could you map out the important milestones?

Sergey: Some core ideas actually go back quite far. End-to-end robot control is a very early idea — for example, the first end-to-end learning-based autonomous driving systems were already being attempted in the 1980s, like the ALVINN project around 1986-87, which used a very small neural network to control a vehicle directly from camera input for highway driving.

The real difficulty has always been how to translate these ideas into working robot learning systems.

Such a system needs to satisfy several conditions simultaneously: first, it must handle concrete application tasks; second, training costs must be controllable, not requiring massive data collection for every new task; third, it must have common sense, able to handle edge cases in long-tail scenarios; fourth, it must be stable, fast, and reliable on the target task.

These requirements stacked together are extremely difficult, because machine learning inherently relies on large-scale data. If you solve problems in the most direct way — say, making a dishwashing robot — the natural approach is to collect lots of dishwashing data. But this doesn't scale; every new task requires repeating the full data collection and training process. This is why general-purpose models are crucial — only when a model can cover multiple tasks does the data required for each new task drop significantly.

But the more critical shift has happened in recent years: how to handle "edge cases." For these long-tail scenarios, the model often has no direct experience and must rely on knowledge obtained from other sources, transferring it to the current situation. Humans are very good at this. For example, when driving, if you encounter a sign you've never seen before warning of a gas leak ahead, you can still judge how to respond based on existing common sense — you don't need to have personally experienced something similar.

For a long time, the core difficulty in robot learning was: where does this common sense come from? The recent breakthrough is that multimodal large language models have shown strong capabilities in "acquiring and expressing knowledge." While they're not good at directly grounding this knowledge in specific physical environments, they do know a lot.

This means there now exists a path to leverage the knowledge embedded in multimodal large models to compensate for robots' judgment capabilities in long-tail scenarios. Of course, this also brings new technical challenges: how to properly interface with this knowledge? You can't simply give the model an image and ask what to do, because it lacks necessary context — for example, what kind of robot it is, what the current environmental state is, etc. This is precisely where current technical efforts are focused. The research community has made some progress, and more importantly, for the first time we see a clear direction: systematically improving long-tail problem handling by introducing large-scale knowledge.

Patrick: When future historians look back at this field, are there nodes along this path comparable to AlexNet or Transformer — major events significant enough to be written as "historical watersheds"?

Sergey: That's a great question. But if I had to give a definitive answer, it's still too early.

Typically, these kinds of "historical nodes" require at least a decade of hindsight to confirm. Looking at the existing thread, one of the earliest milestones was the end-to-end learning systems of the 1980s. Then came deep reinforcement learning in the early 2010s, which was also a key node for robotics because it provided a path for systems to potentially surpass human performance.

As for recent years' progress, it's still hard to judge what will be written into "history." But the emergence of multimodal large language models, and their use for robot control to introduce common sense capabilities, is very significant progress. In the coming years, there will likely be a series of key breakthroughs. What ultimately gets written into history will probably be the advances that emerge from this phase.

Patrick: How did you enter this field? At what stage, and for what reasons, did you first become interested in this direction? And how did you continuously make choices about where to invest your time and energy on specific problems?

Sergey: I started doing robotics research in 2014, right after finishing graduate school, doing a postdoc at UC Berkeley with Pieter Abbeel. Before that I hadn't done any robotics work — I just felt I needed to systematically learn more after graduating, and his lab happened to do robotics, so I tried applying my previous research to this area.

Before that, I mainly worked in computer graphics. But one core problem I'd always wanted to solve was how to make AI systems continuously improve as they "do things." That is, the more a system does, the better it gets — this capability itself is extremely powerful. If this holds, it theoretically has no ceiling, and can keep acquiring new skills.

At first, I had the system start from scratch with no prior knowledge, practicing a specific skill and gradually improving. This works in closed environments but is hard to scale to the open world. Once the environment changes, previously learned capabilities don't transfer well, and the system often has to relearn from scratch.

Next I tried another path. While at Google, I explored whether parallel multi-robot learning could solve this problem — for example, putting 20 robots together and having them learn collectively. This approach did improve efficiency and provided some degree of generalization, but it still struggled with long-tail cases because the model remained fundamentally constrained within a specific task distribution.

So I believe the next step is to combine the ability to "acquire skills through practice" with large amounts of prior knowledge. This is itself an extremely difficult hypothesis, and it's not just a core challenge in robotics — it's a central challenge across all of AI.

If you look back at the past few decades of AI, there are two most important technical threads: one is generative AI, such as large language models; the other is deep reinforcement learning, such as AlphaGo.

Both are extremely powerful, but for completely different reasons. Generative AI's strength lies in its ability to reproduce capabilities humans already possess, like writing and painting. Deep reinforcement learning's strength lies in its ability to discover strategies no human has ever conceived, like "Move 37" in Go. The real difficulty lies in combining these two paths — on one hand, introducing the vast knowledge embedded in generative AI, while on the other, surpassing human-level performance through reinforcement learning. This is precisely the problem I'm now trying to solve at Physical Intelligence. We haven't fully achieved it yet, but we've made some progress in this direction.

Patrick: What specific work have you done, and what are you currently doing to push this forward?

Sergey: Over the past few years, we've built from foundational capabilities, constructing Vision-Language-Action (VLA) models. You can think of it as a large language model adapted for robot control. Its training path roughly divides into three stages: first training on text data, then introducing large-scale image data to give the model visual understanding capabilities, and finally adapting with diverse robot data so it can execute physical actions.

The result of this approach is bringing knowledge from the internet into a model that can control robots, using it as a starting point to produce meaningful behaviors.

On this foundation, we're mainly pushing in two directions: first, how to give the system common-sense judgment in anomalous scenarios, and second, how to use reinforcement learning to let the system continuously optimize its own performance.

For the common-sense component, the core is introducing something like a "chain of thought" mechanism. That is, when a robot enters a scene, it doesn't immediately execute an action — it first reasons about the task. For example, if asked to clean a kitchen, it will first observe the environment, then internally generate intermediate judgments like "I should pick up the plates first," essentially a form of "self-dialogue," before executing actions based on this reasoning. This intermediate reasoning process can mobilize the large-scale knowledge the model acquired during pre-training, thereby improving its ability to handle complex or anomalous situations. Reinforcement learning comes after the system already has preliminary capabilities — through repeated practice of specific tasks, it continuously optimizes through real-world experience. For example, we did an espresso-making demo where the system gradually improved its stability, speed, and overall efficiency through multiple repetitions of this process.

Of course, this is still just a starting point, and the entire system has substantial room for optimization, but the foundational framework is in place.

Making the Real World a Data "Reservoir" for Robots

Patrick: More crucial is your product. If you look at this generation of systems, you'll see cameras on the robot, possibly various sensors distributed in different positions. Essentially, data is collected through sensors deployed at different parts of the robot?

Sergey: Yes, and there's one point about sensors worth emphasizing: the actual number needed may be fewer than intuition suggests, yet still capable of completing quite a lot of tasks. For example, this platform has only three cameras — one on each wrist plus a base camera. No tactile sensors, no force sensors. The overall configuration is very basic, and the cost is quite low.

Of course, adding more sensors would theoretically improve performance. But if the learning method is good enough, it can compensate for insufficient perception to some degree. For instance, wrist cameras can partially substitute for tactile sensors because when contacting objects, you can visually observe local deformation and indirectly obtain "tactile" information.

Patrick: From expert systems in the 80s and 90s to the later empirical shift that "scale is the key" — this was somewhat counterintuitive. You were no longer hand-designing rules for specific tasks, but instead letting capabilities emerge through large-scale data. Within this framework, a key question is: where does the data "reservoir" for robotics come from? How do you build a data source of sufficient scale?

Sergey: Currently, no one really knows exactly how much robot data is needed to achieve embodied intelligence with strong generalization capabilities. But my judgment is that this question itself doesn't need to be precisely answered. What's more critical is pushing the system to a stage where it's "useful enough" to enter the real world and acquire more data on its own.

You can reference Tesla's approach. They don't worry about whether data is sufficient — more often, they face the problem of having too much data. In other words, the focus isn't on pre-calculating the size of some ultimate dataset, but on building a system that can cover multiple tasks and continuously generate data. Once this holds, you can continuously acquire data from the world and keep self-improving.

Patrick: You mentioned Tesla as an example. Its key insight is: even without AI, the system itself is already useful, and humans complete data collection while driving, thereby creating a positive flywheel. So why not start from a single, clearly useful robot form factor, first make a "useful enough" product, and then launch this data flywheel in a similar way?

Sergey: This is actually not a bad approach.

Patrick: Would you consider pushing in this direction?

Sergey: I don't think there's only one correct path here. In some scenarios, having humans remotely control deployed systems makes perfect sense. In others, partially autonomous systems are more suitable. This fundamentally depends on the specific application domain, because robots aren't a single-form-factor product. In home scenarios, users wouldn't want a robot that's constantly controlled by remote personnel. But in certain specific applications, this might not be an issue at all.

Common Sense and Challenges in the Robotics Context

Patrick: Taking Physical Intelligence's founding as a starting point, has there been anything that surprised you up to today, or anything about how research has progressed that exceeded expectations?

Sergey: What surprised me was progress in dexterity — it came much faster than I originally expected.

What was expected was that as data scale continuously expanded, the model would steadily improve in generalization capability, handling more different scenarios and objects. But what surprised me was that without too much targeted design, the system could also achieve quite dexterous manipulation capabilities.

Something similar happened with "cross-embodiment adaptation." Our model can run on many different robots, including multi-fingered hands, systems with different degrees of freedom, and so on. Of course, this process still requires data and some degree of fine-tuning, but the model architecture itself doesn't need to change — it doesn't even need explicit prompting to know which robot embodiment it's currently controlling. This also exceeded my expectations.

I originally thought that to adapt the system to faster, more complex, higher-dexterity tasks and different types of embodied systems, more complex mechanisms would be needed. But the actual result is that these capabilities can naturally generalize to some degree.

Patrick: I've always been very focused on the distribution of capabilities — specifically, which areas have already progressed beyond most people's expectations, and conversely, which capabilities still lag significantly behind what people imagine.

Sergey: Robotics researchers often mention a concept called Moravec's paradox. It holds across all of AI, but is especially pronounced in robotics. We have a cognitive bias where we subconsciously assume that what's easy for us should be easy for machines. For example, most people find calculus difficult but picking up a cup simple, so we think machines should more easily do the latter. But reality is exactly the opposite.

The reason things that are easy for us are easy is that if we couldn't do them well, we wouldn't survive. For instance, we're very good at spotting tigers in jungles because those who weren't good at it got eliminated long ago. Because of this, we develop a cognitive bias, mistakenly assuming certain things should be easy, when they're actually extremely difficult in engineering implementation.

However, machine learning is changing this situation. Using traditional programming to make a system grasp any cup in any environment is very difficult. But using machine learning methods, as long as there's sufficient data, this actually isn't that hard.

I think there will be a future trend where domains where data is easy to acquire will gradually become "easy problems," even if they're physically very complex. But there will also be the opposite situation: problems where data is hard to acquire, that require reliance on common sense, that need reasoning across multiple levels of abstraction, that need combining physical skills learned in other scenarios with knowledge obtained from the internet — these will remain difficult, and this is precisely where future technical breakthroughs are needed.

Patrick: What specifically did you mean by "common sense" earlier?

Sergey: In the context of robot learning, you can understand "common sense" as applying knowledge learned from other domains to current physical tasks through semantic reasoning. You can think of common sense as the opposite of "muscle memory." Muscle memory is when you repeatedly practice a movement in sports until you can automatically complete it with almost no conscious thought.

My understanding of common sense — not necessarily the standard definition — is that you learn certain facts through observation, reading, or hearsay, and when you enter a new situation, those facts happen to be highly relevant to the task at hand. You can make that connection, apply that knowledge to your current environment, and make the right decision.

Patrick: Can you talk about the concept of the "Robot Olympics"?

Sergey: There's a researcher named Benjie Holson who previously worked at Everyday Robots, a now-shuttered Alphabet project. He spent a long time thinking about tasks robots could perform, and wrote a very interesting article about it. He mentioned that China had hosted something like a "robot Olympics," where robots ran and jumped on tracks. But he felt those probably weren't the challenges actually worth paying attention to.

He proposed designing the Olympics around human daily tasks — things humans find simple but robots struggle with, like opening doors, cleaning an oily frying pan, or picking up dog poop with a plastic bag. Not difficult for people, but basically impossible for current robot systems. He listed about a dozen such tasks, and later we tried to take them on. It wasn't a formal research project, more like a test. We built a pipeline that could continuously take on new tasks, and wanted to verify: if given a set of tasks, could we just run them through this pipeline directly and have it work? In a sense, it was more of a test of our internal processes and model training infrastructure.

The result was that we completed almost all of them. The only one we couldn't do was turning a shirt right-side out, because the gripper couldn't fit into the sleeve — probably needs a different gripper design. There was also a "technical failure" with peeling an orange: the task required using fingers, but our fingers didn't have enough strength, so we used a small tool similar to a knife. Other than that, we basically completed everything.

What was more interesting to me: these videos are inherently "cool," but there was no targeted, bespoke development behind them. We just treated these tasks as test cases and ran them through our existing task-onboarding pipeline. This shows that when a system has sufficient generality, it can rapidly cover various seemingly "messy" tasks without adding extra complexity — reflecting the value of general capabilities.

Patrick: I'm curious — beyond dexterity or these "superhuman" capabilities demonstrated under constrained fine control, what other dimensions do you think robots could surpass humans in? And what other trends at Physical Intelligence do you find particularly interesting?

Sergey: We did a task earlier where we had robots plug in various cables — power cords, ethernet cables, things like that. When people do this, without extensive practice, they tend to pause frequently. Because it's not purely a physical motion; it requires cognitive judgments about alignment, angle, and so on. So overall it's quite slow.

If you're teleoperating a robot, the whole process gets even slower because of the added layer of indirect control. But for a machine, it's actually fairly straightforward to identify and eliminate these "pauses," significantly boosting speed and executing more efficiently. One general approach is to optimize this with reinforcement learning, though of course if you're just chasing speed, there are simpler tricks that can achieve similar effects.

This is a typical example. Humans are slow because of an information-processing bottleneck — constantly judging the current state. And increasing processing speed is precisely what computer systems are good at.

Patrick: Another interesting difference is that almost everyone has now used a chatbot, where back-and-forth Q&A is the norm. But now we're also starting to see, in code generation and other scenarios, that you can give it a more complex task and it can execute a long continuous process. The metric becomes "how long can it go without making a mistake." What's the robotic equivalent of this "long-horizon capability"?

Sergey: This is a direction we're actively pushing on. Methodologically, it's actually not fundamentally different from what you just described. Our current model reasons through tasks via something like a "chain of thought" process. Once you have this mechanism, it can support very long-horizon tasks. For example, a robot could complete an entire workflow: taking all dishes out of the dishwasher, putting them in their corresponding cabinets, wiping down the counter, and so on.

One interesting finding: about six months ago, we discovered the model had reached a stage where it could improve continuously through high-level instruction supervision alone. Specifically, put the robot in a new kitchen, have it execute a cleaning task, and it will definitely fail at some point. The traditional approach would be to add more teleoperation data to cover more kitchen scenarios.

But we tried something different: instead of adding teleoperation data, we only added semantic annotations to the robot's existing execution traces. That is, describing its experience in language, without adding any new low-level action data.

It turned out this approach also improved the model's generalization.

This signals a key shift: the system's bottleneck has moved from low-level execution capability to the middle layer — understanding the scene and choosing the next action. And this layer can be supervised with language. This matters because it means people can directly instruct robots through conversation, essentially providing real-time "coaching" optimization.

Multiple Technologies Co-Driving Development; Industrial Deployment Still Has Much Room for Exploration

Patrick: We're already seeing the changes large language models are bringing to work, business, and the economy overall, and the changes in engineering are already quite evident. How do you view a similar phase potentially emerging in robotics? When systems are both general and useful, how will the world deploy them efficiently? And in this early stage, what do you expect to change first?

Sergey: This is an interesting question. Honestly, I'm not sure either. People might have guesses, but it's hard for anyone to predict in advance how large language models will evolve.

I'd return to the core idea: the key is getting more people to try different possibilities. One important characteristic of large language models is their low barrier to application — many people can quickly build an interesting prototype. The underlying layer might just be calling models like ChatGPT, but they can rapidly experiment, iterate, and see what they can create.

The value of this pattern is that when large numbers of smart people are rapidly iterating and trial-and-erroring in different directions simultaneously, it generates strong innovative capacity. This is also why Physical Intelligence places heavy emphasis on community engagement. We've open-sourced our models and want to collaborate with more robotics companies, because we believe getting more people involved, trying more directions, is itself an important force driving the field forward.

Patrick: Elon Musk often says "the factory is the product," and the hardest part of the whole system is scaling — whatever the final form, you have to get it to the hundred-million-unit scale. How do you think about this part? Or is it still relatively distant at this stage, not worth investing too much energy in?

Sergey: No, it's definitely an important question. I'm just not sure it's the most urgent priority right now, but it's certainly one piece of the system. From my earlier answers you probably gathered that I tend toward solving the "hardest parts" first, then opening up more experimental room in other areas.

Manufacturing robots at scale is itself very difficult. If you don't even know what software will ultimately run on them, or whether the robot form factor is correct, scaling becomes even harder. So I think the value of general-purpose AI tools — like robot foundation models — is that they can first reduce many other uncertainties and make the overall solution clearer. When you truly enter the scaling phase, you can at least be more confident in judging whether this is feasible.

Patrick: Many listeners to this episode are entrepreneurs or business operators. A very common question is how traditional companies should start thinking about using large language models, or how to prepare for their continued improvement. If we pose the same question for robotics, how would you answer?

Sergey: This is a very good question, but also very difficult to answer, because the technology is changing so fast.

Let me use a concrete example. One key uncertainty is whether robots will rely more on "demonstration data" or more on "data obtained through autonomous interaction with reinforcement learning." This is very specific, but quite representative.

We're currently exploring both paths simultaneously, and they're clearly both important. But for businesses, the preparation looks completely different depending on which way you bet. If the future mainly relies on teleoperation generating large amounts of demonstration data, supplemented by a small amount of autonomous experience, then companies need to build a data collection and annotation system. But if it's the reverse — only a small amount of demonstration data, with heavy reliance on large-scale autonomous experience — then the entire technical and organizational preparation path looks completely different. The key question is whether the ratio between these two modes is 9:1 or 1:9. We may only gradually see this over the next few years. And this uncertainty has a very large impact on companies' decision paths, which is why this question is difficult to give a standard answer to.

Patrick: From a business perspective, is the reasonable way to think about this first clarifying the "labor cost structure" within your own operations? And how do you view this — how will this technology change the very nature of "labor"?

Sergey: Code tools are a good reference case.

It's not that code tools appeared and we no longer needed software engineers. They increased individual engineers' productivity. To make these tools truly usable, work was needed to make them accessible to people, and continuous technical development was needed to adapt them to different application scenarios. These processes co-evolve, and are still changing — coding agents and code completion tools themselves continue to evolve. It provides a good reference for how AI tools collaborate with humans, improving efficiency while also bringing new challenges.

I think we're likely to see a similar trajectory in robotics. The more realistic picture isn't humanoid robots arriving and humans departing, but work being reconfigured. Some parts can be done by robots alone, some require human-robot collaboration; in some cases, people need to do extra work to improve robot efficiency, and in others, robots enhance human efficiency.

Overall, it's more of a collaborative dance — like what we've already seen with coding tools.

Patrick: Name a robot you personally love that isn't part of the Physical Intelligence ecosystem.

Sergey: I really like Boston Dynamics' robots, especially the latest version of Atlas. In some ways it's very close to human, but in others it's clearly not — the joint design has a larger range of motion, enabling some very interesting movements. It's also extremely agile and capable of impressive demonstrations. I've always admired Boston Dynamics' work.

Patrick: Can you talk about the difference between "cool" and "useful"? Boston Dynamics' robots are undeniably cool — backflips are captivating. But it's hard to identify any real scenario where a robot actually needs to do a backflip.

So I'm curious: how do you trade off between "cool" and "useful"?

Sergey: I don't think our approach is necessarily the only right one, but under the premise of "useful," we try to make it as "cool" as possible. This actually shows up in our papers and videos. Our primary criterion is: what direction best advances the technology toward a truly general, broadly applicable robot foundation model? Under that premise, we deliberately stress-test the system with the hardest tasks — and these hardest tasks often happen to look the "coolest." For example, we didn't set out to build a robot that makes coffee or folds laundry, but in the process of building a general system, we select these tasks as tests because they're sufficiently challenging and representative, helping us determine where the system's capability ceiling lies.

Patrick: Can you infer from the facts that Boston Dynamics has spent years doing very cool demonstrations, but doesn't seem to have delivered genuinely useful products to customers?

Sergey: Good question — and I think in a sense, that applies to many robotics companies.

Overall, I think such demonstrations have value, provided they showcase key challenges on the path toward "useful, productionizable capabilities." Of course, you can also do demonstrations that don't genuinely point toward practicality. If a demonstration serves a clear goal, it can help people understand what might be possible in the future, while also posing a clear challenge. The key is to be honest about what the challenge actually is.

Patrick: To what extent do you think about commercialization? So far, the best-selling robot in the consumer space is actually Roomba — which is itself somewhat surprising. Of course, we may also be on the eve of some kind of Cambrian explosion. How much energy do you spend thinking about: what is the ultimate product form? Is there a product path that can kickstart the data accumulation process?

Sergey: I do spend time thinking about it, though it's hard to converge on a very specific answer right now — I think about feasibility from the "space of possibilities." In fact, much of what we do, whether developing models, trying different tasks, or doing demos like "Robot Olympics," is essentially prototyping: if these capabilities were applied to real scenarios, what would that look like, and what problems would emerge?

Of course, the degree of realism varies across different levels. This is something we continuously think about, though we haven't approached a clear answer yet.

But what is certain is that there exists a "space of possibilities," and many of our 2026 plans involve exploring different directions within this space.

Patrick: Looking back at the history of general-purpose technologies, if this technology ultimately succeeds, it clearly belongs in that category. You typically see a suite of complementary technologies co-evolving around the core technology. Large language models are obviously a direct complement to your work. Beyond that, are there less obvious domains or trends that are similarly supporting your technical direction?

Sergey: One interesting change is that robot hardware has become dramatically cheaper over the past few years.

About a decade ago, when I first started doing robotics research, I used a robot called PR2 that cost roughly $400,000. When I later set up my lab at Berkeley, the robots we used cost around $30,000; now, a single robotic arm on such a device might cost only a tenth of that, with room to fall further.

This isn't driven by a single technology, but by the combined effect of software and hardware. For example, some of the low-cost arms we use now wouldn't actually work in traditional industrial environments, because traditional control methods relying on high precision can't make good use of them. As you mentioned, this is the result of "multiple technologies acting together" — these advances collectively drive down costs and make thinking about general-purpose robots much more realistic today.

Patrick: If someone wants to track important developments in the field in a relatively technical way, where does this information typically appear?

Sergey: A lot of it is in papers, though papers aren't actually an easy channel to extract information from, because it takes effort to sift through them, determine what's "signal," and understand what the results actually mean. Papers are typically written for people already familiar with the field's background, building on past research — but they are indeed an important source.

Another characteristic of robotics, and the broader technology field, is that public-facing content — demo videos or social media showcases — often doesn't accurately reflect the true technical state. Because this content tends to show the "boundary" of capabilities, not the complete capability itself.

To truly understand what these demos mean usually requires deeper analysis. So papers remain the primary channel, and sometimes even that's not enough — you need to talk directly with researchers to get a more accurate picture. This may not be an ideal way to access information, but scientific research often works this way.

Patrick: Your position is interesting — you're at the center of research, yet also interact with people from diverse backgrounds and get asked questions. Is there a question you feel people aren't asking, but should be?

Sergey: The variant of your earlier question about how companies should prepare is actually an important one. For example, if I want to start using autonomous robots in a specific scenario, where should I begin? Should I first build out the operations system? Do I need to modify the task itself to make it more suitable for robot execution? Or should I redesign the hardware — create a new set of equipment to match your software?

Many people make assumptions about these questions. A common one is: since machine learning needs data, I should first figure out how to collect data. But that's often not the best starting point, because the key isn't just whether you have data or whether it's suitable. Some data is easy to obtain, like videos of human operations, but that doesn't mean it's effective for training.

And this depends heavily on the specific scenario, as well as your judgment of the technical path. So I think there are many default assumptions here, and these assumptions themselves may not hold. Even if you ask me, I may not have a standard answer, but what's certain is that there's significant room for exploration.

End-to-End Learning Remains Controversial

Patrick: What are the main controversial issues in your robotics research community right now?

Sergey: Obviously I'm in academia, so "controversy" often just means arguments at conferences (laughs). But from personal experience, the evolution of the debate itself is quite interesting.

Early on, my core disagreement with others was whether "learning" had any value in robot AI. Part of the reason was that in traditional engineering systems, robots and software seemed very different — robots are physical entities that affect their surroundings, involve safety concerns, and enter various complex or even anomalous situations. So it took the research community a long time to gradually accept the idea that you don't necessarily need to explicitly program in physical knowledge — for example, you don't have to put a physics simulator inside the robot for planning; the system can "learn" these laws itself.

By now, people generally accept the importance of "learning" in robotics, but there's still no universal consensus on whether end-to-end learning is the right path. In other words, The Bitter Lesson — that you shouldn't program in how humans think machines "should think," but let it learn from data — hasn't been universally accepted.

Renowned computer scientist and foundational figure in modern computer reinforcement learning

Rich Sutton's 2019 reflective essay on AI research

I think the opposition has reasonable arguments too. A strong counterargument is: if you want reliability in an extremely complex open world, you have to leverage the physical world knowledge we already possess — after all, this content is already well-documented in textbooks, so why not directly incorporate this knowledge into the system?

Patrick: Then what is compositional learning — can you explain that?

Sergey: A more intuitive example comes from one of my students. He had a language model write out "steps to make a sandwich" using the International Phonetic Alphabet. IPA normally only appears in dictionaries to mark pronunciation — it's a very specialized symbol system, and you almost never see anyone write full paragraphs in it. But if you ask a sufficiently capable language model to do it, it can produce complete paragraphs in IPA. That's compositional generalization. The model has never seen this specific mode of expression — using this alphabet system to write paragraphs — but it understands what a "paragraph" is, and it understands that different symbol systems can be combined, so it can complete the task.

Similar capabilities can apply to robotics. A model can first learn a set of foundational skills, then combine and recombine them to solve new problems.

Patrick: This inevitably makes me wonder: what do you think will be the last category of tasks to be achieved for robotic systems?

Sergey: I think changing a baby's diaper would be extremely, extremely difficult. This is fundamentally an expression of Moravec's paradox — humans are incredibly capable at certain skills, like physical manipulation and interacting with other people. These capabilities are core to our survival. So behaviors involving human interaction, like genuinely helping someone, assisting a person out of bed — these are actually much harder than people imagine. I believe tasks like elder care, infant care, these will be very difficult, and likely harder than most people expect.

Patrick: And the risk in these scenarios is extremely high. I'd certainly want my child to be among the last ones entrusted to robot care.

Sergey: Many scenarios have high risk coefficients — this example just happens to be more extreme, which can create the illusion that things are easier than they actually are. Because humans are highly evolved in two areas: interacting with others and physical manipulation. When you help someone up the stairs, or assist them out of bed, you don't really need to consciously think through each step — you just know what to do. I think this is a classic manifestation of Moravec's paradox, perhaps even its most extreme form.

Patrick: If we analogize large language models to a "mind that has already learned almost everything," then when we imagine the "brain" of a robot model, what are its remaining blind spots? What key domains has it not yet truly understood, found difficult to enter, or struggled to learn?

Sergey: Humans have one very prominent capability: using "physical analogies" to understand other situations.

I'm not sure whether large language models possess this, but humans certainly use it extensively — not just in daily life, but in tackling complex problems. For example, when you say "a company has strong momentum" — that's a physical analogy, yet you immediately understand its meaning without additional explanation. But if you think about it carefully, this is actually a very complex concept; the word "momentum" carries a lot behind it.

There's also Richard Feynman's interview where he mentioned using words like "spin" to understand subatomic particles — particles aren't actually rotating like tops, but these analogies help us build understanding. What's more important is that these analogies support reasoning, even leading to new conclusions, and these conclusions often hold true. This is actually quite unusual. Human understanding of the physical world runs so deep that you can both say "a company has strong momentum" in everyday language, and use similar approaches to advance foundational physics theory.

Large language models — it's uncertain whether they truly possess this capability. But for physical interaction, causal structure, and deeper levels of understanding, there remains something distinctive, and humans clearly derive tremendous capability advantages from this.

Patrick: Another surprising fact is that, globally speaking, the people who have genuinely driven LLM progress are actually very few — someone like Ilya Sutskever, for instance. What's the situation in robotics? Worldwide, roughly how many people are truly making an impact on this technical path? And I'd like to press further: what counts as "good research"?

Sergey: I think these kinds of questions are actually very difficult to answer in science. Because when we look back at history, we tend to emphasize certain pivotal moments — in machine learning, you can say AlexNet was an important breakthrough, and that's certainly fine. But equally important is that these advances happened because large numbers of people were constantly trying all sorts of different approaches, many of which were even failures, but these failures themselves were highly instructive.

I mentioned earlier that there was controversy around end-to-end learning. To some extent, without this controversy, robotic learning might not have developed in the same way. We can look at successful cases and point to who achieved important results multiple times. But in the actual process of scientific research, progress isn't driven only by these "moments of success." Many failures, even some ideas that seemed not so good, were also constantly correcting direction and pushing research toward better paths.

Patrick: Yes, the example you gave earlier was quite interesting — insights in research where just giving the system some "guidance" makes it better. Such insights seem very powerful, with high leverage. I'd like to press you on what you've learned about what kind of person can become an excellent researcher.

Sergey: Research and engineering are indeed different, because in research, the key is answering a question, and this often requires making trade-offs in certain places, even "taking shortcuts." One of the most subtle decisions in research is when to try a new direction versus when to persist on the current path — this is extremely difficult to judge. If you judge incorrectly, or don't persist long enough, you might be very close to the answer but stop at the final moment, which would be a great shame. Conversely, you might also invest years on a path that will never succeed, stubbornly "bashing your head against it" without results. So the ability to judge between "pivoting appropriately to explore more possibilities" and "continuing to dig deeper on the current problem because the answer is already close" is often one of the most crucial capabilities. Some people have strong intuition for this judgment, and that's very important.

Patrick: You've been in this field for a long time and are among the top researchers. So at the human level, what do these excellent researchers have in common? Compared to ordinary people, what are their notable differences?

Sergey: Actually, there's no fundamental difference. Those I deeply respect, the truly excellent people — it's hard to summarize a unified set of personality traits. The only commonality might be that there is no true commonality. If forced to say something, perhaps effective research requires some degree of passion. But even this passion can come from completely different sources.

I've seen some highly effective people who are entirely driven by "novelty" — they don't care what the technology can ultimately do, or whether it's useful; they just want to constantly explore new ideas. I've also seen others who are intensely focused on solving a specific concrete problem, equally engaged in building systems, running experiments, repeatedly refining some detail, willing to do whatever it takes as long as the problem gets solved. Both types have very strong execution capability.

---

"I'm on the optimistic end among researchers, but the conservative end among entrepreneurs"

Patrick: When you look ahead at advancing this work, what makes you feel most uncertain?

Sergey: The timeline is very uncertain. To some extent I've actually become more optimistic in my time estimates than when I started, but it remains uncertain because this technology has a kind of "bootstrap challenge" — it needs to reach a sufficiently useful level first before robots can be deployed to perform real tasks and acquire data at scale in open environments.

And once that threshold is crossed, there will be a kind of "phase transition." So the timing itself contains great uncertainty.

At the same time, different technical paths will significantly affect the timeline. As mentioned earlier, whether you collect data through teleoperation, through autonomous systems, or some intermediate form like shared control or "coach-style" interaction — these all change deployment methods and the path to acquiring data in real environments. Because these factors compound together, I think the temporal uncertainty is quite substantial.

Patrick: We've talked a lot about long, uncertain timelines. So what is the most immediate, most concrete, and most visible next problem you need to solve?

Sergey: Without revealing too much detail, I can say that one of our current focuses is better understanding "middle-level reasoning" in this problem. Because we feel that for acquiring low-level physical behavioral capabilities, there's already a relatively clear path. But to make these low-level capabilities truly generalize, you need to introduce large amounts of commonsense knowledge, and the way this knowledge is expressed may itself be crucial.

Large language models are convenient in certain representations — for example, it's very easy to transform text into text. But this isn't necessarily the most suitable representation for embodied systems. In real scenarios, systems sometimes need to think in spatial structures, sometimes in semantic expressions, sometimes perhaps in other forms.

Therefore, a key question is how to organize this internal "thinking process." And the answer to this question is likely different in the field of embodied foundation models than in large language models. This is one direction we're currently pushing forward concretely.

Patrick: If you gathered the hundred most knowledgeable and active researchers in the industry to judge whether this technology will eventually reach near-"unlimited capability," and roughly when this progress might happen, where do you think your view would fall in the distribution?

Sergey: Among traditional robotics researchers, I'm on the more optimistic end; but compared to robotics entrepreneurs, I'm actually on the more conservative end.

This contrast is quite interesting — entrepreneurs are naturally more optimistic, which is easy to understand. As for why I'm relatively optimistic among researchers, it's because robotics has a very long history, but genuinely successful cases are actually quite few. In other words, especially in the robotics AI portion, if you look objectively, most robots that are actually functioning in real work today are still running technology developed in the 1980s. This shows that the problem itself is extremely difficult — it's not just one generation's problem. Because of this, caution is warranted. Even if we've made progress on certain sub-problems, many others remain unsolved.

The reason I'm relatively optimistic is that I have concrete, visceral understanding of where the difficulties have been in the past, and I can also see some possible "puzzle pieces" that could be used to address these problems. But on the other hand, as my co-founder Karol often says: when you climb one mountain, you only then discover there may be more mountains behind it. And in robotics, we've seen far too many such "mountains."

Patrick: Given that this field requires long-term commitment and endurance, who or what inspires you most?

Sergey: I'm actually quite inspired by Boston Dynamics. Their technical approach is certainly debatable, but the sustained demonstration of capabilities once thought impossible is valuable in itself. Even if those demos come with plenty of caveats and constraints, they still shift people's sense of what's possible.

I'm also inspired by organizations that can create an "experimental atmosphere" — some research institutions do this very well. I think OpenAI is one example. It created an environment where individual researchers could try out ideas and push them forward. ChatGPT, for a while, was really more of John Schulman's personal project than a top-down corporate strategy. That mechanism — letting a personal project grow into something with global impact — is deeply inspiring.

At Physical Intelligence, my co-founder and I also hope to create something similar. Though it's genuinely hard to build an organization with that kind of capability.

Patrick: Google used to have similar mechanisms, like letting employees spend a portion of their time on projects they wanted to pursue. Is that the same spirit as what you're describing, or is there a difference?

Sergey: When I first started at Google, I was genuinely shocked by the sense of "leverage" I had. In 2015, I worked on a project with colleagues internally called "Arm Farm." We put dozens of robots in a lab and had them collect data.

This project was entirely bottom-up. I'd heard there was a warehouse full of unused robots, so I asked Jeff Dean and Vincent Vanhoucke if we could move them into a lab. I expected them not to take it seriously — I'd just joined, and I was still a relatively junior research scientist. But Jeff's response was: sure, go for it, what do you need? I remember thinking I'd never imagined having that kind of space and influence.

When a place can unleash creativity and grant that kind of autonomy, it can become something very different.