Physical Intelligence Tech Team: How to Achieve Vibe Coding for the Physical World? | Vital Views

General-purpose robotics is a critical path for AGI to move from the digital world into the physical world, and within the theme of AI robotics, Physical Intelligence (PI) is undoubtedly one of the most technically deep and influential research teams.

In April this year, building on π₀, PI released a new VLA model with open-world generalization capabilities: π₀.₅. PI stated that π₀.₅ can maintain performance comparable to its original training environment even in some previously unseen environments.

So from a technical perspective, what exactly is the relationship between VLA, LLM, and VLM? To build a general-purpose robot brain from scratch, how did PI construct its data pipeline? And how does PI's newly proposed "Knowledge Insulation" mechanism actually work?

This article presents the Physical Intelligence core technical team's interpretation of robotics' past and present technical paths, and shares PI's frontier exploration in data collection, algorithm design, and multi-robot general models.

Oasis Capital curated and recommends this content, sharing the current state of embodied intelligence technology development.

Full read: approximately 10 minutes. Enjoy.

---

While robots have achieved tremendous success in structured industrial production lines, their flexibility and intelligence in open scenarios such as homes, offices, and public spaces still fall far short of human capabilities. Therefore, we must further expand the capability space of LLMs.

VLM Extends LLM with Visual Perception

Visual-Language Models (VLMs) are a class of multimodal models that combine visual and linguistic information, capable of understanding and generating natural language descriptions related to image content, or analyzing images based on text prompts. Compared to LLMs, VLMs integrate visual information on top of language understanding capabilities, enabling them to process not just pure text input but also complex information involving visual content.

For most researchers, the VLM training pipeline is already fairly well-established. Specifically, VLMs take images and text as input, and through a pre-trained Transformer architecture, generate text output autoregressively in response to prompts. The conventional training flow typically involves obtaining a pre-trained model, then scraping general data from the internet supplemented with a small amount of task-specific data, fine-tuning on large-scale clusters, and finally deploying the model to the cloud using off-the-shelf inference and deployment tools.

VLM architecture

VLA's Goal Is to Directly Output Robot Action Commands

VLMs enable machines to understand and generate natural language descriptions, but compared to requirements in other fields, the demands of robotics are fundamentally different. Because robots must not only understand scenes but also generate precise action commands to drive mechanical execution of complex tasks in real time.

Basic principle of VLM processing images and text instructions

Visual-Language-Action models (VLAs) are essentially VLMs applied to robotics. They can receive image and text input, and also integrate the robot's current state information, such as joint angles and arm position. Their goal is no longer to generate linguistic descriptions of scenes, but to directly output action commands that control the robot.

This shift introduces enormous engineering challenges, because robot actions require high-frequency, low-latency, continuous, and precise control—fundamentally different from the demands of traditional text generation models when handling continuous action sequences.

VLA architecture

Over the past three years, multimodal LLMs and VLMs have evolved from initial conversational systems to multimodal reasoning systems integrating visual encoding and reinforcement learning, performing well in tasks such as coding assistance and content creation.

The development path of VLAs is similar to that of VLMs, but started slightly later. Taking RT-2 as an example, early multimodal language models began introducing visual encoders, with some models even specifically trained for robotics tasks.

Although these models primarily existed as proof-of-concept prototypes, they could already direct robots to complete diverse object grasping tasks in familiar environments. Despite limitations in robot data volume, these attempts greatly stimulated industry attention and expectations for general-purpose robot intelligence.

RT-2 (Robotic Transformer 2) is a VLA model introduced by Google Research, extending the VLM architecture. Its training data sources include web text-image pairs, robot execution examples, and other multimodal data.

By the second half of 2024, further multimodal models for robot manipulation emerged, with the industry successively launching solutions such as Gemini for Robotics and NVIDIA GR00T.

However, these systems remain primarily at the proof-of-concept or early demonstration stage.

Evolution of VLMs and VLAs

---

The PI team's core philosophy is: if a task can be completed by human remote control, the model can likely learn it too.

This idea originated from early observations of the PR2 robot. Although PR2's hardware was very advanced for its time, its autonomous decision-making capabilities were limited, still relying on human remote control or preset task planning.

The PR2 robot (Personal Robot 2) was a service robot developed by Willow Garage, widely used in robotics research and development due to its open platform design and rich software ecosystem (such as the ROS robot operating system).

PR2 robot teleoperation example

This reality strengthened the PI team's conviction: the core obstacle to robotics development lies not in hardware performance, but in how to endow robots with human-like intelligence—enabling them to accurately perceive complex environments, understand multimodal information, flexibly plan, and robustly execute actions. PI's goal is to build a unified model based on vision, language, and action, giving robots capabilities similar to humans.

To achieve this goal, PI conducted research based on VLAs.

In the second half of 2024, PI released its first multi-robot general model π₀, capable of executing complex tasks across multiple robot platforms. Physical Intelligence research scientist Jost Tobias Springenberg evaluated this model as one of the most dexterous and practical multi-robot general models available at the time, and it has been open-sourced. π₀ also incorporated architectural optimizations, using a diffusion model to generate high-frequency actions to meet robots' real-time and responsiveness requirements.

π₀ executing complex tasks example

PI Believes Robot Intelligence Development Can Be Understood Through Three Dimensions:

Capability, Generalization, and Performance.

Capability refers to whether a robot can accomplish complex tasks never before achieved. Early robots were mostly limited to repetitive industrial assembly line operations, struggling to handle the complex and variable scenarios of daily life—such as folding clothes, assembling boxes, or organizing desktops.

With the release of π₀, the PI team demonstrated across multiple platforms that robots could complete multi-step manipulation tasks. In actual testing, robots exhibited human-like flexible adaptability, actively adjusting when execution deviated, even retrying multiple times until results were satisfactory. This behavior is entirely different from traditional robots dependent on preset paths, and confirms the team's view: anything achievable through teleoperation, models can similarly learn to accomplish.

However, capability alone is insufficient to make robots practical. If robots can only work in trained environments, their actual value would be greatly diminished. Therefore, the second dimension is Generalization: the ability to correctly execute tasks in completely unfamiliar, unseen environments.

Once robots possess capability and generalization, the remaining greatest challenge is Performance.

PI co-founder and CEO Karol Hausman believes that current models remain "demo ready" rather than "deployment ready." Despite significant progress in capability and generalization, robots at this stage still largely remain at the demonstration level, with failures still occurring frequently.

Performance improvement means not only higher task success rates but also stability, speed, and robustness. For example, in clothes-folding experiments, robots may move slowly and occasionally make errors, but often manage to complete an entire pile through self-correction. This process of "err—correct—persist until success" brings robots closer to human working patterns, rather than the traditional robotic mode of "once an error occurs, complete failure."

But performance improvement cannot rely purely on scaling data volume. Hausman emphasizes that achieving human-level robot performance requires new algorithms and new thinking, not merely more data or larger models. This means future breakthroughs may come from innovations in training paradigms, decision-making mechanisms, and system design.

Therefore, for the PI team, the next stage's focus lies in exploring how model capabilities improve with increased data diversity. Based on this direction, the team developed the VLA model π₀.₅ with open-world generalization, aiming to enable stable and reliable generalization for robots across tasks, environments, and platforms. Preliminary results already show that π₀.₅ can maintain performance comparable to its original training environment in some unknown environments.

π₀.₅ task completion example

π₀.₅ joint training data framework

Notably, among numerous potential application scenarios, the home environment was selected as the first priority landing direction for PI's models. This choice is based on the following considerations:

• Highly complex environment: Homes contain rich and diverse items, with complex and variable spatial structures, and frequent and complex human-robot interaction—this is the ultimate test of robot intelligence;

• Very clear demand: Demand for automated household chores is extremely strong, such as clothes folding, floor cleaning, and item organization, representing massive market potential;

• Easy data collection: Feedback from household users and remote teleoperation enable continuous data accumulation, giving the model access to ever more diverse scenarios and improving generalization and robustness;

Because of this, PI can leverage the rich data and clear demand of home environments to rapidly move its models from lab validation to real-world deployment.

PI Builds Its Robot Data Pipeline from Scratch

Physical Intelligence took a ground-up, systematic approach to robot data collection and model training.

No off-the-shelf solution on the market could simultaneously meet the demands of high robustness and high dexterity, so the PI team built virtually the entire data engine from scratch. The goal was to give robots preliminary autonomous manipulation capabilities in short order, while laying the groundwork for scaled expansion in the years ahead.

Experience on the ground has shown that building and running the data pipeline is one of the most critical parts of the whole process. More than half of the team's effort went into constructing the data system, collecting data effectively, and ensuring data quality.

Specifically, the team designed a dynamically expandable task set to test what actions could currently be performed — things like folding laundry and buying groceries. Human operators control the robot through a purpose-built runtime environment and remote control system. Using a guiding arm, the operator has the robotic arm mimic their own movements; dedicated software then converts the operator's motions into control signals for the robot arm's end effector. Through this method, the robot can not only perform complex, delicate operations but also collect high-quality training data.

PI built its data engine from 0 to 1 to drive robot learning

The entire data collection process is managed by a cloud-based system that monitors task execution in real time and dynamically schedules data collection and annotation work. Once annotated, data is stored in large data buckets, then filtered according to training needs and fed into the model training pipeline. After training, the model masters the strategies needed to perform tasks and demonstrates a degree of autonomous planning and execution capability.

Each small square on the screen represents a specific task executed by an operator on a given Tuesday

This methodology has yielded significant progress in data collection.

At the project's outset, the largest publicly available dataset on the market was the Open Cross Embodiment dataset, containing roughly 3,800 hours of robot manipulation data sourced mainly from static scenarios in robotics research labs around the world.

The static scenarios in the Open Cross Embodiment dataset refer to robot operations in fixed environments — such as robotic arms performing grasping and transport tasks on lab benches or in controlled spaces — where environmental structure and object placement tend to be relatively uniform and lack the complex dynamic variation of real-life settings.

By contrast, within six months of running its own autonomous data collection pipeline, the PI team had accumulated approximately 10,000 hours of robot manipulation clips spanning dozens of different home environments and hundreds of task categories, including folding shirts, transporting objects, and tidying up.

This data was more grounded in real-life scenarios and far richer in task variety than what had come before.

PI's autonomous data collection pipeline

After another six months, the team had not only gathered more data in static environments (such as fixed kitchen and bedroom operation scenarios) but also begun large-scale data collection using mobile manipulation systems. A mobile manipulation system refers to a robotic arm mounted on a mobile base, capable of moving and operating flexibly across larger spaces and capturing dynamically changing environmental data. Data collected during this phase not only covered significantly expanded task scope but also greatly enhanced diversity, specifically in the following ways:

• Environmental diversity: Covering hundreds of distinct scenarios, including kitchens, bedrooms, and living rooms of different layouts, with varying furniture arrangements, lighting conditions, object types, and placements;

• Dynamic variation: Unlike the early static experimental environments, mobile manipulation systems can capture dynamic processes such as objects being moved within the environment and human-robot interaction;

• Task complexity: Expanding from simple grasping to more complex, delicate daily-life operations such as folding laundry, organizing items, and transporting objects.

The growth in data scale and diversity enabled robots to demonstrate stronger autonomous execution capabilities and flexibility across multiple tasks, environments, and platforms.

After scaled data training across diverse scenarios, robots improved their ability to autonomously execute tasks

Notably, to further verify the source of generalization, the team trained models on different types of home environments while holding total training sample size constant, then tested them in a home the model had never encountered.

The results showed that the more homes covered in training, the better the model performed in the new environment — even outperforming models specifically trained in that environment. This finding demonstrates that increasing data diversity is a critical path to improving robot generalization.

The more training scenarios, the stronger the model's generalization in the open world

"Knowledge Insulation" Restructures the Training Pipeline

Beyond its systematic design for robot data collection, the PI team also achieved a modular upgrade to its model architecture. The new model adopts a multimodal fusion design that more effectively integrates visual, language, and action information, enabling more precise environmental perception and task understanding.

On the data side, the model draws not only on diverse static and mobile robot manipulation data but also incorporates web-based image-text data, object detection data, and interaction data with language annotations.

These improvements made model training itself a challenge, because conventional training schemes face a clear dilemma: training the backbone directly with a diffusion model risks degrading the model's ability to understand images and text, while also slowing training speed; fine-tuning on robot data alone, meanwhile, yields poor generalization.

Catastrophic forgetting refers to the phenomenon in neural networks where, during continual learning of new tasks, parameter updates in the face of new tasks cause knowledge learned on old tasks to be rapidly overwritten or lost.

In multimodal or robot learning, if a model acquires strong visual and language understanding capabilities during pre-training but subsequent training on action data lacks carefully designed training strategies, the model may "forget" its original semantic understanding abilities, resulting in low-level control without high-level task comprehension.

To solve this problem, PI's research team proposed the "Knowledge Insulation" mechanism, innovatively restructuring the training pipeline:

First, continuous action sequences are discretized into text-like tokens, allowing the backbone to use natural language processing training paradigms;

Second, gradient backpropagation is truncated when connecting the Action Expert diffusion module, protecting the backbone from interference by diffusion loss.

PI's "Knowledge Insulation" mechanism — related paper

This approach not only prevents the Action Expert diffusion module from disrupting backbone parameters but also increases model training speed by 10x, allowing π₀.₅ to perform on zero-shot tasks on par with fine-tuned models while maintaining its original generalization capabilities.

Specifically, π₀.₅'s architecture uses a pre-trained Transformer at its core, with an extended Action Expert Transformer submodule. The backbone handles perception and scene semantic understanding — it can both predict potential problems in a scene and decompose high-level human instructions into executable subtasks, such as parsing "clean my bedroom" into specific actions like "pick up the pillow."

The Action Expert Transformer receives intermediate outputs from the backbone and runs at a higher frequency, combining a diffusion model with flow-matching objectives to generate continuous, smooth control actions, achieving efficient bridging from semantics to physical execution.

π₀.₅ data architecture

Additionally, the research team proposed a hardware-software decoupling strategy that places intelligent software at the system's core, reducing dependence on specific robot hardware. By building highly portable, reusable software models, the system can rapidly adapt to multiple robot platforms, significantly improving flexibility and efficiency in real-world deployment.

Although PI has made a series of improvements to data collection, model architecture, and training, deploying robots in the open world still faces three core challenges.

Data Gap

While VLMs can rely on massive amounts of publicly available image-text data for rapid iteration, the data required for robot manipulation is far less abundant and accessible than web text.

Unlike autonomous driving's "perception-avoidance" logic on closed roads, robot manipulation requires actual physical contact. Without human intervention, task difficulty grows exponentially. Because every contact can trigger a cascade of uncertainties — such as object deformation, changes in friction coefficients, or disturbances from nonlinear effects — the complexity far exceeds simple path planning.

Moreover, data scarcity isn't just about quantity; it's about quality. Collecting large amounts of incorrect data doesn't help improve robot behavior. What we need is the right data: temporal data that includes closed-loop feedback from physical interactions, such as the correspondence between robotic arm force application and cloth deformation. Without anything resembling the internet as a universal data source, gathering real, effective interaction data in robotics not only has high barriers to entry but is also extremely expensive.

Performance Robustness

Karol Hausman and some team members believe that the biggest challenge facing robot VLAs today is performance robustness — how to truly move from "demo ready" to "deployment ready."

This problem cannot be solved simply by scaling up parameters or increasing data volume; it requires breakthroughs in algorithms and architecture. Although robots still make errors when executing tasks at this stage, they have already demonstrated human-like adaptability. They can self-adjust through "trial-and-error correction," making multiple attempts until the task is completed, bringing robot behavior closer to genuine human operation.

Currently, while we can build VLA models on top of VLMs, robot task execution places extremely high demands on action command frequency and latency, while also requiring the ability to handle physical disturbances and perceptual uncertainty. π₀ has completed cross-domain transfer validation since its open-source release, including aerial grasping with drones and precision operations with surgical robots. However, to achieve commercial deployment, the temporal misalignment in action execution still needs to be resolved.

This is because robots currently typically obtain a single observation before reasoning, but the reasoning process often takes hundreds of milliseconds, during which the model assumes the environment remains static. The real world is dynamic, however. By the time new action predictions are generated, the environmental state may have already changed, leading to execution errors.

To address this challenge, Kevin Zakka proposed an approach similar to image inpainting. Just as diffusion models fill in missing regions of images, in robotics scenarios the model can treat already-executed actions as "known actions" and not-yet-executed portions as "masked regions," then complete and correct future trajectories based on this information. This method requires no retraining of the main model and constitutes a purely algorithmic improvement.

In other words, while executing the current action block, the robot can compute and repair subsequent trajectories in parallel, thereby improving overall robustness. However, Kevin also emphasized that if inference speed can be significantly improved in the future, latency-related problems will be fundamentally alleviated.

Kevin Zakka is a PhD student at U.C. Berkeley's Robot Learning Lab, advised by Pieter Abbeel. His current research focuses on humanoid control, data priors, and simulation.

Hardware Platform Migration Is Highly Complex

Tobias believes that what truly constrains large-scale commercialization and practical application is not performance robustness itself, but cross-hardware platform transfer and generalization. Different robot hardware varies enormously in control protocols, perception systems, and execution mechanisms, lacking a unified interface layer. This makes cross-platform migration of general models highly complex and costly.

Hausman also points out that while mechatronics and sensing technology have advanced considerably, early robot demos already demonstrated that hardware is fully capable of completing complex tasks. As mentioned earlier, what is truly missing are algorithms and models with human-like intelligence. Insufficient intelligence makes it difficult for robots to autonomously adapt and make decisions when facing complex, ever-changing real-world environments, becoming the main obstacle to industry development.

Therefore, if robots are to be deployed at scale in practical applications in the short term — for example, achieving deployment of thousands of units within a year — a critical prerequisite is that the same model can adapt to multiple hardware platforms without requiring tedious, customized adjustments for each individual device.

A typical case: PI needed to deploy a coffee-making robot at a location far from its office. PI had never previously encountered this hardware device, and the model had not undergone any customized training for this particular robot hardware. Yet it was still able to autonomously make a cup of coffee.

The complexity of this challenge also far exceeds that of autonomous driving. The latter mostly trains reinforcement learning models in closed environments with clear objectives and defined paths. Robot tasks, such as cleaning a kitchen, often involve multiple unstructured sub-goals, and the robot's own body structure, control methods, and sensing systems are highly heterogeneous. These differences determine that the development of general robot intelligence cannot simply copy the technical path of autonomous driving, but must redesign task representation and system architecture from the ground up.

There is currently no mature, standardized solution for flexibly deploying such large robot models to different environments, such as robot-local or edge-device deployment. Due to the lack of directly usable general solutions, multi-scenario, multi-device deployment remains in the exploratory stage.

For the three major challenges described above, PI has chosen to focus on breaking through performance bottlenecks over the next 1–2 years.

At the same time, to reduce dependence on task-specific fine-tuning, PI is developing general task "recipes" through a series of algorithmic improvements. Once this "recipe" is achieved, users who obtain the base model will no longer need additional post-training before direct deployment.

PI has built a standardized evaluation system covering multiple scenarios and tasks, hoping to quantify model performance through unified benchmarks to ensure that algorithms and models perform stably and reliably across different environments. PI is also collaborating with hardware manufacturers to optimize overall hardware-software system performance, addressing adaptation and performance issues when deploying robots on different platforms.

This evaluation system was developed by Physical Intelligence for π₀. Its core goal is not merely to tally success rates for single tasks, but to comprehensively measure model generalization across scenarios (different home environments, different object combinations) and across tasks (cleaning, folding, transporting, assembling, etc.).

PI embeds evaluation standards into actual datasets and task interfaces through open benchmark formats, enabling researchers and developers to run π₀ or derivative models under identical conditions, thereby achieving reproducible, comparable quantitative evaluation.

In the future, the team hopes to build a general yet customizable robot intelligence ecosystem where different types of robots can execute diverse tasks. Drawing an analogy to LLM development, PI hopes to create an intelligence platform where users can issue commands in natural language or provide a small number of demonstrations to directly guide robots through complex operations, lowering the barrier to robot use and improving interaction efficiency.

PI also hopes to drive deep hardware-software integration, where users can design and customize hardware while injecting intelligent algorithms into robots as if writing programs. This approach, termed "vibe coding for the physical world," can make the development and iteration of robot intelligence more flexible and efficient. Through this design, PI hopes to create an open, collaborative ecosystem where developers, enterprises, hardware manufacturers, and users jointly advance robotics technology, realizing the application of robots in practical work and daily life.

If these goals can be achieved and scaled, they will have an impact on the robotics industry and everyday life. PI believes that in 5-10 years, general-intelligence home robots may become widespread in households, capable of completing tasks such as folding clothes, cleaning, and organizing items, improving convenience in daily life. In industrial settings, PI's technology could enable robots to quickly adapt to changing production lines, enhancing manufacturing efficiency and flexible production capacity.

This may also give rise to new business models — "Robot Model as a Service" (RMaaS) — providing customized robot solutions for enterprises and individuals through cloud and locally deployed intelligence platforms.

Source: Overseas Unicorn (reprinted with permission)