Astra Scored Just 19% on What May Be the Toughest Benchmark Yet to Crack | RobotWorld Team Responds to Everything

Can LLMs "Take Over" the Physical World? They Need to Pass This Test First

👦🏻 Interview: Ms. Yi

🥷 Edited by: Crossing

🧑‍🎨 Layout: NCon

When general-purpose LLMs can already write code and perform analysis fluently in the digital world, how far are they from truly "taking over" the physical world? Is there a unified evaluation benchmark?

Recently, a team with combined backgrounds in Embodied Artificial Intelligence and foundation models built one. Spending a cumulative total of roughly $13,500, they put five mainstream models — Astra, Opus, Moonshot AI, DeepSeek, and Gemini — on the same testing ground.

It's called RobotWorld[1].

This may be the most challenging robotics test in terms of completion rates to date: Even Astra, currently the strongest AI model for robot manipulation, could only complete 16 out of 84 tasks it designed — a score of just 19%; the overall success rate was only 7.9%.

On October 7, the day after the RobotWorld paper was released, it went straight to the top of the Hugging Face Papers and AlphaXiv trending lists; the code repository was open-sourced simultaneously, with all task recordings and an interactive results panel displayed on the homepage, all 420 evaluation trajectories clearly visible.

63 of the 84 tasks were not solved by any model. Was the bar set too high, or were there other reasons? Why could even something as powerful as GPT-6 only achieve 19%? Based on actual testing, do they think Astra would crush all embodied foundation models? From Computer Use to Robot Use, what capability is currently most scarce? With these questions in mind, we spoke with core members of the RobotWorld team — Zhiqin Yang, Chenxin Li, and Xiaomeng Hu, with Chenxin and Xiaomeng being the project initiators. The three of them together reviewed the original intention behind building this benchmark, task selection principles, challenges faced, key findings and reflections, and what they want to do next. The benchmark is just the starting point.

Below is the complete conversation transcript between Crossing and the RobotWorld team.

---

A Team Combining Embodied AI, Agents, and Foundation Models

🚥 Crossing

First, please introduce the RobotWorld team and tell us a bit about yourselves.

👨🏻‍🚀 Xiaomeng Hu

Our team has quite diverse backgrounds. Some members primarily work on Embodied Artificial Intelligence, with experience at leading embodied AI startups; quite a few others focus on Agents and foundation models, with experience at well-known leading foundation model teams.

This complementary background allows us to combine perspectives from both robotics and general-purpose models: we understand the requirements of robotic tasks themselves, while also thinking about how to approach these problems from the perspective of general-purpose models.

I'm a co-first author and project lead for RobotWorld. My research direction is large language models and general-purpose agents. I previously participated in multiple foundation model R&D projects at Moonshot AI's reinforcement learning team and Alibaba's Qwen post-training team.

Currently, my main focuses are: recursive self-improvement (RSI) of foundation agent models, automated research agents for scientific and engineering problems (Auto Research), and general-purpose physical agents for Robot Use.

👨‍🔧 Zhiqin Yang

I'm a co-first author on the RobotWorld project, currently a PhD student at The Hong Kong University of Science and Technology. I previously conducted research at Tencent and an Embodied Artificial Intelligence startup. My research directions include large language model reasoning, Robot Use Agent, and collaboration between different physical agents.

I previously focused on model preference alignment and the impact of language representations (e.g., code, formal languages) on models and their performance on robots, and I'm a main organizer of the NeurIPS'26 LP4FM workshop. Currently, I'm exploring self-iterative mechanisms for model capabilities and the integration of large language model reasoning with Embodied Artificial Intelligence, pushing toward new paradigms for Physical Agents.

👨 Chenxin Li

I'm a co-first author and project lead for RobotWorld, a PhD at The Chinese University of Hong Kong, and currently a foundation model researcher at Tencent's Hunyuan. I previously conducted research and interned at ByteDance Seed and Meta Superintelligence Labs (MSL).

My main research direction is code agents and Computer Use, focusing on how to improve general-purpose models' ability to solve practical tasks through data, execution environments, and evaluation design. I've worked on visual tool calling, computer and mobile agents, training data generation and self-evolution, and how to use execution trajectories and feedback to improve model capabilities.

Recently, through RobotWorld, I've been exploring the transfer of capabilities from Computer Use to Robot Use, combined with failure case studies to understand the data, training methods, and evaluation systems needed for robotic tasks.

---

A Cross-Domain Experiment from Computer Use to Robot Use

🚥 Crossing

How did you first come up with the idea for this project?

👨‍🔧 Zhiqin Yang

What initially drove us was a question that has become increasingly worth asking: LLMs can already understand images, write code, and use tools — how much of this capability still works in the physical world?

The emergence of models like Astra made this question more concrete.

When faced with an unfamiliar robotic task, can a model first understand the goal and interface, then gradually find a solution through observation, computation, programming, and execution feedback? We wanted to know how far this approach could go, and where it would run into difficulties.

But to answer this, looking at just one type of robot or a few success cases isn't enough. Many existing embodied evaluations revolve around specific robots, task distributions, and training methods. We're more interested in whether the same general-purpose model can show consistent adaptability when facing different forms and tasks.

So we gradually expanded the scope to manipulation, locomotion, driving, and flight, forming RobotWorld.

We hope this work can turn the intuition that "general-purpose models might change robotics" into a systematically testable question: what capabilities do they already have, what are they still missing, and how should we improve them next?

👨 Chenxin Li

Xiaomeng and I were both working on foundation models and general-purpose agents before this.

Even before Astra came out, we noticed that robotic manipulation in simulation had quite a few similarities with Computer Use: both involve understanding tasks in an environment, calling tools, and adjusting based on feedback. Astra's robotics demos brought more attention to this direction, and also made us feel it was worth systematically validating once.

What really confirmed we should do this project was a discussion over a meal between Xiaomeng and me.

Xiaomeng was more focused on subsequent data and training, wanting to explore how to train embodied capabilities into next-generation foundation models; I felt that in the short term, building a challenging benchmark that could plug into foundation model evaluation pipelines was more appropriate. First measure existing capabilities and gaps clearly, then we have concrete grounding for discussing environments, data, and training.

At the time, we referenced OSWorld. It provided a clear evaluation reference for agents in the digital world, and we hoped RobotWorld could serve a similar role for robotic tasks.

We viewed evaluation as the starting point for follow-up research from the very beginning, wanting to create a set of tasks that foundation model teams would want to keep testing against, and that could inspire next steps in training.

🚥 Crossing

How did you come together as a team?

👨 Chenxin Li

Xiaomeng and I co-initiated the project, and together we're responsible for direction and progress. Zhiqin has research and practical experience in the embodied direction, and we already knew each other.

After several initial discussions, we gradually translated ideas into environments and tasks, confirmed the core team, and combined experience from both general-purpose agents and robotics.

On the implementation side, Zhiqin mainly pushed the initial scaffold and environment sandbox debugging, as well as adapting to specifications like Harbor that lean more toward agent evaluation workflows; I was more responsible for batch experiments and high-concurrency runs.

For subsequent analysis and materials, we put them together as a team. Whether the environment runs, how models use interfaces, whether tasks are actually completed — these are all questions we discuss together.

🚥 Crossing

How did you ensure the fairness and impact of the RobotWorld Benchmark?

👨‍🔧 Zhiqin Yang

We mainly considered two aspects: fairness through evaluation rules, and impact through whether it can help the community answer important questions.

On fairness, we first clarify what the model can see, what tools it can call, how much control assistance it receives, and how many steps and how much time it can use.

For example, under the same task, different models follow identical observation, action, and budget settings; across different robots, we preserve each robot's reasonable control interface and make the differences public. Whether a task is completed is judged by executable criteria in the environment, not by the model declaring its own success. We also distinguish task failures from infrastructure failures like interface or runtime environment issues, and we preserve interaction logs and videos so results can be reviewed.

On impact, we hope RobotWorld can answer an increasingly important question: can capabilities learned by generalist models in digital worlds transfer to different robots and physical tasks?

So we put manipulation, locomotion, driving, and aerial vehicle operation into the same evaluation framework. Beyond success rate, we also care about how models use feedback, when they can recover, and why different models perform differently on the same task. These analyses can provide more concrete direction for subsequent training and agent design.

We're also lowering the barrier to entry so other teams can plug in their own models or harnesses. Beyond that, we're continuously refining our framework based on feedback — for example, shifting toward evaluation standards like Harbor.

Impact ultimately depends on community adoption: a benchmark only becomes truly valuable if it helps researchers discover problems that existing evaluations don't expose, and if they improve their models accordingly.

🚥 Crossing

What was the hardest part of this work?

👨‍🔧 Zhiqin Yang

There are three key points in the implementation.

First, integrating different robot tasks into a unified, comparable evaluation pipeline.

Robotic arms, quadrupeds, vehicles, and drones have different action spaces, control frequencies, and termination conditions. We needed to unify task descriptions, interaction logging, and evaluation procedures while preserving each robot type's control characteristics and clearly documenting how much assistance the low-level controller provides.

Second, separating the model's reasoning process from physical execution.

The model can observe images, compute, write programs, then issue action requests; after actions execute, the environment returns new observations and feedback. Here we must accurately record how much physical time has elapsed, how many actions were actually executed, and whether the model sees the current state or historical state — otherwise evaluation results are easily skewed by implementation details.

Third, making results interpretable and reviewable.

Beyond final success rate, we preserve actions, errors, stop reasons, and videos to determine whether a failure stems from model decisions, control execution, or the runtime environment itself.

The hardest part is actually confirming: are we measuring the model's capability?

A seemingly simple interface choice can change task difficulty. For example, if the action interface already handles path planning for the model, success may mainly come from the controller; if feedback is missing or historical images are mishandled, failure may come from the evaluation system itself.

We spent considerable effort checking these boundaries. Getting the environment running is just the starting point; having different models evaluated under clear, consistent, traceable conditions is the harder part.

🚥 Crossing

What best reflects your taste?

👨‍🔧 Zhiqin Yang

I think what best reflects our taste is our choice to make "how generalist models use robots" the object of study.

This meant we didn't design the evaluation around just one robot type or one class of manipulation tasks. Instead, we put robotic arms, quadrupeds, humanoid robots, vehicles, and drones together.

What we want to ask is: faced with different bodies, action interfaces, and physical constraints, can the same model leverage existing knowledge, understand the task, and then use observation, computation, and feedback to get things done? This cross-task, cross-morphology capability is what we care about most.

Another choice was treating the model's problem-solving process itself as a result worth studying.

Does the model actively analyze images, estimate geometric relationships, build dynamics models, generate control programs? After execution errors, can it revise its judgment based on feedback? These behaviors tell us more than a total score about which capabilities start functioning when general intelligence enters the physical world, and which links are still missing.

So if there's any difference we bring, I think it's making this benchmark into a research tool that connects evaluation and training: looking at both how many tasks the model can complete, and hoping to find from its specific successes and failures what should be trained next and how agents should be designed.

We want evaluation to drive capability development, not just record capability rankings.

5 Major Models Compared: Best-Performing Astra at Just 19%

🚥 Crossing

What were your principles for task selection? What did you drop?

👨‍🔧 Zhiqin Yang

Our core principle: does this task let us see generalist models' capabilities in physical interaction, and can we measure that capability reliably?

Specifically, we look at three things:

First, coverage — we want tasks involving different robots and physical constraints. For example, contact manipulation for robotic arms, balance for legged robots, steering for vehicles, and flight control for drones.

Second, whether the task requires the model to make judgments from observations and use feedback to adjust actions, avoiding success that mainly depends on the low-level controller.

Third, whether it can be stably reproduced, with clear initial conditions, control budgets, and executable success criteria.

We drop tasks with high redundancy that don't add new capability dimensions, and we postpone those with missing key assets, unstable runtime, or success conditions that are hard to verify. But difficulty itself isn't a reason to drop; not being able to tell whether failure comes from the model or the environment is the bigger problem. We also don't require every task to demonstrate complex reasoning — simple tasks can help pinpoint which basic link the model fails at.

The more agonizing ones are tasks with strong research value but incomplete evaluation conditions.

For example, aerial payload balancing involves dynamics understanding and control, but the version we had access to lacked key raw assets, so we didn't include it for now. Another example is drone volleyball: dual-drone competition is attractive, but it needs fixed opponent strategy and runtime conditions, otherwise scores are affected by opponent variation. We currently prioritized validating single-drone ball bouncing, and remain cautious about the original competitive setup.

Long-term, we'd also love to add collaborative tasks across drones, robotic arms, and legged robots.

But these tasks still need solutions for what information each robot gets, how they coordinate, and how to attribute failure. We'd rather wait until these conditions are clearly defined before making it formal evaluation.

🚥 Crossing

Is the benchmark set too high? Currently the best-performing Astra only has a 19.0% completion rate.

👨🏻‍🚀 Xiaomeng Hu

Our intention in task selection wasn't to suppress foundation model scores, but to see how far generalist models can go when facing different robots and different tasks.

In selecting tasks, we focused on coverage of robot morphologies and task types, and also on whether tasks posed different capability requirements: precise spatial localization, constrained contact manipulation, coordination in multi-stage tasks, and maintaining balance, tracking, and contacting moving targets. We wanted these tasks to complement each other rather than repeatedly testing the same operation.

Current results show that models still have considerable gaps before becoming reliable generalist physical agents. But the gaps between different models are also clear: Astra and Opus perform markedly better than other evaluated models overall.

Looking deeper at trajectories, these two models don't solve problems the same way either — they invoke different prior knowledge and exhibit different behavioral patterns, such as how they use observations, when they write auxiliary programs, and how they adjust actions based on execution feedback.

For example, in the same block-stacking task, Astra first estimates position from the camera image, completes the grasp, and only later introduces camera parameter fitting to assist localization; Opus first writes a color segmentation program, then uses the robotic arm's probing motion to estimate camera geometry and compute grasp positions.

One acts first based on visual judgment, then supplements with computation; the other first builds computational tools, then acts. This lets us directly see that models can combine visual understanding, programming, and robot knowledge in different ways to solve the same problem.

So we care about not just why the highest score is only 19%, but also how that 19% was achieved, and what capability gaps and corresponding training data needs are reflected in success and failure trajectories.

🚥 Crossing

From what you understand, why does Astra score the highest?

👨🏻‍🚀 Xiaomeng Hu

Intuitively, we think there may be two aspects:

One possibility is that OpenAI included embodiment-related data in training, giving the model some knowledge and capabilities needed for robot tasks.

Another possibility is that capabilities it accumulated in visual understanding, Computer Use, programming, and 3D modeling generalize to Robot Use. For example, visual localization, understanding spatial relationships, completing operations through code, then checking and modifying results based on visual feedback — these capabilities can also play a role in robot scenarios.

Both factors may be at play, but based on evaluation results alone, it's difficult to determine each one's contribution, let alone reverse-engineer the specific training recipe.

Also, Astra did not demonstrate overwhelming dominance in our testing.

It completed 16 tasks, Opus completed 13, and they only overlapped on 8 — each had tasks the other couldn't finish. Opus also showed strong capabilities in quadruped locomotion, suspended load hovering, and drone ball-bouncing.

So Astra's overall lead means it achieved broader success coverage across this task set, not that it's stronger in every individual capability. We're more interested in what each model excels at, what strategies they employ, and what clues these differences offer for future training.

👨‍🔧 Zhiqin Yang

The overall lead may relate to whether the model can better utilize controllers.

For example, in the charger insertion task, Astra adjusts its arm configuration based on planner rejection feedback before retrying.

This shows that existing motion control capabilities only become effective when the model can propose appropriate action requests and understand execution feedback. It reflects coordination between reasoning and control, but a single case isn't enough to explain the full score difference, nor can we infer training data from it.

🚥 Crossing

Within your field of view, have domestic LLM foundation model companies started paying attention to their embodiment-related capabilities?

👨🏻‍🚀 Xiaomeng Hu

Overall, the domestic foundation model teams we've been in touch with are still in relatively early stages of exploring this direction.

After Astra's release, we've clearly felt rising attention. From public job postings, we can also see some leading teams beginning to recruit for related positions.

As far as we know, some teams have indeed incorporated embodiment-related data in earlier stages of new model training, but most teams we interact with are still in the pre-research and evaluation framework setup phase.

Of course, this also relates to each team's priorities. Foundation model teams are pushing many directions simultaneously and need to allocate resources across different capabilities and training tasks.

By comparison, we're more focused, so our progress is faster. Some teams have already reached out to us expressing interest in exchange and collaboration.

What's scarce behind the model "kill threshold"?

🚥 Crossing

You mentioned that many existing agent capabilities aren't actually that far from embodiment-related scenarios, and the barrier is lower than expected. What have you discovered?

👨 Chenxin Li

When I say the gap isn't as large as imagined, I'm speaking from the perspective of general agent evaluation and training in simulation environments.

When we integrated environments, many components felt familiar: sandbox, Docker, tool interfaces, trajectory logging, and returning observations and scores after action execution. The robot scenarios and assets changed, but much of the runtime and evaluation logic could be reused.

For models, there are also similar problem-solving processes. In Computer Use, the model finds buttons on screen, operates them, then checks results; in robot tasks, it decides how to move the arm based on images and state, then checks whether it grasped and placed correctly. The color segmentation and camera geometry calculations in block stacking are applying visual and programming methods to new tasks.

This gives foundation model and agent teams a fairly direct entry point.

Of course, physical interaction brings new requirements — contact, instability, error accumulation all need handling; and since we currently allow simulation to pause during inference, we haven't yet solved real-time deployment on physical hardware. Research entry is easier; deployment reliability still requires separate validation.

🚥 Crossing

In the paper, you dedicated a section to "Where robot closed loops fail" — could you elaborate on this finding and your thinking?

👨🏻‍🚀 Xiaomeng Hu

When analyzing trajectories, we gradually distilled Robot Use's required capabilities into four categories: visual understanding, physics and robotics knowledge, programming and computation, and agent capabilities. "Where robot closed loops fail" essentially means examining where these capabilities didn't activate or connect.

First, visual understanding. The model must not only recognize objects but understand position, orientation, occlusion, motion changes, and what actually happened in the environment after executing an action. After grasping, did it really hold the object? During insertion, did it actually go in despite appearing aligned? Any misjudgment of state can cause subsequent actions to build on false premises.

Second, physics and robotics knowledge. After seeing the scene clearly, it needs to know how the robot should interact with it. Quadruped locomotion involves leg kinematics and gait design; suspended load hovering involves load swing and control; drone ball-bouncing requires coordinating motion states before and after contact. Especially in dynamic tasks, correct position at one moment isn't enough — velocity, timing, and whether the target state can be maintained matter too.

Third, programming and computation. We see models writing their own image segmentation, camera calibration, inverse kinematics, and control programs, turning existing knowledge into computable and executable methods. But successful program execution doesn't mean successful robot task completion. Whether the program's assumptions fit the current environment and whether computed actions are effective still need real execution to verify. In the quadruped case, Opus wrote two additional gait programs during execution, adjusting leg height, stride length, and turning methods.

Fourth, agent capabilities — organizing the previous capabilities: maintaining task goals and historical state, choosing when to observe, compute, or act, how to recover from failure, and what criteria determine task completion. We observed some models continuing to call tools but no longer correcting toward uncompleted goals. They kept executing without continuing to advance the task.

So our core judgment is: Robot Use's difficulty lies both in individual capabilities and in whether these capabilities can coordinate through execution feedback.

We can't simply attribute a failure to poor vision or insufficient physics knowledge. We need to trace through the trajectory: did it correctly perceive state, choose methods, implement them correctly, and adjust when results didn't match expectations?

This also determines subsequent data needs: we need both data supplementing various foundational capabilities and complete interaction trajectories for models to learn how to deploy these capabilities in robot tasks.

---

Crossing

There's a clear "kill threshold" between different models — in your observation, what's behind it?

Chenxin Li

"Kill threshold" is a colloquialism from our discussions.

In this 84-task evaluation, Astra and Opus completed 16 and 13 tasks respectively, while the other three models completed 2, 1, and 1 — performance clearly split into two tiers. This refers to performance gaps under current models and runtime settings; we can't treat it as a universal, strict capability threshold.

What impressed me most was models' familiarity with tasks and interfaces.

In some trajectories, the model knew the goal but never translated it into effective actions. In Kimi's pouring task, after knocking over the bottle, multiple grasping attempts still failed to secure it, and switching arms came too late; Astra, during charger insertion, could adjust arm configuration based on planner rejection and retry.

So I focus more on whether the model can connect instructions, robot interfaces, current state, and execution feedback.

It needs to know what to try next, judge whether it just advanced the goal, and switch approaches after failure. Vision, robotics knowledge, programming, and agent capabilities all matter here; which one is the bottleneck still requires tracing specific trajectories and verifying through ablation experiments.

---

Crossing

What capabilities are scarce when foundation models generalize to embodied scenarios?

Zhiqin Yang

From an embodiment perspective, I'd add: what's scarce isn't just planning capability, but understanding of body constraints and action consequences.

For example, after knocking over a bottle, grasping conditions have changed; planner rejection means the current action may exceed reachable range. The model needs to translate this feedback into new judgments.

Borrowing the System 1 / System 2 framework, the key is whether high-level reasoning can promptly adjust based on low-level execution reality, rather than repeatedly attempting the same approach. This may be one reason capability differences get amplified.

---

Crossing

In the future, will Astra dimensionally crush embodied models?

Crossing

Do you think Astra will dimensionally crush all embodied foundation models? — If the degree of dimensional crushing ranges from light (1) to heavy (10), what score would you give?

Zhiqin Yang

What I value more is that it makes us reconsider: does learning each new robot task require collecting new data and training a new policy?

General models offer another possibility — leveraging existing visual understanding, reasoning, and programming capabilities to observe feedback, adjust plans during execution, even generate control programs. This is also what we wanted to study with RobotWorld: to what extent can capabilities accumulated in the digital world translate into general robot use capabilities.

But broad task coverage doesn't mean faster, more stable performance on each task. High-frequency control, fine contact, and reliability in real environments still require solid embodied technology.

I'm more optimistic about gradual integration of general models with specialized control capabilities.

For embodied teams, the key is judging: which capabilities will come with general model advances, and which still need to be learned from physical interaction.

Future competition will likely shift from "train one model per task" to "adapt existing capabilities to more tasks and robots."

Xiaomeng Hu

For Astra's current capabilities, I don't think it qualifies as dimensionally crushing all embodied foundation models — the two still have fairly distinct tracks at this stage.

But if we take a longer view and look at future foundation Agent models like Astra, I'd be more optimistic. As model capabilities continue to improve, the boundary between them and embodied foundation models may gradually shift.

If we're talking about a "dimensionally crushing" future scenario on a scale of 10, I don't think it's impossible.

Chenxin Li

If we're discussing long-term trends, I'm quite bullish on general-purpose foundation Agent models like Astra.

They can already apply visual understanding, coding, tool use, and reasoning to robotic tasks, and as these capabilities strengthen further, the way some embodied tasks are solved will likely change.

But to say they're already crushing all embodied foundation models — I think that's premature.

Different models face different tasks, observation conditions, and control frequencies. We also haven't systematically compared all VLAs under a unified protocol this time. Success in simulation doesn't directly translate to real-world deployment capability; real-time control, contact, and reliability still have many problems to solve.

I think embodied teams need to consider both the near term and the long term. In the near term, they can tackle tasks that general models aren't yet good at; in the long term, they need to continuously assess how far foundation models will go and how their own technology can integrate with them.

Today's capability gap provides opportunity, but you can't assume this boundary will never move.

---

Crossing

From simulation to real robots, how do you bridge the gap?

Crossing

Many people in embodied AI are doing related work. What are the differences in approach among different teams?

Chenxin Li

I often say we have a "foundation model mindset": we want Robot Use to become a capability that foundation models continuously train and evaluate on, just like Coding and Computer Use.

From the start, we've been thinking about how this set of tasks can fit into existing Agent workflows at foundation model teams, and how evaluation problems can be turned into subsequent data and training objectives.

Some work focuses more on improving control methods or harnesses for a set of robotic tasks; some focuses more on stability under perturbations, or using programmatic solutions to generate robot training data. We care more about whether task coverage is broad enough, whether capability gaps are clearly identified, and whether results can provide useful signals for the next round of foundation model training.

We spend considerable effort on sandboxing, interfaces, and concurrent experiment adaptation, so teams familiar with general Agents can relatively easily start testing.

The tasks themselves are still all robot tasks; we focus on how they serve foundation model capability evaluation and subsequent training. Optimizing control for specific tasks and testing perturbation robustness are also valuable — they address problems at different levels.

---

Crossing

Regarding the "test bank cramming" phenomenon in simulation leaderboards, how did RobotWorld's design account for this?

Xiaomeng Hu

We believe evaluation and models need to co-evolve — it's a dynamic process.

We don't exclude embodied teams or frontier labs from doing targeted training on problems exposed by RobotWorld, because one of evaluation's values is helping people find capability gaps and then pushing models to fill them.

RobotWorld covers different robot morphologies and task requirements. If a model can truly do well on all these tasks, even if the improvement comes from targeted training, we consider that valuable progress. To cover this many tasks, a model needs to master spatial understanding, robot knowledge, programming, and closed-loop execution — capabilities that can also transfer to new scenarios.

Of course, whether generalization actually occurs still needs to be tested with new tasks and new conditions.

As models perform better on existing tasks, we keep expanding tasks, changing scenes and constraints, to see if learned methods transfer.

Going forward, we'll also continuously update and maintain the benchmark and leaderboard based on model capability progress and community needs. We hope to create a cycle of "evaluation finds problems, training improves capabilities, new evaluation tests capabilities" — letting evaluation continuously drive model progress.

---

Crossing

Simulation environment results represent an ideal state. In real factories, it's an entirely different problem — whether a task can be completed might be a binary 0 or 1. How do you view this?

Zhiqin Yang

I agree with this assessment.

In real factories, one task failure might mean line stoppage, material damage, or requiring human takeover. Beyond completion, you also need to look at cycle time, continuous operation stability, and abnormal condition handling. Therefore, simulation results cannot directly equate to industrial deployment capability — contact errors, sensor noise, equipment wear all need validation in real systems.

However, simulation's value lies in providing a repeatable, controllable comparison condition. We can have different models face the same tasks, initial states, and budgets, and analyze where differences actually come from.

Real-robot testing is closer to final application, but making it a benchmark everyone can fairly participate in is very difficult: hardware, calibration, site conditions, and maintenance states are hard to keep completely consistent; each reset, failure handling, and safety check requires human effort. Expanding tasks and test counts also needs sustained funding.

We hope to combine with real-robot validation later. A more realistic path is: use simulation for broad, reproducible evaluation, then use real robots to verify key capabilities and conclusions.

For example, selecting representative manipulation and mobile tasks, under fixed hardware and clear human intervention rules, checking whether capabilities shown in simulation — like feedback utilization, error recovery — are preserved in reality.

We hope to gradually build a credible connection between the two, rather than using a few successful real-robot demonstrations to prove the entire benchmark's validity.

Xiaomeng Hu

We acknowledge the gap between simulation and the real world — perception errors, control delays, contact and friction changes all affect execution, and simulation success rates don't directly equate to real deployment capability.

We're considering selecting some better-performing, relatively stable manipulation scenarios for real-robot validation, but this isn't the highest priority at this stage.

Currently, models still have many foundational capabilities to improve in simulation. We want to solidify these capabilities first, while making simulation conditions closer to real environments, then gradually advance real-robot validation as models progress.

Agent as Brain, Code as Cerebellum

🚥 Crossing

You also started a discussion community. What are some commonly discussed topics from the community?

👨🏻‍🚀 Xiaomeng Hu

Since RobotWorld's community launched, the attention and feedback exceeded our expectations.

Many friends reached out saying they've been working on related things recently. What surprised us was that most of these people weren't from embodied backgrounds before — some weren't even doing Agent work.

The discussions are quite concrete: some suggest what the next version could include, some point out improvements in current designs, and many want to directly participate in subsequent iterations. This feedback has helped us discover issues we hadn't considered.

We've been going through these suggestions recently — some have already been adopted, and we're making changes. A new version will be released soon. We hope this community makes it easier for everyone to communicate and participate in RobotWorld's future iterations together.

🚥 Crossing

What external challenges have you received so far?

👨🏻‍🚀 Xiaomeng Hu

One concentrated criticism has been about latency.

In a group called Agent Hub, people discussed our work, and the next day a bot in the group summarized the discussion as "The Debate Between Embodied Control Latency and Generalization Approaches" — I think that's quite accurate.

Some friends believe that when Agent-controlled robots reach real-world scenarios, especially tasks requiring fast reactions, inference and interaction latency will become a major obstacle. Others believe that as foundation models, inference infrastructure, and model programming and control capabilities improve, this approach still has huge room for imagination.

We generally lean toward the second view.

From ChatGPT's launch until now, we've been involved in foundation model-related work. Coding Agents and GUI Agents were also slow early on, and only gradually became usable as models, harnesses, and infra continuously improved. Robots certainly have stricter real-time requirements, but we don't think today's latency determines this approach's ceiling.

More importantly, what we envision isn't having a large model directly control motors at every step.

It can be responsible for understanding tasks, formulating strategies, writing and adjusting control programs, while programs handle execution requiring high-frequency response — "Agent as brain, Code as cerebellum." This way, the large model's inference frequency and underlying control frequency don't need to be bound together.

A couple days ago we attended an embodied-related offline event in Shenzhen, and many practitioners we spoke with shared similar views.

Also, for "cerebellum" implementation, we currently prefer code over adding another small VLA model.

On one hand, introducing another model increases system complexity and requires considering coordination between two models. By contrast, we're more optimistic about continuously optimizing and scaling around a single model + harness. On the other hand, code has explicit execution logic that's easier to inspect and modify.

🚥 Crossing

Among currently active benchmark teams, how would you categorize them? Which category do you put yourselves in?

👨🏻‍🚀 Xiaomeng Hu

I mainly focus on this recent batch of work at the intersection of Coding Agent and embodied AI. I see them as "post-Astra era" embodied evaluation. Everyone is looking at the intersection of Coding Agent and physical intelligence, but the entry points differ.

For example, EmbodiedSWE has Agents write robot control programs in simulation, using execution feedback to iteratively debug and modify until getting programmatic solutions that complete complex tasks, then expanding these successful solutions into VLA training data.

Its setup grants access to the full simulation state and allows ample time for program development. The emphasis is still on using Coding Agents to scale VLA training data. In our setting, the Agent can also write closed-loop robot control programs, but our approach uses Code Control within smaller closed loops inside the Agent's broader loop to solve real-time subproblems.

Video2World focuses on environment construction: having Coding Agents reconstruct interactive simulation scenes and robot behaviors from demonstration videos of robots or humans. This is also a fairly fundamental starting point, exploring how to provide more environments for embodied research.

RLE-Bench leans more toward Auto Research & Engineering evaluation in robotics, testing whether Coding Agents can complete control, policy training, perception estimation, and mechanical design work like robot engineers.

If I had to position ourselves, I'd say our ambitions are a step larger. We hope that in the future, Robot Use can become a capability for continuous foundation model training and evaluation, just like Coding and Computer Use.

🚥 Crossing

RobotWorld wants to embed itself in the closed loop of foundation model training and evaluation. How far along is it now?

👨🏻‍🚀 Xiaomeng Hu

I come from a foundation model background myself, and my previous focus was mainly on training.

From my perspective, the gap between evaluation and training is often mostly an infrastructure衔接 problem.

RobotWorld's interaction paradigm is quite close to the Coding Agent, GUI Agent, and General Agent evaluation that foundation model teams are already familiar with. So integrating it into existing evaluation pipelines and extending it to training doesn't present major technical obstacles on the path forward.

How far this can actually progress depends more on organizational factors: what priorities different teams have, how many resources they're willing to commit. They need to make that call based on their own judgment. Just as teams invested differently in Coding Agents before — when ROI becomes clearer, teams will naturally double down.

Actually, building a good benchmark already communicates to the outside world our view on training in this direction.

For now, we're focused on refining our own work and keeping up with frontier progress. We won't make it our goal to push every team to incorporate RobotWorld into their training loop. Instead, we hope to contribute more to the community through the actual value of our work.

On Conviction: Waiting for Embodied Scaling's Moment

🚥 Crossing

You mentioned that you firmly believe scaling law will happen again in embodied AI — that's what gave rise to RobotWorld. Behind this long-term directional bet, what do you see that's different?

👨‍🔧 Zhiqin Yang

My judgment comes more from embodied systems themselves.

Robots need to respond to the environment quickly and stably, but they also need to understand goals, plan actions, and reassess after errors. Borrowing the System 1 and System 2 framework, the former is closer to skilled motor abilities and immediate reactions, the latter closer to deliberative decision-making.

What I'm optimistic about is that these two types of capabilities can learn together and reinforce each other.

For example, when a robot walks while carrying a tray, maintaining balance requires continuous, low-latency control — not suitable for having a large model think through every step. But when encountering a narrow passage, whether to change routes, adjust posture, or set the tray down first requires judgment combining task and environment. What's truly difficult is how the two coordinate: when can you rely on existing skills, when should you stop to re-observe, and whether upper-level plans exceed the body's execution capacity.

This is also an important angle for how I understand embodied scaling: as experience accumulates, can actions that originally required repeated deliberation gradually become fast, reliable skills? Can these skills be recombined to solve more complex tasks? Meanwhile, can contact, imbalance, and failure during execution in turn help the model form more accurate physical judgments?

If this loop can hold, what grows isn't just how much the model knows, but how much the entire system can reliably accomplish.

RobotWorld currently mainly tests general models' ability to use robots, and hasn't yet validated the complete dual-system learning path. But I hope that through cross-robot, cross-task evaluation, we can distinguish which bottlenecks come from understanding and planning, which from control and bodily constraints, and whether problems arise at the junction of the two.

👨 Chenxin Li

Both Xiaomeng and I have worked on foundation models and general Agents, so we're quite excited about this path.

We've seen that models can already use visual understanding, programming, and reasoning capabilities in some robot tasks. If these capabilities can continue learning from data and interaction, I'm willing to believe that embodied AI will also benefit from expanded effective data and compute scale.

We also have a concrete entry point where we can get our hands dirty. Sandbox environments, tools, execution feedback, and scoring mechanisms in simulation share quite a few similarities with general Agent evaluation. First measure the capabilities, then design data around failures, improve training and interaction methods — this research path is concrete. RobotWorld is our first step on the evaluation side.

Going forward, I'm more interested in testing how much these training experiences transfer: what data additions make models better at using robot interfaces, what interactive training reduces invalid retries and improves completion judgment, and whether improvements persist when tasks and initial conditions change.

Rigorous scaling laws require controlled experiments to establish, and these questions are also worth our long-term investment.

👨🏻‍🚀 Xiaomeng Hu

Our starting point is still whether this is fundamentally feasible.

One of my basic judgments is: if a capability can be learned through data and interaction, then continuously scaling effective data and compute has a chance to push it further.

I tend to understand Robot Use capability as: visual understanding capability × physics and robotics knowledge × Agent capability × coding capability.

The multiplication signs emphasize that these capabilities need to work together — any shortfall can become a bottleneck.

I currently don't see principled reasons why these capabilities cannot be learned. Conversely, these foundational capabilities are already strengthening in existing models, which is why I'm optimistic about this path.

References

[1] RobotWorld: https://robotworldai.github.io/

Recommended Reading

Join Membership Group

Comment to Win Prizes