Peking University Professor Liu Chang: In Search of the Aladdin's Lamp | Agent Insights

At NVIDIA's 2024 GTC conference, founder and CEO Jensen Huang closed with humanoid robots, stating that building a general-purpose humanoid robot foundation model is "one of the most exciting problems we can solve in AI today."

Among embodied intelligence tasks, one category stands out as particularly challenging: Zero-Shot Object Navigation (ZSON), which requires an agent to navigate in unknown environments. To address the gap between existing datasets and real-world conditions, DOZE — a dataset for open-vocabulary zero-shot object navigation in dynamic environments — was created. Join Peking University professor Chang Liu and an agent on the search for Aladdin's lamp. Enjoy.

DOZE: A Dataset for Open-Vocabulary Zero-Shot Object Navigation in Dynamic Environments

Problem Solved: Previous embodied intelligence research emphasized Zero-Shot Object Navigation (ZSON), which requires an agent to autonomously locate and approach previously unseen objects in unfamiliar environments. However, existing ZSON benchmarks lacked datasets featuring dynamic obstacles, open-vocabulary objects, and diverse scenes. DOZE provides a complex, dynamic, open, high-fidelity world for zero-shot object navigation.

Model Framework: The DOZE simulation environment builds on and modifies ArchitecTHOR, a traditional static goal navigation dataset. The team used Unity 3D (a 3D development engine) to create humanoid obstacles that walk through environments, and text-to-3D large models (such as DreamFusion) to generate open-vocabulary 3D objects.

Results: DOZE evaluated four different methods (Random, Frontier, C-L3MVN, C-LGX) across three levels of ZSON navigation tasks involving open-vocabulary target objects, objects with spatial attributes, and objects with appearance attributes. The results show that existing ZSON navigation methods still fall short in scenes with dynamically moving obstacles, and there remains substantial room for improvement in searching for open-vocabulary target objects.

Applications: Embodied intelligence, text-to-3D, XR

Evaluation Results: From Level 1 to Level 3 across all four goal navigation methods, SPL (Success weighted by Path Length) consistently decreased while collision rates rose. This demonstrates that while existing ZSON methods perform well on traditional ZSON tasks, they show clear deficiencies in scenarios characterized by dynamically moving obstacles.

ZSON Test Results

---

Oasis Capital: Figure 01 recently made headlines for using neural networks for end-to-end video data analysis. What's your take on Figure 01's current technical development?

Professor Liu: Figure 01 essentially puts OpenAI's intelligent decision-making capabilities into a humanoid robot form factor. What's impressive is the high-quality integration of hardware, software, and intelligence. In terms of interaction or intelligence, other systems have achieved similar capabilities, but few combine intelligence and embodied platforms so well — particularly with such dexterous, smooth hand manipulation. From the videos and public materials, Figure 01's hand operations are basically end-to-end, implemented through deep learning neural networks, plus whole-body control. The whole-body control likely uses classical control methods to maintain balance. The videos don't show much bimanual coordination. Handing over an apple, moving a trash bin, placing a plate on a rack — most of the time it's single-hand operation. The algorithm's capabilities in more complex scenarios remain to be evaluated. We haven't seen details about Figure 01's dataset format yet. It's most likely based on human motion capture, though it's unclear whether they used public datasets or created specialized ones from scratch.

Oasis Capital: What are your main research areas?

Professor Liu: Broadly speaking, I've been working on robot decision-making and motion planning. On one hand, this involves traditional control or numerical optimization methods; on the other, integrating large model capabilities to better assist robot decision and control. After my postdoc, I worked full-time in NVIDIA's autonomous driving division on software development, mainly responsible for autonomous driving behavior planning — ensuring smooth driving while guaranteeing safety, comfort, and so on. After returning to Peking University, my research focused on large-scale multi-robot swarm decision-making and planning, as well as embodied intelligence applications related to Object Goal Navigation. I've been working on both the methods themselves and datasets. DOZE is one dataset we created. We've also developed algorithms that help robots better utilize environmental semantic information and key information nodes in maps for decision-making, helping robots more effectively locate relevant objects.

Oasis Capital: Could you elaborate on Object Goal Navigation related to embodied intelligence?

Professor Liu: Object Goal Navigation is one of the earlier applications in embodied intelligence. The task involves telling a robot in an unknown indoor environment, via language or image, what object to find — for example, a red cup — and the robot locates the object with those specific attributes based on the description. There are already some datasets available for training and testing Object Goal Navigation. These fall into two categories. One type is reconstructed from real environments, such as using LiDAR or cameras for 3D scanning and data reconstruction to build scenes — for example, HM3D and Gibson. These reconstructed scenes are very realistic, highly complex, with a wide variety of objects. The downside is that scanning results are still imperfect, often with missing data in places. So many researchers have turned to purely synthetic datasets built in simulation environments. The better-known ones are RoboTHOR and HSSD-200, which have the basic elements we need — indoor environments, common objects — but relatively limited object variety.

Oasis Capital: What specific problem does the DOZE dataset solve?

Professor Liu: DOZE addresses the problem of limited object variety in datasets by adding four categories of objects to existing scenes. The first is objects with different attributes. For example, a cat can be in different spatial positions — on a chair or under a table. These spatially defined objects with different semantic information rarely appeared before. It also includes objects of the same category with different appearances or textures — for instance, a basketball with different colors and patterns. The second category is open-vocabulary objects. Take Aladdin's lamp: during the search process, the robot must first recognize the object and map what it sees to the concept of "Aladdin's lamp." This tests both the robot's localization capability and whether it possesses open-vocabulary object recognition ability. The third category is moving objects. In real life, there are always people or objects moving in a room. Existing datasets are all static, and existing goal navigation methods don't have obstacle avoidance capabilities for moving objects. Additionally, we added hint objects — for example, pieces of paper with partial text information that can help the robot locate target objects faster, testing whether the robot has semantic or text understanding capabilities.

Oasis Capital: How were the humanoid moving obstacles and hint objects created?

Professor Liu: Our simulation environment is based on modifications to the ArchitecTHOR dataset, which has ten rooms and is essentially a static, traditional Object Goal Navigation dataset. On top of this, we added humanoid obstacle information. We used Unity 3D to create humanoid objects, then manually designed their actions and movement trajectories to achieve walking within the environment. For the text information on hint objects, we edited the surface textures of whiteboard objects in the Unity environment, generated images with text, placed them on whiteboards, and positioned them appropriately in the environment. In this process, we also used large models for generating open-vocabulary objects. For more novel objects, we used DreamFusion (a text-to-3D large model). Current text-to-3D capabilities are limited, so generated content needs filtering. If a generated object isn't usable, we supplement with free 3D assets.

Oasis Capital: What are DOZE's advantages? How does its application space differ from previous datasets?

Professor Liu: DOZE's greatest advantage lies in the diversity of its generated objects. The moving obstacles, hint objects, and open-vocabulary generated objects in DOZE have different attributes, appearances, and textures. This object diversity enables robots to develop stronger perception and response capabilities for unknown and dynamic objects — directions that previous datasets couldn't adequately test. Training data in datasets is always finite; there will always be objects in reality that never appeared in past datasets. Open-vocabulary generated objects have long been a significant challenge in computer vision. We hope to address the generalization capability of perception models during the environmental understanding process for target object recognition. In terms of application space, DOZE, like previous datasets, emphasizes indoor environments. But traditional algorithms lose capability when deployed in real environments. For example, finding a pillow with a Doraemon pattern in a real environment — training on existing datasets can't verify whether an algorithm has open-vocabulary generated object recognition capability. This can lead to recognition algorithms that perform very well on traditional data completely failing to find targets in real environments, because they don't know what a Doraemon-patterned pillow is. For DOZE, it provides a validation platform for navigation algorithms with such open-vocabulary capabilities. Navigation algorithms validated on DOZE can also guarantee unknown object recognition capability and stronger dynamic obstacle avoidance in real scenarios. DOZE's core involves perception, decision-making, and interaction, which are reusable in VR environments — including generating new objects through visual capabilities, localization and mapping, and so on.

Oasis Capital: What's your understanding of "Agent"?

Professor Liu: The concept of agent differs between AI and automation. In AI, early agents were more intelligent entities operating at the software level to assist human-computer interaction and decision-making. In automation, an agent is more of an integrated platform that combines intelligent decision-making processes with traditional robotics expertise. Going forward, the software-level intelligence requirements for agents, including large model capabilities, will continue to grow, and there's also substantial room at the hardware level. Current embodied platforms are limited — quadruped robots and humanoid bipedal robots, both biomimetic concepts. But humans have many flexible joints and soft tissues, so there will inevitably be numerous differences in capability compared to robots. Current hardware platforms still focus mainly on fully rigid-body construction, so the development of truly human-like or animal-like robot hardware is also a direction worth watching.

Oasis Capital: What do you see as the biggest change large models bring to agents and embodied intelligence?

Professor Liu: In earlier years, deep learning's impact was concentrated in perception and natural language processing. Back then, we viewed neural networks as black boxes with no safety guarantees and poor interpretability. Later, we discovered that neural networks' most distinctive feature is that generative AI brought stunning improvements to decision-making and control planning. Generative AI's advantage for decision planning and control is that it handles complex, high-degree-of-freedom robot planning and control quite well, especially in situations where environments are difficult to model. For example, walking on uneven outdoor terrain — traditional automation methods require modeling, and the dynamics model is extremely complex. But using deep models with reinforcement learning allows the system to continuously explore the environment without a model and learn superior strategies. This is what I see as the biggest advance from this wave of deep learning. Traditional methods can theoretically guarantee a smaller gap between simulated robot actions and real-world deployment (the Sim2Real gap), but deep learning networks don't necessarily achieve such good action realization effects. At the decision-control level, I think both paths are worth exploring: one fuses neural networks with traditional methods, the other is direct end-to-end approaches. Each currently has advantageous application domains.

Oasis Capital: We've heard differing views on synthetic data. What's your perspective on its development?

Professor Liu: Synthetic data is a fascinating topic. The homogeneity of synthetic data may cause training with large models to hit ceilings. But at least in embodied intelligence, the performance ceiling for training large models with synthetic data for decision and control remains quite high, and we're still far from reaching it. In the near term, we're more focused on how to narrow the gap between data and real scenes. There's still much to mine in synthetic datasets for embodied intelligence, which is why researchers in our field have started adding manipulation elements to datasets. There will certainly be limits, but synthetic data will become increasingly important.

Oasis Capital: How do you view the future of humanoid robot form factors?

Professor Liu: Humanoid is an important form factor, but not necessarily the only direction for future development. Humanoid robots' powerful manipulation capabilities offer great imagination for home environments, including the navigation, manipulation, and interaction capabilities mentioned earlier. If humanoids plus robotic arms work well enough, they can accomplish many tasks. Additionally, the humanoid form satisfies people's expectations for human-like appearance and genuinely expands capability boundaries, so I see it as one important major direction. Of course, humanoids have limitations too — limited payload capacity, weaker locomotion. For example, bipedal stability on rough terrain is inferior to quadrupedal designs. So I think we shouldn't abandon other forms while developing humanoids. For heavy-load requirements or rough outdoor environments, ground-based legged designs are already relatively mature and offer greater advantages over humanoids.

Ji Ma was one of the main contributors to this work and participated in proofreading this transcript.