The Road to AGI | Supercharged by Large Models, Robotics Is Entering a New Era

Multimodal large language models are creating a transformative opportunity for intelligent hardware.

Robots, XR devices, and other hardware products are fundamentally intelligent systems at their core. Their underlying capabilities — perception, control, decision-making — all derive from AI. With the emergence of multimodal large models, AI's ability to understand humans and interact with them will advance significantly. No longer limited to language and text, it will be able to better recognize body language, facial expressions, and tone of voice.

Research shows that when faced with the same task, small models with limited data dimensions achieve only 20%-50% success rates. When large models are deployed, success rates can exceed 75%. This demonstrates that with multimodal data combined with large model capabilities, robots stand to improve comprehensively — in perception, decision-making, human-robot interaction, autonomous navigation in complex home environments, and success rates on complex tasks.

While there's broad consensus that multimodal large models will enhance robot performance, the specific unresolved problems that next-generation AI will solve for intelligent systems remain unclear. What new capabilities will intelligent systems develop under the加持 of multimodal large models? Vague predictions are of limited value. Over the past several months, we've attempted to work through the possible technological breakthroughs and industrial applications. In this installment of "The Road to AGI," Cao Wei, Partner at BlueRun Ventures, joins us to imagine how next-generation AI could evolve the perception, control, decision-making, and interaction layers of intelligent systems.

01

Perception Systems

In the future, the integration of multimodal data collection with new AI model architectures will enable more efficient parallel processing of multidimensional data, allowing for better perception of complex, dynamic external environments and improved task understanding. Why emphasize complex, dynamic environments? Currently, robots are widely deployed in factory settings — simple, bounded scenarios with fixed parameters. In such environments, large models offer limited efficiency gains; traditional directed analysis models are already sufficient. What makes large models most exciting is their potential to help robots penetrate complex, open-ended, exploratory scenarios involving human interaction.

At the specific level of perception, multimodal large models could drive improvements in three directions:

Further Reducing Front-End Hardware Requirements

XR devices and robots operate under relatively harsh constraints for perception systems, with limited energy and compute at the front end. This is a critical bottleneck for intelligent hardware today. Going forward, AI at the perception layer can continuously reduce front-end hardware requirements, expand the adaptive range of perception (anti-interference, noise reduction), and improve robustness and generalization.

Reducing front-end hardware requirements involves two considerations: how to make devices smaller and more flexible, and how to distribute compute between cloud and edge. Currently, the edge side lacks AI capabilities, so compute has been concentrated in the cloud. If AI can directly empower the edge in the future, cloud compute pressure will ease. But the optimal cloud-edge distribution depends on the edge's computational scale and the technical depth that edge AI can deliver.

The edge will handle some real-time end-to-end closed-loop training — for example, feeding labeled data collected by perception systems back to the cloud. But the future likely involves cloud-based large models performing inference based on real-time training across large robot fleets, with more integrated cloud-edge-device compute coordination and potentially very rapid update cycles.

In industrial settings, there's another approach for mobile robots to reduce front-end hardware requirements: adopting 3D visual perception technology. Take Luxshare-ICT as an example. The company's products are equipped with the Luxi-MRDVS mobile robot deep vision system, enabling robots with 3D vision-based localization, navigation, obstacle avoidance, docking, and grasping capabilities. Compared to magnetic strips, QR codes, and reflector markers, 3D visual perception allows Luxshare-ICT's products to deploy easily, build intelligent maps without relying on artificial markers, and operate unaffected by dense human or logistics traffic.

Multimodal large models capable of understanding images will undoubtedly amplify 3D visual perception technology. In the future, mobile robots based on 3D visual perception may achieve semantic understanding, with enhanced environmental recognition and comprehension capabilities. This could gradually replace existing single-line LiDAR navigation robots and popularize 3D vision-based AMRs.

Future vehicles, robots, and other intelligent systems will increasingly resemble humans, no longer requiring arrays of sensors. The optimal system is the human body itself — though we're not there yet.

Further Reducing Manual Annotation Costs

Data annotation and training costs represent a particularly challenging problem in autonomous driving scenarios. Training vehicles collect raw data without labels, which must be completed by third-party annotators or annotation software in assisted or semi-assisted fashion.

But as AI advances, real-time perception and real-time automatic annotation technology may emerge, with superior algorithms addressing data collection cost, quality, and efficiency issues, while feeding labeled real-time data directly into autonomous driving models. With AI models, collected data can be annotated directly at the perception side, without transmitting massive raw data to CPUs and GPUs for annotation — dramatically saving compute overhead and reducing latency.

Revolutionizing Perception Methods

"Foveated rendering algorithms" are currently a clear AI model application for the XR field. Human vision during fixation divides into three regions: the central area, low-resolution zone, and peripheral area. The region spanning roughly 30-40 degrees has the highest resolution, while peripheral areas are blurrier; the brain processes information hierarchically. This affects what users see in XR devices.

The foveated algorithm functions based on fixation point tracking and eye tracking, determining what users are looking at in virtual worlds — rendering the focal point of gaze more sharply while blurring unseen areas.

Take Ruisight Technology as an example. This is a children's myopia prevention and control platform centered on AR optics and digital defocus technology. Its main product is an optical screen with defocus capabilities. When children use the screen for online classes or watching animations, the screen's unique optical path design creates myopic defocus when light images on the retina, achieving myopia control goals. By introducing eye-tracking devices on the perception side to track in real-time what users are viewing, optical defocus and image rendering can be applied specifically to the content at the focal point of gaze, rather than globally — thereby reducing front-end device compute requirements and enabling good performance even on low-compute hardware.

AI improvements to perception systems may also enable intelligent hardware and health devices to form closed loops during perception itself. Consider hearing aids: currently, most products crudely classify all sounds as either speech or noise, preserving speech while suppressing everything else. The result sounds unnatural, with latency exceeding 10ms and poor listening experience.

But with AI assistance in the future, deep neural networks can process front-end external sound input signals, supplemented with omnidirectional sound technology, allowing the brain to hear sounds approaching natural hearing while dramatically reducing sound delay (to 6-8ms). They could even perform personalized sound restoration adaptation based on users' different environments, further enhancing auditory experience.

There's also easily overlooked data perception. Take robotic dogs: each of their four legs has motors. As the four legs walk on the ground, they measure the height between each leg and the ground on one hand, obtaining height data; on the other hand, they determine what friction state exists between the ground and the dog's paw. The robotic dog collects this data, which then informs path decisions and movement planning.

02

Integration with Control, Decision-Making, and Interaction Systems

Control

Large models will drive very significant improvements in machine control.

Chongqing Robotic Technologies (程天科技) is a particularly interesting case. As an external exoskeleton system, it contains complete perception-control-decision loops. It collects IMU data, pressure sensor data, and force control data to assist with step length, speed, gait cycle, and torque output of muscle strength during patient rehabilitation. Throughout this process, all data flows from the perception side to the compute processing platform, where AI algorithms generate gait recommendations based on each patient's rehabilitation status.

In this scenario, fixed pre-trained models are insufficient; fresh data is needed because patients' rehabilitation status changes daily. To achieve optimal therapeutic outcomes, each session's protocol may vary. This is where Transformer architectures become valuable: traditional rule-based systems don't fully exploit robotic decision system value, instead operating as guided action systems. Transformer-based robots can iterate results based on rule guidance and result feedback, deepening rule frameworks to optimize outcomes.

Rule-based robots have historically addressed only simple commercial scenarios, failing in complex environments because rule-based systems cannot anticipate — requiring industry experts to continuously叠加 rules, with long development cycles. If future compute and Transformer models become sufficiently powerful, we may see scenarios where robots are developed in short cycles simply by feeding complex environment data to them.

Google's Robot Transformer utilizes massive multidimensional data. After completing mechanical actuation, it generates corresponding tag systems and evaluation mechanisms to optimize robot performance. Robot Transformer's most direct improvement for robots is solving the dexterous hand problem, dramatically simplifying the entire compute and front-end model, with costs dropping sharply. It doesn't require large compute platforms and can run on ordinary terminals. The future direction for dexterous hands is miniaturization, reduced power consumption, simpler logic, and self-learning capabilities.

Meanwhile, large model-driven robots will achieve higher development in safety flexibility and fault-tolerant adaptability. As intelligent robots move closer to ordinary users, even making direct physical contact in scenarios like massage therapy and human assistance, the absolute safety of robotic manipulators and end effectors and their adaptability to unexpected events become critically important.

Take Wanxun Technology's Pliabot technology and its flexible robotic arms as an example. Through multi-loop control, Wanxun's Nimbo series flexible arms can achieve end-effector loads exceeding their own weight and sub-millimeter end precision matching visual positioning — striking a new balance between safety, operational capability, and economy. Leveraging large model-generated multimodal fusion-based planning commands, flexible arms can execute operation commands accurately and powerfully while providing the final barrier of safety and adaptability at the execution end closest to users.

Furthermore, large models combined with lightweight flexible hardware represented by flexible arms will spawn integrated software-hardware solutions completing the full closed loop from robot understanding, decomposition, planning, actuation, to execution. This essentially matches a flexible, powerful, safe, and affordable "body" to the large model-driven intelligent "brain," rapidly enabling multi-scenario deployment.

OpenAI's latest models have many potential robot applications: five-finger hands, wrestling competitions, open-source deep reinforcement learning platforms, adaptive robot control systems, and robot simulation. But whether these capability improvements can help us打磨 out good products on the industrial and scenario sides remains a question. Industrial deployment ultimately returns to the team's understanding of scenario pain points and challenges.

Planning and Decision-Making

At the planning layer, large models can better embed diverse prior knowledge bases than existing technical solutions, combining with on-site randomness for multimodal intelligent fusion — ultimately obtaining executable robot planning commands that balance accumulated historical experience with on-site random variation. This will greatly accelerate robot development toward general intelligence, enabling intelligent robots to more quickly understand and adapt to open-ended environments and user commands, completing increasingly rich tasks.

At the decision-making layer, there's a distinction between individual intelligence and swarm intelligence. Individual intelligence is result-oriented intelligent decision-making. Swarm intelligence is like intelligentizing an entire factory — appearing as a collection of many individual stations, but fundamentally the factory itself is a giant robot with a single optimization objective.

Factors to consider include whether robotic arms or equipment are fatigued, overheated, or need rest; how to schedule differently based on each product variety's process and intermediate衔接 when producing different batches and varieties; how to ensure specific order quality and achieve true quality closed loops in final inspection stages, achieving swarm decision-making optimal integration goals.

Individual decision-making is currently relatively mature, but there's still substantial optimization space moving from control toward the process side. Traditional process-based decision-making imports workers' process cognition and experience into robot individuals through algorithms or process parameter requirements. AI's advantage lies in its ability to build upon process requirement frameworks, combining with massive data models from process scenarios to continuously form self-training algorithms based on process requirements and optimize them — thereby forming more precise decision systems.

Interaction

The Learning-based methods represented by Transformer will bring entirely new paradigms to robot human-robot interaction interfaces. With the emergence of multimodal large models, new possibilities for human-robot interaction will emerge: ordinary users with no robot programming experience will be able to directly operate robots using natural language and open-ended commands, issuing complex, ambiguous, or even inquiry-style instructions. Large model-driven interaction engines will analyze and understand user objectives, obtain accurate executable purposes, decompose tasks, and backend-distribute them to robot planning layers for task planning and execution.

Two particularly important possibilities exist in the interaction domain. One is Neural Radiance Fields (NeRF). NeRF uses 2D images to train neural networks, predicting images from novel viewpoints to complete scene rendering — an implicit expression of 3D scene information. NeRF brings new 3D interaction methods and new possibilities for 3D content generation. Even with current limitations in training/inference efficiency and handling dynamic objects, the entirely new application scenarios it enables are worth anticipating.

The other is LLMs (large language models), which will help AR glasses become portable AI assistants. AR glasses possess first-person near-eye display capabilities. Empowered by models like ChatGPT, they can achieve voice interaction, real-time translation, navigation, shopping recommendations, and other functions.

Compute Platforms Regarding compute, our conclusion is: different platforms for different scenarios, with corresponding compute platforms for each. For example, glasses correspond to XR2, vehicles to Orin, cloud to A100. This will basically be the future architecture — no single chip or compute product will solve all problems. Because power consumption varies across scenarios, as do power supply and system energy management, task complexity, and scenario states.

---

Industrial Scenario Improvements

We've been imagining whether future industrial scenarios could unify underlying systems on a single platform, with upper layers directly calling ChatGPT for natural interaction and then mining for problems?

Previous industrial internet efforts have had limited results. A core reason is that these weren't information-native — they still relied on externally mounted equipment. If data comes from hardware's built-in AI systems capable of self-forming analytical feedback, valuable predictive analytics becomes possible. This is critical for achieving individual station intelligence in factories; connecting these intelligent stations into lines can form closed-loop whole-factory data chains, enabling whole-factory AI model data architecture. If native data can't be obtained, catching data with aftermarket sensors works too — Geely Auto has a particularly impressive automated production line already using Transformer and ChatGPT-like models to understand where factory problems lie.

AI's improvement of factory predictive maintenance remains limited, because industrial processes involve many interlocking steps, with琐碎 product production processes. The key is achieving adaptivity for both individual factory processes and global complex states. The pain point of industrial internet is that every equipment parameter requires individual testing; one piece of equipment can only do one thing, and beyond that scenario, the equipment algorithm changes. So we've been envisioning future robots operating like assembly lines, with very rapid cognitive alignment — requiring AI that can both control individual manufacturing processes and possess global brain decision-making functions.

Tesla's factory is essentially one giant robot. Sensor data is full-stack, full-chain, full-system shared. But the foundation has sensors, intelligence, and digitization throughout. These all require native data; patching after the fact won't work.

---

A Bonus

Wednesday, May 17, 19:00-21:00, BlueRun Ventures will launch the second installment of its Robot Salon series online, inviting academic experts and outstanding robot entrepreneurs to jointly envision the transformative possibilities that multimodal large models bring to the robotics industry. Scan the QR code to register and join our discussion!

Originating in Silicon Valley, BlueRun Ventures was established in 2005 as a venture capital firm focused on early-stage startups.

Currently, BlueRun Ventures manages multiple USD and RMB dual-currency funds in China, with assets under management exceeding RMB 15 billion, making it one of the largest early-stage funds domestically. Its investment stage focuses on Pre-A and Series A rounds, covering hard technology and innovative interaction, enterprise technology, new consumption, and healthcare. It has cumulatively invested in over 150 startups, including Li Auto, Waterdrop, QingCloud, Guazi.com, Qudian, Songguo Mobility, Ganji.com, Energy Monster, Yuntu Semiconductor, Machenike, Yunsheng Intelligence, Anxin Network Shield, and BioMap.

BlueRun Ventures has been ranked first in Zero2IPO's "China Early-Stage Investment Institutions Top 30" and ChinaVenture's "China Best Early-Stage Venture Capital Institutions TOP30," and was named among Preqin's Top 10 globally consistent high-return VC fund managers.

Additionally, BlueRun Ventures has repeatedly received honors from Forbes China, 36Kr, Cyzone, Caixin Media, CBNweekly, Jiemian, and other media institutions, including "China's Best Early-Stage Institution of the Year," "China's Top Venture Capital Institution," "Most Entrepreneur-Friendly Early-Stage Institution of the Year," and "Most Influential Early-Stage Institution of the Year."