Large Models + Machine = A More Advanced Robot Is Here | BlueRun Ventures Robot Salon Series

AI + machine = robot. When AI advances, the evolution of robots becomes inevitable.

In the last wave of robotics innovation, improvements in AI algorithms enabled a leap in autonomous mobility, ending the era of movement dependent on tracks, QR codes, and other markers. Now, what new innovation cycle will large language models unlock for robots?

Recently, BlueRun Ventures held the second installment of its robotics salon series. This session brought together researchers from the frontier of academia — Dr. Hong Wang, associate professor at Harbin Institute of Technology (Shenzhen) and dual-appointed researcher at Guangdong Jihua Provincial Key Laboratory; Dr. Peng Wang, researcher and doctoral advisor at the Chinese Academy of Sciences Institute of Automation; and Dr. Yuecong Xu, research scientist at the Agency for Science, Technology and Research (A*STAR) and lecturer at Nanyang Technological University — alongside industry practitioners Baoxing Qin, CTO of Gaussian Robotics; and Weijun Zheng, VP of Visual Technology R&D at Bluecore Technology. BlueRun Ventures partner Cao Wei joined them in discussing four practical, critical questions: In what dimensions can large models transform robots; what problems must be solved for their deployment; what models suit different scenarios; and how to acquire and process data. We've distilled the highlights into under 3,000 words, hoping it proves useful for founders.

01 How Large Language Models Can Transform Robots

Hong Wang

Robotics itself is an extremely integrated technology. What distinguishes it most from other mechanical equipment is intelligence. Without a clear breakthrough in intelligence, it's fundamentally no different from ordinary automation equipment. Intelligence has two main applications: task understanding and environmental perception. Currently, what robots can achieve is essentially following trajectories, stopping to avoid obstacles when they encounter them. Only with breakthroughs in language-based task understanding can robots truly qualify as intelligent machines, capable of replacing humans across broader automation applications.

There were previously several approaches to robotic automatic programming. One relied on vision, using 3D models to automatically generate robot work trajectories. Another used natural language, with robots understanding and generating corresponding programming languages for work trajectories. But all these approaches explored rather limited paths.

Yuecong Xu

Previous CNNs and RNNs built up networks through successive stacking, while Transformer can perform global feature extraction in one step. This global feature approach works for natural language, computer vision, and certain sensors — especially LiDAR, which carries a degree of semantic information.

Baoxing Qin

The potential of large models lies mainly in two aspects. First, language, semantics, and images used to be separate, lacking connection. With large models, different states in space can be linked together. Second, generalization capability. In the past, different tasks tended to use different models trained on different data. But large models can use general models or general underlying capabilities to solve problems through zero-shot or one-shot approaches.

Gaussian Robotics currently relies more on exhaustive categorization to solve scene understanding problems. But cleaning robots in relatively open, human-robot coexisting environments encounter numerous long-tail issues — different floor materials, different types of trash. Large models can provide solutions for addressing infinite problem sets with finite problem collections. In the future, there may be more automatically generated action logic to solve problems.

The capabilities of models in environmental perception and scene understanding for robot usability can be illustrated through the automotive industry. The auto industry currently has two approaches: map-light and map-heavy. The map-light approach maintains passenger vehicle autonomous driving functionality without relying on map coverage. Since environments change, the less dependence on pre-labeled map elements from map sources, the better — and labeling map elements requires substantial manual labor. So if you want to improve user experience, reduce demands on users' professional skills, and ensure long-term stable robot operation, semantic understanding of the environment is essential.

Weijun Zheng

Currently, the vast majority of industrial robots are deployed by experts trained by robot manufacturers; very few customers can complete deployment themselves. If robots' own understanding capabilities can be enhanced while maintaining equivalent reliability, that's a huge space for imagination.

02 Problems Yet to Be Solved

Hong Wang

If we want to apply large models to robots, the biggest obstacle may be lightweight deployment. Models need to address computing power and energy consumption on each terminal application platform. Even leading companies now have to conduct large model training in the dead of night, freeing up server resources for model training.

Peng Wang

Robotics isn't like the internet, where you can access everything directly through the cloud. Its applications have certain real-time requirements. How to combine edge computing with the cloud may require breaking down complex tasks — some processed in the cloud, real-time-sensitive ones handled at the edge.

There's also the data acquisition problem. How to acquire and genuinely apply data related to movement and manipulation? There are now many different physics engines that can acquire data through virtual environments, but the actual amount of data obtainable in robotics remains very small.

After all, robots need physical bodies to interact with the environment. Truly applying models to robots requires not just building language models, but combining them with the robot's environment, the robot's own structure and hardware constraints, and task characteristics to select optimal solutions.

The biggest difference between robots and the internet is that robots have physical forms, so requirements for result accuracy and safety are extremely high. ChatGPT sometimes produces logically structured nonsense. If we entrust robots' decision-making, planning, and perception capabilities to it, that could create problems. So large model applications in robotics may need to be layered — perhaps using large models at the top level, while execution, action, and environment interaction may still need refinement using reliable methods, even non-learning-based approaches.

Yuecong Xu

Regarding the issue of一本正经地胡说八道 (speaking nonsense with a straight face), we can also draw on autonomous driving experience for solutions. Autonomous driving initially used deep learning to guide path planning and decision-making. Using traditional model methods or generating many decision or trajectory candidates through deep learning, then using human-written scoring mechanisms or arbiters to judge reliable results. As long as there's discrimination capability, there's no fear of unreliability.

03 Large Models vs. Small Models

Peng Wang

"Large" model is relative. Combined with specific tasks, application scenarios, and robot characteristics, you can localize a large model into a small model for easier deployment and application. You don't necessarily need to pursue model scale from the start, or how many environments and task objects it can cover — as long as it solves the problem. For startups, you still need to combine with industry-specific data accumulation. Without sufficient data accumulation, performance can't reach extremes.

Baoxing Qin

One thing must be emphasized: large models are not the optimal solution in the perception domain. The question of how much public data can actually be used for daily training or model building is crucial. The data needed for robot perception is far less than for traditional tasks or conventional image recognition, but large models' success isn't just the model itself — it's also due to massive data accumulation. With small data volumes, Transformers underperform compared to small models.

Scenarios define products; scenario demands determine model requirements. The more open the scenario and the more diverse the operational forms, the higher the general requirements for large models. Factories, conversely, need models that reach extremes in specific verticals, deploying cerebellum-like tasks to individual robot ends for real-time inference and response. Large models suit the cloud, like a Swiss Army knife; small models suit the machine room, like a kitchen knife. Not every situation needs a Swiss Army knife.

Yuecong Xu

"Large" model is relative. Combined with specific tasks, application scenarios, and robot characteristics, you can localize a large model into a small model for easier deployment and application. You don't necessarily need to pursue model scale from the start, or how many environments and task objects it can cover — as long as it solves the problem. For startups, you still need to combine with industry-specific data accumulation. Without sufficient data accumulation, performance can't reach extremes.

04 Data and Simulation Systems

Hong Wang

Building data systems and establishing simulation environments involves three key points. First, data acquisition. Large amounts of data are needed to build large models, and large models themselves can serve as means of generating data.

Second, evaluating data itself. Currently, much data is generated in virtual environments, then used to build new models that are migrated to actual robot platforms. Much data differs greatly from real-world scenarios, and models trained on it suffer significant performance loss when migrated to robot platforms. But point cloud data is different — because point cloud data generated in virtual environments basically comes with labels. Although there's slight performance loss when migrated to robot platforms, it's basically usable.

Third, data generation. Early robots gathered data from the internet for training, which might be useful for macro-level robot decision-making. But for robots with high safety requirements, data needs to be acquired through actual scenarios, or obtained through evaluation and filtering after virtual environment construction to get high-quality data, thereby obtaining high-quality models.

Peng Wang

Regarding data, there are two tasks: one, data acquisition and collection; two, data digestion. Tesla has implemented many triggers in its data collection chain — abnormal situations can automatically trigger data collection. Additionally, Tesla labels data by extending the time window, using optimized results over longer periods for labeling. The second approach is shadow mode, which more labels vehicle driving behavior in such situations. The third is through 3D reconstruction, building the entire environment through social learning, then using this approach to label each frame in reverse.

Weijun Zheng

Collected point cloud data and texture data can be combined, gathered, and fed into large models to automatically generate more environments. Currently collected data is stored by structure, categorized by various scenarios, and later used to generate more general data.

Simulation is commonly used for task scheduling. All machine data across an entire factory can be placed on a simulation platform, completely simulating actual conditions. Before entering the factory, simulation can determine how many robots complete how many tasks. Factories have many unexpected situations, which can only be generated through models. Discovering unreasonable responses to unexpected situations through simulation is also a closed-loop training process that can improve the entire system.

▶ More Content ◀

Originating in Silicon Valley, BlueRun Ventures was established in 2005 as a venture capital firm focused on early-stage startups.

Currently, BlueRun Ventures manages multiple USD and RMB dual-currency funds in China, with assets under management exceeding RMB 15 billion, making it one of the largest early-stage funds domestically. Its investment stage focuses on Pre-A and Series A, covering hard tech and innovative interaction, enterprise technology, new consumer, and healthcare. It has cumulatively invested in over 150 startups, including Li Auto, Waterdrop, QingCloud, Guazi.com, Qudian, Songguo Chuxing, Ganji.com, Monster Charging, Yuntu Semiconductor, Machenike, Yunsheng Intelligence, Anxin Wangdun, and BioMap.

BlueRun Ventures has been ranked #1 on Zero2IPO's "China Top 30 Early-Stage Investment Institutions" and ChinaVenture's "China Best Early-Stage Venture Capital Institutions TOP30," and was named to Preqin's Global Top 10 VC Fund Managers for Sustained High Returns.

Additionally, BlueRun Ventures has repeatedly received honors from Forbes China, 36Kr, Cyzone, Caixin Media, CBNweekly, Jiemian, and other media organizations, including "China Best Early-Stage Institution of the Year," "China Top Venture Capital Institution," "Most Founder-Friendly Early-Stage Institution of the Year," and "Most Influential Early-Stage Institution of the Year."