ChatGPT for Robots: Large Models Enter the Physical World, DeepMind's Breakthrough | BlueRun Ventures Share
Is Embodied Intelligence Close?
Robotic Transformer is finally here. Point to DeepMind.
BlueRun Ventures has twice explored large models + robotics = ? Now we see the first answer: using simple natural language to replace complex instructions and form concrete action plans — for instance, hearing "extinct animal" and going to find a dinosaur, or recognizing a photo of Taylor Swift. What does this mean? Here's today's BlueRun Ventures share —
We know that after mastering language and images on the internet, large models will eventually enter the physical world. "Embodied intelligence" should be the next direction of development.
Connecting large models to robots, using simple natural language to replace complex instructions and form concrete action plans, all without additional data or training — this vision looks appealing, but also somewhat distant. After all, robotics is notoriously difficult.
Yet AI is evolving faster than we imagined.
This Friday, Google DeepMind announced RT-2: the world's first Vision-Language-Action (VLA) model for controlling robots.
Now, without complex instructions, robots can be manipulated directly like ChatGPT.
How intelligent has RT-2 become? DeepMind researchers demonstrated with a robotic arm. Told to select an "extinct animal," the arm extended, its gripper opened and descended — and it grabbed the dinosaur plush.
Previously, robots could not reliably understand objects they had never seen, let alone perform reasoning tasks like connecting "extinct animal" to "plastic dinosaur plush."
Tell the robot to give a can of Coke to Taylor Swift:
You can tell this robot is a true fan. Good news for humanity.
The development of large language models like ChatGPT is sparking a revolution in robotics. Google has installed its most advanced language models on robots, finally giving them an artificial brain.
In a newly submitted paper, DeepMind researchers state that the RT-2 model is trained on web and robotics data, leveraging advances from large language models like Bard and combining them with robotics data. The new model can also understand instructions in languages other than English.
Google executives call RT-2 a major leap in how robots are built and programmed. "Because of this change, we had to rethink our entire research planning," said Vincent Vanhoucke, head of robotics at Google DeepMind. "So much of what we had been doing has become completely useless."
01
How was RT-2 achieved?
DeepMind's RT-2 breaks down to Robotic Transformer — a transformer model for robots.
Making robots understand human speech like in sci-fi movies and demonstrate survival capabilities is no easy task. Compared to virtual environments, the real physical world is complex and disordered. Robots typically require complex instructions to do simple things for humans. Humans, by contrast, instinctively know what to do.
Previously, training robots took a long time. Researchers had to build separate solutions for different tasks. With RT-2's powerful capabilities, robots can now analyze more information themselves and infer what to do next.
RT-2 builds upon Vision-Language Models (VLMs) and introduces a new concept: the Vision-Language-Action (VLA) model. It can learn from web and robotics data and translate that knowledge into general instructions robots can control. The model can even use chain-of-thought prompting, such as determining which drink is best for a tired person (energy drink).
In fact, Google launched the RT-1 robot version last year. With just a single pre-trained model, RT-1 could generate instructions from different sensory inputs (such as vision, text, etc.) to perform multiple tasks.
As a pre-trained model, building it well naturally requires massive amounts of data for self-supervised learning. RT-2 builds upon RT-1 and uses RT-1's demonstration data, collected by 13 robots in office and kitchen environments over 17 months.
DeepMind created the VLA model
As mentioned, RT-2 builds upon VLMs. VLM models have already been trained on web-scale data and can be used to perform tasks such as visual question answering, image captioning, or object recognition. Additionally, researchers adapted two previously proposed VLM models — PaLI-X (Pathways Language and Image model) and PaLM-E (Pathways Language model Embodied) — as the backbone for RT-2, naming these vision-language-action versions RT-2-PaLI-X and RT-2-PaLM-E.
To make vision-language models capable of controlling robots, one step remained: action control. The study adopted a remarkably simple approach: they represented robot actions as another language — text tokens — and trained them together with web-scale vision-language datasets.
The encoding of robot actions was based on the discretization method proposed by Brohan et al. for the RT-1 model.
As shown below, the study represents robot actions as text strings, which can be sequences of robot action token numbers, such as "1 128 91 241 5 101 127 217."
This string begins with a flag indicating whether the robot should continue or terminate the current episode, followed by commands for the robot to change the end effector's position and rotation, as well as gripper commands.
Since actions are represented as text strings, executing action commands becomes as simple as executing string commands. With this representation, we can directly fine-tune existing vision-language models and convert them into vision-language-action models.
During inference, text tokens are decomposed into robot actions, enabling closed-loop control.
02
Experiments
Researchers conducted a series of qualitative and quantitative experiments on the RT-2 model.
The figure below shows RT-2's performance in semantic understanding and basic reasoning. For the task "put the strawberry in the correct bowl," RT-2 requires not only representational understanding of strawberries and bowls, but also reasoning within the scene context to know that the strawberry should be placed with similar fruits. For the task "pick up the bag about to fall from the table," RT-2 needs to understand the physical properties of bags to disambiguate between two bags and identify the object in an unstable position.
It should be noted that all interaction processes tested in these scenarios were never seen in the robot data.
The figure below shows that RT-2 outperforms previous RT-1 and visual pretraining (VC-1) baselines on four benchmark tests.
RT-2 retains the robot's performance on original tasks and improves performance in previously unseen scenarios, rising from 32% for RT-1 to 62%.
The series of results demonstrate that Vision-Language Models (VLMs) can indeed be transformed into powerful Vision-Language-Action (VLA) models. By combining VLM pre-training with robotics data, robots can be directly controlled.
Similar to ChatGPT, if such capabilities were applied at scale, the world would likely undergo significant changes. However, Google has no immediate plans to deploy RT-2 robots, stating only that researchers believe these human-language-understanding robots will not remain at the level of capability demonstrations.
Simply imagine: robots with built-in language models could be placed in warehouses, help you fetch medicine, or even serve as home assistants — folding laundry, removing items from dishwashers, tidying up around the house.
It may truly open the door to using robots in human environments, taking over all directions requiring physical labor — covering the portion that OpenAI's report on predicting ChatGPT's impact on jobs identified as beyond large models' reach.
Is embodied intelligence not far from us?
Recently, embodied intelligence has been a direction that numerous researchers are exploring. Earlier this month, Fei-Fei Li's team at Stanford University demonstrated some new results: through large language models plus vision-language models, AI can analyze and plan in 3D space to guide robot actions.
Zhiyuan Jun's general-purpose humanoid robot startup "AgiBot" released a video last night also demonstrating large language model-based automatic robot behavior orchestration and task execution capabilities.
In August, Zhiyuan Jun's company is expected to showcase some recent achievements.
Clearly, in the large model domain, major developments are still to come.
This article is reprinted from Synced. Reference materials: https://www.deepmind.com/blog/rt-2-new-model-translates-vision-and-language-into-action https://www.blog.google/technology/ai/google-deepmind-rt2-robotics-vla-model/ https://www.theverge.com/2023/7/28/23811109/google-smart-robot-generative-ai https://www.nytimes.com/2023/07/28/technology/google-robots-ai.html https://www.bilibili.com/video/BV1Uu4y1274k/
Originating in Silicon Valley, BlueRun Ventures was established in 2005 as a venture capital firm focused on early-stage startups.
Currently, BlueRun Ventures manages multiple USD and RMB dual-currency funds in China, with assets under management exceeding 15 billion RMB, making it one of the largest early-stage funds domestically. Its investment stage focuses on Pre-A and Series A, covering hard tech and innovative interaction, enterprise technology, new consumption, and healthcare. It has cumulatively invested in over 150 startup companies, including Li Auto, Waterdrop, QingCloud, Guazi Used Car, Qudian, Songguo Mobility, Ganji.com, Monster Charging, Yuntu Semiconductor, Machenike, Cloud Saint Intelligence, Anxin Network Shield, and BioMap.
BlueRun Ventures has been ranked #1 on Zero2IPO's "China's Top 30 Early-Stage Investment Institutions" and ChinaVenture's "China's Best Early-Stage Venture Capital Institutions TOP30," and was named among Preqin's Top 10 VC Fund Managers Globally for Sustained High Returns.
Additionally, BlueRun Ventures has received consecutive awards from Forbes China, 36Kr, Cyzone, Caixin Media, CBNweekly, Jiemian, and other media organizations for titles including "China's Best Early-Stage Institution of the Year," "China's Top Venture Capital Institution," "Most Entrepreneur-Friendly Early-Stage Institution of the Year," and "Most Influential Early-Stage Institution of the Year."