Attent!on! ChatGPT Hands Humanity an Apple! | Yunqi Kepu

Late on March 13 Beijing time, humanoid robot star company Figure AI posted a video on X. In it, the company's Figure 01 robot, integrated with an OpenAI large model, could hold conversations with humans, understand instructions, and execute tasks. According to the company, OpenAI's model provides high-level visual and language intelligence, while Figure supplies fast, low-level, dexterous robotic actions for the neural network.

Figure 01 stands roughly 170 cm tall and weighs 60 kg. It can carry a 20 kg payload, moves at 1.2 meters per second, and runs for up to five hours on a charge.

Veteran robotics expert Eric Jang predicted not long ago: "ChatGPT happened overnight. I think intelligent robotics will too."

Does Figure 01's release mean that prediction has already come true? Is large-model intelligence entering the physical world just around the corner? Immediately following the Figure 01 launch, this edition of "Yunqi Kepu" explores the latest technical advances behind it, along with OpenAI's layout in embodied intelligence.

P.S. Our AGI+ salon series is still recruiting. AI practitioners with passion, insight, and conviction are welcome to join us in exploring how to push new technology to its limits and build AI Ace products (click the link on the left for details).

Author | Li Yuan Editor | Zheng Xuan

OpenAI and Figure collaborative robot demo video | Source: Figure

Over the past year of embodied intelligence progress, you may have seen similar demonstrations of robots making autonomous decisions and grasping objects. But in this video, the fluency of Figure's humanoid robot in conversation, the sense of intelligence it displays, and the fluidity of its movements at near-human operating speed are absolutely top-tier. Figure specifically emphasized that the entire video has no speedup and no editing — it was shot in a single take. At the same time, the robot was acting completely autonomously, with no teleoperation — seemingly a subtle dig at the Stanford cooking robot that went viral not long ago for showing off cool mechanical capabilities without much actual intelligence.

More striking than the robot's intelligent performance is that this represents only a modest effort from OpenAI — from OpenAI's announcement of its collaboration with Figure to advance the frontier of humanoid robotics to the release of this video, only thirteen days passed.

The intelligence behind this Figure humanoid robot comes from an end-to-end large language-vision model, currently a very cutting-edge area in embodied intelligence. Last year, GeekPark reported on Google's progress in a similar field. Google's end-to-end robot control model was hailed by some industry insiders as the GPT-3 moment for robot foundation models.

At that time, however, Google's robot model could only perform some grasping based on conversation. It couldn't hold conversations with humans, nor could it explain to humans why it was doing what it did. And Google itself had more than five years of robotics research experience, dating back to Everyday Robotics.

Figure, by contrast, was founded in 2022. From OpenAI's announcement of its involvement in a collaboration with the company to their joint release of a robot capable of autonomous conversation and decision-making — just 13 days.

The development of robot intelligence is clearly accelerating.

01. End-to-End Large Model Drives Robots to Near-Human Speed

Figure founder Brett Adcock and AI team lead Corey Lynch explained on X the principles behind the robot interactions shown in the video.

This breakthrough came from OpenAI and Figure working together. OpenAI provides visual reasoning and language understanding, while Figure's neural network supplies fast, low-level, dexterous robotic actions.

All behaviors the robot performs come from learned, internalized capabilities, not from teleoperation.

Researchers feed images from the robot's cameras and text transcriptions from speech captured by onboard microphones into a multimodal model (VLM) trained by OpenAI that can understand both images and text. This model processes the entire history of the conversation, generates a language response, and replies to the human via text-to-speech.

The same model also decides which learned closed-loop behaviors to run on the robot to execute a given command, loading specific neural network weights onto the GPU and executing the policy.

This is why this robot qualifies as "end-to-end" robot control.

Starting from language input, the model takes over all processing and directly outputs both language and behavioral results, rather than outputting intermediate results that then get processed by other programs.

Figure's onboard cameras capture images at 10 Hz, and the neural network outputs 24 degrees-of-freedom actions at 200 Hz.

The Figure founders noted that this represents a significant speed improvement for the robot, beginning to approach human speed.

Image source: Corey Lynch's X

The multimodal capabilities of OpenAI's model are key to the robot's ability to interact with the world. We can see many similar moments in the video demonstration, such as:

• Describing its surroundings.
• Using commonsense reasoning when making decisions. For example, "The dishes and cups on the table will likely go into the drying rack next."
• Translating ambiguous high-level requests like "I'm hungry" into contextually appropriate behaviors, such as "hand the person an apple."
• Explaining in simple English "why" it performs a particular action. For example, "This is the only edible item I can give you from the table."

The model's capabilities are strong enough to also maintain short-term memory. For instance, when shown in the video: "Can you put them over there?" What does "them" refer to? Where is "there"? Answering correctly requires the ability to reflect on memory.

The specific hand movements can be understood in two steps:

First, an internet-pretrained model performs commonsense reasoning on images and text to arrive at a high-level plan. As shown in the video: Figure's humanoid robot quickly formed two plans: 1) place the cup in the dish rack, 2) place the plate in the dish rack.

Second, the large model generates 24-DOF actions (wrist poses and finger joint angles) at 200 Hz, serving as high-speed "setpoints" for a higher-rate whole-body controller to track. The whole-body controller ensures safe, stable dynamics, such as maintaining balance.

All behaviors are driven by a neural network visuomotor Transformer policy, mapping pixels directly to actions.

02. GPT — Sora — Robotics: OpenAI Claims "Intelligence"

In the summer of 2021, OpenAI quietly shut down its robotics team. At the time, OpenAI announced it was indefinitely halting exploration of the robotics field, citing a lack of data needed to train robots to move and reason using AI, which had impeded R&D progress.

But clearly, OpenAI never stopped paying attention to this domain.

In March 2023 — a year ago — GeekPark reported that OpenAI had invested in Norwegian robot manufacturer 1X Technologies. Its vice president was none other than Eric Jang, whom I mentioned at the beginning of this article, who believes embodied intelligence will arrive suddenly.

And not coincidentally, 1X Technologies' technical direction also involves end-to-end neural networks for robot control.

In early March this year, OpenAI joined other investors in Figure's Series B round, giving the two-year-old company a $2.6 billion valuation.

And it was following this funding round that OpenAI announced its collaboration with Figure.

Figure founder Brett Adcock is a serial entrepreneur who "knows how to put deals together," having founded at least seven companies over his career. One went public at a $2.7 billion valuation; another was acquired for $110 million.

After founding the company, he recruited research scientist Jerry Pratt as CTO and former Boston Dynamics/Apple engineer Michael Rose as head of robot controls. Corey Lynch, the AI team lead who shared details this time, was previously an AI researcher at Google DeepMind.

Figure announced it had recruited hardcore design talent across motors, firmware, thermal management, electronics, middleware OS, battery systems, actuator sensors, and mechanical structures.

The company has indeed moved fast. Before collaborating with OpenAI, it had already notched several achievements. In January 2024, Figure 01 (Figure's first humanoid robot) learned to make coffee. The company said this involved introducing an end-to-end neural network, with the robot learning to correct its own mistakes over ten hours of training.

Figure 01 uses AI to learn coffee-making | Image source: Figure

In February, the company showed Figure 01's latest progress. In the video, the robot had learned to move boxes and transport them onto a conveyor belt, though at only 16.7% of human speed.

It had even taken its first commercial step: Figure announced a commercial agreement with BMW Manufacturing to integrate AI and robotics into automobile production, deploying at BMW's Spartanburg, South Carolina plant.

And in today's video announcement, Figure declared its goal is to train a world model, eventually selling billions of model-driven humanoid robots.

However, despite the smooth progress of OpenAI's collaboration with Figure, it appears OpenAI isn't putting all its chips on one robotics company.

On March 13 Beijing time, Physical Intelligence, a new robotics AI company founded by researchers from Google Research, UC Berkeley, Stanford University professors, and others, was reported by Bloomberg to have also received funding from OpenAI.

Unsurprisingly, this company is also researching AI that could become general-purpose robot systems.

Making multiple bets in the robotics field, collaborating for just 13 days to produce a leading robot foundation model — OpenAI's intentions in robotics are attracting significant attention.

The future of intelligent humanoid robots is no longer just about Elon Musk.

---