Astribot Releases SmoothRL, Enabling Online Reinforcement Learning to Keep Pace with LLMs' Asynchronous Inference | Yunqi Partners

As VLAs and World-Action Models grow larger and inference times stretch longer, robots in real-world deployments have long since switched to asynchronous inference — executing task A while the model computes task B in the background. But this "deployment reality" poses a fresh challenge for online reinforcement learning: the actions the model "planned" and the actions the robot "actually performed" no longer match. Who gets credit for success, and who takes the blame for failure?

Recently, the foundation model team at Astribot, an embodied intelligence company seed-led by Yunqi Capital, offered an answer: the release of SmoothRL, an online reinforcement learning framework capable of asynchronous execution. The framework was validated for the first time on real-world high-dynamic throwing tasks: dynamic throw success rates improved from 39% to 94%, pen capping from 8% to 83%, and package opening from 30% to 90%.

From "can it do it" to "more accurate, more stable, more reliable" — pretraining teaches robots how to act, and real-world post-training hones those capabilities to deployment-ready precision. We continue to watch Astribot advance its full-stack approach of "AI model — embodied OS — tendon-driven hardware" toward scaled deployment of Physical AI.

Real robots don't have a "pause button."

Today's VLAs and World-Action Models typically generate action sequences — action chunks — spanning future time windows. As models scale up, so do inference times. But robots can't simply halt every few hundred milliseconds to wait for the next chunk of actions. In high-dynamic tasks like throwing, a single pause can bleed away accumulated velocity and doom the entire attempt.

So real deployments more commonly rely on asynchronous inference: the robot keeps executing its current actions while the model computes the next segment in the background. In short, hands doing A, model already computing B.

But this creates a new problem for online reinforcement learning. The model generates an action chunk, yet only a portion of it actually enters the physical world. The actions the model "planned" and the actions the robot "actually performed" are no longer identical.

Recently, Astribot's foundation model team released SmoothRL (Online Reinforcement Learning During Asynchronous Execution), an online RL framework designed for asynchronous execution. It addresses a critical question: as large-model asynchronous inference becomes the deployment norm, which actions should online RL actually learn from?

Technical report + unedited videos:

www.astribot.com/en/research/SmoothRL

SmoothRL marks the first time such asynchronous online RL has been validated on real-world high-dynamic throwing tasks, further demonstrating that online learning can serve not only high-precision correction but also dynamic operations involving continuous acceleration, precise release, and zero tolerance for pausing.

SmoothRL overall framework: the robot continuously generates real-world data during asynchronous execution; the base policy provides general capabilities, and online RL updates the action policy based on real execution outcomes.

After Asynchronous Execution,

Why Does RL "Mispost" Credits and Debits?

Traditional online RL typically assumes a tidier rhythm: the model computes actions, the robot executes them, and the policy updates based on results. What the model generates, the robot performs.

Asynchronous execution breaks this correspondence. By the time a new action chunk is ready, the robot has already advanced partway through the previous instruction; and before this new chunk fully executes, the next inference result may arrive and replace its latter half. So within a single action chunk, some actions are too late to change, some are actually executed, and some never happen at all.

If reinforcement learning still optimizes the entire chunk uniformly, a direct problem arises: actions the robot couldn't change or never performed may still get "credited" for success or "blamed" for failure.

SmoothRL untangles this accounting first. It divides the action chunk into three regions based on execution status:

• Committed region: Actions already "committed" by the previous inference round, unchangeable and certain to execute.

• Execution region: Actions from the current newly generated chunk that the robot will actually perform.

• Discarded region: Actions the model generated but the robot will never execute, as the next inference round will overwrite them.

These three regions cannot be treated equally during training. SmoothRL's core insight: reinforce only what the robot actually executed — that is, let gradients pass only through the execution region.

This ensures the optimization objective aligns with the process the robot actually experiences at runtime. The second step: making training and deployment follow the same temporal rhythm. SmoothRL runs asynchronous inference directly during training rollouts: model computation and robot execution happen in parallel, and the replay buffer records trajectories produced under this real-time relationship. The team distills this principle as Reinforce in Deployment — not training in an idealized synchronous world and then switching to asynchronous deployment, but confronting the execution dynamics the robot will actually face from the start of training.

In the paper's concrete implementation, the team uses fine-tuned π0.5 as the base policy per task, builds on the RLT framework, and employs a lightweight TD3-style actor-critic to predict residual corrections in the original action space. The S1 robot executes actions at 30 Hz with inference requests at 5 Hz — a new action chunk every 200 ms. The base policy predicts 32-frame action chunks; under a fixed latency budget, Committed and Execution together occupy 12 frames, with 6 frames belonging to the current round's actual Execution Region, while the remaining 20 frames get overwritten by subsequent chunks before execution.

SmoothRL divides asynchronous action chunks into Committed / Execution / Discarded regions, allowing value gradients to pass only through the actually-executed Execution Region.

From High-Speed Throwing to Millimeter-Scale Insertion:

Testing Two Categories of Real-Deployment Challenges

The team tested three real-hardware tasks on Astribot's tendon-driven S1 platform, covering two common and complementary scenarios: high-speed dynamic manipulation and high-precision dual-arm manipulation.

Particularly for high-dynamic tasks: robot movements change rapidly, and waiting for model computation at every step would cause stuttering, making asynchronous inference essential.

Dynamic throwing: 39% → 94%. The robot must grasp objects from random positions and throw them into target bins at varying locations. This task doesn't end at reaching a target pose; it requires the arm to continuously build velocity during swinging and release at the precise moment. The paper specifically notes that a pause at action chunk boundaries can nearly bring the arm to rest, making it difficult to recover the release velocity in the remaining swing. Thus, it's particularly sensitive to asynchronous execution. At the evaluation checkpoint of 250 cumulative rollout episodes, success rates improved from 39% with the base policy to 94%.

Pen capping: 8% → 83%. Both arms must precisely align pen and cap, with allowable relative pose error around 5 mm. The base policy typically reaches the general vicinity but adapts poorly to minor variations in cap position. Final success rates improved from 8% to 83%.

Package opening: 30% → 90%. The robot must insert a blade approximately 1 mm wide into a box-lid seam only 2–3 mm wide, then slice through tape along the seam. The base policy exhibited a consistent leftward bias. Notably, its learning curve wasn't monotonically improving: at 150 rollout episodes, success temporarily dropped from 30% to 20%, then recovered to 40%, ultimately reaching 90%. Real online exploration doesn't necessarily improve steadily.

Success rate changes across three real-hardware tasks as cumulative rollout episodes increase.

More revealing than success rates is how the robot's failures changed.

Before RL, all three tasks showed clear systematic biases: throws failed to modulate release velocity correctly for target distance; the blade consistently drifted left during package opening; pen capping produced overly similar motions regardless of cap position. What online RL did was leverage real execution outcomes to gradually correct these fixed deviations.

By the final checkpoint, failures had visibly contracted toward success positions. In package opening, failed samples showed blade deviation from the seam of roughly 1–2 mm left or right; pen capping failures were mainly slight misalignments near the correct position.

So the RL here isn't reteaching robots "what package opening means" from scratch. It's doing something closer to a deployment problem: taking a policy that already basically knows how, and training it to sufficient accuracy.

Moreover, greater accuracy must coexist with motion smoothness. Through smoothness constraints in the policy, during one real autonomous throwing rollout after online RL, the right end-effector's acceleration RMS decreased by 52% and jerk by 47%. The robot not only succeeds more often but also exhibits fewer sudden accelerations, decelerations, and direction changes — the entire throwing motion becomes smoother and more continuous.

Velocity / Acceleration / Jerk comparison in dynamic throwing

After the Robot Starts Working, Training Isn't Over

In recent years, robot foundation models have primarily addressed "can it do it": more data, stronger models, broader capabilities. But when robots actually enter real environments, the new problems are rarely complete inability — they're off by a few millimeters, half a beat, or unstable when object positions shift.

SmoothRL targets the layer beyond this: a pretrained policy that already possesses base capabilities, entering the real world, and continuing to correct itself based on real execution outcomes.

The current paper uses sparse task success/failure rewards, and operators can intervene when necessary during training. These intervention actions belong to the raw action space and can be directly used for subsequent model training. Thus, this isn't yet fully unattended "autonomous evolution," but rather a more efficient online post-training mechanism oriented toward real deployment.

It also has clear boundaries. The current implementation requires each chunk-level inference to complete within a preset latency budget; the lightweight residual policy's expressiveness is also constrained by the frozen base policy. If the base policy itself is too far from target behavior, local residual correction can't magically rescue it.

Next steps, the team plans to explore larger-scale policy updates, broader task distributions, and the combination of asynchronous execution with end-to-end optimization of generative policies.

SmoothRL points toward a problem becoming increasingly concrete: pretraining teaches robots how to act, and real-world post-training hones those capabilities to be more accurate, more stable, and more reliable.

Behind SmoothRL lies Astribot's distinctive long-term conviction about robot models: action should not be a byproduct that visual or language models happen to generate at the end, but rather a "first-class citizen modality" within robot intelligence. Because robots ultimately exist not merely to understand the world, but to act correctly in the physical world. The Lumo series of foundation models has consistently focused on "why move this way," from Lumo-1's explicit reasoning to Lumo-2's prediction of how the world changes due to action — all developing along this line of thinking.

Along this main thread, Astribot has also formed a complete model architecture spanning front-end Agent, middle-layer foundation model, and RL post-training: the Agent handles interaction, memory, task understanding, and capability invocation; the foundation model gradually develops generalizable manipulation capabilities in open scenarios; and RL corrects the policy from real execution successes, failures, and feedback, enabling continuous evolution.

As models develop comprehensively, Astribot's "AI model — embodied OS — tendon-driven hardware" full-stack system also continues iterating, driving application and scaled deployment of Physical AI.

SmoothRL was led by Jianan Wang as project lead at Astribot, with Guang Gao and Yuxuan Nong as co-first authors, and Baifu Huang contributing to the research.