We’re building a unified autonomous software stack for robot learning.
Moving a trained policy to another simulator requires reproducing its control loop. A locomotion policy can output offsets from a nominal joint pose, which a separate controller converts into torques several times between predictions. Importing the robot model doesn’t recreate that exact behavior. For an engineer to interpret a failed gait, they must distinguish physics differences from interface differences.
A robot hand can have several joints driven by the same tendon, and contact with objects change how motion is distributed across them. If training gives the policy independent control of every joint, it learns to use a hand that doesn’t exist. Even with the right transmission model, tendon stretch can make motor readings a poor estimate of finger position under load (dealing with material degradation, hysteresis, etc.). Replacing a finger can change both the motion a policy produces and the state it believes the hand is in.
Recalibration is routine hardware development. The fitted model must also train at scale. A hand that behaves correctly in a single simulation may need simpler collision geometry to run efficiently across thousands of copies. That approximation can change which surfaces make contact, and therefore whether a grasp holds or slips. After a hardware revision, engineers must check that the accelerated model still reproduces the measured behavior.
Rollout time puts another limit on learning. Visual rollouts can spend much of their time rendering images or waiting for predictions. Some pipelines render intermediate frames the policy never reads. A batch can also hold ready environments behind slower ones. Adding environments can simply move the queue to the GPU running the model.
Asynchronous reinforcement learning (RL) lets rollout collection continue during model updates. When those rollouts reach the learner, the policy that collected them may already be out of date. Training must account for how far that policy has fallen behind, and on the other hand, discarding too much experience can erase the gain from collecting it faster.
We aim to produce usable rollouts faster and make parallel, asynchronous RL easier to run on new hardware. That work will include robots of our own, so changes to the training system can be tested on the machine.