Roll Model
A robotic arm that picks up whichever die I point at. orange-bot runs the arm, and friday, a planner, decides what to pick, checks each attempt and tries again when it fails.
orange-bot is the policy that drives my SO-101 arm, a SmolVLA I fine-tuned on demonstrations I recorded myself. Mark a die with an X and it picks that one up. Now it can put it down where it’s told, too.
friday is the part that thinks. It looks at the table, chooses which die to mark and where it should go, watches each attempt and retries the ones that failed. It never moves a motor itself.
The posts build on each other. First, getting a policy to work at all and measuring it properly. Then adding friday. Then letting the arm practise on its own, and finally world models.
Writings
-
00 Meet orange-botHow I set up the arm so a policy could learn, and the three camera setups it took. I keep this one updated as the rig changes. coming
-
01 Point, pick, place (almost)Teaching an SO-101 to pick up the die I mark and put it down where I mark. coming
-
02 Practice without meRewards, self-play and reinforcement learning on the real arm. coming
-
03 Think before you pickGetting the robot to predict what will happen before it acts, on hardware this small. coming