Lab Notes

SO-101: teaching a cheap robot arm to do useful things

I've been experimenting with an SO-101, an open-source robot arm that I built and program myself. There are two parts: the leader arm, which I call Brachius Rex, and the follower arm, which I call Brachiomimus. These let me use an approach called imitation learning: the follower arm (Brachiomimus) can learn a task by watching me demonstrate it over and over via teleoperation (controlling Brachius Rex), rather than by following rules I write out by hand. This page tracks my journey. I start with "episodes", or recordings of task demonstrations. Anyone can use these to train "policies", or trained models that decide what the arm should do at each moment. Below, I shared runs in simulation and on the real arm, my lessons learned, and a quick infographic to ties the concepts together

Progress

This is my journey so far, in order: recording the episodes (task demos), my first training attempt that failed, and the my v2 training that worked.

Step 1 · Morning one

Recording 50 episodes

I spent the first morning teleoperating the SO-101; I controlled the Leader arm (Brachius Rex) by hand while computer recorded everything it does. I repeated my task (moving a yellow block to a blue landing zone) until I had fifty episodes. Each episode was an episode is just one complete recording of the arm doing the task from start to finish, with a reset in-between. Each episode logs the arm's joint positions, the camera images, and what action came next; this raw data is what every later step depends on. View the dataset →

Home workbench with the SO-101 follower arm on a tape-marked mat, a recording laptop, and maker gear on the shelf above
This is my workbench: Brachiomimus (Follower arm), the tape-marked workspace, and the laptop I record episodes from.
Laptop showing a terminal encoding episodes next to the Rerun viewer with joint/action plots and the front camera feed
Episode ten of fifty, with the joint and action data streaming live into Rerun, the tool I use to watch what the arm is doing.
Step 2 · First attempt

I trained a policy, and it glitched

I tried my first training pass on the fifty episodes over 5,000 steps (see infographic); this showed an Action Chunking Transformer (ACT) model the data I recorded so it can learn a policy of its own. It seemed to move in the right direction, but stuttered in place more than it fulfilled the goal. I did some research to learn what might cause it: insufficient steps in the training job. View this policy (v1) →

Step 3 · Retrain, with tracking

I logged the run to Weights & Biases, and it worked

I reran training, but this time I logged everything to Weights & Biases (W&B) so I could track track a training run over time; it also charted metrics like loss, a measure of how wrong the policy's predictions are, so I could verify the model was improving. I trained for 30,000 steps, roughly 17 full passes over the whole dataset; the training charts and the infographic below walk through that math in more detail. This time the policy held the task together on the real arm. View this policy (v2) →

Training charts

This report shows three curves from the v2 training run, the one that worked. - Loss is the overall metric for how wrong the policy's predictions are; it should trend downward as training goes on. - L1 measures that same kind of error more directly, as the average distance between the arm position the policy predicted and the real one. - KLD (KL divergence) is a more technical measure of how much the model's internal representation is drifting; it matters less for this story, but flat and stable is a good sign, while wild swings would not be. I didn't track my v1 policy run , so this is not a before-and-after comparison. Just nice to visualize.

The infographic

This is the training behavior in plain language: how fifty recorded episodes turn into 28,000 individual frames of data. It's also what "30,000 steps" means, broken down into nudges and epochs, where an epoch is one complete pass through the entire dataset.

Infographic explaining the training math: 28,000 frames from 50 episodes, batch of 16 = one step, 1,750 steps = one epoch, 30,000 steps is about 17 epochs

Resources

Here is the dataset, both saved versions of the trained policy, each one called a checkpoint, and the training run.

What's next

I'm still at work on the bench!

In progress
Sim-to-real for episode efficiency. Sim-to-real means training or testing a policy in a simulated version of the world before trying it on the real arm. The idea here is to see how few real demonstration episodes I can get away with, by comparing policies trained on fewer real episodes plus simulated data against the current real-only baseline.
Planned
Migrate to Gr00t N1.7. The plan is to swap the current policy, which was trained entirely from scratch on my own data, for NVIDIA's Gr00t N1.7. I'll compare how much data it needs, and how good its performance ompares to my from-scratch policy.
Exploring
Broader task set. Beyond the current pick-and-place task, I want to test whether the pipeline can generalize to new objects and manipulation without having to record an entirely new dataset from scratch.