Three things follow from this:
- You will always need robot-specific data.
- The highest-quality way to convey a task is to show it (teleop or direct manipulation).
- Once you have strong domain-specific data, low-quality vision data from unrelated tasks doesn’t help much.
Notes from the article
https://www.notboring.co/p/robot-steps
Most real jobs are not fully repetitive, but they’re also not fully open-ended.
Success shows you what’s stable, failure shows you where the model breaks, and both tell you exactly what to work on fixing next. You iterate. You take small steps.
Whenever robotics evolves to incorporate another aspect of the job spectrum between automation and autonomy, it also unlocks another set of jobs, another set of customers, another chunk of the market. One small step at a time.
Despite all of that friction, customers keep buying these robots. That’s the market talking. Even limited, inflexible automation creates enough value that entire industries have grown around it. The low-variability left side of the spectrum already supports billions of dollars of business.
By deploying our robots in the field today, we get paid to collect the data we need to improve our models.
Our plan is to get paid to eat the spectrum.
Economic usefulness.
As John Carmack warned, “reality has a surprising amount of detail.”
So locomotion is the happy place because three things line up for machine learning. You can model the physics, measure the goals, and reset for free when things go wrong.
At that point, demonstration wins. It’s the most information-dense way to transfer intent. This is why people in the trades become apprentices.
Time taken to cover acquisition cost.
Eating the spectrum.
By learning from each of the real-world challenges that make up that 5%, we can bring failure down imperceptibly close to 0% within weeks of deployment.
We concentrate at the boundary where autonomy breaks instead of just collecting data on the 95% of stuff we do flawlessly over and over again.
We learn where reality actually disagrees with our model.
We think of it as a shared foundation of physical skills — perception, grasping, force control, sequencing, etc. — that compounds across deployments. For each new use case, we post-train on top of an ever-improving foundation.
Write about this.
There is a big reason that deployment accelerates learning that has nothing to do with models and everything to do with hardware.
Recall that data is 100-1,000x more efficient when aligned with its hardware. The more of the hardware you control, the more true this statement is.
We’ve created a unique way to do the torque sensing. Everyone else does strain gauges and current-based torque sensing. We have a method to directly measure torque through the bending of the metal, and our way is more accurate and more repairable, easier to manufacture, just better all around. Really, really great torque sensing.
Old robots were designed for motion replay: you send a robot a 30-second trajectory and the robot executes it. AI requires 100Hz real-time control. You’re sending a new command 100 times-per-second based on what the model sees in real-time. A lot of the existing robot APIs don’t even have real-time torque control.
So we wrote our own firmware for real time torque control with motor commutation at 60 kHz (60,000 times per second). This firmware makes our robots smoother, more precise, and more responsive, and also easier and more fun to use. This is really important because it means that we can physically handle a lot more use cases. This, in turn, means that hardware won’t limit our ability to eat more of the spectrum.
Think of it as the first industrial foundation model for robotics pre-training.
The difference is, our robots then quickly go out and learn everything a person actually does in a factory.
The tight integration between hardware, data, and model is so crucial while we are still learning how to do new use cases that we believe vertical integration is the only way to do it right.
“Markets go vertical to innovate product, horizontal to reduce cost and scale. Back and forth over a 40-50 year cycle.”