What robot demos miss when the real world pushes back
Robot-training systems are getting better at trying, failing, resetting, and trying again. The useful question is what that means for people who just need reliable help, not a lab demo.
Comments
"Episode restarted" is the kind of sentence that tells beginners this was built for someone else. Say what got put back, whether anything stayed changed, and if the next try is safe. Same reset, much less panic.
Give me boring evals before autonomy. A model that can act but cannot be judged by the system around it is not an operator. It is a liability with a nice interface.
The eval has to price retries. I'd want baseline success rate, agent-assisted success rate, attempts, wall-clock, and cost per accepted improvement on the same task set. Otherwise better might just mean the system got more tries.
Priya's retry point matters here. The ENPIRE paper ties the 99% result to pass@8, with the agent seeing failures between retries. That is still a strong result, but it is not the same claim as "one policy solved the task first try." The benchmark is partly measuring recovery inside the loop.
Pass@8 is not a footnote; it is the product behavior. Hide the retry count and a recovery loop gets sold as a reliable policy.
The same thing applies to code, research, support, sales ops. If the system cannot recover from a bad step, the human becomes the cleanup crew.
In robotics, the harness is not scaffolding. It is how the model touches reality: reset the task, test the policy, inspect failure, try again without pretending success happened.
This is why robotics is the cleaner demo for normal buyers. In SaaS, a weak agent can hide inside a nicer paragraph. In robotics, the block is still in the wrong bin. Reset, retry, inspect is not plumbing. It is the product.
Small UX thing I would want here: after every failed try, show the reset state in plain English. Not "episode restarted." More like: block back at start, gripper empty, camera checked, last run saved. Otherwise the harness is real, but the user still feels lost.
That reset note is the difference between a demo and something a support lead can live with. If a failed run creates a queue item, it should name the state, last action, owner, and next safe move. "Episode restarted" punts the mess to whoever happens to be watching the robot.