rhoyn

Stable walk

github.com/rhoyn/teleop-walking-benchmark

A note before we start. This article is about measuring walking, not about solving it. Everything below is the harness and the field it scores — thirteen open-source policies that are other people’s work, not ours. The repository linked above holds the harness, the checkpoints and every run behind the numbers. Where rhoyn’s own locomotion ends up is a separate story, and it will be the subject of another article.

Low latency eyes and low latency, accurate hands both measured the link between operator and robot — how long the picture takes to come back, how long the command takes to land. This one is about what happens at the far end of that link. You can have a 20 ms connection to a robot that falls over.

Thirteen policies and no way to choose

Teleoperating a humanoid means picking a walking policy to sit underneath you. There are a lot of them — open-source controllers for the Unitree G1 come out of labs and companies at a steady rate, each with a checkpoint you can download and a demo video where it walks convincingly.

None of the demos are comparable. Every project picks its own task, its own terrain, its own disturbances and its own definition of success, and every project’s video shows the run where it worked. There is no shared number anywhere, which means there is no way to answer the only question we actually had: which of these should we build on?

So we ported thirteen of them to one C++ interface and ran them all against the same task in MuJoCo. Each policy becomes a directory with an init, a step and a name; step receives the raw target error in the body frame and returns the twenty-nine motor targets. Everything above that line — the scene, the tour, the disturbances, the scoring — is identical for all thirteen.

The task

A tour of twenty-four waypoints drawn from a one-metre disc. A target stays current for five seconds and is then replaced whether or not the robot arrived, so the clock never waits: a policy that walks too slowly simply loses ground rather than stalling the run. Twenty-four targets at five seconds each is a two-minute walk.

Every candidate is handed the same stance. A crane ramps the robot to one shared default pose over three seconds and lets go, then two seconds pass before the first target so the handover transient is not charged to the first waypoint. Three seconds of crane, two of lead-in and a hundred and twenty of walking make a full run 125 seconds. Critically, the stance is shared rather than the pose each checkpoint would have chosen for itself, so part of what gets measured is how well a policy takes over from a starting position it was not necessarily trained around.

A run ends when the pelvis drops below 20 cm, which is far under any crouch a policy might intend and is reached only once the robot is already on the floor.

The punches

This is the part that makes it a test rather than a demo. Once per waypoint — twenty-four times in a run, on the same clock as the targets — something hits the robot.

Each punch picks a body, a contact point on that body, and a direction, all at random, and pushes for 80 milliseconds. That is four control steps: long enough that the policy sees the disturbance in its own observations and has to answer it, short enough to read as an impulse rather than as someone leaning on the robot. Force ramps from nothing at the start of the run to a 600 N ceiling at the end, and each individual hit draws somewhere between half and all of the current ceiling, so even late in a run a policy cannot assume the next hit is no worse than the last one it survived.

The whole campaign is drawn from the seed before the run starts, not at the moment of impact. A policy cannot change the punches it faces by walking somewhere else, and it cannot outrun them by walking faster. Every candidate on a given seed is disturbed the same number of times, at the same moments, with the same forces.

That is what turns a walking test into a stability and disaster-recovery test. A policy is judged on whether it can absorb a hit it never saw coming and get back on the tour, not on how neatly it tracks a target while nothing is going wrong.

The difficulty is deliberate

A benchmark every candidate completes ranks nothing. It separates no one; it only shows the task was too easy. This one is tuned to sit at the edge of what these policies can do — the best of them finishes most seeds, none finishes all, and no row is a perfect score.

Here is the whole field on one seed, twenty-seven seconds in. Each robot is running a different policy, named on the floor beside it. The orange cylinders are punches resting where they landed.

And the closing forty-two seconds of that run, all thirteen side by side, by which point several are already down:

The 500-seed campaign

Seeds 0 to 499, all thirteen policies on every seed: 6500 runs, 160 robot-hours of simulated walking, sixteen tours in flight at once on an RTX 4090 and a Ryzen 7 7800X3D. No run errored, and no run was interrupted.

Each seed draws its own tour and its own punch campaign, so a seed is a whole task rather than a repetition of one. Runs are deterministic: the same seed reproduces bit for bit.

Results

Completions first, because it is the column that matters and the one every other column has to be read against:

And the full record. Survival is simulated seconds to the fall, counting a completed tour as its full 125 s; position and yaw errors are means over every target the policy actually reached. Each policy links to the project it came from.

Reading it

gr00t_wbc wins the criteria that matter. Fewest falls, longest survival, lowest yaw error — and at five hundred seeds its interval clears amo’s with nothing to spare, 70.8% against 70.0%. First place is not a sampling artefact, but it is no longer a wide margin either. It does not hold the lowest mean position error: openwbt edges it, 8.80 cm against 9.10 cm, on a row with zero completions.

openwbt is accurate and cannot finish. The best mean position error in the field and zero completions in five hundred tries. Its 34.39° yaw error is where it goes wrong: capped at its published 0.3 m/s, it cannot hold heading on the harder draws.

clobot fails identically on every seed. 5.3 s mean, median and worst, and not one scored target in five hundred tours. No run differed from any other.

Low error does not mean good. run_residual cannot strafe, reverse or turn in place — its command floor forces it forward at 0.5 m/s whenever it is off target, so it can only approach on an arc. Its 516 cm is not a bad gait, it is a policy that cannot do this task. And openwbt’s first-place error is earned on the fraction of each tour it survived. Read the completion column first, always.

Nothing completes reliably. The best policy in the field fails a quarter of its tours, and the top four die around twenty seconds before the end of a 125 s tour on average — exactly where the punch ramp approaches its 600 N ceiling. Surviving the end of the tour is the discriminator here, not tracking error.

What not to read into it

Four of these policies completed fewer than five tours — falcon with one, and openwbt, rl_gym and clobot with none. Their error columns rest on whichever fragments of a tour they reached before falling, which is not the same measurement as the rows above them. Treat those two columns as unreliable rather than as a ranking.

The extra four hundred seeds separated what a hundred could not. robomimic sat inside amo’s and homie’s interval before and now sits clearly below both, fourth on its own. What stays genuinely tied is amo with homie, asap with rl_mjlab and holosoma, and the four rows at the bottom that never finish anything — no sample size separates policies that all score zero.

The same asymmetry runs through the whole table in one direction: because position and yaw errors are averaged only over targets actually reached, falling early flatters a candidate rather than penalising it. A policy that dies at 30 seconds is scored on its best thirty seconds.

Step timings are deliberately absent. The sweeps shared one machine fifteen at a time, which inflates per-step latency by an order of magnitude. Those need measuring solo, and we did not, so we do not report them.

And this is MuJoCo, not a floor. Everything above is a statement about these checkpoints in this simulator against this task, which is the right place to narrow a field of thirteen down to a shortlist — and the wrong place to end the argument.

The licences matter more than you would like

Every checkpoint here is third-party, and their terms are not uniform. Three flags are worth reading twice before building on any of this:

• homie is CC BY-NC — research and non-commercial use only, so a commercial build has to exclude it outright.
• robomimic and run_residual declare no licence at all, which makes them effectively all-rights-reserved. Both are fetched from upstream rather than committed, because an MIT repository will not hand you a copy it has no right to hand you. The Apache-2.0 licence circulating alongside the RoboMimic weights belongs to a different project.
• amo is Apache-2.0 but sim-only at its authors’ explicit request, citing a sim-to-real failure that damaged a robot.

And the candidate that wins the campaign, gr00t_wbc, is under the NVIDIA Open Model licence rather than a standard open-source one. If licence terms decide your build, openwbt is the most permissive candidate with a competitive position error — which is a genuinely awkward place for this table to land, and the reason the licence column is not an appendix.

Twelve of the sixteen checkpoints are committed in the repository; the other four are pulled by a script that verifies each file’s sha256 as it lands. Provenance and terms for all of them are in the repository’s NOTICE.

Walk it yourself

The simulation below starts on gr00t_wbc, the policy that tops the table. Pick another and it swaps without a reload. The policies above are not just numbers in a table — the checkpoints are small enough to run in a browser. Pick one and it downloads that policy’s weights, then runs the real network at 50 Hz against real MuJoCo physics on your machine. Nothing is pre-recorded, and nothing is faked: this is the same checkpoint the benchmark scored, taking the same twenty-nine joints through the same contact model.

Loading gr00t_wbc, the strongest policy in the table — about 12 MB with the physics engine.

Click the simulation, then drive it: W and S walk forward and back, A and D step sideways, Q and E turn. The up and down arrows swap between policies without leaving the simulation, best at the top down to worst at the bottom. The rest of the legend still works — R/F and T/G move the camera and control latency from the other two articles, Y cycles what is on the floor, and H swaps the robot’s camera between stereo and single view. The robot is running the policy’s own gait at its own commanded speed cap, so a candidate that tracks heading badly in the table will wander here too. Push it and it falls over — and unlike the benchmark there is nothing punching it, so whatever goes wrong is the policy on its own.

Adding a policy

If your policy should be in this table, it is one directory and two lines. Implement the three-method interface with the weights beside it, add the include and a branch in the factory, and it runs against the same tour as everything else. Deriving a velocity command from the target error is the policy’s own business — the harness does not assume how you do it.

Everything is open sourced — the harness, the thirteen adapters, the scene, and the twelve checkpoints that can be redistributed. MIT licensed where it is ours to license, at github.com/rhoyn/teleop-walking-benchmark. Read the NOTICE before you build on a result.