rhoyn

Low latency, accurate hands

github.com/rhoyn/teleop-hands-latency-analysis

A note before we start. This article is about measuring latency, not about solving it. Everything below is the instrument — the rig, the ten algorithms, and the numbers that came out of them. The repository linked above holds the raw recordings of every experiment behind those numbers. How we actually got rhoyn down to 20 ms is a separate story, and it will be the subject of another article.

Low latency eyes measured the half of the teleoperation loop that comes back to you: light leaves the room, and some number of milliseconds later it lands on the headset display. This is the other half — the half that goes out. You move your hand; some number of milliseconds later the robot’s hand moves. Here is that delay, dialled up, on the same simulation:

Accuracy is why the number matters, and why an article with accuracy in its title spends its length on milliseconds. A person steering anything runs a closed loop: look, correct, look again. That loop assumes the thing you are steering responds now. Put a delay in it and you stop correcting toward where the target is and start correcting toward where it was — the textbook recipe for overshoot. You push, nothing happens, you push harder, and then everything you asked for arrives at once. At 20 ms you don’t notice. At 90 ms you are placing the gripper by trial and error, and no amount of care in the hand itself buys the precision back.

So: how long does a humanoid take to reproduce a movement of your hand? Two obvious ways to find out, both of which fail.

Filming it and counting frames

Point a camera at the operator and the robot, then measure how many frames apart the two hands start moving. This is what the field actually does. The ExtremControl paper estimates the end-to-end latency of ten humanoid teleoperation systems by running optical flow over each project’s released videos — a genuinely clever way to get numbers for systems you don’t have in the room.

The trouble is resolution. A 30 fps clip cannot resolve better than about 33 ms, and 33 ms is more than the whole latency we are chasing. It is the same failure as the millisecond counter on a monitor from the eyes article: the instrument is coarser than the thing being measured, so the measurement comes back as a shrug in the shape of a number.

Timestamping your own software

Log the moment a pose leaves the tracker, log the moment the joint command is issued, subtract. Two problems, and both of them are fatal.

First, it stops at the last line of your code. The command still has to reach a motor, and the motor still has to move a limb that has mass and compliance and gearing. Every millisecond of that is delay the operator feels, and none of it appears in your log.

Second, the tracker’s clock is not the robot’s clock. Subtracting a timestamp on one machine from a timestamp on another gives you the latency plus however wrong your clock synchronisation is, with no way to tell the two apart. At 20 ms of true latency, a few milliseconds of NTP drift is a large fraction of your answer.

Measuring the hands instead

Stop trying to time the software. Time the two hands, with one clock.

Two MPU6050 breakouts. One rides on the operator’s hand, the other on the robot’s. Both are wired to the same ESP32, one per I2C bus, because both boards answer to the same address and neither will give it up. The firmware samples both at 1 kHz and stamps every sample with one micros() call.

That last detail is the entire trick. Two sensors, one clock, so there is no device-to-device offset to estimate away and nothing to synchronise. Every millisecond of measured lag is physical delay, in the hardware, between two hands.

Mounting orientation does not matter, which is what makes the rig this dumb. Every algorithm works on gyroscope magnitude, |ω| = √(ω_x² + ω_y² + ω_z²), and that value is identical for any two rigidly related frames. Tape the boards on crooked and the numbers are unchanged. There is no relative rotation to calibrate, no common frame to establish, and because it is the gyro rather than the accelerometer, gravity never enters the signal at all. The only rule is to stay inside the MPU6050’s default ±250 dps, since a clipped peak distorts every estimator equally.

Recording a session

Drive the robot through whatever teleoperation stack you are measuring, and move in bursts with rests between them. Sharp starts and stops are what carry timing information; smooth continuous motion carries much less of it, and completely still stretches are gated out of the analysis entirely. Ten to twenty reaches over five to twenty seconds makes a good take.

A session comes off the wire as a CSV, one row per millisecond, and that CSV is the whole measurement. Every session was also filmed, which the analysis never looks at — the latency comes entirely from the two IMUs — but it is what lets you go back and see what the hands were actually doing on any take whose numbers look odd.

The operator’s hand is the one entering from the left; the robot’s is the black gloved one, with an IMU taped to the back of it. Watch either clip at speed and you will see them move together. That is exactly the problem — at 20 ms, the eye cannot tell you which one is behind, let alone by how much.

Ten algorithms, one median

Here is the part we care most about. A single delay estimator is easy to fool: pick the wrong one for a recording and it will hand you a confident, wrong number, and nothing in its output will tell you so. So the tool does not pick one. Ten independent algorithms run over every recording, and the reported latency is the median across them.

That is what makes the number worth trusting. No single method decides the answer, the ten do not share a failure mode, and a method that goes wrong on a given take cannot move a median far. An algorithm is also allowed to abstain on a recording it cannot measure — the run continues, its tile reads “rejected”, and the median forms from the rest. A refusal is a result, not an error.

Every run writes one image: a two-column grid holding the result and one 800×800 tile per algorithm, each showing its own internals in three panels. Nothing is hidden behind a summary statistic — if a number looks wrong, the picture of how it was computed is right there. Four of those tiles, from run 0 on the rhoyn stack:

peak_matching — 18.4 ms. The one to show someone who does not want the signal processing. Every reach has a moment where the hand is moving fastest; the algorithm finds that moment on both hands and measures the gap. The middle panel zooms a single reach and brackets it: two curves, two peaks, 20 ms apart. There is no model and no assumption in that panel, just two humps and the distance between them.

group_delay — 18.3 ms. The same recording read in the frequency domain. A pure delay shows up as phase falling linearly with frequency, so the slope of that line is the answer and its intercept is thrown away. This one matters more than its number suggests: the robot does not reproduce your motion exactly, it also shrinks and smooths it, and every method that assumes a clean delayed copy is biased by that. This one reads phase only, so amplitude can do whatever it likes.

kalman_bank — 19.6 ms. Two hundred and one Kalman filters run at once, one per delay hypothesis from 0 to 200 ms, each scoring how well its own assumption explains what actually arrived. The bottom panel is the one to read: it shows which hypothesis is winning as the recording plays. It thrashes for the first half second, then locks on and stays there for the remaining five.

The result. All ten, sorted, with the median on top. This run: 20.6 ms, spread from 18.3 to 26.0, MAD 0.9 ms. Ten methods that share no assumptions landing within a few milliseconds of each other is the actual claim being made here — not that any one of them is right, but that they agree.

Results

Twenty-one recordings across three teleoperation stacks, same two IMUs, same ten algorithms. Each cell is the median of the ten estimators for that recording; each column’s last row is the median across its recordings, which is that stack’s number. Everything is how far the robot’s hand lags the operator’s, in milliseconds.

The full names are XRoboToolkit PICO VR + Genesis ExtremControl and XRoboToolkit PICO VR + NVIDIA GR00T low latency weights, both run on the same robot in the same room as ours. That makes them 2.2× and 4.4× slower than rhoyn respectively, and the three sets never touch: rhoyn’s slowest recording is 21.6 ms, and the fastest recording from either competitor is 37.2 ms. There is no overlap to argue about.

The per-algorithm breakdown behind every cell above — all ten estimators for all twenty-one recordings — is in the repository, along with the report grid and the video for each take.

Against the published field

The ExtremControl paper’s Table I, re-sorted by latency and reduced to the latency column, with rhoyn added — Ziyan Xiong, Lixing Fang, Junyun Huang, Kashu Yamazaki, Hao Zhang, Chuang Gan, ExtremControl: Low-Latency Humanoid Teleoperation with Direct Extremity Control, arXiv:2602.11321:

The two instruments are not the same, and the gap is wide enough that it matters. Every row but the first is optical flow read off a published video, bounded by that video’s frame rate — the failure this article opened with. The rhoyn row is the 20.3 ms from the table above, rounded to the precision the other rows can support. Read the column as an order-of-magnitude ranking, not as eleven numbers measured the same way.

One row overlaps our own measurements, which is the useful part. The stack we measured as XRoboToolkit PICO VR + Genesis ExtremControl came out at 44.5 ms, against the ~54 ms MoCap and ~64 ms VR the paper reports for ExtremControl. Two instruments, one system, and the IMU number is the faster of the two — which is the direction that should reassure you about the method here rather than the reverse. An instrument that flattered us would have gone the other way.

What the number does and doesn’t include

It includes everything mechanical and computational between the two boards: tracking, network, controller, actuation. That is the point — it is the latency the operator actually feels — but it means a slow or compliant joint counts here exactly as much as a network hop does. It is not a measurement of any single link in the chain, and it cannot tell you which link to fix.

The robot also attenuates and low-passes your motion rather than reproducing it exactly, so methods that assume a pure delayed copy are biased by that; group_delay is the one that isn’t, because it reads only phase. And the tool reports a magnitude, never a negative lag: the robot is taken to follow always, which is the only direction a working teleoperation link can produce.

Everything is open sourced — the ten algorithms, the ESP32 firmware, the host-side recorder, and all twenty-one recordings with their reports and videos. MIT licensed, at github.com/rhoyn/teleop-hands-latency-analysis.

Play with it

The simulation again, nothing switched off. T/G moves the control latency — the delay this article measures — and R/F the camera latency from the eyes article, with the arrow keys moving both. Y cycles what is on the floor and H toggles the robot’s camera between stereo and single view. Set the control slider to 20 ms, then to 90 ms, and try to place a hand exactly where you meant to. That difference is the whole argument.