A field guide to sim-to-real transfer that actually transfers.

The most common failure mode of an RL project isn't the algorithm. It's the gap between a clean reward curve in simulation and a policy that actually behaves on the robot in front of you. Bridging that gap is its own discipline — one with more folklore than literature.

This guide collects what we've learned shipping policies onto real hardware: which techniques are non-negotiable, which are theatre, and what order to do them in.

The gap nobody plans for

Your simulator is a model. Your robot is the territory. The difference between them — the sim-real gap — is a function of three things:

• •Dynamics mismatch — friction, mass distribution, latency, actuator delay, sensor noise.
• •Distribution shift — the real world contains states your simulator never samples.
• •Observation gap — cameras have rolling shutters, encoders drift, depth has holes.

A policy can be brilliant in sim and useless in deployment because of any one of these. Most teams underestimate the third one until the day they ship.

Three failure modes, in order

When a sim-trained policy fails on hardware, it almost always fails in one of three ways. They're ordered here by how often we see them and how subtle they are to diagnose.

You can ignore the second column for a while. You can't ignore the third — that's where most teams give up.

Randomization vs. calibration

There are two schools of thought. Randomize the simulator until the policy is robust enough to span the real world. Or calibrate the simulator to look as much like reality as possible. In practice you need both, but in a specific order.

Calibrate first — even a sloppy calibration. Then randomize around the calibrated point. Randomizing without calibration produces a policy that's robust to a distribution that has nothing to do with your actual deployment. You get a policy that survives a wide range of nonsense scenarios and fails on the one you care about.

Doing randomization well

The temptation is to randomize everything. The instinct is wrong. A useful randomization scheme:

1. 1.Is gradient-aware — the policy should see what changed and adapt, not pattern-match.
2. 2.Targets parameters you can measure — if you can't measure it on the robot, don't randomize over it.
3. 3.Has a stopping rule — otherwise wider distributions slow training without improving transfer.
4. 4.Is auditable — if a policy fails in production, you should be able to ask which parameter combination it never saw.

# A reasonable starting schedule
schedule = DomainRandomizationSchedule(
    friction       = ("uniform",  0.6, 1.4),   # measured at the lab
    mass_scale     = ("uniform",  0.9, 1.1),
    actuator_delay = ("uniform",  0.00, 0.05),
    camera_noise   = ("gaussian", 0.0, 0.02),
    auto_widen_until = "transfer_score >= 0.9",
    log_distribution = True,                   # for post-hoc audit
)

In our experience, the right baseline is to randomize over maybe six parameters. We've seen teams set up forty and end up with a policy that spends 80% of its capacity on robustness to noise that never occurs in their physical setup.

Real-to-sim calibration

Ten minutes of teleop is enough to fit the dynamics parameters of most articulated robots within tolerable error. The pipeline:

• •Record a short trajectory under teleop — deliberate, varied, with stops.
• •Fit simulator parameters to minimize the residual between sim and replayed real.
• •Hold out 20% of the trajectory to validate the fit.
• •Lock those parameters. Don't let training adjust them.

Calibration isn't glamorous. It's the highest-leverage thing you'll do all month.

One nuance: calibrate against trajectories that include the regime your policy will operate in. Calibrating a bipedal locomotion policy against a robot standing still gets you a great standing-still model and a useless walking model. The data you collect should look like the work the policy will do.

Modeling sensor noise

The single most common bug we see at deployment time isn't in the algorithm or the dynamics — it's in the observation pipeline. Sim sensors are clean. Real sensors aren't.

Three patterns worth implementing on day one:

• •Quantization noise — encoders return discrete values. Don't feed your policy 14 decimal places it'll never see in the field.
• •Asynchronous updates — cameras run at 30Hz, IMUs at 200Hz, joint encoders at 1kHz. If your sim feeds all of them at the policy step rate, you've built the wrong observation.
• •Calibrated outliers — every sensor returns garbage occasionally. Train the policy to handle it; otherwise it'll act on garbage when it sees it.

# Match the actual sensor characteristics
obs_pipeline = SensorPipeline(
    rgb_camera   = Camera(rate=30, jitter_ms=4, dropout=0.001),
    depth_camera = Camera(rate=15, holes_prob=0.02),
    joint_pos    = Encoder(rate=1000, quantization=2**-12),
    imu          = IMU(rate=200, bias_drift_std=1e-4),
)

On-robot adapter heads

Even with perfect calibration, the real world drifts. Hardware ages. Workpieces vary. The answer isn't to retrain — it's to add a small adapter layer that can fine-tune in production without overwriting the policy you carefully trained.

Adapter heads have three properties worth defending:

• •Bounded by design — strict gradient norm clip, strict step budget, strict KL divergence from the base policy.
• •Trivially revertible — you can disable them in one command if telemetry drifts.
• •Independently versioned — adapter v17 might be running on top of base policy v3.2.4. Treat them like dependencies.

Concrete sizing

A useful adapter head adds under 5% of the base policy's parameters. More than that and you're basically retraining, which means you've lost the property that made the base policy trustworthy in the first place.

Evaluation gates

Before any policy touches a real robot beyond shadow mode, it should pass an ordered series of automated gates. None of these are optional.

1. 1.Reproducibility gate. — Re-run the training from pinned commit. Final policy hash must match.
2. 2.Regression gate. — Behavior on 200+ pinned scenarios must not regress more than the agreed delta.
3. 3.Adversarial gate. — A curated set of edge cases your team has accumulated over months. They're a moat.
4. 4.Counterfactual replay gate. — Take last quarter's production logs, perturb them, replay against the new policy.
5. 5.Shadow gate. — 7 days minimum running alongside the production system. Compare action distributions.

If any gate fails, the merge is blocked. Not flagged — blocked. The whole point is that the gates are a contract: the engineer who built the policy doesn't decide which failures are acceptable; the gates do.

The migration checklist

If you're bringing a policy from sim to real, work through this in order:

1. 1.Calibrate the simulator from 10 minutes of teleop. Validate on held-out trajectory.
2. 2.Characterize each sensor on the bench. Match the sim observation pipeline to those numbers.
3. 3.Train with gradient-aware domain randomization around the calibrated point.
4. 4.Pin a regression suite. Block the merge if behavior regresses on any scenario.
5. 5.Deploy in shadow mode. Compare predicted vs. baseline action distributions.
6. 6.Add a bounded adapter head. Version it independently from the base policy.
7. 7.Ship behind a feature flag. Roll out to 5%. Watch the eval suite, not the dashboard.
8. 8.Have a rollback runbook your on-call team has rehearsed. The first time you use it should not be in production.

None of this is novel. All of it is necessary. If any one step is missing, the other seven won't save you.

Anti-patterns we've seen

The same mistakes show up across teams, across robotics domains, across years. If you recognize yourself in any of these, it's worth pausing.

• •Tuning hyperparameters to close the gap. — If transfer is bad, the answer is usually in calibration or observation matching — not the learning rate.
• •Believing the reward curve. — Reward going up doesn't mean the policy got better at the task; it sometimes means the policy got better at exploiting the reward.
• •One big randomization range. — Wider isn't better. Targeted is.
• •Skipping shadow mode. — “We’ll catch it in QA” is an expensive sentence.
• •Rebuilding the simulator. — Eight weeks of physics work rarely produces more transfer than one week of careful sensor modeling.