Randall Balestriero, Yann LeCun and Kerda Varaku propose giving AI world models a useful form of hindsight: estimating what a different action would have produced under the same hidden disturbance. Their preprint, submitted to arXiv on November 18, describes a correction that requires no retraining of the existing predictor. The extra ingredient is already available: how the original action actually turned out. Source
Across four rendered simulators, the authors report counterfactual R² rising from a range of 38–46% to 86–96%, using the same world models. R² measures explained variation in the alternative outcomes—not the percentage of decisions the system gets right. Source Source
Same current, different heading
Take the paper’s boat example. A skipper chooses a heading, but an unseen current pushes the boat off course. The counterfactual question is where the boat would have ended up under another heading, with that very same current. Quietly swapping in nicer water would rather defeat the exercise. Source
The authors’ first-order approximation starts with the observed outcome and adjusts it by the model’s predicted difference between the two actions: Source
Estimated alternative outcome ≈ observed outcome + prediction for the alternative action − prediction for the original action.
The observed result carries evidence about the particular disturbance that occurred. The predictor supplies the estimated effect of changing the action. In his separate explainer, Paolo Di Prodi describes the calculation as two calls to the predictor and an addition, without needing to identify the hidden disturbance itself. Source
The tested JEPA world models predict image features—numerical representations of visual inputs—rather than pixels, as Di Prodi explains. Their DINOv2 and masked-autoencoder image encoders turn images into those features and stay fixed during predictor training. This is not a demonstration of generating an alternative-outcome movie. The authors say the correction needs no counterfactual training data, extra parameters or retraining. That does not mean the underlying models never needed training. The shortcut comes afterward. Source Source
Physics gets a veto
The authors’ rationale is that when actions and disturbances act as separate forces, their interaction appears at second order in time. That supports a first-order approximation over short transitions under the paper’s conditions. Smooth motion alone is not a universal permission slip. Source Source
The preprint identifies failures involving abrupt changes in the dynamics, certain representations of outcomes and disturbances that scale the command itself. In particular, a disturbance that changes how strongly an action works can undermine the correction: its effect cannot simply be carried unchanged from one action to another. Source Source
What the hindsight-planning test shows
The authors also report a practical test in hindsight planning: choosing an action that would have reached a goal in the same event. Their approach succeeds in 35–69% of events, versus 14–23% for an oracle planner that knows the simulator but not the observed outcome. The counterfactual model can use that outcome, so the comparison doesn't establish superiority over a fully informed simulator. Source
Nor does hindsight automatically become foresight. Di Prodi notes that the observed prediction error describes this particular event; carrying it into later transitions helps only if the disturbance persists. The reported evidence is simulated and does not establish reliable operation on physical robots. Source Source
Comments