Predicting how deformable objects evolve under robotic manipulation is a longstanding challenge. Existing approaches typically rely on per-object optimization to fit material parameters, which can be slow and cannot generalize, while end-to-end learned alternatives extrapolate poorly and often violate basic physical structure. We present PhysCoRe, a physics-corrected residual world model that couples a differentiable Material Point Method (MPM) simulator with two feed-forward neural networks. A material refinement module, Material from Motion (MfM), infers per-particle elasticity from visual observations, grounding the simulator in object-specific physics. A residual correction module, Residual from Dynamics (RfD), learns the discrepancy and predicts corrections to the simulator's internal dynamics, absorbing systematic biases that the analytical model cannot capture. This design also supports online material identification on novel objects. MfM adapts from limited interactions, and its predictive uncertainty steers further exploration toward the regions where its estimate is least confident. Experiments on real deformable-object manipulation sequences show that PhysCoRe outperforms state-of-the-art baselines in prediction accuracy, and that its predicted confidence forms a reliable distribution across the object's geometry, providing a natural signal for future confidence-guided exploration.
Along with the material itself, MfM predicts a per-particle confidence. We render this confidence onto the object with 3D Gaussian Splatting (3DGS) and overlay it on real manipulation sequences. The confidence is never supervised directly. As more of the object's motion is observed, MfM becomes more confident, and its high-confidence region concentrates on the parts that have actually deformed.
Confidence uses the viridis colormap, where blue denotes normalized low confidence and yellow denotes normalized high confidence.
Drag the slider on each pair below to wipe between the RGB observation and the confidence map overlay.
Lifting one corner of a towel
Lifting two corners of a towel
Lifting one end of a rope
Pushing the middle of a rope
Stretching a toy bear
Squeezing a piece of Play-Doh
The same confidence also supplies a signal for active exploration. A KUKA arm probes each object online and interacts next with the regions whose material estimate is least confident. Confidence starts low across the object and rises only after the arm has deformed a region, and it stays low everywhere else. The map therefore indicates where further interaction is still needed.
As in the visualization above, confidence uses the viridis colormap, with blue denoting normalized low confidence and yellow denoting normalized high confidence.
Drag the slider on each pair below to wipe between the RGB observation and the confidence map overlay.
Exploring a rope
Exploring a cloth
Exploring a toy bear
We use PhysCoRe as the dynamics backend of a goal-conditioned planner. The planner starts from a single RGB-D observation and a goal configuration authored in the simulator. At each control step it moves the gripper along the direction that reduces the predicted distance to the goal the most.
We demonstrate two tasks, lowering a suspended rope onto the table to form half of an “S” and folding a towel along its diagonal. The green region marks the goal configuration.
Planning a rope into a target shape
Planning a cloth into a target shape
@inproceedings{yin2026physcore,
title = {PhysCoRe: Physics-Corrected Residual World Models for Material-Aware Deformable Dynamics},
author = {Yin, Haocheng and Tao, Shuohan and Chen, Yongsheng and Gan, Lu},
booktitle = {Conference on Robot Learning (CoRL)},
series = {Proceedings of Machine Learning Research},
publisher = {PMLR},
year = {2026}
}