Relational object rearrangement (ROR) tasks (e.g. put plate in rack) require a robot to manipulate objects with precise semantic and geometric reasoning. Existing approaches either rely on pre-collected demonstrations that struggle to capture complex geometric constraints or generate goal-state observations to capture semantic and geometric knowledge, but fail to explicitly couple object transformation with action prediction, resulting in errors due to generative noise. To address these limitations, we propose Imagine2Act, a 3D imitation-learning framework that incorporates semantic and geometric constraints of objects into policy learning to tackle high-precision manipulation tasks. We first generate imagined goal images conditioned on language instructions and reconstruct corresponding 3D point clouds to provide robust semantic and geometric priors. This imagined goal point clouds serve as additional inputs to the policy model, while an object–action consistency strategy with soft pose supervision explicitly aligns predicted end-effector motion with generated object transformation. This design enables Imagine2Act to reason about semantic and geometric relationships between objects and predict accurate actions across diverse tasks. Experiments in both simulation and real world demonstrate that Imagine2Act outperforms previous state-of-the-art policies.

Before robot execution, the semantic–geometric constraint generation module produces an imagined point cloud conditioned on the initial observation. During training, this imagined point cloud is used as an additional input to the policy. Furthermore, by introducing Object–Action Consistency Learning, we compute the transformation between the initial and imagined object poses, which serves as an auxiliary prior input and contributes a loss term that enforces the strong correlation between object transformation and end-effector motion.