Reinforcement learning has shown its strength in challenging sequential decision-making problems. The reward function in RL is crucial to the learning performance, as it serves as a measure of the task completion degree.  In real-world problems, the rewards are mostly human-designed, which requires laborious tuning, and is easily affected by human cognitive biases. 

To achieve automatic auxiliary reward generation, we propose a novel representation learning approach ：TDRP，which can measure the "transition distance" between states. Built upon these representations, we introduce an auxiliary reward generation technique for both single tasks and skill-chaining scenarios. 

The proposed approach is evaluated in a wide range of manipulation tasks and navigation task. We conduct experiments in two simulated robot navigation tasks and five manipulation tasks. The navigation tasks, Maze2D and AntMaze, are sourced from the D4RL benchmark. Table-wiping task (Wipe) and Nut-assembly task (NutAssemblySquare) are from the Robosuite benchmark  . The Pick-nut (Pick), Place-nut-on-bolt (Place), and Screw-nut (Screw) tasks are from the Factory benchmark, which includes a Franka Panda arm, a table, a nut, and a bolt. All components have high-quality simulations and rich meshes, as the complex control required to solve the tasks demands high-fidelity simulations. The experiment results demonstrate the effectiveness of measuring the "transition distance" between states and the induced improvement by auxiliary rewards, which not only promotes better learning efficiency but also increases the convergent stability.