Robotic and biological systems evolve continuously and require coordinated control across diverse timescales:
Some commands can remain nearly unchanged for extended periods.
Others require rapid adjustments to track movement or maintain balance.
These different temporal requirements must be coordinated across many actuators.
Standard RL selects actions at fixed intervals, leaving this temporal structure largely implicit.
In this paper, we aim to develop scalable and sample-efficient reinforcement learning for high-dimensional continuous control by learning how actions evolve over time.
Figure W1. Command tracking with different temporal demands.
Dashed curves denote references and shaded bands indicate tolerances. Discrete-time RL tracks references through frequent updates without explicitly modeling temporal structure. Hard persistence holds all components over shared intervals, limiting tracking of faster references. Soft persistence allows temporal coherent action evolution at different rates while maintaining tracking.
Continuous-time RL methods can introduce action persistence by learning both control commands and their execution durations. Our analysis reveals a trade-off:
Longer persistence can reduce the amplification of value-estimation errors.
Longer commitment delays corrections based on new observations.
Shared execution durations can become restrictive when different actuators require different rates of adjustment.
This motivates soft persistence: allowing bounded action revision can extend the feasible control horizon while accommodating different temporal requirements.
We propose Soft Persistence Regularization (SoPer), which learns action dynamics by maximizing task return under an action-drift budget.
Temporal coherence: The budget regulates the average squared rate of action change.
Flexible coordination: Action revision can be allocated across states and actuators according to task demands.
Feedback-based adaptation: Commands remain responsive to new observations, with different components evolving at different rates.
Practical integration: Discretizing the continuous-time objective yields a regularizer compatible with standard actor-critic learning. (See our implementation )
Task-based high-dimensional continuous control
Figure W2. Evaluation returns on HumanoidBench H1-hand tasks.
Lines show the mean and shaded regions show one standard deviation across 5 seeds.
Imitation-based high-dimensional continuous control
Figure W3. Evaluation performance and representative motion sequences.
(a) Forward Lunge, (b) Squat, and (c) Single Foot Balance in LocoMuJoCo, and (d) MS-Human-700 Walk in MS-Gym. Lines show the mean and shaded regions show one standard deviation across 5 seeds. Frames above each plot show motions controlled by SoPer.
Forward Lunge
Squat
Single-Foot Balance
MS-Human-700 Walk
Figure W4. Imitation performances of SoPer.
Timescale analysis
Figure W5. Algorithm analysis.
(a) HumanoidBench Run performances, with mean and one standard deviation across 5 seeds. (b) Action timescales distribution in Run task. (c) Action timescales distribution across different task. Densities retain the common log-space KDE bandwidth of the source analysis.
ε = 0.1
ε = 0.3
ε = 0.5
Figure W6. Dynamical behavior of SoPer under different soft perststence intensity.
Real-world deployment
Figure W7. Real world humanoid control.
SoPer successfully controls an adult-size humanoid robot to perform full-body dance motions.