Efficiency in robot learning is closely linked to hyperparameter selection. Most approaches to hyperparameter optimization require either sequential or parallel repetition, strongly increasing environment interactions and computational cost. We propose a training method that only relies on a single experiment. This is achieved by training mixture policies with diverse components, enabling us to reduce the impact of design choices - providing low-level policies with different sets of hyperparameters and distribution types. Our approach yields robust and data-efficient learning by letting the agent select controller structure conditioned on the task, while exploiting synergies between individual controller designs .

Continuous control enables representation of intricate transitions in state-action space to yield highly-optimized behaviors through local exploration or to generate smooth references for a low-level tracking controller. Asymmetric policies can guard against undesirable state-action half-spaces while exploring the remaining interactions. Here, we consider Gaussian and Kumaraswamy policy heads. The probability density of the latter approximates the Beta distribution, while being easier to reparameterize.

Discrete control can leverage reduced resolution for coarse exploration and easily encodes bang-bang responses to switching dynamics . We consider Categorical and Discrete Gaussian policy heads. The latter is implemented as a Categorical with probabilities parameterized by a Gaussian distributions to embed relational structure. In mixed continuous-discrete distributions, we introduce action tolerances (bin width) to map out-of-distribution samples into the support. This improves sharing of gradient information by enabling discrete distributions to train samples generated by continuous distributions.