Uncertainty-Aware Safety Propagation Critics for Safe Reinforcement Learning

Kutay Demiray, Efe Eren Ceyani, Özgür S. Öğüz

LiRA Lab, Department of Computer Engineering, Department of Electrical and Electronics Engineering

Bilkent University, Ankara, Türkiye

Contact: kutay.demiray@bilkent.edu.tr

Abstract

Safe reinforcement learning (RL) aims to optimize long-term performance while satisfying safety constraints, a requirement that is critical in many applications but difficult to guarantee when cost estimates are inaccurate or data is limited. In model-free actor-critic methods, cost critics are often unreliable in poorly explored regions, leading to constraint violations during both training and deployment. In this work, we propose a novel uncertainty-aware approach in safe RL called USPC, which constructs conservative cost surrogates using epistemic uncertainty. Our method trains an ensemble of cost critics to estimate uncertainty and uses these estimates to build an upper confidence bound on predicted costs. We then introduce a safe set network that approximates a pessimistic surrogate of the cost action-value function inspired by safe Bayesian optimization, enabling scalable safety propagation in continuous state-action spaces. Replacing standard cost critics with this surrogate in existing off-policy safe RL algorithms yields policies that are less likely to violate cost constraints. We show empirically across multiple Safety Gymnasium benchmark tasks that our approach reduces both the frequency and magnitude of constraint violations in most tasks while maintaining competitive reward performance compared to the baselines. We further provide theoretical justification for our approach, including a conservativeness guarantee for the safe set network, and an approximation result for the anchor-based scheme.

Figure 1: An illustration of one step of USPC training on a simple one-dimensional example. (a) Initial state of the critic ensemble and safe set network. Shaded blue region shows the confidence interval for the ensemble. (b) A query point (for safe set network update) is selected, and local and global anchors are sampled to create the anchor set. Anchor points are then filtered according to the current safe set network and the cost threshold, and unsafe points are discarded. (c) Safe anchor points are evaluated on the UCB, then shifted upwards by the Lipschitz distance term to obtain candidate targets. The minimum of the candidates (denoted by star) is selected as the target for the safe set network update. (d) Safe set network is updated at the query point via regression towards the target selected in the previous step.

Figure 2: A high level overview of USPC.

USPC safe set network training algorithm pseudocode.

Figure 3: Training reward and cost curves for baselines and our method, each column representing a task. Solid lines represent the mean and shaded regions represent the 95% confidence interval, across 5 random seeds. Cost threshold is 25 for all tasks.

Fİgure 4: Evaluation reward and cost curves for baselines and our method, each column representing a task. Solid lines represent the mean and shaded regions represent the 95% confidence interval, across 5 random seeds. Cost threshold is 25 for all tasks.

Table 1: Train reward and cost (mean ± std over last 50 episodes). Unsafe task-algorithm pairs are colored purple, and safe task-algorithm pairs with the best reward for each task are highlighted in blue. Cost threshold is 25 for all tasks.

Figure 5: Box plots representing violation rates over episodes during training and evaluation episodes, for experiment runs across 5 random seeds. Shaded boxes represent an interquartile range whereas the lower and upper extremes represent the minimum and maximum, respectively.

Figure 6: Pareto frontier plots of our experiment runs. Reward and cost values are averaged for the last 10 training episodes for each seed. To reduce visual clutter, best (in terms of reward-cost tradeoff) 4 seeds are taken for each method.

Figure 7: Ablation studies for USPC-CVPO on PointGoal (training episodes) for Lipschitz constant and confidence interval width. (a) Effect of the Lipschitz constant L. (b) Effect of the uncertainty scaling parameter β.

Figure 8: Ablation studies for USPC-CVPO on PointGoal (training episodes) for anchor set parameters. (a) Effect of the anchor size. Each run has equal amount of local and global anchors. (b) Effect of anchor set distribution. Each variant contains the same amount of total

Figure 9: Ablation study for USPC-CVPO on PointGoal (training episodes) for the ensemble size.

Figure 10: Ablation study for USPC-CVPO on PointGoal (training episodes) for the ensemble size.

Page updated

Google Sites

Report abuse