ProSh: Probabilistic Shielding for Model-free Reinforcement Learning

Abstract

not assume access to a simulator or environment abstraction, and Safety is a major concern in reinforcement learning (RL): we aim at is compatible with continuous environments. developing RL systems that not only perform optimally, but are also We also show that optimizing over shielded policies in the aug- safe to deploy by providing formal guarantees about their safety. mented space is sufficient for constrained optimality in the base arXiv:2510.15720v2 [cs.LG] 21 Oct 2025 To this end, we introduce Probabilistic Shielding via Risk Aug- CMDP in the deterministic setting. We implement ProSh as an mentation (ProSh), a model-free algorithm for safe reinforcement off-policy deep RL algorithm and evaluate it on standard Safe RL learning under cost constraints. ProSh augments the Constrained benchmarks. Notably, ProSh turns out to be safe during training MDP state space with a risk budget and enforces safety by apply- in the experiments. To summarize, the contributions of the paper ing a shield to the agent’s policy distribution using a learned cost are as follows. critic. The shield ensures that all sampled actions remain safe in (1) We introduce ProSh, a shielding method for CMDP that expectation. We also show that optimality is preserved when the assumes no knowledge of the environment’s dynamics. environment is deterministic. Since ProSh is model-free, safety (2) We provide formal safety guaranties during training, depend- during training depends on the knowledge we have acquired about ing only on the accuracy of the learned cost critic. the environment. We provide a tight upper-bound on the cost in (3) We show that optimizing across shielded policies in the Risk- expectation, depending only on the backup-critic accuracy, that is Augmented MDP suffices for constrained optimality (in the always satisfied during training. Under mild, practically achievable deterministic case). assumptions, ProSh guarantees safety even at training time, as (4) We implem