Pure-Past Action Masking

Abstract

by (default or sampled) safe ones; action projection, e.g., (Cheng et al. 2019), where unsafe actions are replaced by We present Pure-Past Action Masking (PPAM), a lightweight approach to action masking for safe reinforcement learning. projecting them to the safe action space; and action masking, In PPAM, actions are disallowed (“masked”) according to e.g., (Krasowski, Wang, and Althoff 2020), where the agent specifications expressed in Pure-Past Linear Temporal Logic is able to choose actions only from the safe action space. (PPLTL). PPAM can enforce non-Markovian constraints, i.e., However, most provably safe approaches to RL are Marko- constraints based on the history of the system, rather than vian, that is, the set of actions which are considered “safe” is just the current state of the (possibly hidden) MDP. The fea- based only on the current state. There has been some work in tures used in the safety constraint need not be the same as which safety constraints are expressed in temporal logic. For those used by the learning agent, allowing a clear separation example, shields (Alshiekh et al. 2018; ElSayed-Aly et al. of concerns between the safety constraints and reward spec- 2021) is an action masking approach in which safety con- ifications of the (learning) agent. We prove formally that an straints are expressed in Safety LTL. The use of temporal agent trained with PPAM can learn any optimal policy that satisfies the safety constraints, and that they are as expressive logic means safety constraints can be specified over execu- as shields, another approach to enforce non-Markovian con- tion traces rather than states, allowing a much richer set of straints in RL. Finally, we provide empirical results showing non-Markovian safety constraints to be used, e.g., “do not how PPAM can guarantee constraint satisfaction in practice. close the valve if it was opened within the last three sec- onds” (Alshiekh et al. 2018). However, generating a shield