Safe Multi-Agent Reinforcement Learning via Shielding

Abstract

A recent work [1] developed a shielding framework for single- Multi-agent reinforcement learning (MARL) has been increasingly agent reinforcement learning (RL), which synthesizes a shield to used in a wide range of safety-critical applications, which require enforce the correctness of safety specifications in linear temporal guaranteed safety (e.g., no unsafe states are ever visited) during logic (LTL) [17]. The shield guarantees safety during learning by the learning process. Unfortunately, current MARL methods do monitoring the RL agent’s actions and preventing the exploration not have safety guarantees. Therefore, we present two shielding of any unsafe action that violates the LTL safety specification. In approaches for safe MARL. In centralized shielding, we synthesize a this paper, we adapt the shielding framework to the multi-agent single shield to monitor all agents’ joint actions and correct any un- setting. Guaranteeing safety for multiple agents with potentially safe action if necessary. In factored shielding, we synthesize multiple competing goals is more challenging than the single-agent setting, shields based on a factorization of the joint state space observed because safety is an emergent property that concerns the coupling by all agents; the set of shields monitors agents concurrently and of all agents. In addition, the combinatorial nature of MARL (i.e., the each shield is only responsible for a subset of agents at each step. joint state space and joint action space increase exponentially with Experimental results show that both approaches can guarantee the the number of agents) poses scalability issues to the computation safety of agents during learning without compromising the quality of shields. of learned policies; moreover, factored shielding is more scalable in We present in this paper the first work to provide safety guaran- the number of agents than centralized shielding. tees (expressed as LTL specifications) for MARL. Our co