Constrained Policy Optimization

Abstract

In reinforcement learning (RL), agents learn to act by trial and error, gradually improving their performance at the For many applications of reinforcement learn- task as learning progresses. Recent work in deep RL as- ing it can be more convenient to specify both sumes that agents are free to explore any behavior during a reward function and constraints, rather than learning, so long as it leads to performance improvement. trying to design behavior through the reward In many realistic domains, however, it may be unacceptable function. For example, systems that physically to give an agent complete freedom. Consider, for example, interact with or around humans should satisfy an industrial robot arm learning to assemble a new product safety constraints. Recent advances in policy in a factory. Some behaviors could cause it to damage it- search algorithms (Mnih et al., 2016; Schul- self or the plant around it—or worse, take actions that are man et al., 2015; Lillicrap et al., 2016; Levine harmful to people working nearby. In domains like this, et al., 2016) have enabled new capabilities in safe exploration for RL agents is important (Moldovan & high-dimensional control, but do not consider Abbeel, 2012; Amodei et al., 2016). A natural way to in- the constrained setting. We propose Constrained corporate safety is via constraints. Policy Optimization (CPO), the first general- purpose policy search algorithm for constrained A standard and well-studied formulation for reinforcement reinforcement learning with guarantees for near- learning with constraints is the constrained Markov Deci- constraint satisfaction at each iteration. Our sion Process (CMDP) framework (Altman, 1999), where method allows us to train neural network poli- agents must satisfy constraints on expectations of auxil- cies for high-dimensional control while making liary costs. Although optimal policies for finite CMDPs guarantees about policy behavior all throughout with known models c