Dealing with Non-Stationarity in Multi-Agent Deep Reinforcement Learning

Abstract

policies during the training procedure. This non-stationarity stems from breaking the Markov assumption that governs most Recent developments in deep reinforcement learning single-agent RL algorithms. Since the transitions and rewards are concerned with creating decision-making agents depend on actions of all agents, whose decision policies keep which can perform well in various complex domains. changing in the learning process, each agent can enter an end- A particular approach which has received increasing less cycle of adapting to other agents. attention is multi-agent reinforcement learning, in Additional problems in multi-agent systems include multi- which multiple agents learn concurrently to coordi- agent credit assignment, partial observability, and heterogene- nate their actions. In such multi-agent environments, ity. In the first problem, only a subset of agents contribute additional learning problems arise due to the contin- to a reward, and we need to identify and reward them while ually changing decision-making policies of agents. avoiding punishing agents that acted optimally. Partial ob- This paper surveys recent works that address the servability consists of agents having access only to their local non-stationarity problem in multi-agent deep rein- observations and not the actual state of the environment, which forcement learning. The surveyed methods range can significantly hinder training performance. Heterogeneity from modifications in the training procedure, such refers to the fact that agents may have different sensing and act- as centralized training, to learning representations ing capabilities, and that their learning algorithms may differ of the opponent’s policy, meta-learning, commu- (Albrecht and Ramamoorthy, 2012). nication, and decentralized learning. The survey Multi-agent RL is a widely studied topic with surveys rang- concludes with a list of open problems and possible ing from modelling other agents (Albrecht and S