Partially Observable Multi-agent Inverse Reinforcement Learning

A major challenge of reinforcement learning is approximating the reward structure, which inverse reinforcement learning addresses by learning the reward structure from expert demonstrations.

This method has shown a lot of success, however, few literature exists that focuses on inverse reinforcement learning methods for multi-agent systems, although the use of multi-agent systems is on the rise.

Current inverse reinforcement learning methods for multi-agent systems model the systems in a fully observable environment, which can be impractical to use in a real world environment.

In a partially observable environment, we assume that data that is observed can be noisy or even incorrect, which better models real systems.

To address this knowledge gap, the main research question that will be addressed by this research is what algorithms can be used for environments that can be modelled as multi-agent partially observable Markov decision processes? To extend this question, we will evaluate the performance of these algorithms in simulated and practical environments.

To solve these questions, we will first design algorithms using the theoretical understanding of the problem. Then, we will apply the algorithms to simulated games, and evaluate the performance against other algorithms. These games will be standard benchmark games for partially observable multi-agent reinforcement learning.

The same will be done with a practical example using real data. The goal of using real world data is to see how accurately we can model human behaviour in a multi-agent setting.

Finally, we will explore how the use of multi-agent inverse reinforcement learning to better improve traditional multi-agent reinforcement learning methods by having a more systematic way of modelling reward structures for these problems. These new methods will similarly be evaluated on simulated and practical environments.

By studying multi-agent reinforcement learning in a partially observable environment, it allows us to much better apply inverse reinforcement learning to real world systems. One of the main potential benefits lies in the development of human compatible AI systems. Whenever a human interacts with an AI, we can classify that as a multi-agent system.

In order to appropriately design the AI system for the benefit of the human, we need to model the human behaviours and needs accurately, which can be done with inverse reinforcement learning. Another important application is the estimation of human needs.

In economics, psychology, and engineering, it is important to understand what humans need in order to design systems that address those needs.

This research will allow us to accurately model these human behaviours in realistic scenarios in the presence of other intelligent beings.

Additionally, by using these estimations and models, we can then also design better engineering systems by simulating human behaviour and test humans will interact with the new systems. This research will align with the EPSRC engineering and artificial intelligence research areas.

One of the main factors for integrating artificial intelligence in engineered systems, such as future smart cities, is the ability for humans to interact intelligently with such systems.

This research will directly address that by having a more accurate way of modelling human behaviour and being able to train artificially intelligent agents to behave like humans. In this way, we can design systems that will address the human needs in future engineering systems.