Applications of Reinforcement Learning in Long Term Planning for Robotics

Research Context Whilst Reinforcement Learning (RL) has been shown to be effective when training occurs in environments identical or very similar to those in which agents are evaluated, the transfer of agent performance to unseen environments tends to be poor. Methods such as Unsupervised Environment Design (UED) have attempted to improve the generalisation ability of RL agents but

have only shown somewhat limited success. Similarly, whilst RL has been very successful at short timescale robot control tasks, longer term control tasks in environments with large state and action spaces have presented a much greater challenge, especially when trying to apply model-free approaches. Training agents using model-free policy gradient methods in these

contexts tend to require large amounts of computation, and fail to take advantage of prior environment knowledge that model-based approaches such as Monte Carlo Tree Search (MCTS) are able to use. However, online planning methods such as MCTS tend to require much greater computation at runtime than pre-trained policies.

Aims and Objectives The aim of this research is to examine whether it is possible to use RL methods to train policies that are capable of generalising to a wider scope of robotics environments and problem solving tasks, whilst also using long term planning methods to improve long term decision-making by

selecting optimal pre-trained policies based on a given environment state. Novelty of the research methodology Similar research has been done in relation to using policy selection rather than action selection in long term planning, such as with the use of 'options' in MCTS. However, these options tend to

be task-specific, sometimes hand-designed policies, and so fail to adapt to more diverse sets of environments. This research aims to train more general policies are more robust to different environments. Alignment to EPSRC's strategies and research areas This research aligns with the EPSRCS's Artificial Intelligence and Robotics research theme. This

research aims to use machine learning techniques to improve task planning in robotics environments.