Beyond pixels and words: computational generation and understanding of spatial language in action

The project falls within language technology and connects linguistics and psychology with machine learning, computer vision and robotics.

We will create novel human-centred interactive machine learning methods that enable situated agents to learn and use spatial descriptions referring to their environment by engaging in situated dialogue with humans.

This is important when applying AI technology based on deep neural networks in our daily lives as it allows it to adapt to and learn continuously from limited interactions with humans who are not expert programmers and their ever-changing environment.

We will solve the problem of the agent´s limited exposure to learning observations in its lifetime by (i) bootstrapping the training of deep learning models of spatial language semantics using offline pre-training (ii) use methods from transfer learning and active learning to enable agents to incrementally update their initial models through dialogue games where an agent can query their conversational partner for feedback and new examples of spatial descriptions.

The length of the project will be 3-years during which these tasks will be explored iteratively in stages, increasing the complexity of situations and learning.

The project involves Dobnik (computational models of spatial language and interaction), a postdoctoral researcher, a research programmer (system implementation), and Kelleher (spatial language and machine learning) as an international collaborator.