Semantic Parsing of Multimodal Data

We address the problem of semantic parsing of multimodal data.

In other words, the translation of composite media items such as a video with audio tracks and subtitles, or a news article with text and images, into structured representations that capture central aspects of the combined media. The problem appears in many technological areas.

In robotics, it takes the form of language grounding, where the linguistic constituents of a natural language command are linked to real-world objects, attributes, and relations. In media asset management, it is the basis for advanced workflow automation.

It is also of inherent value in machine learning, because it allows us to transfer knowledge between different modalities. The outcome of the project is a theory of graph-based computation tailored for multimodal parsing.

This consists of formal graph languages to represent the data, together with transition-based computation models that operate on such representations.

The over-arching theoretical framework is weighted hyperedge replacement grammars, a computational model capable of expressing probability distributions over graphs, complemented by continuous-state semantic embeddings.

The project pushes the boundaries of how much and what kind of structure can be managed with efficiency within the framework of supervised machine learning.

We aim to advance the state of art both in unimodal (i.e., textual) semantic parsing, and in the broader field of multimodal semantic parsing.