III: Small: From Spatial Language to Spatial Data - a simulation-based approach

User experience related to spatial information is currently directly linked to the representation of the data; that is, geographic co-ordinates pinpoint locations on maps and routing algorithms determine the best route based on distance and time. In contrast, human interaction with the world is based on experience, learning and reasoning on qualitative factors, including spatial concepts, such as near/far, behind, next to, inside.

Considering how spatial information is conveyed in natural language, there is no unique mapping between the spatial expressiveness and quantifiable spatial concepts. For the most part this is attributed to the highly contextualized nature of human language; that is, what human language is interpreted in part by who and where it was said and what other words surrounded the comment.

The challenge in this project is on devising means to better understand people's perception of space by deciphering such spatial language terms. This will lead to novel text and audio-based interfaces for the consumption of geospatial data such as when asking for or giving directions in a way that is intuitive to people or for systems that more effectively assist the visually impaired.

The technical aims of the project are divided into two thrusts. The first thrust develops a simulation to crowdsource geospatial language expression data by having users interact in a virtual environment. The spatial language expressions and interactions are captured using quantitative models.

The second thrust then uses these user-generated descriptions to evaluate several modeling approaches that include (i) studying specific urban settings to identify contextual factors and (ii) exploring neural approaches to modeling the problem of grounding language to this spatial context. The resulting models can then be used to automatically translate language to geospatial information and, in the reverse direction, to train dialogue agents that can generate enriched, contextualized route and scene descriptions with natural, useful geospatial language expressions.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.