Semantics of tactile and visual texture perception

People use lines and points on transit maps to navigate new environments, colors and patterns on weather maps to track perilous storms, and tones and vibrations in navigation systems to avoid obstacles. These forms of communication are more successful when the displays are designed to match people’s expectations (e.g., tones getting louder as an obstacle gets closer).

This project examines the idea that some tactile and visual textures convey specific emotional, cultural, or cognitive meaning (in other words, they have semantic content). Rapid developments in tactile and haptic interfaces are promising for rendering complex information on desktop and mobile displays and in new wearable technologies, opening possibilities for simulation and training in many applications (e.g., students at school, surgeons in training, etc.) and for increasing accessibility through touch for individuals with visual impairments.

Such devices could also be used to explore data dynamically in real-time. However, for this technology to be successful, it is critical to deepen the understanding of how to design meaningful displays to communicate through touch.

This project aims to understand (1) semantics of visual texture for visual communication and (2) semantics of tactile texture for haptic communication. The overall goal is to make communication through vision and touch more effective and efficient. The approach is motivated by a theory of color semantics, called the color inference framework, in which people continually form and update associations between colors and concepts through experiences in the world, representing these associations in a network that connects all colors to all concepts.

Inference operations are performed on this network to produce different kinds of judgments, including those that interpret color meaning in data visualizations. Critically, the inferred meaning of a color in the context of a data visualization can contradict the conceptual associations of that color in isolation. Understanding these contextual effects is key to designing visualizations to match people’s expectations.

This project will test the extensibility and limitations of this framework as a general account of perceptual semantics by studying a wide range of textures humans experience through both vision and touch. Participants will rate the association between each texture and a wide range of concepts, including concrete concepts (e.g. banana, raspberry) and abstract concepts (justice, peace).

Later experiments will assess factors that influence how people perform assignment inference to interpret the meaning of visual and tactile textures. The aim is to identify and quantify factors to prioritize when optimizing visual and tactile design elements to enhance communication.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.