Collaborative Research: FMitF: Track I: Usable Synthesis-based End-User Programming with Rich Interaction Modalities

Programming, defined as the process of transforming a mental plan of actions into a representation that can be executed by a computer, has been recognized as a powerful, desirable, and even necessary skill for an increasing number of people beyond professional programmers. In parallel, it has been a longstanding challenge in computer science to provide easier, more supportive interfaces for programming, which may be especially useful for computer users who want to leverage the power of programming but have little or no formal training.

This project aims to develop software development tools and methods that help people to interactively create programs for data science and data visualization, using suggestions and feedback provided by the tools. The tools will allow users to express their goals through gestures, voice, and other familiar ways people interact with the world, providing rapid and ongoing feedback to help people zero in on the programs they are looking to create.

To achieve this goal, this project is combining techniques from human-computer interaction, programming languages, and formal methods. On the one hand, this project is contributing a program-synthesis framework that supports rich interaction modalities, including accepting multi-modal specifications and providing feedback during the synthesis process.

On the other hand, this project is also contributing novel user interfaces that allow end-users to naturally express their intent in different modalities, to better understand the characteristics of the program(s) generated on their behalf, and to more easily recognize and update their expression of intent. By synergistically combining strengths of these two areas, the project aims to lay the formal methods foundation for usable human-centered synthesis-based interactive systems.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.