Collaborative Research: HCC: Small: Optimizing the Human-Machine System for Citizen Science

This project investigates strategies that combine both human and machine attention for improving human-machine collaboration for the analysis of large, complex data sets. Open-source platforms for online citizen science, such as the Zooniverse, now provide infrastructure to incorporate machine learning alongside volunteer classifiers enabling human-in-the-loop techniques.

Citizen Science is an established method for carrying out distributed analysis of large quantities of data in which online volunteers help with tasks requiring human pattern recognition. Examples include identifying shapes of galaxies in the Galaxy Zoo project, determining animal species in images taken by remote cameras in the Snapshot Serengeti project, or helping transcribe handwritten texts from historical documents such as the Civil War Bluejackets project.

However, much larger datasets are looming on the horizon. Designing a human-machine system to accelerate labeling of known classes at the same time as solving the problem of detecting interesting anomalies (suggesting new phenomena) requires answering several crucial research questions about how humans and machines best complement one another. One promising direction is showing what the machine “thought” was anomalous within a given image to volunteers for further inspection.

Additional benefits of this project include engaging nearly 3 million members of the public who participate in citizen science through Zooniverse and giving them the opportunity to learn more about how machine learning really works, engaging young women in University of Minnesota computer science coding camps, and providing capstone projects for Data Science Masters program students to engage in real-world research while preparing them for careers in data science.

Taking advantage of the exceptionally large labeled datasets available through Zooniverse projects and the fact that the majority of Zooniverse projects are image-based, the research effort will investigate training and deploying Vision Transformers (ViTs). Specifically, the impact of combining human and machine attention and anomaly detectors on a real-world citizen science platform will be explored with the two objectives: (1) Classification Efficiency Studies to optimize the classification efficiency of known-known classes, including sparsely represented classes, across multiple domains and task types; and (2) Systematized Serendipity Studies to increase the efficiency of detecting diverse, scientifically interesting anomalies beyond the usual statistically inferred ones.

Both objectives will be carried out through experiments implemented on Correct-a-Machine infrastructure which will enable machine proposals as well as machine attention and anomaly maps to be displayed to volunteers. A Leveling-up Strategy for Volunteers infrastructure will also be deployed to enable a human-driven approach to anomaly detection. The research team will carry out experiments that include simply testing improved classification efficiency with humans correcting implicit machine attention (i.e., are models such as vision transformers just significantly better at object detection and therefore the efficiency gains when coupled in a human-machine system are worth the complexity of implementation and training) through to more complex experiments that explicitly combine machine and human attention (that is, does asking volunteers to correct an explicit machine attention map improve performance of the machine or further, can machine attention and anomaly maps aid in human identification of anomalies in the data?).

The test projects implemented through this work will be immediately scientifically useful, but importantly will also underpin the human-machine infrastructure required to enable multiple research disciplines to make best use of the ever-increasing amounts of data.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.