CISE-MSI: DP: SaTC: CyIndiBee - CyberInfrastructure for video analysis of individual bee behavior

The goal of the CyIndiBee project is to build an innovative cyberinfrastructure showcasing the use of modern Artificial Intelligence (AI) approaches in the study of pollinator behavior in the Caribbean. Pollinators are a key part of our food production system and play a critical role in the life cycle of plants and their ecosystems. Climate change and other anthropogenic activities are endangering pollinators and their habitats, causing behavioral changes, and leading to serious consequences for humans, especially in areas with scarce ecological diversity.

A keen understanding of the complex effects that environmental changes, contaminants, and other factors produced on pollinator behavior and their biological mechanisms is needed urgently. This project will develop new computer vision, software and data analysis tools to expand our capacity to measure the individual behavior of a large number of pollinators in order to gather more detailed data over longer periods of time.

The contributions will lead to mature and robust tools for automatic video monitoring of insects, which can be readily used and deployed in the field. The project will consolidate the partnership between the Computer Science department and the biology department at UPR Rio Piedras (UPR-RP), a Hispanic Minority Serving Institution (MSI). It will integrate undergraduate and graduate student training into research-intensive activities, thus creating a critical mass of closely interacting professors and students conducting state-of-the-art transdisciplinary research.

The project will increase UPR-RP's capacity in the Computer Science field to promote innovation and growth in AI related research. This project is a first step towards transforming UPRRP into the reference research center in the Caribbean on the topic of Artificial Intelligence applied to transdisciplinary research in climate change.

This project will develop an integrated cyberinfrastructure for pollinator video monitoring that combines (i) new deep learning models for detection and characterization of bee behavior, phenotype, and identity, (ii) a computational platform to collect monitoring data and perform behavior analysis with graphical interfaces usable by both the biology end-user and AI researchers, (iii) tools for the analysis of long-term individual behavior. The deep learning models leverage the Vision Transformer architecture to provide powerful pre-training using Masked Image Modeling on unannotated video data, thus reducing the need for large annotation efforts when deploying new collection setups.

They will also provide flexibility with an extendable system of trainable query tokens to extract various types of information (pose, tag, marking, morphology, presence of pollen…) from the same latent representations. The computational platform will integrate a web application for video visualization and behavior annotation, with interactive dashboards for the analysis of individual behavior and phenotype, from data collected from long-term video monitoring.

These tools will be applied to individual analyses to recognize patterns of shift-work and seasonal activity in both marked and unmarked bees. The large-scale analysis enabled by the new cyber-infrastructure will be demonstrated by the creation of two new curated datasets: a large-scale honeybee re-identification dataset, and a foraging behavior dataset combining multiple colonies and foraging trips at the individual and colony level including the presence of pollen and morphology of the bees.

This project is jointly funded by the CISE MSI Research Expansion Program and the Established Program to Stimulate Competitive Research (EPSCoR).

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.