RI: Small: A Hierarchical Framework for Naturalistic Multi-animal Behavior Understanding

Understanding social animal behaviors has become a major focus in computational neuroscience due to its ability to reveal complex brain-behavior connections. Despite advances in studying animal behavior, challenges remain in comprehending natural and social behaviors over long periods. Imagine the behavior of two mice moving around in an open field over several days.

Many behaviors and interactions can happen over shorter and longer time scales. In just a few seconds or minutes, a mouse might groom itself or dart around, or two mice might sniff each other. Over hours or days, they might look for water or food together, or one might grab food from the other.

Each behavior reflects different cognitive processes and has potential to connect with neural functions. The goal of behavior analysis is to extract and parse such complicated multi-animal behavior from days of video recordings, accounting for multiple decision-making processes and actions involving multiple individuals. This project seeks to provide accurate and reliable machine learning methods for identifying short-term actions, understanding long-term decision-making, and analyzing multi-animal interactions.

These tools will offer new insights into natural and social animal behaviors and enhance the understanding of the relationships among brain regions, neural functions, and behaviors, especially in naturalistic and real-world contexts.

This project introduces advanced computational methods designed to analyze complex multi-animal behaviors. Aim 1 focuses on developing new self-supervised learning techniques for behavior segmentation, breaking down long behavior sequences into short, distinct actions like grooming or sniffing. Aim 2 addresses animal behavior as a series of decisions influenced by goals and rewards.

A novel inverse reinforcement learning framework will be developed to explore how animals make decisions and how these decisions change over time. Aim 3 introduces a new spatiotemporal graph to capture interactions between animals and to study short-term actions and long-term decision-making in social contexts. These aims are interconnected to understand how behaviors are generated across different time scales and spatial contexts.

The ultimate goal is to create a practical set of accurate, interpretable, and reliable machine learning models for analyzing multi-animal behaviors. Insights gained will reveal neuro-behavioral correlations across brain regions and functions, potentially impacting fields such as AI, robotics, cognitive science, and psychology.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.