RAPID/Collaborative Research: Datasets for Uncrewed Aerial System (UAS) and Remote Responder Performance from Hurricane Ian

This Grants for Rapid Response Research (RAPID) project will curate, supplement, and analyze data collected over a period of intensive uncrewed aerial system (UAS) operations, carried out as part of the State of Florida’s response to Hurricane Ian. From September 27, 2022, just before Hurricane Ian made landfall, and continually for the next nine days, teams from Florida State University and Texas A&M University helped coordinate 24 UAS pilots flying 16 different models of fixed-wing and rotorcraft UAS over Charlotte, Lee, and Hardee counties.

These missions obtained aerial imagery to survey wind and flood damage, direct ground response, support strategic planning and resource allocation, monitor threats to public safety, and provide documentation for subsequent emergency relief funding. Under this award, the research team will curate 55,000 images and videos collected during the disaster, comprising over 750 gigabytes of data, and supporting material such as flight schedules and log files.

The curated data, and derived products such as aerial maps and edited video, will be made publicly available for open-source use. The project will analyze the mission logs and data products to assess pilot performance over time, and will document variables potentially influencing pilot performance, including pilot skill, prior training and experience, operations tempo, and fatiguing conditions, supplemented by individual and collective interviews with the UAS pilots.

The image dataset and derived products will help the computer vision/machine learning (CV/ML) community design better algorithms for identifying threats to public safety, damaged structures, and people in distress. The pilot performance dataset will be made available to the research community, to characterize human-robot performance, formulate best practices, and to understand deviation in behaviors and sources of mission error.

The resulting insights into proper matching of vehicles, pilots, missions, and operational parameters will increase the ability of UAS platforms and pilots to save lives and accelerate economic recovery after a disaster. The datasets can help the domestic UAS industry improve products for response to a broad class of natural disasters, including wildfires and flooding, and for use in extreme environments, such as in oil and gas exploration and extraction and for in nuclear reactors and nuclear waste sites.

The project will support the creation of better workflow procedures to reduce human error, increasing trust in the technology by UAS operators and other first responders, and facilitating adoption of UAS for emergency response. The project will broaden participation in science, with three women out of the four co-PIs, and will engage STEM students to help annotate the UAS imagery.

This project will curate vehicle and pilot data from the deployment of uncrewed aerial systems (UAS) during Hurricane Ian by Florida State University and Texas A&M University for the robotics, computer vision/machine learning (CV/ML), human-robot interaction, and geospatial land-use communities. The project has the following three objectives: 1) Curate the data (imagery, log files, flight schedules, etc.) and data products (images, video, orthomosaic maps, digital surface maps) collected during the disaster and make available for open-source use; 2) Interview the UAS pilots individually and collectively in order to capture human-robot performance, best practices, deviation in behaviors, and sources of error; and 3) Analyze the mission logs and data products for performance (quality or completeness) and document the quality over time by pilots, prior training, and frequency of flying the missions in normative conditions, the operations tempo, and fatiguing conditions.

From a robotics perspective, it will contribute to the emerging model of how multiple agents may be used during disasters, and the consequences for design, performance specifications, the role of artificial intelligence, and wireless communications. Such a model can greatly increase the competitiveness of the domestic drone industry, as well as motivate novel directions in swarm research.

Research stemming from this project will generate guidelines for data collection in future disasters, setting the stage for advances in engineering and computing for disasters. It will increase the availability of training data for CV/ML and serve as a testbed for transfer of learned features from other disasters; both of which could lead to fundamental advances in machine learning.

From a human-factors perspective, it will generate a new methodology for creating human-robot datasets that combine on-site direct data (with no experimenters in the field) with post-event data. This methodology would overcome current barriers in conducting empirical investigations into scientific questions on extreme work environments because of the prohibition on embedded experimenters.

This methodology is expected to transfer to other extreme work environments, such as nuclear, space, oil and gas industry, and the military. The human-robot data itself could lead to major findings in human error and workforce training. From a geospatial perspective, the data can help establish the impact of rising sea levels, the built environment, and prior storm surge and flooding mitigations.

Overall, the project will benefit society by increasing the ability to save lives and accelerate economic recovery after a disaster with UAS and is expected to create findings and methods that will generalize to new technologies for extreme environments.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.