Collaborative Research: LightningBug, An Integrated Pipeline to Overcome The Biodiversity Digitization Gap

Insects are the largest and most diverse class of animals on our planet where they play essential roles in ecosystems and the services those provide to society. Entomologists have long been engaged in collecting, preserving and depositing nearly one billion insect specimens at natural history museums around the globe. These collections form the basis for much of our knowledge about insects and provide critical information about the past from which scientists can assess current and future global change impacts.

To fully realize the value of these collections, data from insect specimens must first be digitized. However, their small size, delicate structures, and traditional storage and labeling methods creates enormous challenges for large-scale digitization. Consequently, at present, only 5% of specimens have transcribed labels and less than 1% of specimens are imaged.

The LightningBug project will break through this digitization bottleneck by establishing a semi-automated workflow involving advancements in robotic multi-view imaging, information extraction and 3D reconstruction. Results from this work will provide researchers with the unprecedented capability to capture specimen metadata representing time, place and taxonomic identity along with accurate three-dimensional surface morphology representing color and shape.

These investigators expect LightningBug and related technologies will promote ecomorphological studies at a scale that has not been possible to date.

The LightningBug project seeks to create an end-to-end pipeline for high-throughput data acquisition from pinned insects in entomological collections. To accomplish this goal, it will: (1) further develop an existing hardware and software platform to capture multi-view imagery of both labels and specimens; (2) build robust algorithms to automatically process fragmentary views of multiple labels into separate integrated “virtual labels;" (3) connect virtual labels to structured text extraction services; and (4) apply photogrammetric analysis to assemble the 3D shape and structure of specimens.

Guided by real-world science use cases that highlight the use of specimen-based multi-view imaging in studies of global change and functional morphology, the entomological collections of the Yale Peabody Museum and the Harvard Museum of Comparative Zoology will be used in rigorous test-case implementations. Results will include robust sets of annotated multi-view images, 3D models of specimens (point clouds, textured meshes), 2D reconstructed “virtual labels” and digitized specimen metadata generated from those labels.

These digital specimens will present new challenges for data preservation and access, but they will also catalyze new solutions for large-scale storage and delivery of research imagery. This challenge will be addressed via a partnership with MorphoSource to develop a linked institutional repository model for data access to large digital assets such as those produced by multi-view imaging.

Ultimately, the ability to capture multi-view image suites and generate virtual specimens at scale will permit new avenues for remote access to research resources, and enable the application of computer vision and machine learning to trait identification and evolution, species recognition and new species discovery. Label data from pinned insects will give researchers access to critical temporal and geospatial information necessary for relating changes in biodiversity to other biotic and environmental variables.

It will also provide collections staff with a complete digital portrait of their holdings, which can enable historical research, streamline collections use and tracking, and improve data quality control. Results from this project will also have applications beyond the natural history collections and research communities, such as computer graphics, product imaging, motion pictures, 3D animation, virtual and augmented realities, and education. More information and results from this project can be found at http://lightningbug.tech

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.