RI: Small: Adaptive synaptic dynamics neural networks: a novel hypothesis for temporal computations in artificial neural networks

One of the most dramatic differences between artificial neural network (ANN) models, as currently used in machine learning, and biological neural networks pertains to how they process information about time. Specifically, the difference is in how the brain tells time and how it uses its inherent neural dynamics to process time-related information. Because of the importance of time and prediction in natural behaviors, the brain is an inherently time-based computational device.

For example, during speech recognition people continuously and unconsciously extract information about the durations of syllables and the intervals between words in order to understand the meaning and emotional content of speech. Current experimental and theoretical results suggest that the processing of time-varying information, such as speech, relies on the inherent short-term dynamics of synapses on the scale of milliseconds to seconds.

This phenomenon is referred to as short-term synaptic plasticity (STP). The standard view is that the temporal characteristics of STP are essentially a hard-wired property of specific classes of synapses. Inspired by recent neuroscientific findings, this project proposes that STP itself undergoes long-term plasticity in a manner that adapts to and optimizes the processing of the timing of information and experience.

This project will develop a novel class of feedforward networks referred to as adaptive synaptic dynamics neural networks, which can process temporal information in the absence of recurrency, delay lines, or the spatialization of time. STP will be incorporated into feedforward ANNs and, in addition to standard training of ANN weights, the time constants that determine the temporal profile of STP will also be trained.

Performance of networks with adaptive-STP will be contrasted to standard feedforward and recurrent ANNs on tasks including Morse Code and speech recognition. The primary hypothesis to be tested in this project extends a fundamental tenet of computational neuroscience—that information is stored in the population of weights of a neural network—into the temporal domain by suggesting that synapses may also learn to govern their short-term dynamics in order to optimize the processing of temporal information.

Overall, this project will lead to novel biological principles being applied towards machine learning, and further advance the ability to emulate the brain’s computational strategies.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.