Adaptive Neural Networks: An Occam’s Razor in Deep Learning to Obviate Hand-Crafted Architectures

Traditional computational approaches require hand-crafting analytical models, estimation functions, and/or algorithmic solution designs.

Neural networks have shown great promise as universal function approximators in solving complex tasks, where the network weights and biases are learned using optimization schemes.

This however has in practice led to shifting the hand-crafting process to the selection of network design, layer types, connectivity, activation functions, and other hyper-parameter choices.

Consequently, instead of computer scientists who can design algorithms, we are now training NN architects!Biological systems in the brain, nevertheless, do not learn only by weighting existing axon-dendrite signaling, but also via synaptic plasticity, where new axon-dendrite synapses form between different neurons or older ones disappear.

Happening at different rates throughout aging, such plasticity is indeed a large part of biological learning in the human brain.

Through an ongoing architectural adaptation, the brain achieves as simple as possible and as complex as necessary connectivity, for optimal use of resources and effective generalization. Despite evolutionary, pruning, and compression approaches, growing effective networks is still a challenge. This project will study theoretical, numerical, and implementation aspects of such artificial neuroplasticity.

Successful methods will contribute to and help shape the next revolution in deep learning.