Generalization in data-driven models of primary visual cortex

Konstantin-Klemens Lurz · Mohammad Bashiri · Konstantin Willeke · Akshay Jagadish · Eric Wang · Edgar Walker · Santiago Cadena · Taliah Muhammad · Erick M Cobos · Andreas Tolias · Alexander S Ecker · Fabian Sinz


Keywords: [ visual perception ] [ Computational Biology ] [ Network Architecture ] [ cognitive science ] [ neuroscience ] [ multitask learning ] [ representation learning ] [ transfer learning ]

[ Abstract ]
[ Slides [ Paper ]
Tue 4 May 1 a.m. PDT — 3 a.m. PDT
Spotlight presentation: Oral Session 1
Mon 3 May 3 a.m. PDT — 6:05 a.m. PDT


Deep neural networks (DNN) have set new standards at predicting responses of neural populations to visual input. Most such DNNs consist of a convolutional network (core) shared across all neurons which learns a representation of neural computation in visual cortex and a neuron-specific readout that linearly combines the relevant features in this representation. The goal of this paper is to test whether such a representation is indeed generally characteristic for visual cortex, i.e. generalizes between animals of a species, and what factors contribute to obtaining such a generalizing core. To push all non-linear computations into the core where the generalizing cortical features should be learned, we devise a novel readout that reduces the number of parameters per neuron in the readout by up to two orders of magnitude compared to the previous state-of-the-art. It does so by taking advantage of retinotopy and learns a Gaussian distribution over the neuron’s receptive field position. With this new readout we train our network on neural responses from mouse primary visual cortex (V1) and obtain a gain in performance of 7% compared to the previous state-of-the-art network. We then investigate whether the convolutional core indeed captures general cortical features by using the core in transfer learning to a different animal. When transferring a core trained on thousands of neurons from various animals and scans we exceed the performance of training directly on that animal by 12%, and outperform a commonly used VGG16 core pre-trained on imagenet by 33%. In addition, transfer learning with our data-driven core is more data-efficient than direct training, achieving the same performance with only 40% of the data. Our model with its novel readout thus sets a new state-of-the-art for neural response prediction in mouse visual cortex from natural images, generalizes between animals, and captures better characteristic cortical features than current task-driven pre-training approaches such as VGG16.

Chat is not available.