A tale of two tails: Preferred and anti-preferred natural stimuli in visual cortex

A fundamental quest in neuroscience is to find the preferred stimulus of a sensory neuron. This search lays the foundation for understanding how selectivity emerges in the primate visual stream---from simple edge-detecting neurons to highly-selective "grandmother" neurons---as well as for the architectures and activation functions of deep neural networks. The prevailing notion is that a visual neuron primarily responds to a single preferred visual feature like an oriented edge or object identity, resulting in a "one-tailed" distribution of responses to natural images. However, surprisingly, we instead find "two-tailed" response distributions of neurons in higher-order visual cortex (macaque V4), suggesting that V4 neurons have both preferred and anti-preferred stimuli. We ran further experiments to validate the existence of anti-preferred stimuli in V4. We find that these anti-preferred stimuli help to shape a neuron's tuning: Only a small number of preferred and anti-preferred images are needed to estimate the rest of a neuron's tuning. Moreover, in a psychophysics task, humans rely on anti-preferred images to interpret and predict V4 stimulus tuning; this was not the case for hidden units from a deep neural network. We find that the preferred and anti-preferred visual features, while clearly distinguishable for individual neurons, are not easily distinguishable across neurons. Thus, the V4 population seemingly encodes anti-preferred stimuli to double its capacitity for feature selectivity. To encourage future experiments searching for anti-preferred stimuli, we release a tool called ImageBeagle to efficiently "hunt" for a neuron's preferred and anti-preferred stimuli by traversing the nearest neighbor graph of 30 million natural images. Overall, we establish anti-preferred stimuli as an important encoding property of V4 neurons. Our work embarks on a new quest in neuroscience to search for anti-preferred stimuli along the visual stream as well as update our deep neural network models of visual cortex to account for the two-tailed response distributions of neurons.