Visual Coding with Spikes

Compared to static natural images, decoding dynamics videos from visual neuronal spikes is more difficult. Here, we develop an approach to fulfill this goal.

By using a population of the retinal ganglion cells (~100), we can decode the given stimulus with a neural network approach. Such a decoder is named as spike-image decoder since it is an end-to-end decoding model with one end as a population of neuronal spikes, and the other end as the pixel images. See the details in Ref. 1.

Video 1. What's in the box: Dynamics videos reconstructed from neuronal spikes. Here the neuronal signal is a population of spikes reordered form the ganglion cells of the Salamander retina (see Ref 2 for experimental data descriptions). As a comparison, videos reconstructed from the neuronal signal of fMRI is listed (see Ref 3 for fMRI reconstruction).

Video 2. What's in the box: Arbitrary images and videos obtained with simulated spikes. Here neural spikes are generated by an encoder, and then another decoder is to reconstruct visual scenes with these spikes. The encoder and decoder are based on a model of the retinal ganglion cells .

Video 3. What's in the box: Real-time encoding and decoding of videos. Similar to Video 2, here a few stacked encoders and decoders are used to obtain high-resolution videos.

References:

Ref 1: Zheng Y, Jia S, Yu Z, Huang T, Liu J.K., Tian Y. Probabilistic Inference of Binary Markov Random Fields in Spiking Neural Networks through Mean-field Approximation. Neural Networks. 126:42-51 (2020) [PDF] DOI: 10.1016/j.neunet.2020.03.003

Ref 2: Onken A., Liu J. K., Karunasekara C. R., Delis I., Gollisch T. and Panzeri S. Using Matrix and Tensor Factorizations for the Single-Trial Analysis of Population Spike Trains. PLoS Comput Biol 12(11): e1005189 (2016) [PDF]. Data Availability: http://dx.doi.org/10.5061/dryad.4ch10

Ref 3: Nishimoto, S., Vu, A. T., Naselaris, T., Benjamini, Y., Yu, B., Gallant, J.C. Reconstructing visual experiences from brain activity evoked by natural movies Curr Biol. 21(19): 1641–1646 (2011). Demo: http://gallantlab.org/index.php/publications/nishimoto-et-al-2011/