Extended NN architectures

Semi-Supervised Learning with Ladder Networks

Training Very Deep Networks ("Highway networks")



Unsupervised Feature Learning by Deep Sparse Coding


Nonlinear dictionary learning based deep neural networks
How to Train Your Deep Neural Network with Dictionary Learning


Deep dictionary learning




Residual networks:

Deep Residual Learning for Image Recognition

Residual Network Tutorial

Residual Networks explanation


Sparse Coding

Elvis Dohmatob, Arthur Mensch, Gael Varoquaux, Bertrand Thirion.  Learning brain regions via large-scale online structured sparse dictionary learning NIPS 2016



  1. Karol Gregor and Yann LeCun, Learning Fast Approximations of Sparse Coding. In Proc of ICML 2010.

  2. C.J. Rozell, D.H Johnson, R.G. Baraniuk, and B.A. Olshausen. Sparse coding via thresholding and local competition in neural circuits. Neural Computation, 20(10):2526-2563, October 2008. [ .pdf | Abstract ]

  3. R. Jenatton, J. Mairal, G. Obozinski, F. Bach. Proximal Methods for Hierarchical Sparse Coding. Technical report, HAL : inria-00516723, 2010. [pdf]

  4. A. Joulin, F. Bach, J.Ponce. Efficient Optimization for Discriminative Latent Class ModelsAdvances in Neural Information Processing Systems (NIPS), 2010. [pdf]

  5. M. Hoffman, D. Blei, F. Bach. Online Learning for Latent Dirichlet AllocationAdvances in Neural Information Processing Systems (NIPS), 2010. [pdf]

  6. R. Jenatton, J. Mairal, G. Obozinski, F. Bach. Proximal Methods for Sparse Hierarchical Dictionary LearningProceedings of the International Conference on Machine Learning (ICML), 2010. [pdf] [slides]

  7. J. Mairal, F. Bach, J. Ponce, G. Sapiro. Online Learning for Matrix Factorization and Sparse Coding. Journal of Machine Learning Research, 11, 10-60, 2010. [pdf] [code]

  8. J. Mairal, F. Bach, J. Ponce and G. Sapiro. Online dictionary learning for sparse codingInternational Conference on Machine Learning (ICML), 2009. [pdfMATLAB code

  9. Michal Aharon, Michael Elad, Alfred M. Bruckstein. On the uniqueness of overcomplete dictionaries, and a practical way to retrieve them. Linear Algebra and its Applications 416 (2006) 48–67.

  10. B. Olschausen and D. Field, Sparse Coding  with an Overcomplete Basis Set: A Strategy Employed by V1?

  11.    David P. Wipf, Sparse Estimation with Structured Dictionaries , NIPS 2011 

  12. Onur Dikmen, Cédric Févotte, Nonnegative dictionary learning in the exponential noise model for adaptive music signal representation, NIPS 2011 

  13.      13. Zhen James Xiang, Hao Xu, Peter J. Ramadge,  Learning Sparse Representations of High Dimensional Data on Large Scale Dictionaries , NIPS 2011 
  14.      14. Shiva Prasad Kasiviswanathan, Huahua Wang, Arindam Banerjee, Prem Melville, Online L1-Dictionary Learning with Application to Novel Document Detection , NIPS 2012
  15.      15. Mingyuan Zhou, Haojun Chen, John Paisley, Lu Ren, Guillermo Sapiro, Lawrence Carin, Non-Parametric Bayesian Dictionary Learning for Sparse Image Representations , NIPS 2009
  16.      16. Samy Bengio, Fernando Pereira, Yoram Singer, Dennis Strelow, Group Sparse Coding, NIPS 2009
  17.      17. J. Andrew Bagnell, David M. Bradley, Differentiable Sparse Coding, NIPS 2009
  18.      18. Pierre Garrigues, Bruno Olshausen, Group Sparse Coding with a Laplacian Scale Mixture Prior, NIPS 2010.
  19.      19.  Taehwan Kim, Gregory Shakhnarovich, Raquel Urtasun , Sparse Coding for Learning Interpretable Spatio-Temporal Primitives, NIPS 2010
  20.      20. Nobuyuki Morioka, Shin'ichi Satoh, Generalized Lasso based Approximation of Sparse Coding for Visual Recognition, NIPS 2011
  21.      21. arthur d. szlam, Karol Gregor, Yann LeCun, Structured sparse coding via lateral inhibition, NIPS 2012
  22.      22. Jacquelyn Shelton, Philip Sterne, Joerg Bornschein, Abdul-Saboor Sheikh, Joerg Luecke, Why MCA? Nonlinear sparse coding with spike-and-slab prior for neurally plausible image encoding, NIPS 2012
  23.      23. Adam Coates, Andrew Ng, The Importance of Encoding Versus Training with Sparse Coding and Vector Quantization   ICML 2011
  24.       24. Y. Xie, J. Ho, B. Vemuri, On a nonlinear generalization of sparse coding and dictionary learning, ICML 2013.
  25.      25. Smooth sparse coding via Marginal Regression,  ICML 2013
  26.      26. Sparse PCA through low-rank approximations, ICML 2013
  27.      27. Sparsity based generaliuzation bounds for predictive sparse coding, ICML 2013
  28.      28. Sparse coding for multitask and transfer learning, ICML 2013
  29.      28. Online L1-Dictionary Learning with Application to Novel Document Detection NIPS 2012

Supervised Dictionary Learning / Supervised Dimensionality Reduction


J. Mairal, F. Bach, J. Ponce, G. Sapiro and A. Zisserman. Supervised Dictionary LearningAdvances in Neural Information Processing Systems (NIPS), 2008. [pdf]

I. Rish, G. Grabarnik, G. Cecchi, F. Pereira, G. Gordon Closed-form supervised dimensionality reduction with generalized linear models ICML 2008

J. Mairal, F. Bach, J. Ponce. Task-DrivenDictionary Learning. Technical report, HAL : inria-00521534, 2010. [pdf]

Gangeh, M. J., Ghodsi, A., & Kamel, M. S. (2013). Kernelized supervised dictionary learning. IEEE Transactions on Signal Processing, 61(19), 4753-4767.


S. Kong and D. Wang. Online Discriminative Dictionary Learning for Image Classification Based on Block-Coordinate Descent Method. arXiv 2012

G Zhang, Z Jiang, LS Davis. Online semi-supervised discriminative dictionary learning for sparse representation, Asian Conference on Computer Vision, 2012

Yuxin Zhang and Bo Yuan. Supervised Online Dictionary Learning for Image Separation Using OMP, ICIC-2016