Michael Zibulevsky

Adjunct Professor

Department of Computer Science
Technion - Israel Institute of Technology
Haifa 32000, Israel

Email: mzib@cs.technion.ac.il, mzibul@gmail.com

Room 436

We have open positions for  talented M.Sc. and Ph.D.  students in areas of Deep Learning, Numerical Optimization and Biomedical Imaging

Research Interests

Numerical optimization, deep learning, sparse signal representations, independent component analysis, inverse problems in medical imaging

Teaching

•   CS-236330: Introduction to Optimization and Deep Learning  <==> YouTube video lectures

Discussion group
      Deep Learning Club Technion

YouTube video 

Neural Networks
• Intro to Neural Networks
• Convolutional Neural Networks in 10 minutes
  
•  Convolutional Neural Networks in 7 min (summary)
• An easy way to compute Jacobian and gradient with forward and back propagation in a graph
• Gradient of Neural Network in matrix form: Part 1, Part 2  
• Elad Hoffer, Deep Learning course, Technion, 2016 (in Hebrew):  Lecture 2,  Lecture 3, Lecture 4, Lecture 5  
• End-to-End Deep Learning: Applications in Speech, by Yedid Hoshen
• Deep Learning on Graphs and Manifolds, by Michael Bronstein
• Raja Giryes, How structure can improve the theory and practice in neural networks, Part 1, Part 2, Part 3  
  
  Optimization
• Introduction to Optimization, video course
• Intro to Neural Networks
• Lecture 2-3: Gradient and Hessian of Multivariate Function (enhanced)
• Easy way to compute Jacobian and gradient with forward and back propagation in graph
• Gradient of Neural Network in matrix form: Part 1, Part 2  
• Fixed Point Iteration
• Lecture 4-5: Convex sets and functions (enhanced)
• Lecture 6 (enhanced).Local and global minimum. Sufficient and necessary optimality conditions
• Bisection method for finding root and minimum of 1D function 
• Golden section method of 1D minimization
• Quadratic interpolation method of 1D minimization
• Cubic interpolation method of 1D minimization  
• Multidimensional optimization with line search 
• Gradient descent method (steepest descent)   
• Newton method for multidimensional minimization. Part 1 Part 2 
• Newton and Gauss-Newton methods for nonlinear system of equations and least squares problem
• Conjugate gradient method  
• Conjugate gradients 1: Introduction
• Conjugate gradients 2: Inner product; Gram-Schmidt orthogonalization
• Conjugate gradients 3: Conjugate directions; Expanding manifold
• Conjugate gradients 4: Derivation of CG method
• Preconditioning: Gradient Descent, Conjugate Gradients and SESOP 
• SESOP - Sequential Subspace Optimization. 
• Part 1: Description of the method; 
•  Part 2: Convergence properties  
• Preconditioning: Gradient Descent, Conjugate Gradients and SESOP  
• Accelerating SESOP: Preconditioning, Coordinate Descent, Majorization-Minimization(Iterative Thresholding)  
•  Boosting stochastic optimization with SESOP 
• Quasi-Newton Optimization Methods    (BFGS, L-BFGS, etc.)
• Penalty function and Augmented Lagrangian methods 2013 (Introduction)
• Penalty Multiplier Method (Augmented Lagrangian) 1   
• Penalty Multiplier Method (Augmented Lagrangian) 2: Dual Interpretation 
• Lagrange Duality: Conic Programming vs Nonlinear one
• Conic Lagrange Multipliers via Gradient of Penalty Function   
• SESOP - sequential subspace optimization method
  
  
• 
  Image / Signal Processing
  
  
• Denoising, deconvolution and computed tomography using total variation penalty
• Maximum likelihood blind source separation (ICA)
• Intensity-Modulated Radiation Therapy via Conic Programming
• Blind source separation by sparse decomposition
  
  School Math 
  
• График квадратичной функции: урок за 9 минут
• Build Graph of Quadratic Function: Lesson in 9min
• Graph of Quadratic Function (in Hebrew) גרפ של פונקציה ריבועית
• My short video lessons in secondary school mathematics
  
  Society
  
• Как сделать экономику устойчивой к шоку
• Зеленая энергия для Ирана вместо атомной
• Green energy for Iran instead of nuclear power 
• Экологические Поселки, Роботы и Интернет
• 
  
  Others
  
•  Michael Unser, Wavelets Demystified
  
• Michael Unser, Beyond the Digital Divide

Presentations (slides)

•   Blind source separation by sparse decomposition + Relative Newton + Method of multipliers, Jerusalem 2004
•   Blind source separation, deconvolution and localization using sparse representations, 2004
•   SESOP: Sequential Subspace Optimization Method for large-scale optimization problems (including SESOP-TN), 2012

Matlab Code

• PCD-CG: Parallel coordinate descent merged with conjugate gradients 
  
•  SESOP_PACK - large-scale unconstrained optimization tool, version 10.05.2010 Includes L1-L2 optimization via  PCD-SESOP (parallel coordinate descent), SSF-SESOP, FISTA, etc
•   PBM - penalty/barrier multiplier algorithm for large-scale nonlinear optimization with functional constraints
•   Newton method with "frozen" Hessian for unconstrained optimization
•   Modified Cholesky Factorization (modified Brian Borchers' code)
•   Image Total Variation, its gradient and Hessian-vector products (simple and fast code)
•   Adjoint convolution 2D convolution, efficient mex implementation: Version 29.11.2010, compiled at windows 7, 64 bit
•   Relative Newton Method for Blind Source Separation
•   Sparse ICA code (by A&M Bronstein): Blind Source Separation based on Sparse Representations
•   Extraction of a single source from multichannel data using template and sparse decomposition
  

Selected Publications

See the latest publications at my Google Scholar page:

• Sorted by time
• Sorted by citation 

Book Chapters:

•   M. Zibulevsky, B. A. Pearlmutter, P. Bofill, and P. Kisilev, "Blind Source Separation by Sparse Decomposition", chapter in the book: S. J. Roberts, and R.M. Everson eds., Independent Component Analysis: Principles and Practice, Cambridge, 2001. gzipped ps file , pdf file

•   M. Zibulevsky,"Relative Newton and Smoothing Multiplier Optimization Methods for Blind Source Separation", chapter in the book: S. Makino, T.W. Lee and H. Sawada eds., Blind Speech Separation, Springer Series: Signals and Communication Technology XV, 2007 pdf file

•   R. Gribonval and M. Zibulevsky. Sparse Component Analysis, in Pierre Comon and Christian Jutten (Editors), Handbook of Blind Source Separation: Independent Component Analysis and Applications, ELSEVIER 2010, pp.367-420

Papers and Reports:

•   Michael Zibulevsky (2010) How to inhibit destructive positive feedback in time of economic crisis

•   M. Zibulevsky and M. Elad, L1-L2 Optimization in Signal and Image Processing, IEEE Signal Processing Magazine, Vol. 27 No. 3, Pages 78-88, May 2010.

•   J. Shtok, M. Elad, and M. Zibulevsky, Spatially-Adaptive Reconstruction in Computed Tomography Based on Statistical Learning, Submitted to IEEE Transactions on Image Processing

•   Yehuda Pfeffer and Michael Zibulevsky (2010) Sampling and Noise in Compressive Sensing

•   Yehuda Pfeffer and Michael Zibulevsky (2010) A Micro-Mirror Array based System for Compressive Sensing of Hyperspectral Data
Master thesis of Yehuda Pfeffer "Compressive Sensing for Hyperspectral Imaging", Technion, 2010

thesis_yehuda_pfeffer

•   Ron Rubinstein, Michael Zibulevsky, Michael Elad, Double Sparsity: Learning Sparse Dictionaries for Sparse Signal Approximation, IEEE Transactions on Signal Processing, Vol. 58, No. 3, Pages 1553-1564, March 2010

•   Eliyahu Osherovich, Michael Zibulevsky, and Irad Yavneh (2009) IMAGE RECONSTRUCTION FROM NOISY FOURIER MAGNITUDE WITH PARTIAL PHASE INFORMATION

•   Michael Zibulevsky (2009) How to prevent economic crisis in time of disaster: consumer insurance bonds

•   Michael Zibulevsky (2008), On convex formulation of 1D scaling problem

•   Michael Zibulevsky (2008), SESOP-TN: Combining Sequential Subspace Optimization with Truncated Newton method

•   Alexander M. Bronstein, Michael M. Bronstein and Michael Zibulevsky, "On Separation of Semitransparent Dynamic Images from Static Background",

•   L. Dascal, M. Zibulevsky and R. Kimmel, "Signal denoising by constraining the residual to be statistically noise-similar", Technical Report, 2008 pdf file

•   A.M. Bruckstein, M. Elad, and M. Zibulevsky, On the Uniqueness of Nonnegative Sparse Solutions to Underdetermined Systems of Equations, IEEE Transactions on Information Theory, Vol. 54, No. 11, Pages 4813-4820, November 2008

•   Sarit Shwartz, Yoav Y. Schechner and Michael Zibulevsky (2008), Blind separation of convolutive image mixtures, To be published in Neurocomputing, Special issue on Advances in Blind Signal Processing.

•   M. Elad, B. Matalon, and M. Zibulevsky, "Coordinate and Subspace Optimization Methods for Linear Least Squares with Non-Quadratic Regularization", Applied and Computational Harmonic Analysis, Vol. 23, pp. 346-367, November 2007. pdf file

•   D. Model and M. Zibulevsky (October 2006), Learning Subject-Specic Spatial and Temporal Filters for Single-Trial EEG Classi/cation, NeuroImage, Vol 32, Issue 4, pp 1631-1641 pdf file

•   Model D. and Zibulevsky M. (2006), Signal Reconstruction in Sensor Arrays using Sparse Representations, Signal Processing, Vol 86, Issue 3, pp 624-638 pdf file

•   Shwartz S., Zibulevsky M., and Schechner Y.Y. (2005), Fast kernel entropy estimation and optimization, Signal Processing, Vol 85, pp. 1045-1058 pdf file

•  Narkiss, G. and Zibulevsky, M. (2005). "Sequential Subspace Optimization Method for Large-Scale Unconstrained Problems", Tech. Report CCIT No 559, EE Dept., Technion. pdf file

•  Narkiss, G. and Zibulevsky, M. (2005). "Support Vector Machine via Sequential Subspace Optimization", Tech. Report CCIT No 557, EE Dept., Technion. pdf file

•  Zibulevsky, M. (2005). "Blind Source Separation using Relative Newton Method combined with Smoothing Method of Multipliers", Tech. Report CCIT No 556, EE Dept, Technion. pdf file

•   A.M. Bronstein, M.M. Bronstein, M. Zibulevsky and Y.Y.Zeevi, "Sparse ICA for blind separation of transmitted and reflected images", Intl. Journal of Imaging Science and Technology (IJIST), Vol. 15/1, pp. 84-91, 2005.

•   A.M. Bronstein, M.M. Bronstein, M. Zibulevsky and Y.Y.Zeevi, "Blind Deconvolution of Images using Optimal Sparse Representations", IEEE Trans. on Image Processing, 14(6):726-736, June 2005. pdf file

•   Alexey Polonsky, Michael Zibulevsky: MEG/EEG Source Localization Using Spatio-temporal Sparse Representations. ICA 2004: 1001-1008 pdf file
Thesis of Alexey Polonsky

•   A. Bronstein, M. Bronstein and M. Zibulevsky (2003), "Relative optimization for blind deconvolution", IEEE Trans. on Signal Processing, to appear. pdf file

•   A. Bronstein, M. Bronstein and M. Zibulevsky (2003), "Blind source separation using block-coordinate relative Newton method" pdf file

•   P. Kisilev, M. Zibulevsky, Y.Y. Zeevi (2003). "Multiscale framework for blind source separation", JMLR, in press. pdf file

•  Zibulevsky, M. "Blind Source Separation with Relative Newton Method", Proceedings ICA2003, pp. 897-902

•  Zibulevsky, M. and Pearlmutter, B.A. (2000). "Second order blind source separation by recursive splitting of signal subspaces", Proceedings ICA2000. gzipped ps file

•   Bronstein M., Bronstein A. and Zibulevsky M. (2002). ``Iterative reconstruction in diffraction tomography using non-uniform FFT'' pdf file

•   Bronstein A., Bronstein M., Zibulevsky M. and Zeevi Y.Y. (2002). ``Optimal nonlinear estimation of photon coordinates in PET'' pdf file

•   M. Zibulevsky (2003). "Smoothing Method of Multipliers for Sum-Max Problems" gzipped ps file

•   Bofill P., Zibulevsky, M. (2001). Underdetermined Blind Source Separation using Sparse Representations, Signal Processing, Vol.81, No 11, pp.2353-2362. pdf file

•  Zibulevsky, M. and Pearlmutter, B.A. (1999). "Blind Source Separation by Sparse Decomposition ", Neural Computations 13(4), 2001 gzipped ps file , zipped ps file , OLD html verson, Demo

•   M. Zibulevsky, Y.Y. Zeevi (2002). "Extraction of a single source from multichannel data using sparse decomposition", Neurocomputing 49, pp 163-173. gzipped ps file

•   M. Zibulevsky, P. Kisilev, Y.Y. Zeevi, B.A. Pearlmutter (2000). "Blind source separation via multinode sparse representation", NIPS-2001, gzipped ps file

•  Levkovitz R., Falikman D., Zibulevsky M., Ben-Tal A., Nemirovski A. (2001) ``The design and implementation of COSEM, an iterative algorithm for fully 3D listmode data'', IEEE Trans. Med. Imaging, v. 20, #7, pp. 633-642 pdf file

•  Zibulevsky, M. and Pearlmutter, B.A. (2000). "Recovering shape and distribution of delays of repetitive responses in strong noise" Technical Report CS00-1, Computer Science Department, University of New Mexico. gzipped ps file

•   Akaysha C. Tang, Barak A. Pearlmutter, and Michael Zibulevsky. Blind source separation of neuromagnetic responses. Computational Neuroscience 1999, proceedings published in Neurocomputing. In press.

•  Mosheyev, L. and Zibulevsky, M. (2000). "Penalty/Barrier Multiplier Algorithm for Semidefinite Programming", Optimization Methods and Software, vol.13, No 4, pp. 235-261. pdf file , gzipped dvi file

• Zibulevsky, M. (1998). "Pattern Recognition via Support Vector Machine with Computationally Efficient Nonlinear Transform ". gzipped ps

•  Zibulevsky, M. (1998). "ML Reconstruction of Dynamic Pet Images from Projections and Clist ", Technical Report CS98-3, Computer Science Department, University of New Mexico. gzip ps

•  Ben-Tal, A. and Zibulevsky, M. (1997). ``Penalty/Barrier Multiplier Methods for Convex Programming Problems",SIAM Journal on Optimization v. 7 # 2, pp. 347-366, gzip ps

•  Zibulevsky M. (1996) Penalty/Barrier Multiplier Methods for Large-Scale Nonlinear and Semidefinite Programming. Ph.D. Thesis. gzip ps , gzip dvi

•  Kochvara, M., Zibulevsky, M. and Zowe, J. (1996). "Mechanical Design Problems with Unilateral Contact", MAN - Mathematical Modeling and Numerical Analysis, v.32, no 3, pp. 255-282

•   Akaysha C. Tang, Barak A. Pearlmutter, Michael Zibulevsky, Tim A. Hely, and Michael Weisend. An MEG study of response latency and variability in the human visual system during a visual-motor integration task. In Advances in Neural Information Processing Systems*99. Morgan Kaufmann, 2000, to appear.

•  Greig, D., Siegelman, H. and Zibulevsky, M., (1996). "A New Class of Neural Network Activation Functions That Don't Saturate". Report. gzipped ps file

•  Mosheyev, L. and Zibulevsky, M. (1996). "Penalty/Barrier Multiplier Algorithm for Semidefinite Programming: Dual Bounds and Implementation". Research Report #1/96} , Optimization Laboratory.

• Zibulevsky M. (1995) "New Penalty/Barrier and Lagrange Multiplier Approach for Semidefinite Programming". Research Report #5/95, Optimization Laboratory:

•  A. Ben-Tal, I. Yuzefovich, and M. Zibulevsky (1992). "Penalty/barrier multiplier methods for minimax and constrained smooth convex problems". Research Report 9/92, Optimization Laboratory, Faculty of Industrial Engineering and Management, Technion, Haifa, Israel.

•  Kuprienko, A. and Zibulevsky, M. (1987). " Efficient Data Transmission over High-noised Telephone Channels", NIIASS, Kiev, USSR.

---

Pointers to Other Pages

AMPL: A Modeling Language for Optimization

ICA - Independent Component Analysis

---

• Vision Research and Image Sciences Laboratory
• Optimization Laboratory

---

Some Bibliography on ICA and sparse decomposition

ICA MATLAB ASSIGNMENT

---

www.mathtools.net - Scientific computing links for MATLAB, C/C+, Fortran and others

NetLib: free numerical libraries

Wavelet Digest

CONNECTIONISTS: neural computation mailing list

Book: Convex Optimization by Boyd and Vandenberghe

Slides to the book book Convex Optimization by Boyd and Vandenberghe

ICA page-papers,code,demo,links by Paris Smaragdis at MIT

ICA (independent component analysis by Allan Barros, Site in Japan )

ICA page of SALK Computational Neuroscience Laboratory

ICA CENTRAL: web page + mailing list (by Jean-François Cardoso)