Ryo Karakida

Recent works

• "MLP-Mixer as a Wide and Sparse MLP",  Tomohiro Hayase & Ryo Karakida, accepted in ICML 2024 [arXiv]

• "Self-attention Networks Localize When QK-eigenspectrum Concentrates", Han Bao, Ryuichiro Hataya & Ryo Karakida, accepted in ICML 2024 [arXiv]

• "On the Parameterization of Second-Order Optimization Effective Towards the Infinite Width",  Satoki Ishikawa & Ryo Karakida, accepted in ICLR 2024 [arXiv]

International conference and workshop

Refereed proceedings:

• "Understanding Gradient Regularization in Deep Learning: Efficient Finite-Difference Computation and Implicit Bias",  Ryo Karakida, Tomoumi Takase, Tomohiro Hayase & Kazuki Osawa,  ICML 2023 [arXiv]

• "Learning Curves for Continual Learning in Neural Networks: Self-Knowledge Transfer and Forgetting", Ryo Karakida  & Shotaro Akaho,  ICLR 2022 [arXiv]

•  "Understanding Approximate Fisher Information for Fast Convergence of Natural Gradient Descent in Wide Neural Networks", Ryo Karakida & Kazuki Osawa,  NeurIPS 2020 (oral presentation)  [arXiv]

• "The Normalization Method for Alleviating Pathological Sharpness in Wide Neural Networks",  Ryo Karakida, Shotaro Akaho, Shun-ichi Amari, NeurIPS 2019 [arXiv]

• "Universal Statistics of Fisher Information in Deep Neural Networks: Mean Field Approach", Ryo Karakida, Shotaro Akaho, Shun-ichi Amari,  AISTATS2019, PMLR 89:1032-1041  [link]

• "Fisher Information and Natural Gradient Learning in Random Deep Networks",  Shun-ichi Amari, Ryo Karakida, Masafumi Oizumi,  AISTATS2019, PMLR 89:694-702  [link]

• "Information geometry of Wasserstein divergence" Ryo Karakida, Shun-ichi Amari,  Geometric Science of Information (GSI 2017) [preprint] [link] 

• "Adaptive natural gradient learning algorithms for unnormalized statistical models"  Ryo Karakida, Masato Okada, Shun-ichi Amari, International Conference on Artificial Neural Networks (ICANN 2016) [preprint] [link]

• "Adaptive Natural Gradient Learning Based on Riemannian Metric of Score Matching", Ryo Karakida, Masato Okada, Shun-ichi Amari, 4th International Conference on Learning Representations - Workshop Track (ICLR 2016) [pdf]

• "Maximum likelihood learning of RBMs with Gaussian visible units on the Stiefel manifold",  Ryo Karakida, Masato Okada, Shun-ichi Amari, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN), 2016  [pdf] 

• "Analysing Feature Extraction by Contrastive Divergence Learning in RBMs", Ryo Karakida, Masato Okada, Shun-ichi Amari,  Deep Learning And Representation Learning Workshop: NIPS2014

Others:

• Talk on Natural Gradient Descent in NTK regime at Mathematical Machine Learning Seminar MPI + UCLA [Slides]

• Americal Institute of Mathematics (AIM) workshop "Boltzmann machines", "Theoretical analysis of RBMs with Gaussian visible units - Dynamical analysis and Riemannian optimization-", San Jose, Oct 2018 

• "On the capability of restricted Boltzmann machine learning to extract appropriate input features", Object Vision in Human, Monkey, and Machine, Tokyo, Nov 2015

• "Equilibrium Analysis of Representation Learning in Gaussian RBMs",  New Frontiers in Non-equilibrium Physics 2015 , Kyoto, July 2015

Full papers

• "Attention in a family of Boltzmann machines emerging from modern Hopfield networks", Toshihiro Ota & Ryo Karakida,  Neural Computation, 2023 [arXiv]

• "Deep Learning in Random Neural Fields: Numerical Experiments via Neural Tangent Kernel", Kaito Watanabe, Kotaro Sakamoto, Ryo Karakida, Sho Sonoda,& Shun-ichi Amari, Neural Networks, 2023 [arXiv]

•  "Pathological spectra of the Fisher information metric and its variants in deep neural networks",  Ryo Karakida, Shotaro Akaho & Shun-ichi Amari, Neural Computation, vol.33(8), pp.2274-2307, 2021  [arXiv]

•  "Self-paced data augmentation for training neural networks" ,  Tomoumi Takase, Ryo Karakida & Hideki Asoh, Neurocomputing, vol.442, pp.296-306, 2021 [link]

•  "Collective dynamics of repeated inference in variational autoencoder rapidly find cluster structure", Yoshihiro Nagano, Ryo Karakida & Masato Okada, Scientific Reports, vol.10, 16001, 2020 [link]

• "Statistical Neurodynamics of Deep Networks: Geometry of Signal Spaces", Shun-ichi Amari, Ryo Karakida, Masafumi Oizumi,  Nonlinear Theory and Its Application (NOLTA), Vol.10(4), 322-336, 2019 [link]

• "Statistical Mechanical Analysis of Learning Dynamics of Two-Layer Perceptron with Multiple Output Units" , Yuki Yoshida, Ryo Karakida, Masato Okada, Shun-ichi Amari,  Journal of Physics A, 52(18), 184002, 2019 [link]

• "Information Geometry for Regularized Optimal Transport and Barycenters of Patterns",  Shun-ichi Amari, Ryo Karakida, Masafumi Oizumi, Marco Cuturi, Neural Computation, 31(5), 827-848, 2019 [link]

• "Information Geometry Connecting Wasserstein Distance and Kullback-Leibler Divergence via the Entropy-Relaxed Transportation Problem", Shun-ichi Amari, Ryo Karakida, Masafumi Oizumi,  Information Geometry, 1(1), 13-37, 2018 [pdf]

• "Dynamics of Learning in MLP: Natural Gradient and Singularity",  Shun-ichi Amari, Tomoko Ozeki, Ryo Karakida, Yuki Yoshida, Masato Okada, Neural Computation, Vol.30(1), 2018 [link]

• "Statistical Mechanical Analysis of Online Learning with Weight Normalization in Single Layer Perceptron”,  Yuki Yoshida, Ryo Karakida, Masato Okada, Shun-ichi Amari, Journal of the Physical Society of Japan, 86, 044002, 2017 [pdf]

• "Input Response of Neural Network Model with Lognormally Distributed Synaptic Weights",  Yoshihiro Nagano, Ryo Karakida, Norifumi Watanabe, Atsushi Aoyama, Masato Okada,  Journal of the Physical Society of Japan, 85, 074001, 2016 [pdf] 

• "Dynamical analysis of contrastive divergence learning: Restricted Boltzmann machines with Gaussian visible units",  Ryo Karakida, Masato Okada, Shun-ichi Amari, Neural Networks,79, 78-87, 2016 [preprint] [link]

• "Inter-Layer Correlation in a Feed-Forward Network with Intra-Layer Common Noise", Ryo Karakida, Yasuhiko Igarashi, Kenji Nagata and Masato Okada, Journal of the Physical Society of Japan, 82, 064007, 2013 [pdf]  

Educations

March 2017: Doctor of Science, Department of Complexity Science and Engineering, Graduate School of Frontier Sciences, University of Tokyo（Supervisor: Prof. Masato Okada), PhD Thesis "Geometric Theories of Weight Dynamics for Learning in Hierarchical Neural Networks"

March 2014: Master of Science, Department of Complexity Science and Engineering, Graduate School of Frontier Sciences, University of Tokyo（Supervisor: Prof. Masato Okada）

March 2012: Bachelor of Science, Department of Physics, University of Tokyo 

Teaching 

Part-time Lecturer, "Introduction to AI" in Probability and Statistics,  Department of Precision Engineering, University of Tokyo, Fall '17, '18, '20, '21, '22

Service
Reviewer:  NeurIPS, ICML, AISTATS, JMLR, Neural Networks, Neurocomputing, Neural Computation, Information Geometry, ...

Awards

• Best Student Paper Award, 24th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN2016), Belgium, April, 2016. 

• JNNS Best Paper Award, "Dynamical analysis of contrastive divergence learning: Restricted Boltzmann machines with Gaussian visible units ." 2017 

• JNNS Young Presenter Award, 24st Annual Conference of the Japanese Neural Network Society (JNNS2014), Hokkaido, Japan, August, 2014.

• IEEE CISJ Young Researcher Award, Technical Committee on Neurocomputing, Tokyo, Japan, March, 2015.

and others