Principal Scientist and Investigator
A*STAR Centre for Frontier AI and Research (A*STAR CFAR)

Adjunct Associate Professor
College of Computing and Data Science (CCDS), Nanyang Technological University (NTU)

Supported by JST PRESTO (Math-structure area, Oct. 2019 -- Mar. 2023)

Ph.D. in Information Science and Technology. The University of Tokyo. Supervisor: Taiji Suzuki

Research interests: Stochastic Optimization, Mean-field Optimization, Optimization-based Learning Theory, Machine Learning, Deep Learning

E-mail: atsushi_nitanda [at] cfar.a-star.edu.sg

Address: 1 Fusionopolis Way, #16-16, Connexis (North Tower), Singapore 138632

A*STAR scholarship and internship

A*STAR offers various scholarships and internship opportunities for students who want to study at A*STAR and universities in Singapore.  For details, see the call for applicants for scholarships. 

For Singaporean / local university students:

• ACIS: A program that encourages students to pursue Ph.D. degree at a university in Singapore. A monthly stipend, tuition,  and support oversea attachment are provided.

• ARIA: An internship program for Singaporean undergraduate students. Awardees will receive a monthly allowance from $1,600 to $2,000

For international students: 

• SINGA: A program that allows students to earn a Ph.D. degree at a university in Singapore (NTU, NUS, SUTD, or SMU) while studying at NTU, NUS, SUTD, SMU, or A*STAR. Tuition and living expenses are also subsidized.

• ARAP: A program for Ph.D. students outside of Singapore to support research at A*STAR for 1~2 years. Living expenses are subsidized.

• SIPGA: A program that supports 3rd, and 4th-year undergraduate and master's students outside of Singapore with 2~6 months of research activities at A*STAR, with a monthly grant of S$2,000.

Please contact me if you are interested in these opportunities under my supervision.

     * denotes alphabetical ordering below

Conference Papers (Refereed)

• 1. Atsushi Nitanda, Ryuhei Kikuchi, Shugo Maeda, and Denny Wu. Why is Parameter Averaging Beneficial in SGD? An Objective Smoothing Perspective. The 27th International Conference on Artificial Intelligence and Statistics (AISTATS2024), Proceedings of Machine Learning Research, volume 238, pages 3565–3573, 2024. [arXiv]

  2. Yuka Hashimoto, Sho Sonoda, Isao Ishikawa, Atsushi Nitanda, and Taiji Suzuki. Koopman-Based Bound for Generalization: New Aspect of Neural Networks Regarding Nonlinear Noise Filtering. The 12th International Conference on Learning Representations (ICLR2024), 2024.  [arXiv] [openreview]

  3. Atsushi Nitanda*, Kazusato Oko*, Taiji Suzuki*, and Denny Wu*. Improved Statistical and Computational Complexity of the Mean-field Langevin Dynamics under Structured Data. The 12th International Conference on Learning Representations (ICLR2024), 2024. [openreview]

  4. Taiji Suzuki, Denny Wu, Kazusato Oko, and Atsushi Nitanda. Feature Learning via Mean-field Langevin Dynamics: Classifying Sparse Parities and Beyond. The 37th Annual Conference on Neural Information Processing Systems (NeurIPS2023), In Advances in Neural Information Processing Systems, 36:34536--34556, 2023. 

  5. Taiji Suzuki, Denny Wu, and Atsushi Nitanda. Convergence of Mean-field Langevin Dynamics: Time and Space Discretization, Stochastic Gradient, and Variance Reduction. The 37th Annual Conference on Neural Information Processing Systems (NeurIPS2023), In Advances in Neural Information Processing Systems, 36:15545--15577, 2023. (Spotlight) [arXiv]

  6. Atsushi Suzuki, Atsushi Nitanda, Taiji Suzuki, Jing Wang, Feng Tian, and Kenji Yamanishi. Tight and Fast Generalization Error Bound of Graph Embedding in Metric Space. The 40th International Conference on Machine Learning (ICML2023), Proceedings of Machine Learning Research, 202:33268--33284, 2023. 

  7. Atsushi Nitanda, Kazusato Oko, Denny Wu, Nobuhito Takenouchi, and Taiji Suzuki. Primal and Dual Analysis of Entropic Fictitious Play for Finite-sum Problems. The 40th International Conference on Machine Learning (ICML2023), Proceedings of Machine Learning Research, 202:26266--26282, 2023. [arXiv]

  8. Taiji Suzuki, Atsushi Nitanda, and Denny Wu. Uniform-in-time Propagation of Chaos for the Mean Field Gradient Langevin Dynamics. The 11th International Conference on Learning Representations (ICLR2023), 2023.  [openreview]

  9. Naoki Nishikawa, Taiji Suzuki, Atsushi Nitanda, and Denny Wu. Two-layer Neural Network on Infinite Dimensional Data: Global Optimization Guarantee in the Mean-field Regime. The 36th Annual Conference on Neural Information Processing Systems (NeurIPS2022), In Advances in  Neural Information Processing Systems, 35:32612--32623, 2022. 

  10. Kazusato Oko, Taiji Suzuki, Atsushi Nitanda, and Denny Wu. Particle Stochastic Dual Coordinate Ascent: Exponential convergent algorithm for mean field neural network optimization. The 10th International Conference on Learning Representations (ICLR2022), 2022. [openreview]

  11. Atsushi Nitanda, Denny Wu, and Taiji Suzuki. Convex Analysis of the Mean Field Langevin Dynamics. The 25th International Conference on Artificial Intelligence and Statistics (AISTATS2022), Proceedings of Machine Learning Research, 151:9741--9757, 2022. [arXiv]

  12. Atsushi Suzuki, Atsushi Nitanda, Jing Wang, Linchuan Xu, Kenji Yamanishi, and Marc Cavazza. Generalization Bounds for Graph Embedding Using Negative Sampling: Linear vs Hyperbolic. The 35th Annual Conference on Neural Information Processing Systems (NeurIPS2021), In Advances in Neural Information Processing Systems, 34:1243--1255, 2021. 

  13. Atsushi Nitanda, Denny Wu, and Taiji Suzuki. Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis. The 35th Annual Conference on Neural Information Processing Systems (NeurIPS2021), In Advances in Neural Information Processing Systems, 34:19608--19621, 2021. [arXiv], [openreview]

  14. Taiji Suzuki and Atsushi Nitanda. Deep Learning is Adaptive to Intrinsic Dimensionality of Model Smoothness in Anisotropic Besov Space.  The 35th Annual Conference on Neural Information Processing Systems (NeurIPS2021), In Advances in Neural Information Processing Systems, 34:3609--3621, 2021. (Spotlight) [arXiv]

  15. Atsushi Suzuki, Atsushi Nitanda, Jing Wang, Linchuan Xu, Kenji Yamanishi, and Marc Cavazza. Generalization Error Bound for Hyperbolic Ordinal Embedding. The 38th International Conference on Machine Learning (ICML2021), Proceedings of Machine Learning Research, 139:10011--10021, 2021.

  16. Atsushi Nitanda and Taiji Suzuki. Optimal Rates for Averaged Stochastic Gradient Descent under Neural Tangent Kernel Regime. The 9th International Conference on Learning Representations (ICLR2021), 2021. (Outstanding Paper Award) [arXiv], [openreview] (8 papers out of 860 accepted papers, 2997 submissions)

  17. Shun-ichi Amari*, Jimmy Ba*, Roger Grosse*, Xuechen Li*, Atsushi Nitanda*, Taiji Suzuki*, Denny Wu*, and Ji Xu*. When Does Preconditioning Help or Hurt Generalization?. The 9th International Conference on Learning Representations (ICLR2021), 2021. [arXiv], [openreview]

  18. Shingo Yashima, Atsushi Nitanda, and Taiji Suzuki. Exponential Convergence Rates of Classification Errors on Learning with SGD and Random Features. The 24th International Conference on Artificial Intelligence and Statistics (AISTATS2021), Proceedings of Machine Learning Research, 130:1954--1962, 2021. [arXiv]

  19. Atsushi Nitanda and Taiji Suzuki. Functional Gradient Boosting for Learning Residual-like Networks with Statistical Guarantees.  The 23rd International Conference on Artificial Intelligence and Statistics (AISTATS2020), Proceedings of Machine Learning Research, 108:2981--2991, 2020.  

  20. Atsushi Suzuki, Jing Wang, Feng Tian, Atsushi Nitanda, and Kenji Yamanishi.  Hyperbolic Ordinal Embedding. The 11th Asian Conference on Machine Learning (ACML2019), Proceedings of Machine Learning Research, 101:1065--1080, 2019. 

  21. Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Data Cleansing for Models Trained with SGD. The 33rd Annual Conference on Neural Information Processing Systems (NeurIPS2019), In Advances in Neural Information Processing Systems, 32:4213--4222, 2019. [arXiv]. 

  22. Atsushi Nitanda, Tomoya Murata, and Taiji Suzuki. Sharp Characterization of Optimal Minibatch Size for Stochastic Finite Sum Convex Optimization. 2019 IEEE International Conference on Data Mining (ICDM2019), pp. 488--497. 2019. (Regular, Best Paper candidate for KAIS publication) [slide]

  23. Atsushi Nitanda and Taiji Suzuki. Stochastic Gradient Descent with Exponential Convergence Rates of Expected Classification Errors. The 22nd Artificial Intelligence and Statistics (AISTATS2019), Proceedings of Machine Learning Research, 89:1417--1426, 2019.  (Oral presentation) [arXiv] [slide]

  24. Atsushi Nitanda and Taiji Suzuki. Functional Gradient Boosting based on Residual Network Perception. The 35th International Conference on Machine Learning (ICML2018), Proceedings of Machine Learning Research, 80:3819--3828, 2018. [arXiv] [code] [slide]

  25. Atsushi Nitanda and Taiji Suzuki. Gradient Layer: Enhancing the Convergence of Adversarial Training for Generative Models. The 21st International Conference on Artificial Intelligence and Statistics (AISTATS2018), Proceedings of Machine Learning Research, 84: 1008--1016, 2018. [arXiv]

  26. Atsushi Nitanda and Taiji Suzuki. Stochastic Difference of Convex Algorithm and its Application to Training Deep Boltzmann Machines. The 20th International Conference on Artificial Intelligence and Statistics (AISTATS2017), Proceedings of Machine Learning Research, 54:470--478, 2017.

  27. Atsushi Nitanda. Accelerated Stochastic Gradient Descent for Minimizing Finite Sums. The 19th International Conference on Artificial Intelligence and Statistics (AISTATS2016), Proceedings of Machine Learning Research, 51:195--203, 2016. [arXiv]

  28. Atsushi Nitanda. Stochastic Proximal Gradient Descent with Acceleration Techniques. The 28th Annual Conference on Neural Information Processing Systems (NeurIPS(NIPS)2014), In Advances in Neural Information Processing Systems, 27:1574--1582, 2014.

Journal Articles

• 29. Naoki Nishikawa, Taiji Suzuki, Atsushi Nitanda, and Denny Wu. Two-layer Neural Network on Infinite-dimensional Data: Global Optimization Guarantee in the mean-field regime. Journal of Statistical Mechanics: Theory and Experiment (JSTAT), 2023(11):114007, 2023. (Journal version of NeurIPS2022 paper)

  30. Atsushi Nitanda, Denny Wu, and Taiji Suzuki. Particle Dual Averaging: Optimization of Mean Field Neural Networks with Global Convergence Rate Analysis. Journal of Statistical Mechanics: Theory and Experiment (JSTAT), 2022(11):114010, 2022. (Journal version of NeurIPS2021 paper)

  31. Atsushi Nitanda, Tomoya Murata, and Taiji Suzuki. Sharp Characterization of Optimal Minibatch Size for Stochastic Finite Sum Convex Optimization. Knowledge and Information Systems (KAIS), 63(9):2513--2539, 2021. (Journal version of ICDM2019 paper)

  32. Atsushi Nitanda. The Growth of the Nevanlinna Proximity Function. Journal of Mathematical Sciences, 16(4):525--543,  2009.

Technical Reports

1. Atsushi Nitanda. Improved Particle Approximation Error for Mean Field Neural Networks. 2024. [arXiv]

2. Yuto Mori, Atsushi Nitanda, and Akiko Takeda. BODAME: Bilevel Optimization for Defense Against Model Extraction. 2021. [arXiv]

3. Linchuan Xu, Jun Huang, Atsushi Nitanda, Ryo Asaoka, and Kenji Yamanishi. A Novel Global Spatial Attention Mechanism in Convolutional Neural Network for Medical Image Classification. 2020. [arXiv]

4. Shintaro Fukushima, Atsushi Nitanda, Kenji Yamanishi. Online Robust and Adaptive Learning from Data Streams. 2020. [arXiv]

5. Atsushi Nitanda, Geoffrey Chinot,  and Taiji Suzuki.  Gradient Descent can Learn Less Over-parameterized Two-layer Neural Networks on Classification Problems. 2019. [arXiv]

6. Atsushi Nitanda and Taiji Suzuki. Stochastic Particle Gradient Descent for Infinite Ensemble. 2017. [arXiv]

Notes

Atsushi Nitanda. Note: Noise Conditions and Convergence Analysis of SGD under Polyak-Lojasiewicz Inequality. 2022. [link]

Doctor Thesis

Efficient Machine Learning from Gradient Method Perspective in Finite and Infinite Dimensional Spaces

Professional Activities

Reviewer: NIPS, NeurIPS, ICML, AISTATS, ICLR, IJCAI, ICPR, JMLR, IEEE TNNLS/TSP/SPL, IEICE, Neural Networks, Signal Processing.
Selected as a top reviewer at NeurIPS 2019 and ICML 2020. Selected as a TMLR expert reviewer in 2023.

Action Editor: TMLR (Jul. 2023--).

Editorial Board Reviewer: JMLR (Aug. 2020--).

Editorial Board: IEICE Trans. (Jun. 2023--).

Committee: IBISML (Jun. 2022--Jun. 2024).

Organizing Committee: IBIS WS Program Committee (2020, 2024), CAI Workflow Co-chair (2024).

Awards

1. Atsushi Nitanda and Taiji Suzuki. Outstanding Paper Award, The Ninth International Conference on Learning Representations (ICLR), 2021.

2. Atsushi Nitanda, Tomoya Murata, and Taiji Suzuki. ICDM '19 Best Paper Candidate for KAIS Publication, IEEE International Conference on Data Mining (ICDM), 2019.  

3. Satoshi Hara, Atsushi Nitanda, and Takanori Maehara. Best Presentation Award, The 22nd Information-Based Induction Sciences Workshop (IBIS2019), 2019.

4. Atsushi Nitanda. Dean's Awards, Graduate School of Information Science and Technology, the University of Tokyo, 2019.

5. Atsushi Nitanda. Dean's Awards, Graduate School of Mathematical Sciences, the University of Tokyo, 2009.

Bio

Atsushi Nitanda is a Principal Scientist at A*STAR CFAR and an adjunct faculty at NTU. Prior to his current position, he was an Associate Professor at the Kyushu Institute of Technology and an Assistant Professor at the University of Tokyo. Previously, he worked at NTT DATA Mathematical Systems Inc. (MSI) as a researcher. He obtained his Ph.D. in Information Science and Technology from the University of Tokyo in 2018. His research interests include stochastic optimization, mean-field optimization, statistical learning theory, kernel method, and deep learning. He received the Outstanding Paper Award at ICLR in 2021 and the Dean’s Awards for doctoral and master's theses from the University of Tokyo in 2019 and 2009.