Hidenori Tanaka

Group Leader, NTT Research at Harvard University

email: hidenori_tanaka [at] fas.harvard.edu, twitter

"Physics of Intelligence" for Trustworthy and Green AI

News: Excited to announce the launch of the

"CBS-NTT Program in Physics of Intelligence" at Harvard's Center for Brain Science!
News article: [link]

Welcome! Our research group aims to pioneer the "Natural Science of Artificial Intelligence" by integrating theory and empirical methods to develop AI that is both trustworthy and environmentally sustainable. We also study and derive insights from the brain in collaboration with psychologists and neuroscientists.

Goals: Our research enhances AI systems in key areas:

Mechanistic Interpretability
Fairness and Robustness
Harnessing Emergent Abilities in Generative AI
Loss Landscapes and Learning Dynamics
AI Alignment
Advanced Reasoning.

Approaches: We bring a unique approach to the rapidly evolving field of artificial intelligence by applying the methods of the natural sciences to artificial neural networks. Our interdisciplinary approach involves collaborations with physicists, psychologists, neuroscientists, and computer scientists to blend scientific insight with practical engineering impact.

News:

Apr. 2024: Excited to establish new "CBS-NTT Program in Physics of Intelligence" at Harvard's Center for Brain Science!

Press releases: Harvard Gazette, Detailed Coverage, Harvard Crimson

Sep. 2023: 2 papers accepted at NeurIPS main conference, 5 papers accepted at NeurIPS workshops!

Mar. 2023: a paper accepted at ICML

Jan. 2023: a paper accepted at ICLR

Nov. 2022: Talk at CMSA Colloquium, Center of Mathematical Sciences and Applications, Harvard University

Sep. 2022: Talk at Harvard ML Foundations Seminar

Mar. 2022: Our group has physically relocated to the Center for Brain Science at Harvard University for an industry-academia collaboration. Visit us in room 180.01 of the Northwest building!

Mar. 2022: excited to join Harvard Center for Brain Science as an Associate member.
Jan. 2022: excited to welcome Dr. Gautam Reddy to collaborate with Harvard Center for Brain Science [news]!

Sep. 2021: 2 papers accepted at NeurIPS

Jan. 2021: a paper accepted at ICLR

Sep. 2020: a paper accepted at NeurIPS

Sep. 2019: a paper accepted at NeurIPS

Current/Past Group:

I have been fortunate to work closely with the following researchers who spend time in our group.

- Ekdeep Singh Lubana, PhD student (thesis co-advisor), EECS, U of Michigan
- Jeffrey Andrade, Ph.D. student, Systems Biology, Harvard
- Mikail Khona, Ph.D. student, Physics, MIT
- Rahul Ramesh, Ph.D. student, CS, U of Pennsylvania
- Kento Nishi, B.S. student, CS, Harvard
Ziyin Liu, PhD student, Physics, U of Tokyo
Fatih Dinc, PhD student, Applied Physics, Stanford
Max Aalto, PhD student, EECS, MIT
Daniel Kunin, PhD student, Computational & Mathematical Engineering, Stanford

Former colleagues in Intelligent Systems/Neural Network Group, NTT PHI Lab:

Gautam Reddy, Assistant Professor, Physics, Princeton University
Logan G. Wright, Assistant Professor, Applied Physics, Yale University

Our NTT Research at Harvard Group is young and growing! Please feel free to contact us for discussions, collaborations, and internships. Visit our website to learn more about the NTT Physics & Informatics Lab.

Brief CV:

2020 - : Group Leader & Senior Research Scientist, NTT Physics & Informatics Laboratories, CA, USA

2022 - : Associate, the Center for Brain Science, Harvard University, MA, USA

2024 - : Affiliate, IAIFI at Massachusetts Institute of Technology (MIT), MA, USA

2023 - : Visiting Researcher, Institute for Physics of Intelligence, the University of Tokyo, Japan

2018 - 2019: Masason Postdoctoral Fellow, Stanford University advisors: Surya Ganguli and Daniel Fisher

2014 - 2018: Ph.D. & M.S. in Applied Physics, Harvard University advisors: David Nelson and Michael Brenner

2010 - 2014: B.S. in Physics, Kyoto University, Kyoto, Japan

Born and grew up in Tokyo, Japan.

Selected Publications:

M. Okawa*, E.S. Lubana*, R.P. Dick, H. Tanaka*

"Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task"

NeurIPS (Advances in Neural Information Processing Systems) (2023) [link]

H. Tanaka, D. Kunin

"Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks"

NeurIPS (Advances in Neural Information Processing Systems) (2021) [pdf] [tweet-print]

H. Tanaka, A. Nayebi, N. Maheswaranathan, L. McIntosh, S.A. Baccus, S. Ganguli

"From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction"

NeurIPS (Advances in Neural Information Processing Systems) (2019) [pdf]

Publications:

"Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks"

R. Ramesh, E.S. Lubana, M. Khona, R.P. Dick, H. Tanaka

ICML (International Conference on Machine Learning) (2024)

"Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model"

M. Khona, M. Okawa, J. Hula, R. Ramesh, K. Nishi, R. Dick, E.S. Lubana*, H. Tanaka*

ICML (International Conference on Machine Learning) (2024)

"Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks"

S. Jain*, R. Kirk*, E.S. Lubana*, R.P. Dick, H. Tanaka, T. Rocktäschel, E. Grefenstette, D. Krueger

ICLR (International Conference on Learning Representations) (2024)

E.J. Bigelow, E.S. Lubana, R.P. Dick, H. Tanaka, T.D. Ullman

"In-Context Learning Dynamics with Random Binary Sequences"

ICLR (International Conference on Learning Representations) (2024)

M. Okawa*, E.S. Lubana*, R.P. Dick, H. Tanaka*

"Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task"

NeurIPS (Advances in Neural Information Processing Systems) (2023)

F. Dinc*, A. Shai*, M. Schnitzer, H. Tanaka

"CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics"

NeurIPS (Advances in Neural Information Processing Systems) (2023)

N. Maheswaranathan*, L.T. McIntosh*, H. Tanaka* (theoretical co-first author), S. Grant*, D.B. Kastner, J.B. Melander, A. Nayebi, L. Brezovec, J. Wang, S. Ganguli, S.A. Baccus

"Interpreting the retinal neural code for natural scenes: from computations to neurons"

Neuron (2023)

D. Kunin*, J. Sagastuy-Brena*, L. Gillespie, E. Margalit, H. Tanaka, S. Ganguli, D.L.K. Yamins

"Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations and anomalous diffusion"

Neural Computation (2023)

E.S. Lubana, E.J. Bigelow, R.P. Dick, D. Krueger, H. Tanaka

"Mechanistic Mode Connectivity"

ICML (International Conference on Machine Learning) (2023)

L. Ziyin, E.S. Lubana, M. Ueda, H. Tanaka

"What shapes the loss landscape of self-supervised learning?"

ICLR (International Conference on Learning Representations) (2023)

G. Reddy, L. Desban, H. Tanaka, J. Roussel, O. Mirat, C. Wyart (2022)