Science of Natural and Artificial Intelligence Group
CBS-NTT Program in Physics of Intelligence, Harvard University
Hidenori Tanaka
(Google Scholar, email: hidenori_tanaka [at] fas.harvard.edu, X)
Embedded Files
Welcome! Our research group studies the Science of AI for Alignment,
"Nothing in life is to be feared, it is only to be understood. Now is the time to understand more, so that we may fear less."
- Marie Curie
Goals: Our research aims to develop experimental model systems and draw mathematical representations to better understand the generalization behaviors of AI. Our current interests include:
Understanding and aligning AI's emergent abilities
Drawing mathematical pictures of AI's generalization behaviors
Integrating AI and psychology to explore applications in education and psychiatry
Approaches: We bring a unique perspective to the rapidly evolving field of artificial intelligence by applying natural science methods to artificial neural networks. Our interdisciplinary approach involves collaborations with physicists, psychologists, neuroscientists, and computer scientists, blending scientific insights with practical engineering impact.
News:
News:
Jan. 2025: 5 works accepted at ICLR 2025!
Nov. 2024: Joining MATS (ML Alignment & Theory Scholars) program as a mentor
Sep. 2024: A paper accepted at NeurIPS Spotlight
Apr. 2024: Excited to establish new "CBS-NTT Program in Physics of Intelligence" at Harvard's Center for Brain Science!
Press releases: Harvard Gazette, Detailed Coverage, Harvard Crimson
Sep. 2023: 2 papers accepted at NeurIPS main conference, 5 papers accepted at NeurIPS workshops!
Mar. 2023: a paper accepted at ICML
Jan. 2023: a paper accepted at ICLR
Nov. 2022: Talk at CMSA Colloquium, Center of Mathematical Sciences and Applications, Harvard University
Sep. 2022: Talk at Harvard ML Foundations Seminar
Mar. 2022: Our group has physically relocated to the Center for Brain Science at Harvard University for an industry-academia collaboration.
Current/Past Group:
Current/Past Group:
I have been fortunate to work closely with the following researchers who spend time in our group.
Ekdeep Singh Lubana, PhD student (thesis co-advisor), EECS, U of Michigan
Sam Bright-Thonney, IAIFI PostDoc Fellow, Physics, MIT
Fatih Dinc, PhD student, Applied Physics, Stanford
Maya Okawa, Visiting Research Scientist, NTT
Bo Zhao, PhD student, Computer Science, UCSD
Eric Bigelow, PhD student, Psychology, Harvard
Core Francisco Park, PhD student, Physics, Harvard
Bhavya Vasudeva, PhD student, CS, University of Southern California
Yongyi Yang, PhD student, CSE, U of Michigan
Mikail Khona, PhD student, Physics, MIT
Rahul Ramesh, PhD student, CS, U of Pennsylvania
Kento Nishi, BA & MS student, CS, Harvard
Ziyin Liu, PhD student, Physics, U of Tokyo
Max Aalto, PhD student, EECS, MIT
Daniel Kunin, PhD student, Computational & Mathematical Engineering, Stanford
Former colleagues in Intelligent Systems/Neural Network Group, NTT PHI Lab:
Gautam Reddy, Assistant Professor, Physics, Princeton University
Logan G. Wright, Assistant Professor, Applied Physics, Yale University
Our NTT Research at Harvard Group is young and growing! Please feel free to contact us for discussions, collaborations, and internships. Visit our website to learn more about the NTT Physics & Informatics Lab.
Brief CV:
Brief CV:
2020 - : Group Leader & Senior Research Scientist, NTT Physics & Informatics Laboratories, CA, USA
2022 - : Associate, the Center for Brain Science, Harvard University, MA, USA
2024 - : Affiliate, IAIFI at Massachusetts Institute of Technology (MIT), MA, USA
2023 - : Visiting Researcher, Institute for Physics of Intelligence, the University of Tokyo, Japan
2018 - 2019: Masason Postdoctoral Fellow, Stanford University advisors: Surya Ganguli and Daniel Fisher
2014 - 2018: Ph.D. & M.S. in Applied Physics, Harvard University advisors: David Nelson and Michael Brenner
2010 - 2014: B.S. in Physics, Kyoto University, Kyoto, Japan
Born and grew up in Tokyo, Japan.
Publications:
Publications:
D. Wurgaft*, E.S. Lubana*, C.F. Park, H. Tanaka, G. Reddy, and N. Goodman "In-Context Learning Strategies Emerge Rationally"
C.F. Park, A. Lee, E.S. Lubana, Y. Yang, M. Okawa, K. Nishi, M. Wattenberg, H. Tanaka
"ICLR: In-Context Learning of Representations"
ICLR (International Conference on Learning Representations)
C.F. Park*, E.S. Lubana*, I. Pres, H. Tanaka
"Competition Dynamics Shape Algorithmic Phases of In-Context Learning"
ICLR (International Conference on Learning Representations)
Y. Yang, C.F. Park, E.S. Lubana, M. Okawa, W. Hu, H. Tanaka
"Dynamics of Concept Learning and Compositional Generalization"
ICLR (International Conference on Learning Representations)
E. S. Lubana*, K. Kawaguchi*, R. P. Dick, H. Tanaka
"A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language"
ICLR (International Conference on Learning Representations)
E. Bigelow, A. Holtzman, H. Tanaka*, T. Ullman*
"Forking Paths in Neural Text Generation"
ICLR (International Conference on Learning Representations)
F. Dinc*, E. Cirakman*, Y. Jiang, M. Yuksekgonul, M.J. Schnitzer*, H. Tanaka*
"A ghost mechanism: An analytical model of abrupt learning"
K. Nishi, M. Okawa, R. Ramesh, M. Khona, H. Tanaka*, E.S. Lubana*
"Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing"
R. Gould, H. Tanaka
"Continuous-Time Analysis of Adaptive Optimization and Normalization"
C. F. Park*, M. Okawa*, A. Lee, H. Tanaka*, E.S. Lubana*
"Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space"
NeurIPS (Advances in Neural Information Processing Systems) (2024)
"Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks"
R. Ramesh, E.S. Lubana, M. Khona, R.P. Dick, H. Tanaka
ICML (International Conference on Machine Learning) (2024)
"Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model"
M. Khona, M. Okawa, J. Hula, R. Ramesh, K. Nishi, R. Dick, E.S. Lubana*, H. Tanaka*
ICML (International Conference on Machine Learning) (2024)
"Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks"
S. Jain*, R. Kirk*, E.S. Lubana*, R.P. Dick, H. Tanaka, T. Rocktäschel, E. Grefenstette, D. Krueger
ICLR (International Conference on Learning Representations) (2024)
E.J. Bigelow, E.S. Lubana, R.P. Dick, H. Tanaka*, T.D. Ullman*
"In-Context Learning Dynamics with Random Binary Sequences"
ICLR (International Conference on Learning Representations) (2024)
M. Okawa*, E.S. Lubana*, R.P. Dick, H. Tanaka*
"Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task"
NeurIPS (Advances in Neural Information Processing Systems) (2023)
F. Dinc*, A. Shai*, M. Schnitzer, H. Tanaka
"CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics"
NeurIPS (Advances in Neural Information Processing Systems) (2023)
N. Maheswaranathan*, L.T. McIntosh*, H. Tanaka* (theoretical co-first author), S. Grant*, D.B. Kastner, J.B. Melander, A. Nayebi, L. Brezovec, J. Wang, S. Ganguli, S.A. Baccus
"Interpreting the retinal neural code for natural scenes: from computations to neurons"
Neuron (2023)
D. Kunin*, J. Sagastuy-Brena*, L. Gillespie, E. Margalit, H. Tanaka, S. Ganguli, D.L.K. Yamins
"Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations and anomalous diffusion"
Neural Computation (2023)
E.S. Lubana, E.J. Bigelow, R.P. Dick, D. Krueger, H. Tanaka
"Mechanistic Mode Connectivity"
ICML (International Conference on Machine Learning) (2023)
L. Ziyin, E.S. Lubana, M. Ueda, H. Tanaka
"What shapes the loss landscape of self-supervised learning?"
ICLR (International Conference on Learning Representations) (2023)
G. Reddy, L. Desban, H. Tanaka, J. Roussel, O. Mirat, C. Wyart (2022)
"A lexical approach for identifying behavioural action sequences"
E.S. Lubana, R.P. Dick, H. Tanaka
"Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning"
NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]
D. Kunin*, J. Sagastuy-Brena, S. Ganguli, D.L.K. Yamins, H. Tanaka*
"Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics"
ICLR (International Conference on Learning Representations) (2021) [tweet-print] [StanfordAI Blog]
H. Tanaka, D. Kunin
"Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks"
NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]
H. Tanaka*, D. Kunin*, D. Yaimns, S. Ganguli
"Pruning neural networks without any data by iteratively conserving synaptic flow"
NeurIPS (Advances in Neural Information Processing Systems) (2020) [pdf] [tweet-print]
H. Tanaka, A. Nayebi, N. Maheswaranathan, L. McIntosh, S.A. Baccus, S. Ganguli
"From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction"
NeurIPS (Advances in Neural Information Processing Systems) (2019)
H. Tanaka, D.R. Nelson
"Non-Hermitian quasi-localization and ring attractor neural networks"
Physical Review E, selected for the Editors' Suggestion (2018)
H. Tanaka, H.A. Stone, D.R. Nelson
"Spatial gene drives and pushed genetic waves"
PNAS (Proceedings of the National Academy of Sciences) (2017)
H. Tanaka, A.A. Lee, M.P. Brenner
"Hot particles attract in a cold bath"
Physical Review Fluids (2017)
H. Tanaka, Z. Zeravcic, M.P. Brenner
“Mutation at expanding front of self-replicating colloidal clusters”
Physical Review Letters, selected for the Editors' Suggestion (2016)
D. Shibata, H. Tanaka, S. Yonezawa, T. Nojima, Y. Maeno
"Quenched metastable vortex states in Sr2 RuO4"
Physical Review B (2015)
Publications:
Publications:
D. Wurgaft*, E.S. Lubana*, C.F. Park, H. Tanaka, G. Reddy, and N. Goodman "In-Context Learning Strategies Emerge Rationally"
C.F. Park, A. Lee, E.S. Lubana, Y. Yang, M. Okawa, K. Nishi, M. Wattenberg, H. Tanaka
"ICLR: In-Context Learning of Representations"
ICLR (International Conference on Learning Representations)
C.F. Park*, E.S. Lubana*, I. Pres, H. Tanaka
"Competition Dynamics Shape Algorithmic Phases of In-Context Learning"
ICLR (International Conference on Learning Representations)
Y. Yang, C.F. Park, E.S. Lubana, M. Okawa, W. Hu, H. Tanaka
"Dynamics of Concept Learning and Compositional Generalization"
ICLR (International Conference on Learning Representations)
E. S. Lubana*, K. Kawaguchi*, R. P. Dick, H. Tanaka
"A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language"
ICLR (International Conference on Learning Representations)
E. Bigelow, A. Holtzman, H. Tanaka*, T. Ullman*
"Forking Paths in Neural Text Generation"
ICLR (International Conference on Learning Representations)
F. Dinc*, E. Cirakman*, Y. Jiang, M. Yuksekgonul, M.J. Schnitzer*, H. Tanaka*
"A ghost mechanism: An analytical model of abrupt learning"
K. Nishi, M. Okawa, R. Ramesh, M. Khona, H. Tanaka*, E.S. Lubana*
"Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing"
R. Gould, H. Tanaka
"Continuous-Time Analysis of Adaptive Optimization and Normalization"
C. F. Park*, M. Okawa*, A. Lee, H. Tanaka*, E.S. Lubana*
"Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space"
NeurIPS (Advances in Neural Information Processing Systems) (2024)
"Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks"
R. Ramesh, E.S. Lubana, M. Khona, R.P. Dick, H. Tanaka
ICML (International Conference on Machine Learning) (2024)
"Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model"
M. Khona, M. Okawa, J. Hula, R. Ramesh, K. Nishi, R. Dick, E.S. Lubana*, H. Tanaka*
ICML (International Conference on Machine Learning) (2024)
"Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks"
S. Jain*, R. Kirk*, E.S. Lubana*, R.P. Dick, H. Tanaka, T. Rocktäschel, E. Grefenstette, D. Krueger
ICLR (International Conference on Learning Representations) (2024)
E.J. Bigelow, E.S. Lubana, R.P. Dick, H. Tanaka*, T.D. Ullman*
"In-Context Learning Dynamics with Random Binary Sequences"
ICLR (International Conference on Learning Representations) (2024)
M. Okawa*, E.S. Lubana*, R.P. Dick, H. Tanaka*
"Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task"
NeurIPS (Advances in Neural Information Processing Systems) (2023)
F. Dinc*, A. Shai*, M. Schnitzer, H. Tanaka
"CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics"
NeurIPS (Advances in Neural Information Processing Systems) (2023)
N. Maheswaranathan*, L.T. McIntosh*, H. Tanaka* (theoretical co-first author), S. Grant*, D.B. Kastner, J.B. Melander, A. Nayebi, L. Brezovec, J. Wang, S. Ganguli, S.A. Baccus
"Interpreting the retinal neural code for natural scenes: from computations to neurons"
Neuron (2023)
D. Kunin*, J. Sagastuy-Brena*, L. Gillespie, E. Margalit, H. Tanaka, S. Ganguli, D.L.K. Yamins
"Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations and anomalous diffusion"
Neural Computation (2023)
E.S. Lubana, E.J. Bigelow, R.P. Dick, D. Krueger, H. Tanaka
"Mechanistic Mode Connectivity"
ICML (International Conference on Machine Learning) (2023)
L. Ziyin, E.S. Lubana, M. Ueda, H. Tanaka
"What shapes the loss landscape of self-supervised learning?"
ICLR (International Conference on Learning Representations) (2023)
G. Reddy, L. Desban, H. Tanaka, J. Roussel, O. Mirat, C. Wyart (2022)
"A lexical approach for identifying behavioural action sequences"
E.S. Lubana, R.P. Dick, H. Tanaka
"Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning"
NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]
D. Kunin*, J. Sagastuy-Brena, S. Ganguli, D.L.K. Yamins, H. Tanaka*
"Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics"
ICLR (International Conference on Learning Representations) (2021) [tweet-print] [StanfordAI Blog]
H. Tanaka, D. Kunin
"Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks"
NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]
H. Tanaka*, D. Kunin*, D. Yaimns, S. Ganguli
"Pruning neural networks without any data by iteratively conserving synaptic flow"
NeurIPS (Advances in Neural Information Processing Systems) (2020) [pdf] [tweet-print]
H. Tanaka, A. Nayebi, N. Maheswaranathan, L. McIntosh, S.A. Baccus, S. Ganguli
"From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction"
NeurIPS (Advances in Neural Information Processing Systems) (2019)
H. Tanaka, D.R. Nelson
"Non-Hermitian quasi-localization and ring attractor neural networks"
Physical Review E, selected for the Editors' Suggestion (2018)
H. Tanaka, H.A. Stone, D.R. Nelson
"Spatial gene drives and pushed genetic waves"
PNAS (Proceedings of the National Academy of Sciences) (2017)
H. Tanaka, A.A. Lee, M.P. Brenner
"Hot particles attract in a cold bath"
Physical Review Fluids (2017)
H. Tanaka, Z. Zeravcic, M.P. Brenner
“Mutation at expanding front of self-replicating colloidal clusters”
Physical Review Letters, selected for the Editors' Suggestion (2016)
D. Shibata, H. Tanaka, S. Yonezawa, T. Nojima, Y. Maeno
"Quenched metastable vortex states in Sr2 RuO4"
Physical Review B (2015)
Videos:
Videos:
Google Sites
Report abuse