We have moved to this new website: https://physicsintelligence.org/

News:

Jan. 2025: 5 works accepted at ICLR 2025!

Nov. 2024: Joining MATS (ML Alignment & Theory Scholars) program as a mentor

Sep. 2024: A paper accepted at NeurIPS Spotlight

Apr. 2024: Excited to establish new "CBS-NTT Program in Physics of Intelligence" at Harvard's Center for Brain Science!

Press releases: Harvard Gazette, Detailed Coverage, Harvard Crimson

Sep. 2023: 2 papers accepted at NeurIPS main conference, 5 papers accepted at NeurIPS workshops!

Mar. 2023: a paper accepted at ICML

Jan. 2023: a paper accepted at ICLR

Nov. 2022: Talk at CMSA Colloquium, Center of Mathematical Sciences and Applications, Harvard University

Sep. 2022: Talk at Harvard ML Foundations Seminar

Mar. 2022: Our group has physically relocated to the Center for Brain Science at Harvard University for an industry-academia collaboration.

• • Ekdeep Singh Lubana, PhD student (thesis co-advisor), EECS, U of Michigan

  • Sam Bright-Thonney, IAIFI PostDoc Fellow, Physics, MIT

  • Fatih Dinc, PhD student, Applied Physics, Stanford

  • Maya Okawa, Visiting Research Scientist, NTT

  • Bo Zhao, PhD student, Computer Science, UCSD

  • Eric Bigelow, PhD student, Psychology, Harvard

  • Core Francisco Park, PhD student, Physics, Harvard

  • Bhavya Vasudeva, PhD student, CS, University of Southern California

  • Yongyi Yang, PhD student, CSE, U of Michigan

  • Mikail Khona, PhD student, Physics, MIT

  • Rahul Ramesh, PhD student, CS, U of Pennsylvania

  • Kento Nishi, BA & MS student, CS, Harvard

• Ziyin Liu, PhD student, Physics, U of Tokyo

• Max Aalto, PhD student, EECS, MIT

• Daniel Kunin, PhD student, Computational & Mathematical Engineering, Stanford

Former colleagues in Intelligent Systems/Neural Network Group, NTT PHI Lab:

• Gautam Reddy, Assistant Professor, Physics, Princeton University

• Logan G. Wright, Assistant Professor, Applied Physics, Yale University

Our NTT Research at Harvard Group is young and growing! Please feel free to contact us for discussions, collaborations, and internships. Visit our website to learn more about the NTT Physics & Informatics Lab.

Hidenori's Brief CV:

2020 - : Group Leader & Senior Research Scientist, NTT Physics & Informatics Laboratories, CA, USA

2022 - : Associate, the Center for Brain Science, Harvard University, MA, USA

2024 - : Affiliate, IAIFI at Massachusetts Institute of Technology (MIT), MA, USA

2023 - : Visiting Researcher, Institute for Physics of Intelligence, the University of Tokyo, Japan

2018 - 2019: Masason Postdoctoral Fellow, Stanford University advisors: Surya Ganguli and Daniel Fisher

2014 - 2018: Ph.D. & M.S. in Applied Physics, Harvard University advisors: David Nelson and Michael Brenner

2010 - 2014: B.S. in Physics, Kyoto University, Kyoto, Japan

Born and grew up in Tokyo, Japan.

Publications:

D. Wurgaft*, E.S. Lubana*, C.F. Park, H. Tanaka, G. Reddy, and N. Goodman "In-Context Learning Strategies Emerge Rationally"

arXiv

C.F. Park, A. Lee, E.S. Lubana, Y. Yang, M. Okawa, K. Nishi, M. Wattenberg, H. Tanaka

"ICLR: In-Context Learning of Representations"

ICLR (International Conference on Learning Representations)

C.F. Park*, E.S. Lubana*, I. Pres, H. Tanaka

"Competition Dynamics Shape Algorithmic Phases of In-Context Learning"

ICLR (International Conference on Learning Representations)

Y. Yang, C.F. Park, E.S. Lubana, M. Okawa, W. Hu, H. Tanaka

"Dynamics of Concept Learning and Compositional Generalization"

ICLR (International Conference on Learning Representations)

E. S. Lubana*, K. Kawaguchi*, R. P. Dick, H. Tanaka

"A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language"

ICLR (International Conference on Learning Representations)

E. Bigelow, A. Holtzman, H. Tanaka*, T. Ullman*

"Forking Paths in Neural Text Generation"

ICLR (International Conference on Learning Representations)

F. Dinc*, E. Cirakman*, Y. Jiang, M. Yuksekgonul, M.J. Schnitzer*, H. Tanaka*

"A ghost mechanism: An analytical model of abrupt learning"

arXiv

K. Nishi, M. Okawa, R. Ramesh, M. Khona, H. Tanaka*, E.S. Lubana*

"Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing"

arXiv

R. Gould, H. Tanaka

"Continuous-Time Analysis of Adaptive Optimization and Normalization"

arXiv

C. F. Park*, M. Okawa*, A. Lee, H. Tanaka*, E.S. Lubana*

"Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space"

NeurIPS (Advances in Neural Information Processing Systems) (2024)

"Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks"

R. Ramesh, E.S. Lubana, M. Khona, R.P. Dick, H. Tanaka

ICML (International Conference on Machine Learning) (2024)

"Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model"

M. Khona, M. Okawa, J. Hula, R. Ramesh, K. Nishi, R. Dick, E.S. Lubana*, H. Tanaka*

ICML (International Conference on Machine Learning) (2024)

"Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks"

S. Jain*, R. Kirk*, E.S. Lubana*, R.P. Dick, H. Tanaka, T. Rocktäschel, E. Grefenstette, D. Krueger

ICLR (International Conference on Learning Representations) (2024)

E.J. Bigelow, E.S. Lubana, R.P. Dick, H. Tanaka*, T.D. Ullman*

"In-Context Learning Dynamics with Random Binary Sequences"

ICLR (International Conference on Learning Representations) (2024)

M. Okawa*, E.S. Lubana*, R.P. Dick, H. Tanaka*

"Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task"

NeurIPS (Advances in Neural Information Processing Systems) (2023)

F. Dinc*, A. Shai*, M. Schnitzer, H. Tanaka

"CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics"

NeurIPS (Advances in Neural Information Processing Systems) (2023)

N. Maheswaranathan*, L.T. McIntosh*, H. Tanaka* (theoretical co-first author), S. Grant*, D.B. Kastner, J.B. Melander, A. Nayebi, L. Brezovec, J. Wang, S. Ganguli, S.A. Baccus

"Interpreting the retinal neural code for natural scenes: from computations to neurons"

Neuron (2023)

D. Kunin*, J. Sagastuy-Brena*, L. Gillespie, E. Margalit, H. Tanaka, S. Ganguli, D.L.K. Yamins

"Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations and anomalous diffusion"

Neural Computation (2023)

E.S. Lubana, E.J. Bigelow, R.P. Dick, D. Krueger, H. Tanaka

"Mechanistic Mode Connectivity"

ICML (International Conference on Machine Learning) (2023)

L. Ziyin, E.S. Lubana, M. Ueda, H. Tanaka

"What shapes the loss landscape of self-supervised learning?"

ICLR (International Conference on Learning Representations) (2023)

G. Reddy, L. Desban, H. Tanaka, J. Roussel, O. Mirat, C. Wyart (2022)

"A lexical approach for identifying behavioural action sequences"

PLoS Computational Biology

E.S. Lubana, R.P. Dick, H. Tanaka

"Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning"

NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]

D. Kunin*, J. Sagastuy-Brena, S. Ganguli, D.L.K. Yamins, H. Tanaka*

"Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics"

ICLR (International Conference on Learning Representations) (2021) [tweet-print] [StanfordAI Blog]

H. Tanaka, D. Kunin

"Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks"

NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]

H. Tanaka*, D. Kunin*, D. Yaimns, S. Ganguli

"Pruning neural networks without any data by iteratively conserving synaptic flow"

NeurIPS (Advances in Neural Information Processing Systems) (2020) [pdf] [tweet-print]

H. Tanaka, A. Nayebi, N. Maheswaranathan, L. McIntosh, S.A. Baccus, S. Ganguli

"From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction"

NeurIPS (Advances in Neural Information Processing Systems) (2019)

H. Tanaka, D.R. Nelson

"Non-Hermitian quasi-localization and ring attractor neural networks"

Physical Review E, selected for the Editors' Suggestion (2018)

H. Tanaka, H.A. Stone, D.R. Nelson

"Spatial gene drives and pushed genetic waves"

PNAS (Proceedings of the National Academy of Sciences) (2017)

H. Tanaka, A.A. Lee, M.P. Brenner 

"Hot particles attract in a cold bath"

Physical Review Fluids (2017)

H. Tanaka, Z. Zeravcic, M.P. Brenner

“Mutation at expanding front of self-replicating colloidal clusters”

Physical Review Letters, selected for the Editors' Suggestion (2016)

D. Shibata, H. Tanaka, S. Yonezawa, T. Nojima, Y. Maeno

"Quenched metastable vortex states in Sr2 RuO4"

Physical Review B (2015)

Publications:

D. Wurgaft*, E.S. Lubana*, C.F. Park, H. Tanaka, G. Reddy, and N. Goodman "In-Context Learning Strategies Emerge Rationally"

arXiv

C.F. Park, A. Lee, E.S. Lubana, Y. Yang, M. Okawa, K. Nishi, M. Wattenberg, H. Tanaka

"ICLR: In-Context Learning of Representations"

ICLR (International Conference on Learning Representations)

C.F. Park*, E.S. Lubana*, I. Pres, H. Tanaka

"Competition Dynamics Shape Algorithmic Phases of In-Context Learning"

ICLR (International Conference on Learning Representations)

Y. Yang, C.F. Park, E.S. Lubana, M. Okawa, W. Hu, H. Tanaka

"Dynamics of Concept Learning and Compositional Generalization"

ICLR (International Conference on Learning Representations)

E. S. Lubana*, K. Kawaguchi*, R. P. Dick, H. Tanaka

"A Percolation Model of Emergence: Analyzing Transformers Trained on a Formal Language"

ICLR (International Conference on Learning Representations)

E. Bigelow, A. Holtzman, H. Tanaka*, T. Ullman*

"Forking Paths in Neural Text Generation"

ICLR (International Conference on Learning Representations)

F. Dinc*, E. Cirakman*, Y. Jiang, M. Yuksekgonul, M.J. Schnitzer*, H. Tanaka*

"A ghost mechanism: An analytical model of abrupt learning"

arXiv

K. Nishi, M. Okawa, R. Ramesh, M. Khona, H. Tanaka*, E.S. Lubana*

"Representation Shattering in Transformers: A Synthetic Study with Knowledge Editing"

arXiv

R. Gould, H. Tanaka

"Continuous-Time Analysis of Adaptive Optimization and Normalization"

arXiv

C. F. Park*, M. Okawa*, A. Lee, H. Tanaka*, E.S. Lubana*

"Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Space"

NeurIPS (Advances in Neural Information Processing Systems) (2024)

"Compositional Capabilities of Autoregressive Transformers: A Study on Synthetic, Interpretable Tasks"

R. Ramesh, E.S. Lubana, M. Khona, R.P. Dick, H. Tanaka

ICML (International Conference on Machine Learning) (2024)

"Towards an Understanding of Stepwise Inference in Transformers: A Synthetic Graph Navigation Model"

M. Khona, M. Okawa, J. Hula, R. Ramesh, K. Nishi, R. Dick, E.S. Lubana*, H. Tanaka*

ICML (International Conference on Machine Learning) (2024)

"Mechanistically analyzing the effects of fine-tuning on procedurally defined tasks"

S. Jain*, R. Kirk*, E.S. Lubana*, R.P. Dick, H. Tanaka, T. Rocktäschel, E. Grefenstette, D. Krueger

ICLR (International Conference on Learning Representations) (2024)

E.J. Bigelow, E.S. Lubana, R.P. Dick, H. Tanaka*, T.D. Ullman*

"In-Context Learning Dynamics with Random Binary Sequences"

ICLR (International Conference on Learning Representations) (2024)

M. Okawa*, E.S. Lubana*, R.P. Dick, H. Tanaka*

"Compositional Abilities Emerge Multiplicatively: Exploring Diffusion Models on a Synthetic Task"

NeurIPS (Advances in Neural Information Processing Systems) (2023)

F. Dinc*, A. Shai*, M. Schnitzer, H. Tanaka

"CORNN: Convex optimization of recurrent neural networks for rapid inference of neural dynamics"

NeurIPS (Advances in Neural Information Processing Systems) (2023)

N. Maheswaranathan*, L.T. McIntosh*, H. Tanaka* (theoretical co-first author), S. Grant*, D.B. Kastner, J.B. Melander, A. Nayebi, L. Brezovec, J. Wang, S. Ganguli, S.A. Baccus

"Interpreting the retinal neural code for natural scenes: from computations to neurons"

Neuron (2023)

D. Kunin*, J. Sagastuy-Brena*, L. Gillespie, E. Margalit, H. Tanaka, S. Ganguli, D.L.K. Yamins

"Rethinking the limiting dynamics of SGD: modified loss, phase space oscillations and anomalous diffusion"

Neural Computation (2023)

E.S. Lubana, E.J. Bigelow, R.P. Dick, D. Krueger, H. Tanaka

"Mechanistic Mode Connectivity"

ICML (International Conference on Machine Learning) (2023)

L. Ziyin, E.S. Lubana, M. Ueda, H. Tanaka

"What shapes the loss landscape of self-supervised learning?"

ICLR (International Conference on Learning Representations) (2023)

G. Reddy, L. Desban, H. Tanaka, J. Roussel, O. Mirat, C. Wyart (2022)

"A lexical approach for identifying behavioural action sequences"

PLoS Computational Biology

E.S. Lubana, R.P. Dick, H. Tanaka

"Beyond BatchNorm: Towards a Unified Understanding of Normalization in Deep Learning"

NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]

D. Kunin*, J. Sagastuy-Brena, S. Ganguli, D.L.K. Yamins, H. Tanaka*

"Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics"

ICLR (International Conference on Learning Representations) (2021) [tweet-print] [StanfordAI Blog]

H. Tanaka, D. Kunin

"Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks"

NeurIPS (Advances in Neural Information Processing Systems) (2021) [tweet-print]

H. Tanaka*, D. Kunin*, D. Yaimns, S. Ganguli

"Pruning neural networks without any data by iteratively conserving synaptic flow"

NeurIPS (Advances in Neural Information Processing Systems) (2020) [pdf] [tweet-print]

H. Tanaka, A. Nayebi, N. Maheswaranathan, L. McIntosh, S.A. Baccus, S. Ganguli

"From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction"

NeurIPS (Advances in Neural Information Processing Systems) (2019)

H. Tanaka, D.R. Nelson

"Non-Hermitian quasi-localization and ring attractor neural networks"

Physical Review E, selected for the Editors' Suggestion (2018)

H. Tanaka, H.A. Stone, D.R. Nelson

"Spatial gene drives and pushed genetic waves"

PNAS (Proceedings of the National Academy of Sciences) (2017)

H. Tanaka, A.A. Lee, M.P. Brenner 

"Hot particles attract in a cold bath"

Physical Review Fluids (2017)

H. Tanaka, Z. Zeravcic, M.P. Brenner

“Mutation at expanding front of self-replicating colloidal clusters”

Physical Review Letters, selected for the Editors' Suggestion (2016)

D. Shibata, H. Tanaka, S. Yonezawa, T. Nojima, Y. Maeno

"Quenched metastable vortex states in Sr2 RuO4"

Physical Review B (2015)

Videos: