LT25: Language and Thought

Topic leaders

Steve Abreu (University of Groningen) s.abreu@rug.nl
Nicole Dumont (University of Waterloo) ns2dumon@uwaterloo.ca

Co-organizers and invited speakers:

Guido Zarrella (MITRE) guido@mitre.org
Ivana Kajić (Google DeepMind) kivana@google.com
Alessandro Pierro (Intel Labs) alessandro.pierro@intel.com
Johannes von Oswald (Google Research) jvoswald@google.com
Anand Subramoney (Royal Holloway, University of London) anand.subramoney@rhul.ac.uk
Emre Neftci (FZ Juelich) e.neftci@fz-juelich.de
Chris Eliasmith (University of Waterloo) celiasmith@uwaterloo.ca

Goals

With this topic area, we explore how neuromorphic principles can enhance the performance, efficiency, and robustness of state-of-the-art machine learning models. While foundation models excel in many tasks, they face significant challenges, such as high computational cost, hallucinations, lack of continual learning and knowledge editing, and limited reasoning abilities.

We aim to bridge disciplines and collaborate between fields - mainstream machine learning, robotics, neuromorphic engineering, neuroscience, cognitive science, and psychology - by focusing on sparsity and always-on reasoning as convergence points.

Neuromorphic hardware offers energy-efficient, adaptive alternatives to GPUs/TPUs, enabling scalable and sustaina AI systems for real-world tasks. Participants will collaborate to develop models leveraging neuromorphic hardware for efficient training, inference, and applications.

We will also connect sparsity to interpretability, using sparse autoencoders as tools for mechanistic interpretability while drawing parallels to neuroscience methods for functional understanding of complex systems. Insights from cognitive science, such as human reasoning mechanisms, will complement these efforts, synergizing with neuromorphic techniques to create adaptable, robust AI whose inner workings we can analyze and intervene on.

Cognitive science and always-on learning for improved reasoning

Improve reasoning abilities of neuromorphic language models through adaptive computation, always-on continual learning, and efficient use of resources during inference-time reasoning, in ways inspired by OpenAI’s recent o1 model.
Explore agent interactions to shape behaviors through online learning with multi-agent reinforcement learning (MARL) principles like cooperation, competition, and communication, to learn collaboratively and improve reasoning.

Edge intelligence: sparse state space models for real-time applications

Use sparse state space models (SSMs) to solve real-time edge tasks like keyword spotting or robotic control at the frontier of efficiency and performance
Train and deploy ultra-sparse SSMs on neuromorphic hardware on novel applications, to showcase neuromorphic computing’s potential for energy-efficient, adaptable edge systems

Sparse insights: enhancing interpretability through dynamic sparsity

Improve our understanding of how LLMs make decisions, for greater transparency and accountability, using concepts from neuroscience (sparse coding) and cognitive science (feature binding).
Use sparse autoencoders (e.g., GemmaScope) to analyze the structure of representations in large foundation models and isolate relevant features.
Investigate compositionality in LLM representations using sparse autoencoders – identifying and distinguishing linear composition, tensor product representations, etc. – and compare these to binding operations in vector symbolic architectures and other cognitive science models.

Bridging scale and efficiency

Use knowledge distillation techniques to transfer capabilities of open-source LLMs (e.g., Llama-3, Gemma-2) into neuromorphic LLMs and SSMs, enabling the power of state-of-the-art AI in energy-efficient, scalable systems optimized real-world applications

Always-on continual learning with neuromorphic principles

Integrate in-context learning (ICL) or Low Rank Adaptation (LoRA) for state space models (SSMs) with biologically plausible mechanisms such as local three-factor learning rules.
Apply to continual learning tasks (e.g., learning from streaming data, control in dynamics environments) to demonstrate real-time adaptation and efficient inference in spiking AI systems.

Participants will have access to a variety of tools, including Lava-DL, Nengo, NIR, snnTorch, JAX, PyTorch, TensorRT. Topic leaders have experience in these tools and will provide tutorials and guidance.

In addition, multiple codebases will be provided as starting points, for example SSMs with event-based data, sparse auto-encoders with GemmaScope, sparse neural networks for Loihi 2 using lava-dl, and two new codebases will be released before the workshop for (1) sparse and quantized SSMs optimized for neuromorphic hardware and (2) a neuromorphic language model based on the matmul-free LLM running on a multi-chip Loihi 2 system.

LT25: Language and Thought

Topic leaders

Co-organizers and invited speakers:

Goals

Recommended reading