L&T24: Language and thought 

Goals

The goal of this topic area is to explore how cognitive-science inspired techniques can improve the performance, efficiency, creativity and evaluation of large foundational models. Large language models (LLMs) have achieved remarkable results on various natural language processing tasks, but they also face significant challenges and limitations, such as the high computational cost, the lack of generalization and robustness, the difficulty of incorporating or updating prior knowledge and reasoning, and the ethical and social implications of their use. This topic area aims to address these challenges by applying neuromorphic, brain-inspired computing principles that move the frontiers of today’s AI towards better power-efficiency, faster learning, low latency, and improved cognitive abilities. This topic area also connects from the area of neuromorphic computing and hardware engineering to other mainstream areas driving the future of artificial intelligence research.

Projects

• Biological brains are very efficient at processing structured representations derived from natural environments, whether it involves making sense out of complex visual scenes or deriving meaning from a sentence in a language we are learning. In this project we propose to explore integration of different types of structured representations (for example, Vector Symbolic Architectures) into contemporary AI models and algorithms. We will explore how representations that have been successful in modeling various cognitive functions can be incorporated into RL algorithms and LLMs.

• Exploring cognitive science inspired evaluations of frontier LLMs and Visual Language Models (VLMs), with possible applications in designing and testing self-reflection and self-rewarding mechanisms, such as meta-learning and intrinsic motivation, to enhance self-awareness, self-improvement, and self-regulation.

• Investigating and implementing neuromorphic techniques for LLMs, such as sparsity, synaptic plasticity, and neuromodulation, that can reduce their resource requirements, increase their adaptability, and facilitate their synchronization and coordination. This could be in the form of spiking neural networks (e.g. SpikeGPT) or by exploring how Mixture of Experts (MoE) models use very coarse forms of sparsity that could be informed by bio-inspired techniques.

• Studying and modeling memory consolidation processes in LLMs, such as rehearsal and sleep, that can improve their long-term retention and generalization of learned knowledge and skills. This could include exploring new techniques for developing and evaluating memory-augmented LLMs, such as Retrieval Augmented Generation, that can retrieve and integrate relevant information from external sources to enhance their performance on various tasks.

Materials, Equipment, and Tutorials:

Hardware: 

Software: 

Preparation material, literature:

The field of LLMs is extremely fast-moving but the organizers are directly working at the frontier of the field and will prepare preparatory and tutorial materials informed by the state-of-the-art in June 2024. The lectures will cover the background, concepts, and methods of the topic area, such as the basics of LLMs, cognitive science, and neuromorphic computing, and the open challenges and opportunities laid out above. There will be ample opportunities for hands-on learning thanks to potential cloud computing credits and the ability of organizers to provide locally accessible models that can be inferenced and updated on-premises. Due to a minimal reliance on on-site hardware it is also likely that this will be a remote-friendly portion of the workshop, and every effort will be made to engage virtual participants in working with the same models.