The first webinar of Season 13 will start on September 7, 2026 at 20:00 GMT.
CHECK THE UPCOMING EVENTS TOWARDS THE END OF THIS PAGE!
Abstract:
Large language models (LLMs) struggle with reliable rule-based reasoning, particularly in mathematical domains requiring precise computation. Recent neurosymbolic approaches have shown promise by encoding LLM hidden states into neurosymbolic representations to perform rule-based reasoning, but current methods rely on linear encoders applied to a single token’s hidden state, limiting representational capacity and robustness to input variations. This work extends previous neurosymbolic LLM methods by replacing linear encoders with transformer-based architectures that encode multiple hidden states from the model’s forward pass, enabling the model to capture information distributed across longer input sequences. We evaluate the method on arithmetic prompts in a low-data 3-digit setting with varied question formats and in 4/6-digit scaling experiments. The transformer encoder achieves 93.1% average accuracy, compared to 53.6% for the linear encoder, and remains substantially more robust as input length increases.
Abstract:
Disentanglement, the separation of factors of variation in data using neural networks, remains a long-standing challenge in machine learning. Prior work has addressed this problem with variational autoencoders and generative adversarial networks that incorporate ideas from variational inference and information-theoretic constraints. In contrast to methods that rely on continuous representations, we propose a design that treats disentangled representations as symbolic structures, motivated by the compositional relationships among the concepts that make up samples from a distribution. However, learning discrete symbolic structures with neural networks while maintaining differentiability is difficult and often requires complex architectures. To address this, we introduce an unsupervised learning algorithm that uses holographic reduced representations (HRR) for neural disentanglement. We show that the HRR unbinding operation provides an inductive bias for separating factors and yields competitive results against baselines, as measured by latent traversals and disentanglement metrics. We complement these empirical findings with an information-theoretic analysis of the HRR unbinding channel. We prove that unbinding induces approximately independent symbol-value pairs and derive a per-slot capacity bound that quantifies how many distinct symbolic concepts can be reliably encoded, giving a quantitative account of the inductive bias toward disentanglement. The resulting representations differ from standard autoencoder-based models, in that their latent units are vectors that are summed together, rather than scalar dimensions of a low-dimensional latent vector. We show that this HRR representation is more robust to noise than other disentangled representations and maintains reconstruction quality across a range of SNRs.
Abstract:
In this talk, we show the possibility of a direct injection of algorithms into neural network architecture. We focus on a complex algorithm, that is, CockeYouger-Kasami (CYK) for parsing context-free grammars in Chomsky Normal Form and we propose CYKNN, a simple recurrent neural network architecture for encoding the CYK algorithm in trainable matrix-vector multiplications.We experimented with a very simple grammar with 4 variations showing that our approach outperforms existing LLMs with more than 20B parameters with an in-context learning setting and smaller LLMs of the Qwen family fine-tuned with LoRA. Our attempt paves the way to a different approach to neuro-symbolic methodologies.
Abstract:
TBA
Abstract:
In visual scene understanding tasks, it is essential to capture both invariant and equivariant structure. While neural networks are frequently trained to achieve invariance to transformations such as translation, this often comes at the cost of losing access to equivariant information - e.g., the precise location of an object. Moreover, invariance is not naturally guaranteed through supervised learning alone, and many architectures generalize poorly to input transformations not encountered during training. Here, we take an approach based on analysis-by-synthesis and factoring using resonator networks. A generative model describes the construction of simple scenes containing MNIST digits and their transformations, like color and position. The resonator network inverts the generative model, and provides both invariant and equivariant information about particular objects. Sparse features learned from training data act as a basis set to provide flexibility in representing variable shapes of objects, allowing the resonator network to handle previously unseen digit shapes from the test set. The modular structure provides a shape module which contains information about the object shape with translation factored out, allowing a simple classifier to operate on centered digits. The classification layer is trained solely on centered data, requiring much less training data, and the network as a whole can identify objects with arbitrary translations without data augmentation. The natural attention-like mechanism of the resonator network also allows for analysis of scenes with multiple objects, where the network dynamics selects and centers only one object at a time. Further, the specific position information of a particular object can be extracted from the translation module, and we show that the resonator can be designed to track multiple moving objects with precision of a few pixels.
Abstract:
The brain faces the feature binding problem: how are multiple stimulus features and variables combined into coherent representations that support flexible behavior? A key finding from neuroscience is that some brain regions employ factorized representations, where distinct features are encoded in neural state space in such a way that enables independent readout and robust generalization. Various algebraic operations have been proposed to model multi-variable representations, but despite extensive study of their theoretical properties (e.g., capacity, noise robustness), it remains unclear which operations produce the representational geometries observed in neural recordings. We systematically evaluate six binding operations implemented in recurrent spiking neural networks performing a working memory task. We find that only superposition and binding with slot-filler structure produce factorized geometry with favorable scaling, while the alternatives do not. These results provide a taxonomy linking algebraic binding operations to neural representational signatures, offering guidance for both computational modelers and experimentalists.
Abstract: Distributional and neural approaches to natural language semantics have been built almost exclusively on conventional linear algebra: vectors, matrices, tensors, and the operations that accompany them. These methods have achieved remarkable empirical success, yet they face persistent structural limitations in compositional semantics, type sensitivity, and interpretability. I argue in this paper that geometric algebra (GA) – specifically, Clifford algebras – provides a mathematically superior foundation for semantic representation, and that a Functional Geometric Algebra (FGA) framework extends GA toward a typed, compositional semantics capable of supporting inference, transformation, and interpretability while retaining full compatibility with distributional learning and modern neural architectures. I develop the formal foundations, identify three core capabilities that GA provides and linear algebra does not, present a detailed worked example illustrating operator-level semantic contrasts, and show how GA-based operations already implicit in current transformer architectures can be made explicit and extended. The central claim is not merely increased dimensionality but increased structural organization: GA expands an n-dimensional embedding space into a 2 n multivector algebra where base semantic concepts and their higher-order interactions are represented within a single, principled algebraic framework
Abstract:
Humans cannot always intuit what scenarios are most challenging to LLMs. Hoping to capture challenging edge cases, developers either design problems to be difficult for humans or curate extensive benchmarks. What if we could instead anticipate which scenarios a model will fail on? In this paper, we use an LLM's representational geometry to predict which concept combinations it will fail on. We attribute this compositional failure to interference between salient features. In tasks that require systematic composition - toy programmatic settings, multihop reasoning, multilingual factual recall - we find that when a pair of concepts is encoded near-orthogonally, the model reliably composes them. When their linear encodings are close, producing interference, the model fails to compose them. Our method reliably anticipates failure modes across different compositional tasks, without evaluating specific inputs. These results lay the groundwork to use representational geometry to identify high-risk examples, construct targeted stress tests, and provide a scalable foundation for active learning in real-world deployment.
Abstract:
TBA