LLMs are trained on a self-supervised problem such as next token prediction, which enables them to acquire a form of “intelligence.” Our theoretical work will seek to identify the kind of internal representation that should be leveraged during the pre-training phase to further their intellectual ability.

Based on our theoretical analysis of efficient representation learning, we will propose a new learning methodology and a new architecture with the objective of going beyond the current form of foundation models

Capturing the epistemic uncertainty in LLMs has been recognized as a key component for ensuring the reliability of their outputs. We will explore the use of generalised uncertainty representations to better capture this type of uncertainty.