Abstract: The capabilities of Transformer-based models are far exceeding our expectations, and in many cases, our understanding of these models lags behind their success. In this talk, we will explore the hidden world of the internal representations learned by Transformer models, with a specific emphasis on the mechanism at the heart of their architecture: attention. We will begin by showcasing a novel approach that transforms attention maps into reliable explanations, shedding light on the reasoning behind the predictions of various Transformer models. Shifting gears, we will demonstrate how these interpretations serve not only as valuable visualization aids but also as powerful tools for controlling and correcting model behavior. We will demonstrate how applying intuitive desiderata to Vision Transformer (ViT) interpretation maps can mitigate critical issues such as reliance on biases and spurious correlations. Next, we will illustrate how the same approach enhances generative models, significantly improving text fidelity, all without the need for additional data and with very limited computational resources.

Abstract: Self-attention performs well in long context but have quadratic complexity. RNN layers have linear complexity, but their performance in long context is limited by the expressive power of their hidden state. We propose a new class of sequence modeling layers with linear complexity and an expressive hidden state. The key idea is to make the hidden state a machine learning model itself, and the update rule a step of self-supervised learning. Since the hidden state is updated by training itself even on test sequences, our new class of layers is called TTT layers, where TTT stands for test-time training. We consider two simple instantiations: TTT-Linear and TTT-MLP, whose hidden state is a linear model and a two-layer MLP respectively. We evaluate our instantiations at the scale of 125M to 1.3B parameters, comparing with a state-of-the-art Transformer and one of the most advanced RNNs called Mamba. TTT-Linear and TTT-MLP scale better than the baselines. They can also effectively take advantage of 32k context like Transformers, while Mamba cannot. With preliminary systems optimization, TTT-Linear is already faster than Transformers and Mamba in wall-clock time at 2k context length. TTT-MLP still faces challenges in memory I/O, but shows larger potential in long context, pointing to a promising direction for future research.

Abstract: Large-scale training of self-supervised transformers has allowed us to obtain robust data representations on nearly any domain. Features computed with DINOv2 show solid performance on many benchmarks, meeting the quality of CLIP-like models on categorization tasks and setting a new bar for dense prediction ones (segmentation, depth estimation). At the same time, those models show outstanding out-of-domain robustness, allowing one to run inference on drastically different inputs. In this talk, I will present two recent advances in this domain. First, I will discuss the surprising effect of using additional learnable input tokens (registers) to alleviate artifacts in feature maps. We observed that such tokens allow to obtain clear feature maps, for models trained with DINO, but also CLIP or even supervised criterions. Second, I will present our recent work challenging the importance of data augmentation for training self-supervised transformers. Our study shows that previous conclusions don’t necessarily hold as long as one trains models on sufficiently large datasets.