Task-Agnostic Benchmarking of Pretrained Representations using Synthetic Data

Abstract: 

Fine-tuning large models pretrained at scale on broad data for solving downstream tasks has made considerable success in recent years. There seems to be indeed an ongoing paradigm shift in deep learning from task-centric model design to task-agnostic representation learning and task-specific fine-tuning. Specifically, the representations of pretrained models are used as a foundation for different downstream tasks. We propose a new task-agnostic framework, SynBench, to measure the quality of pretrained representations using synthetic data. To address the challenge of task-agnostic data-free evaluation, we design synthetic binary classification proxy tasks with class-conditional Gaussian mixtures. This way we probe and compare the robustness-accuracy performance on pretrained representations and input synthetic data. SynBench offers a holistic quantitative evaluation, informs the model designers of the intrinsic performance, and spares efforts on task-specific finetuning with real-life data. Evaluated with various pretrained models, the experimental results show that SynBench score matches well the actual linear probing performance of the pretrained model when fine-tuned on downstream tasks using real-life data. Finally, SynBench can also be used in robust linear probing to mitigate the robustness-accuracy tradeoff in downstream tasks.