Tamara Broderick

Associate Professor

MIT

Dropping Just a Handful of Preferences Can Change Top Large Language Model Rankings

Practitioners will often analyze a data set with the goal of applying any conclusions to a new population. For instance, one might hope that the top large language model (LLM) in a widely-used ranking system would outperform other ranked LLMs on future tasks. Typically the tasks that comprise the data used for ranking are not a perfect random sample from the population of future tasks --- but researchers might feel comfortable generalizing anyway so long as deviations from random sampling are small, and the corresponding impact on conclusions is small as well. Conversely, researchers might worry if a very small proportion of the data set was instrumental to the original conclusion (e.g. which LLM is top ranked). So we propose a method to assess the sensitivity of conclusions to the removal of a very small fraction of the data set. Manually checking all small data subsets is computationally infeasible, so we propose an approximation based on the classical influence function. Our method is automatically computable for common estimators, including those used in popular LLM rankings. We provide error bounds on approximation performance and a low-cost exact lower bound on sensitivity. We find that sensitivity is driven by a signal-to-noise ratio in the inference problem, does not disappear as data accrues, and is not decided by misspecification. Empirically we find that many data analysis conclusions are robust, but dropping just 0.003% of human preferences can change the top-ranked model on Chatbot Arena.

Bio

Tamara Broderick is an Associate Professor in Electrical Engineering and Computer Science at MIT, where she is a member of LIDS and IDSS. Her research focuses on the foundations of Bayesian inference, uncertainty quantification, and scalable, interpretable machine learning. She develops methods to rigorously characterize uncertainty and improve the reliability of data-driven decisions in complex models. Her contributions have been recognized with the NSF CAREER Award, the ONR Young Investigator Award, and the COPSS Emerging Leader Award. Tamara earned her Ph.D. in Statistics from UC Berkeley and her A.B. in Mathematics from Princeton University.

Mengdi Wang

Professor

Princeton

LabOS: The AI-XR Co-Scientist That Sees and Works With Humans

Modern science advances fastest when thought meets action. LabOS represents the first AI co-scientist that unites computational reasoning with physical experimentation through multimodal perception, self-evolving agents, and Extended-Reality(XR)-enabled human-AI collaboration. By connecting multi-model AI agents, smart glasses, and robots, LabOS allows AI to see what scientists see, understand experimental context, and assist in real-time execution. Across applications -- from cancer immunotherapy target discovery to stem-cell engineering and material science -- LabOS shows that AI can move beyond computational design to participation, turning the laboratory into an intelligent, collaborative environment where human and machine discovery evolve together.

Bio

Mengdi Wang is Co-Director of Princeton AI for Accelerated Invention, and Professor of the Department of Electrical and Computer Engineering and the Center for Statistics and Machine Learning at Princeton University. Her research focuses on machine learning, reinforcement learning, generative AI, large language models, and AI for science. Mengdi received her PhD in Electrical Engineering and Computer Science from Massachusetts Institute of Technology in 2013, where she was affiliated with the Laboratory for Information and Decision Systems and advised by Dimitri P. Bertsekas. She serves as a Program Chair for ICLR 2023 and Senior AC for Neurips, ICML, COLT, associate editor for Harvard Data Science Review, Operations Research. Her research is supported by NSF, AFOSR, NIH, ONR, Google, Microsoft C3.ai, FinUP, RVAC Medicines, MURI, GenMab.

Coz Ududec

UK AI Safety Institute

From Scores to Process Evidence: Automated Information Triage for Long-Horizon Agent Evaluations

Long-horizon agent evaluations are increasingly important for safety and capability assessment, but they strain current benchmarking practice. Runs are costly, sample sizes are small, and transcripts are too large for consistent expert review. Final outcomes may also miss key evidence about progress, failure modes, shortcut use, and harness effects.

I present Automated Information Triage (AIT), a method and review interface for turning long agent trajectories into expert reviewer-facing process evidence. AIT starts with expert calibration of the evaluation claim, success criteria, milestones, artefacts, and review rubrics. It then uses transcript scanners, artefact analysis, and LLM-judge enrichment to produce key decision points, run narratives, process signals, and source-linked evidence.

Using insights from an AI R&D case study, I discuss open questions around signal validity, transfer across tasks, and the boundary between reviewer assistance and automated judgement.

Bio

Cozmin Ududec leads the Science of Evaluation team at the UK AI Security Institute in London. His work focuses on methods for evaluating frontier AI systems and making stronger empirical claims from evaluation results, including evaluation validity, log analysis, inference scaling, LLM personas, and changes in propensities and capabilities over long-horizon tasks. He joined AISI early in its life and previously co-led its pre-deployment testing programme. He holds a PhD in physics from the University of Waterloo and the Perimeter Institute.