Abstract: To address knowledge-rich tasks such as question answering, NLP models should combine knowledge from multiple sources – memorized knowledge in the language model (LM), and passages retrieved from an evidence corpus. Prior work assumed information stored in various sources is consi- stent with each other. Yet, a mismatch in knowledge sources is common in the real world: some sources are updated while others remain stale, and different sources can interpret the same question differently or propose differing opinions. How should we resolve such complex knowledge conflicts? In this talk, I will describe our recent work on (1) an evaluation framework for updating knowledge in LMs and (2) how state-of-the-art models behave under knowledge conflicts. I will conclude my talk with paths for handling real-world scenarios, continual learning of models and calibrating them to avoid answering when provided with incomplete or conflicting information.

Bio: Eunsol Choi is an assistant professor in the Computer Science department at the University of Texas at Austin and a visiting researcher at Google AI. Her research area spans natural language processing and machine learning. She is particularly interested in interpreting and reasoning about text in a rich real- world context. She received a Ph.D. from University of Washington and B.A from Cornell University. She is a recipient of Facebook research fellowship, Google faculty research award, and outstanding paper award at EMNLP 2021.
https://www.cs.utexas.edu/~eunsol/

Abstract: Language changes are shaped by world events and social structures. With an increasingly rich set of resources for studying text over time, this raises the possibility of reverse-engineering language change to uncover new insights about the world. This talk will survey work on diachronic corpora of social media, historical newspapers, and scientific research papers. We use these corpora to build models of cultural transmission between US cities, the effects of social media platforms’ policies and norms, the leaders of the movement to abolish slavery in the United States, and which ACL papers are likely to get the most citations in the future. Collaborators include Sandeep Soni, Umashanthi Pavalanathan, Lauren F. Klein, Kristina Lerman, and David Bamman.

Bio: Jacob Eisenstein is a research scientist at Google, where he is focused on making language technolo- gy more robust and trustworthy. He was previously on the faculty of the Georgia Institute of Technology, where he supervised six successful doctoral dissertations, received the NSF CAREER Award for research on computational sociolinguistics, and wrote a textbook on natural language processing. He completed his Ph.D. at MIT, winning the George M. Sprowls award for a dissertation on computational models of speech and gesture. Thanks to his brief appearance in the documentary film If These Knishes Could Talk, Jacob has a Bacon number of 2.
https://jacobeisenstein.github.io/

Abstract: The field of automatic question answering has been rapidly advancing recently. The existing QA approaches are however generally working on synchronic document collections such as Wikipedia, Web data or short-term news corpora. This talk is about our latest efforts in automatic question answe- ring over diachronic news collections composed of articles published over several decades. We will first discuss an unsupervised re-ranking approach that works by utilizing temporal information embedded in the temporal document collection. Next, we will introduce a solution for finding the occurrence dates of events described in input questions based on the underlying news dataset. Finally, we will introduce ArchivalQA - a large-scale question answering dataset which has been automatically created from a two decades’ long news article collection, and which contains over 500k question-answer pairs. The dataset has been processed to remove temporally ambiguous questions for which more than one correct answer exist.

Bio: Adam Jatowt is a Professor at the Department of Computer Science and Digital Science Center at the University of Innsbruck. He received his Ph.D. in Information Science and Technology from the University of Tokyo, Japan in 2005. Before moving to Austria, Adam worked at Kyoto University as an Assistant and later as an Associate Professor. His research interests include information retrieval, natural language processing, digital libraries, and digital humanities. Adam has published over 180 research papers in international conferences and journals. He is an editorial board member of IP&M, JASIST, IJDL, JIIS, and IEEE JoSC journals. Adam has received the Vannevar Bush Best Paper Award at JCDL2021, the best short paper award at ECIR2018, and the best demo award at ECIR2019.
https://adammo12.github.io/adamjatowt/

Abstract: We envision continual learning systems to interact with humans, with each other, and the physical world through time – and continue to learn and adapt as they do. On the other hand, we develop and benchmark continual learning algorithms, hoping they will be the future founding blocks of such sy- stems. I will argue that going from algorithmic to system thinking requires carefully questioning various assumptions. Consider the common assumption of storage being fixed and limited through the agent’s lifetime. The economics of data storage suggests that the cost of storing data decreases over time and is negligible compared to the cost of computing. Following this economics, computing systems around us are not constrained by storage but rather by computation. This motivates us to explore online continual learning with computation constraints instead of storage constraints. Using this new setting, we set a new state of the art on the largest large-scale continual learning datasets.

Bio: Ozan Sener is interested in machine learning and its applications in computer vision and ro- botics. Specifically, Ozan is interested in designing machine learning algorithms which can process large-amount of multimodal information with no/weak supervision. Ozan is a research scientist at Apple. Ozan received his PhD from Cornell University, advised by Ashu- tosh Saxena. Following the advisor’s move, Ozan spent three beautiful years at Stanford AI Lab as a visiting PhD student (2015-2016) and a postdoc (2017) working with Silvio Savarese. Ozan obtained his BSc and MSc degrees from METU.
https://ozansener.net/

Abstract: Language models trained using transformers dominate the NLP model landscape, making Hugging Face (HF) the de facto hub for sharing, benchmarking, and evaluating NLP models. The HF hub provides a rich resource for understanding how language models evolved, opening up research que- stions on how factors such as model age and documentation affect their usage. We conducted a systematic study of the hub to glean insights into the ever-evolving model landscape and what factors affect model usage. We also studied model documentation for semantic drifts, and observed an evolution in the use of specific keywords (such as “train,” “evaluate,” “impact”), indicating a paradigm shift from model-centric to more data-centric ML development. In this talk, I will give a macro-level view of this evolving model landscape and discuss the results from our systematic study of 75K HF models.

Bio: Nazneen is a Research Lead at HuggingFace, a startup with a mission to democratize ML, leading data-centric ML research which involves systematically analyzing, curating, and automatically annota- ting data. Before HF, she worked at Salesforce Research with Richard Socher and led a team of resear- chers focused on building robust natural language generation systems based on LLMs. She completed her Ph.D. in CS at UT-Austin with Prof. Ray Mooney. Nazneen has over 30 papers published at ACL, EMNLP, NAACL, NeurIPs, and ICLR and has her resear- ch covered by Quanta magazine, VentureBeat, SiliconAngle, ZDNet, and Datanami. She is also teaching a course on interpreting ML models with Corise – http://corise.com/go/nazneen. More details about her work can be found here https://www.nazneenrajani.com/