In an era of information explosion, the study of information, including how information is acquired, aggregated, and communicated, has never been more crucial. This workshop explores the interplay between Information Economics and Large Language Models (LLMs), focusing on how LLMs can enhance our ability to acquire, aggregate, and communicate information, and conversely, how the economic principles of information can optimize the training and usage of LLMs.

Researchers have studied information acquisition, aggregation, and communication through a mathematical lens. However, much real-world communication occurs through language, which has been difficult to analyze formally. The recent breakthroughs in generative AI, particularly LLMs, provide new opportunities to bridge this gap, enabling us to examine how language generation tools can contribute to the theoretical study of information. 

Bringing together senior and junior researchers across diverse disciplines, including economics, computer science, and operations research, this workshop will explore four topics at the intersection of information design and LLMs:

• Information acquisition × LLM

• Information aggregation × LLM

• Information design × LLM

• Other topics on information economics × LLM

Through this interdisciplinary dialogue, we aim to push forward both the theoretical boundaries and real-world applications of information economics in the modern era.

We are thrilled to announce the first EC Workshop on Information Economics × Large Language Models.

The workshop will be held at Stanford University on the morning of July 10, 2025. The workshop explores the interplay between Information Economics and Large Language Models (LLMs), focusing on how LLMs can enhance our ability to acquire, aggregate, and communicate information, and conversely, how the economic principles of information can optimize the training and usage of LLMs.

We are soliciting contributions of papers and posters that fit the theme of the workshop. Topics of interest include, but are not limited to:

• Information acquisition × LLM

• Information aggregation × LLM

• Information design × LLM

• Other topics on information economics × LLM

Submission Guidelines:

Submissions are non-archival. We accept a wide range of submission formats, e.g., extended abstracts, working papers, conference papers, journal papers, etc. The primary contribution and the relevance to the workshop should be clearly identifiable within the first six pages (or the first three pages for double-column papers). We will implement a light review process based on criteria including but not limited to the paper’s soundness, merits, and credibility. A limited number of accepted contributions will be selected for contributed talks, and other accepted works will be included in a poster session.

To submit, please email your manuscript to ec25workshop-info-econ-llm@googlegroups.com. Further details are available at our website: https://sites.google.com/view/ec25-information-economics-llm.

Important Dates:

Submissions open: Now

Submission deadline: June 7, 2025 (AoE) 

Decision Notification Deadline: June 14, 2025

Workshop: July 10, 2025

9:00 - 9:45 AM        Invited talk 1: Jason Hartline  

          Title:  Proper Scoring Rules for Text and Super-alignment

  Abstract: 

From statistics, scoring rules evaluate probabilistic forecasts of an unknown state against the realized state and are a fundamental building block in the incentivized elicitation of information. Proper scoring rules are ones where the forecaster is incentivized to report their true belief about the unknown state.

The talk will review the classical and recent work on (statistical) scoring rules.  This review will include the geometry of proper scoring rules (classical) and the optimization of scoring rules (to induce more effort of the forecaster to learn about the state).

The talk will generalize these statistical scoring rules to give an algorithm for scoring elicited text against ground truth text using domain-knowledge-free queries to a large language model (specifically ChatGPT).

A key application for text scoring is in peer grading where students grade each other's work. In peer grading, a challenge is in incentivizing high quality peer reviews.  The textual scoring rule can be used to grade a textual peer review by comparing it to the textual instructor review.  This example parallels super alignment, specifically, the more intelligent students are scored using the less intelligent domain-knowledge-free queries that are sent to a large language model.

9:45 - 10:00 AM     Contributed talk 1: Benchmarking LLM's Judgments with No Gold Standard

  Authors: Shengwei Xu, Yuxuan Lu, Grant Schoenebeck, Yuqing Kong

Abstract: 

We introduce the GEM (Generative Estimator for Mutual Information), an evaluation metric for assessing language generation by Large Language Models (LLMs), particularly in generating informative judgments, without the need for a gold standard reference. GEM broadens the scenarios where we can benchmark LLM generation performance-from traditional ones, like machine translation and summarization, where gold standard references are readily available, to subjective tasks without clear gold standards, such as academic peer review.

GEM uses a generative model to estimate mutual information between candidate and reference responses, without requiring the reference to be a gold standard. In experiments on a human-annotated dataset, GEM demonstrates competitive correlations with human scores compared to the state-of-the-art GPT-4o Examiner, and outperforms all other baselines. Additionally, GEM is more robust against strategic manipulations, such as rephrasing or elongation, which can artificially inflate scores under a GPT-4o Examiner.

We also present GRE-bench (Generating Review Evaluation Benchmark) which evaluates LLMs based on how well they can generate high-quality peer reviews for academic research papers. Because GRE-bench is based upon GEM, it inherits its robustness properties. Additionally, GRE-bench circumvents data contamination problems (or data leakage) by using the continuous influx of new open-access research papers and peer reviews each year. We show GRE-bench results of various popular LLMs on their peer review capabilities using the ICLR2023 dataset.

10:00 - 10:15 AM   Contributed talk 2: How AI Aggregators Affect Knowledge Sharing

  Authors: Daron Acemoglu, Tianyi Lin, Asuman Ozdaglar, James Siderius

Abstract: 

Recent advancements in artificial intelligence (AI) have brought great promise to more efficiently aggregate and deliver information, but also raise concerns about their tendency to exacerbate the existing biases entrenched in society. We formalize this tension by extending the DeGroot model of network learning to incorporate AI aggregators, which are modeled as specific nodes in the network that take as input beliefs from the population (“training data”) and communicate synthesized beliefs (“answers to queries”). We fully characterize the wisdom gap, a proxy for the degree of mislearning in society, both pre-AI and post-AI. Our results demonstrate the feedback loop between AI input and output tends to amplify the opinion of majority group, often degrading learning in the post-AI environment. We derive comparative statics with respect to the network structure, the training parameters, and the influence of the AI aggregators on society. Lastly, we contrast the case of a single global aggregator (e.g., ChatGPT) to that of multiple local aggregators (e.g., newspapers or Internet forums) to uncover simple yet sufficiently general conditions under which the aggregators help or hinder wisdom in society.

Break (10:15 - 10:45 AM) 

10:45 - 11:30 AM   Invited talk 2: Aranyak Mehta

          Title:  GenAI, Advertising, and Markets

  Abstract: 

I will present recent results at the intersection of AI and markets: Envisioning the evolution of advertising auctions in LLM summarization and conversational contexts; and designing markets with multiple AI API providers.

11:30 - 11:45 AM   Contributed talk 3: Human-AI Interactions and Societal Pitfalls

  Authors: Francisco Castro, Jian Gao, Sebastien Martin

Abstract: 

When working with generative artificial intelligence (AI), users may see productivity gains, but the AI-generated content may not match their preferences exactly. To study this effect, we introduce a Bayesian framework in which heterogeneous users choose how much information to share with the AI, facing a trade-off between output fidelity and communication cost. We show that the interplay between these individual-level decisions and AI training may lead to societal challenges. Outputs may become more homogenized, especially when the AI is trained on AI-generated content. And any AI bias may become societal bias. A solution to the homogenization and bias issues is to reduce human-AI interaction frictions and enable users to flexibly share information, leading to personalized outputs without sacrificing productivity.

11:45 - 12:00 [TBD]

Poster Session

1:30 -2:45 PM      The Effect of State Representation on LLM Agent Behavior in Dynamic Routing Games

  Authors: Lyle Goodyear, Rachel Guo, Ramesh Johari

                              Tokenized Bandit for LLM Decoding and Alignment

  Authors: Suho Shin, Chenghao Yang, Haifeng Xu, MohammadTaghi Hajiaghayi

                              Learning from a Mixture of Information Sources

  Authors: Nicole Immorlica, Brendan Lucier, Clayton Thomas, Ruqing Xu

                              An Interpretable Automated Mechanism Design Framework with Large Language Models

  Authors: Jiayuan Liu, Mingyu Guo, Vincent Conitzer

                              Tell me Why: Incentivizing Explanations

  Authors: Siddarth Srinivasan, Ezra Karger, Michiel A. Bakker, Yiling Chen

                              Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives

  Authors: Ander Artola Velasco, Stratis Tsirtsis, Nastaran Okati, Manuel Gomez-Rodriguez

                              Natural Language Mechanisms via Self-Resolution with Foundation Models

  Authors: Nicolas Della Penna

                              How AI Aggregators Affect Knowledge Sharing

  Authors: Daron Acemoglu, Tianyi Lin, Asuman Ozdaglar, James Siderius

                              Verbalized Bayesian Persuasion

  Authors: Wenhao Li, Yue Lin, Xiangfeng Wang, Bo Jin, Hongyuan Zha, Baoxiang Wang

                              STEER-ME: Evaluating LLMs in Information Economics

  Authors: Narun Raman, Taylor Lundy, Kevin Leyton-Brown, Jesse Perla

                              Proper Dataset Valuation by Pointwise Mutual Information

  Authors: Shuran Zheng, Xuan Qi, Rui Ray Chen, Yongchan Kwon, James Zou

Break (2:45 - 3:15 PM) 

3:15 - 4:30 PM.   Cost-Aware Sequential Testing for Human-in-the-Loop LLM Tasks

  Authors: Renyuan Xu, Luhao Zhang, Haotian Zong

                              Benchmarking LLM's Judgments with No Gold Standard

  Authors: Shengwei Xu, Yuxuan Lu, Grant Schoenebeck, Yuqing Kong

                              Understanding LLMs’ Economic Rationality through Sparse Autoencoders

  Authors: Chuanhao Li, Xiaozhi Wang, Shuhuai Zhang, Shu Wang, Zijun Yao, Tracy Xiao Liu, Songfa Zhong

                              The Value of Costly Signaling in Interactive Alignment with Inconsistent Preferences

  Authors: Ali Shirali

                              Bayesian Persuasion as a Bargaining Game

  Authors: Yue Lin, Shuhui Zhu, William A Cunningham, Wenhao Li, Pascal Poupart, Hongyuan Zha, Baoxiang Wang

                              Framing and Signaling: An LLM-Based Approach to Information Design

  Authors: Paul Duetting, Safwan Hossain, Tao Lin, Renato Paes Leme, Sai Srivatsa Ravindranath, Haifeng Xu, Song Zuo

                              Human-AI Interactions and Societal Pitfalls

  Authors: Francisco Castro, Jian Gao, Sebastien Martin

                              Incentives for Digital Twins: Task-Based Productivity Enhancements with Generative AI

  Authors: Catherine Wu and Arun Sundararajan

                              Persuasive Calibration

  Authors: Yiding Feng, Wei Tang

                              Fairness Behind the Veil: Eliciting Social Preferences from Large Language Models

  Authors: Yizhuo Alexandra Dong, Muzhi Ma, Natalia Trigo Tomasevich, Niuniu Zhang

 Contact

 Email: ec25workshop-info-econ-llm@googlegroups.com