Call for Papers

LLMs have achieved state-of-the-art performance in several textual inference tasks and are gaining popularity. There is a significant focus on their integration with web and online applications, including web search, thus allowing them to reach millions of users. LLMs can influence various information tasks in our everyday lives, ranging from personal content creation to education, financial advice, and mental health support (Augenstein, 2023). However, with their vast linguistic capabilities and opaque nature, LLMs can inadvertently generate or amplify false information. There is growing concern about the factuality of LLM-generated content and its potential adverse impact on our information ecosystem (Chen, 2023; Peskoff, 2023).

Thus, the need for reliable methods to assess the factuality of information is more critical than ever. This is where the synergy of AI, Natural Language Processing (NLP), and Human-Computer Interaction (HCI) becomes essential. AI and NLP techniques can be employed to analyze and identify the factuality of information through various tasks (Augenstein, 2023), such as fact-checking, stance detection, claim verification, and misinformation detection. These techniques can sift through the vast amounts of data to spot inconsistencies, biases, or inaccuracies that could indicate misinformation. Still, these approaches often use language models themselves, and epistemological questions arise when one LLM is fact-checked using another (or itself). Meanwhile, HCI plays a vital role in designing interactions and tools that enable humans to effectively oversee, interpret, and correct the outputs of LLMs. This human-in-the-loop approach ensures a critical evaluation and context-sensitive understanding of the factuality of information, which pure algorithmic methods might overlook. The combination of NLP's analytical capabilities and HCI's focus on human-centric design is instrumental in creating a digital ecosystem where LLMs can be utilized safely and responsibly, minimizing the risks of false information while maximizing their potential for user-centric applications.

The goals of the 1st ICWSM workshop Reliable Evaluation of LLMs for Factual Information (REAL-Info) are to facilitate discussion around such new LLM evaluation approaches, metrics, and benchmarks for factuality assessment tasks within the community, to inform the scope, biases, and blindspots of LLMs. It will spark interdisciplinary conversations from academic and industry researchers in computational social sciences (CSS), natural language processing (NLP), human-computer interaction (HCI), data science, and social computing. The workshop will solicit, research, and position papers with novel ideas, including but not limited to:

New evaluation methods and metrics for evaluating LLM’s factuality considering diverse social context, e.g., source and domain of data, language, temporal generalization of information, or hallucination in generated/summarized content.
Human-centered design approaches to aid LLMs in detecting and mitigating false information, e.g., human experts in the loop, and variation in prompting.
New LLM-powered tools, methods, and applications for improving factuality assessment in social computing and computational social science.
Biases and blindspots of LLMs in factuality assessment, including approaches for error analysis and model diagnostics.
Limitations of existing benchmarks for tasks relevant to factuality assessment, e.g., claim verification, fact-checking, stance detection, and misinformation detection.
Improve datasets and evaluation quality, e.g., avoidance of selection bias, addressing subjective judgments and biases in crowd-sourced annotation.
Comparative evaluation and implications of open source and commercial LLMs for tasks relevant to factuality assessment.
How does the reliability and factuality of LLM impact users (e.g. journalists, software engineers, artists..) and communities?

Submission Format and Instructions

We invite the following formats:

research papers (4-8 pages)
position papers (4 pages)

Submission link: https://easychair.org/conferences/?conf=realinfo1

References and appendices (if applicable) are excluded from this page count, but the length of the entire paper including references must not exceed eleven pages in the case of full papers, 8 in the case of short papers. Submissions must be original and should not have been published previously or be under consideration for publication while being evaluated for this workshop. Submissions will be evaluated by the program committee based on the quality of the work and its fit to the workshop themes. All submissions must be double-blind, and a high-resolution PDF of the paper should be uploaded to the EasyChair submission site before the paper submission deadline. The accepted papers will be published as Proceedings of the ICWSM Workshops. Please use AAAI two-column, camera-ready style.

Workshop Submissions

While affiliating the workshop with the ICWSM ensures visibility within the field of computational social science (CSS) and social computing, the workshop organizers also strongly encourage contributions from other disciplines, such as computational social sciences (CSS), natural language processing (NLP), human-computer interaction (HCI), and industry, to ensure that we cover the complexity of factuality assessment and capture diverse, multidisciplinary perspectives. Submissions will be evaluated by a program committee composed of researchers from relevant disciplines.