CEDAR: Probing Commonsense Explanation in Dialogue Response Generation
Pei Zhou, Pegah Jandaghi, Hyundong Cho, Bill Yuchen Lin
Jay Pujara, Xiang Ren
University of Southern California and Information Sciences Institute
Overview:
CEDAR is a probing framework that aims to understand why response generation (RG) models respond as they do by probing RG model’s understanding of commonsense reasoning that elicits proper responses. We formalize the problem by framing commonsense as a latent variable in the RG task and using explanations for responses as textual form of commonsense.
CEDAR contains 6k annotated explanations justifying responses from four dialogue datasets verified by humans. Two probing settings are proposed to evaluate RG models’ CSR capabilities. Probing results show that models fail to capture the logical relations between commonsense explanations and responses and fine-tuning on in-domain data and increasing model sizes do not lead to understanding of CSR for RG. We hope our study motivates more research in making RG models emulate the human reasoning process in pursuit of smooth human-AI communication.
Links (coming up soon): [Paper], [Github]
Probing Settings:
