How a unified mathematical approach to entropy can help us understand human intuitions and design better experiments
By Jonathan Nelson, Vincenzo Crupi, Björn Meder, Gustavo Cevolani & Katya Tentori (Max Planck Institute for Human Development)
Abstract: Optimal Experimental Design (OED) approaches have proven influential in a variety of domains. Examples include the design of mini experiments in the Adaptive Design Optimization (ADO) framework, theory of the diagnostic value of medical tests, identification of features for use in machine perception, and understanding human intuitions about which questions (tests, or experiments) are most useful. In many of these domains, minimization of Shannon entropy has been the primary goal (utility) function. In mathematics and physics, however, a wide variety of entropy functions have been proposed, from before Shannon up through the present. When tested in particular domains in machine learning, other entropy functions have in some cases led to better performance than Shannon entropy. A similar finding has occurred in psychology, where minimizing error predicts human intuitions better than minimizing Shannon entropy in some contexts.
We introduce a mathematical framework, the Sharma-Mittal family of entropy measures, that unifies Shannon entropy, error entropy, Hartley entropy, quadratic entropy, and several families of entropy measures, including Rényi, Tsallis, and Arimoto. The Sharma-Mittal family of entropy measures can each be thought of as quantifying expected surprise, where different entropy measures define surprise of an event, and the expectation of a distribution, in different ways. This decomposition of the Sharma-Mittal entropy measures is intuitive, and leads to insight into why particular measures behave as they do. It offers a way to identify entropy measures that might be as faithful as error entropy in quantifying human intuitions and behavior, but without some counterintuitive properties of error entropy itself. We have used the Sharma-Mittal framework to design psychological experiments to identify how people intuitively characterize the value of information. We discuss how it may also prove useful in future development of the Adaptive Design Optimization framework.