Topic -Specific Synopses and Publications: Theories of Visualization | Video Visualization | Glyph-based Visualization | Volume graphics, Visualization, and Deformation | Perception, Cognition and Empirical Studies | Visual Analytics | Real World Applications
See also: References for Theoretic Research in Visualization
Drawn to Information Theory
I became interested in this topic in 2007 and 2008 when I was part of the organization team of VisWeek Workshop on Knowledge-assisted Visualization (KAV 2007 and KAV2008). The elusive concepts of "information" and "knowledge" in visualization fascinated me, while their broad, and often inconsistent, semantics and connotations made me curious about how much do we really knew how visualization worked. This initial quest led to an article [1] that attempted to clarify the notions of "data", "information", and "knowledge" in the context of visualization. This was also the time when I started to read about Information Theory with visualization in mind. I first encountered Information Theory briefly in my undergraduate course on Coding Theory around 1981/82, and late in my background reading for preparing my course on Data Communication in the early 1990s. I was excited about its relevance to the subject of visualization, but I did not have the confidence to write down anything seriously, until Heike Leitte (nee Jaenicke) arrived at Swansea.
As Heike worked on information-theoretic measures in her PhD programme, we shared the common languages of both visualization and information theory. Heike is a great thinker as well as a doer. We had many brainstorm meetings, which led to two pieces of work on this topic [2, 3]. The work on salience-based metric [3] was based on our information-theoretic agenda. It was intentional that information theory was not mentioned in the paper, partly because the writings on commonly-used metric for visual salience by Itti et al. usually do not mention information theory and partly because we were still writing [2] and felt inadequate to make a serious claim about information theory. When Heike was in Swansea, we also worked on a reconstructability metric [4] but did not consider this under the information-theoretic framework. I only realised the connection during the writing of the cost-benefit measure paper [11]. I wrote to Heike about this connection excitedly in 2015 after the paper was submitted to SciVis 2015.
Hunger for Explaining Why Visualization
Heike left Swansea for Heidelberg in 2010 and I left Swansea for Oxford in 2011. Our joint research agenda in the direction of information theory was put on hold. At Oxford, I met Luciano Floridi -- a leading expert on philosophy of information theory and Bob Coecke -- a quantum-information theorist. We had some most inspiring discussions, which led to a few of discourse articles on, for example, causality [5, 6] and information in visualization [8]. During that period, I also worked with others on sources of uncertainty in visualization [7] and the phenomenon of visual multiplexing [9]. Both tentatively suggested the possible connections between information theory and the perception and cognition in visualization, while uncomfortably pointing out that visualization is not an accurate process. I was wondering that visualization may not be intended to be an accurate process. Perhaps the most contentious discourse was the article on the question "what is visualization really for?" [10]. At that time, I suspected that it would not be acceptable to any visualization venue. Even today (2017), the idea that visualization is primarily for saving time is yet to be widely accepted by colleagues in the visualization community. While the practical evidence is everywhere and in front of everyone, it would still need some time and some more elegant evidence, such as mathematics, to convince many of us.
In 2013, Luciano introduced me to Amos Golan -- an economist and information-theorist in Washington DC. Amos invited me to attend several workshops organised by his multidiscplinary think tank, Info-Metrics Institute, where scientists and scholars from various disciplines met to discuss some fundamental questions about information. I learnt a great deal from these workshops. In the autumn of 2014, Amos invited me to stay in the DC for three weeks, working on one of such fundamental questions. "The resolution of stock market data is now in microseconds, and will reach nanoseconds soon," said he. He then asked, "What will be an appropriate resolution, such as seconds or days, for an economist?" After a few days of intensive discussions, I realised that the question could be extended to my puzzle about visualization being an inaccurate process. This led to our joint work on an information-theoretic measurement of the cost-benefit of visualization processes and indeed any processes in a data intelligence workflow [11]. Our paper was disappointedly rejected by SciVis 2015. The four reviewers scored the paper 1, 2, 2.5 and 3 (out of 5) respectively, and the SciVis co-chairs could not do anything with such low scores. I was confident that the mathematics was both correct and meaningful, and made the paper available on arXiv. Following some SciVis co-chairs' encouragement, I resubmitted it to TVCG directly. The first decision by TVCG was a major revision, and some reviewers asked for an empirical study to back the theory. Amos suggested doing some studies first. I knew that the theory could not be confirmed easily by a single empirical study, and we would have to submit our revision without such a study. I jokingly said to Amos, "it would be intriguing for a paper mathematically showing that visualization is useful to be rejected by the visualization community twice." Our revision was accepted, and TVCG published it online on 31 December 2015.
Until now, one conventional belief in visualization has been to assume that visualization must be accurate, since it would be much more difficult to suggest that visualization could be inaccurate. Most visualization scientists and practitioners made passionate arguments for accuracy in their teaching. In a small way, this is a bit like the early 19th century or before, when it was much easier to assume that the earth was flat. However, many visualization researchers have privately doubted the completeness or correctness of this doctrine because it cannot explain some obvious phenomena in visualization. For example, why would a time series plot be accurate enough for a series of data points in the range of [0, 10,000] despite that humans can manage to distinguish perhaps no more than 500 data values per data point? How could displaying such a time series on a powerwall with 20,000x40,000 pixels hardly bring about any more benefit than a desktop display with 1,000x2,000? Why should the highly-distorted London underground map be considered as one of the best visualizations in the history?
The cost-benefit measure [11], which is not easy to appreciate, uses information theory to explain that visualization is not always accurate but helpful. It does not focus on the traditional measure of mapping accuracy or decision accuracy. Instead, it introduces reconstructability (or potential distortion) as part of the measurement for accuracy, while presenting an argument that alphabet compression is the main causal factor of benefit (i.e., being imprecise is helpful). To prove the correctness of this measure is not trivial. At the end of my presentation of the paper [11] during VAST 2016, the brilliant Helwig Hauser asked about the difficulties in obtaining empirical proofs. I had to justify my lack of knowhow by borrowing the story of his fellow countryman, Ludwig Boltzmann, who encountered a lot of difficulties in having an empirical proof for his entropic modelling of ideal gases. The two subsequent papers [14, 15] are first empirical observations of this measure in practice, but many more studies will be needed to show the correctness of this measure. However, for the time being, we may assume that the measure is correct until it is falsified with a counter-example. We have not found such a counter-example yet. If we are courageous enough, we may make further hypotheses. For example, could the cost-benefit measure be the fitness function in the development or evolution of some cognitive capabilities, such as visual search, selective attention, gestalt grouping, heuristics, and memory (see the second short video for [11])?
Han-wei Shen (Ohio) is a leading expert on applying information theory to flow visualization. When we gave a tutorial on information theory in visualization together in EG2016, he told me that he received some $1M funding for supporting this part of his work, and was surprised that it had been difficult for me to receive funding in the UK for such research. With a few colleagues, we submitted a large proposal in January 2015 (here is the proposal). It was rejected at the outline stage, and conventionally, no reason was given. I then tried two small proposals. Both were rejected. One of them is a proposal to Oxford Martin School’s Call for Proposal: Great Transitions. The proposal ambitiously connects the cost-benefit analysis of data intelligence with operations management. It passed the outline stage. The final submission received mostly positive reviews. We were initially invited to make a presentation to the final evaluation panel but the proposal was rejected before it took place. (Here are the proposal, reviews and our feedback.) The latest small proposal to EPSRC received some very positive reviews and high scores (6, 6, 6, 5, 2) where 6 is the highest score available to the reviewers. The proposal was still rejected in January 2018. Readers may find interesting to read the proposal, reviews, and my feedback. The Dagstuhl 2018 event Foundation of Data Visualization (18041) is perhaps the first major event focusing mainly on theories of visualization. I might be the only attendee of this event who has to cover the airfare and accommodation using one's own saving.
The organisers of the Dagstuhl 2018 event, Helwig Hauser, Gerik Scheuermann, and Penny Rheingans, kindly asked me to coordinate the discussions on theory development and evaluation through empirical studies, and later invited me to join the co-editor team for the Springer book [24]. During that time, I was reading Roger Penrose's Shadows of the Mind, and writing lecture slides for a new course entitled "Human and Data Intelligence". I was inspired by Penrose's discourse on the four different viewpoints (first proposed by Philip Johnson-Laird) on whether thinking is computation, and wondered what are the major viewpoints in the field of visualization and whether organising these viewpoints into schools of thought or some kinds of isms would stimulate qualitative research to deliberate and evidence, quantitative research to evaluate and compare, and hopefully mathematical research to prove or falsify such viewpoints. I sought help from Darren Edwards (a psychologist in Swansea) and another colleague at Dagstuhl 2018 to relate the viewpoints in visualization with the isms in psychology and philosophy of mathematics. Darren and I managed to deliver part of our objectives [26]. I hope that there will be a study on isms in visualization and philosophy of mathematics in the future.
At Dagstuhl 2018, as all my scheduled meetings were with colleagues in the group on empirical studies, I did not talk much about the information-theoretic work, such as the cost-benefit measure [11]. It was perhaps a sheer chance when I joined an informal chat among Ivan Viola and a few others after a dinner. Ivan was mentioning the challenge in formulating an ovearching theory for visual abstraction. I offered a suggestion: information theory could explain visual abstraction. Ivan then invited me and six other colleagues to work on this topic. Ivan, myself and Tobias Isenberg eventually delivered a chapter [25]. As Ivan worked on visualization techniques based on information theory, he was able to appreciate my rather mathematical explanation of visual abstraction using the cost-benefit measure in [11]. It was rewarding to find that the cost-benefit analysis sails through the theoretic test on this important concept.
Advancing towards Measurement and Prediction
An ideal theory should feature the power of (i) explanation, (ii) measurement, and (iii) prediction. By 2018. The explanation power of the cost-benefit analysis became evident, at least to me. I began working on the other two properties. As the power of measurement was yet confirmed, I focused the power of prediction on the qualitative formula of the cost-benefit measure [11], and the very coarse measurement of the three abstract components: alphabet compression, potential distortion, and cost. I discovered that given a visual analytics workflow, one can invoke a systematic procedure to improve the workflow by introducing different human-centric and/or machine-centric processes. I first presented this systematic procedure as part of a VIS2018 Tutorial [18] and hoped to find a number of application case studies and a few colleagues to help consolidate the procedure. However, the tutorial unfortunately failed to attract many attendees. I then sought application case studies from a few colleagues. David Ebert kindly offered some five application cases and his time to scrutinise the systematic methodology. In our EuroVis 2019 paper [23], this systematic methodology was presented as an iterative procedure for analysing the shortcomings in an existing workflow (symptoms), the underlying reasons of the shortcomings (causes), the potential solutions (remedies), and the undesirable secondary effects of the solutions (side-effects).
Like many who are interested in information theory, I have been fascinated by the history of thermodynamics. In particular, the historical evolution of various measurements in thermodynamics has made me appreciate that most measuring schemes are not necessarily the "ground truth" facts, but are definitions appropriate to the right time, right application, and right people. When Amos and I were working on the cost-benefit measure, we spent many days pondering the unboundedness and asymmetry of the KL-divergence. At the end, we were allured by the de facto status of the KL as an information divergence measure and simplicity in explaining the resultant cost-benefit measure. For modelling and simulating economic phenomena, Amos was used to handle the unboundedness of the KL by using a probability range slightly narrower than [0, 1]. I used this approach in an empirical study for measuring knowledge [15], and realised that probability range has to be much narrower in dealing with simple phenomena, such as visualizing a TRUE binary value as FALSE. As there is no ground truth fact as to the measure that quantify the divergence between two probability distributions [100% TRUE, 0% FALSE] and [100% FALSE, 0% TRUE], the use of KL is not necessarily incorrect, but may not be the most appropriate.
Mateu Sbert, who is perhaps the most knowledgeable about information theory in the field of computer graphics, put an author team together in 2014 to write the book "Information Theory Tools for Visualization" [12]. In 2016, Mateu and I started to address the unboundedness of KL, initially by attempting to formulate a standardised algorithm for narrowing a probability range to avoid the unboundedness. In 2017, we discovered a proof that the KL for a finite alphabet should be conceptually bounded [27], though we continued the path of narrowing the probability range. In July 2019, when I was visiting Repsol Technology Lab. in Madrid as part of an EU project, Mateu paid a short visit to Madrid over a weekend. I showed Mateu my latest analysis on some bounded divergence measures, including the JS-divergence and a new one that I cooked up but had not yet found anything similar in the literature. We devoted the whole weekend to compare these measures conceptually and visually (i.e., by plotting out their curves in different value ranges), apart from enjoying wonderful Spanish meals in local restaurants that Mateu knew how to find. At that time, we were marginally in favour of the JS-divergence.
Back from my Madrid trip, I sought help from Alfie Abdul Rahman and Deborah Silver to collect some data in two practical contexts to see how these candidate measures would quantify the divergence in visualization processes in a more meaningful and intuitive manner. Interestingly, we found that one variant of the new divergence measure seemed to return more meaningful values. Since we were looking for a more appropriate bounded measure to replace the KL-divergence, this was a fairly significant finding. We submitted a paper to EuroVis 2020 [28 v1], which was not accepted but with encouraging scores (5, 4.5, 2.5, 2). We then improved the paper [28 v2] and submitted it to SciVis 2020, which was firmly rejected with much lower scores (3.5, 2, 2, 1.5). The reviewers seemed to want more explanations and more clarifications as well as some “ground truth” proof to confirm the best measuring scheme.
It many ways, the negative reviews provided me with motivation for writing more explanatory texts about the original cost-benefit measure [11]. I had been reluctant to critique the failure of some existing theoretical postulations in explaining the phenomena of "inaccurate" visualization, such as metro maps, volume rendering, and glyph-based visualization. Being able to point out such a failure, openly and gently, was comforting. I said to Mateu: "We just carry on writing more text and improving the paper, while waiting to see how many rejections it will accumulate." We thus wrote more explanatory texts, split the paper into two papers [29, 30], and submitted them to EuroVis 2021.
Meanwhile, Mateu and I were continuing our study on some mathematical properties of the new divergence measure. During the Christmas holiday of 2020, we discovered that under some conditions, the measure is a distance metric [31].
In the field of visualization, our effort for laying a theoretical foundation is still at its early stage. The VIS panel in 2016 [13], and the subsequent CGA article [17] encourage all researchers in the field to take part in this endeavour. As the panellists called for in [17], "Building a theoretical foundation for visualization is the collective responsibility of the visualization community. ... We hope every visualization researcher can find at least one pathway in this article through which to explore unanswered questions, known problems, and identified deficiencies in the theoretical foundation of visualization."
M. Chen, D. Ebert, H. Hagen, R. S. Laramee, R. van Liere, K.-L. Ma, W. Ribarsky, G. Scheuermann and D. Silver. Data, Information and Knowledge in Visualization. IEEE Computer Graphics and Applications, 29(1):12-19, 2009. DOI., PDF(0.2M)
M. Chen and H. Jaenicke. An Information-theoretic Framework for Visualization. IEEE Transactions on Visualization and Computer Graphics, 16(6):1206-1215, 2010. (Presented in IEEE VisWeek 2010.) DOI, VisWeek slides(4.6M).
H. Jaenicke and M. Chen, A salience-based quality metric for visualization. Computer Graphics Forum, 29(3):1183-1192, 2010. (Presented in EuroVis 2010.) DOI.
H. Jaenicke, T. Weidner, D. Chung, R. S. Laramee, P. Townsend and M. Chen. Visual reconstructability as a quality metric for flow visualization. Computer Graphics Forum, 30(3):781-790, 2011. (Presented in EuroVis 2011.) DOI.
M. Chen, A. Trefethen, R. Banares-Alcantara, M. Jirotka, B. Coecke, T. Ertl and A. Schmidt. From data analysis and visualization to causality discovery. IEEE Computer, 44(10):84-87, 2011. DOI.
M. Chen, T. Ertl, M. Jirotka, A. Trefethen, A. Schmidt, B. Coecke and R. Banares-Alcantara. Causality discovery technology. The European Physical Journal Special Topics, 214:461-479, 2012. DOI.
A. Dasgupta, M. Chen and R. Kosara. Conceptualizing visual uncertainty in parallel coordinates. Computer Graphics Forum, 31(3):1015–1024, 2012. (Presented in EuroVis 2012.) DOI.
M. Chen and L. Floridi. An analysis of information in visualisation. Synthese, 190(16)3421-3438, 2013. (Errata: Due to a typesetting error, the word "in" in the title, which was in all versions of the manuscript, got lost in the printed version.)
M. Chen, S. Walton, K. Berger, J. Thiyagalingam, B. Duffy, H. Fang, C. Holloway, and A. E. Trefethen. Visual multiplexing. Computer Graphics Forum, Wiley 33(3):241-250, 2014. DOI. (Presented in EuroVis 2014.) Slides).
M. Chen, L. Floridi, and R. Borgo. What is visualization really for? The Philosophy of Information Quality. Springer Synthese Library, Volume 358, pp 75-93 . 2014. DOI, arXiv (earlier version).
M. Chen and A. Golan. What May Visualization Processes Optimize? IEEE Transactions on Visualization and Computer Graphics, 22(12):2619-2632, 2016. DOI. (Presented in IEEE VIS 2016, Slides.) Early submission: arXiv:1506.02245, Short Videos: Cost-Benefit Analysis of Data Intelligence, Can Information Theory Explain Concepts in Cognitive Science?
M. Chen, M. Feixas, I. Viola, A. Bardera, H.-W. Shen, M. Sbert. Information Theory Tools for Visualization. A K Peters/CRC Press, 2016. ISBN: 9781498740937 - CAT# K26715.
M. Chen, G. Grinstein, C. R. Johnson, J. Kennedy, T. Munzner, and M. Tory. Pathways for Theoretical Advances in Visualization. IEEE VIS Panel, Baltimore, 23-28 October, 2016. (IEEE VIS 2016 Best Panel Award.) PDF(450K).
G. K. L. Tam, V. Kothari, and M. Chen. An analysis of machine- and human-analytics in classification. IEEE Transactions on Visualization and Computer Graphics, 23(1):71-80, 2017. DOI. (Presented in IEEE VIS 2016, VAST2016 Best Paper Award.)
N. Kijmongkolchai, A. Abdul-Rahman, and M. Chen. Empirically measuring soft knowledge in visualization. Computer Graphics Forum, 36(3):73-85, 2017. DOI. (Presented in EuroVis 2017, Slides.)
P. A. Legg, E. Maguire, S. Walton, and M. Chen. Glyph visualization: A fail-safe design scheme based on quasi-Hamming distances, IEEE Computer Graphics and Applications, 37(2):31-41, 2017. DOI. (Presented in VIS2017, Slides.)
M. Chen, G. Grinstein, C. R. Johnson, J. Kennedy, and M. Tory. Pathways for Theoretical Advances in Visualization. IEEE Computer Graphics and Applications, 37(4):103-112, 2017. DOI.
M. Chen. Cost-benefit Analysis in Visualization: Theory and Practice. IEEE VIS Tutorial, 2018. Proposal (138K). (A special web page will be created before the tutorial.)
M. Chen, K. Gaither, N. W. John, and B. McCann. Cost-benefit analysis of visualization in virtual environments. IEEE Transactions on Visualization and Computer Graphics, 25(1):32-42, 2019. DOI. (Presented in IEEE VIS 2018.) Early version: arXiv:1802.09012, 2018.
D. Sacha, M. Kraus, D. A. Keim, and M. Chen. VIS4ML: An ontology for visual analytics assisted machine learning. IEEE Transactions on Visualization and Computer Graphics, 25(1):385-395, 2019. DOI. (Presented in IEEE VIS 2018.)
M. Chen. Cost-Benefit Analysis of Data Intelligence -- Its Broader Interpretations. In Advances in Info-Metrics: Information and Information Processing across Disciplines, Oxford University Press, 2020, ISBN 9780190636685. (Pre-accepted version available at arXiv:1805.08575, First version was completed on 20 May 2018.)
M. Chen. The value of interaction in data intelligence. (available at arXiv:1812.06051, First version was completed on 21 September 2018.)
M. Chen and D. S. Ebert. An ontological framework for supporting the design and evaluation of visual analytics systems. Computer Graphics Forum, 38(3):131-144, 2019. DOI. (presented at EuroVis 2019.) Supporting web site: IVAS (Improving Visual Analytics Systems).
M. Chen, H. Hauser, P. Rheingans, and G. Scheuermann (eds.), Foundations of Data Visualization, Springer, 2020, ISBN 978-3-030-34444-3.
I. Viola, M. Chen, T. Isenberg. Visual Abstraction. In Foundations of Data Visualization, Springer, 2020. Preprint available at arXiv:1910.03310
Chen and D. J. Edwards. "Isms" in Visualization. In Foundations of Data Visualization, Springer, 2020, ISBN 978-3-030-34444-3. Preprint PDF(631K).
M. Chen and M. Sbert. On the upper bound of the Kullback-Leibler divergence and cross entropy. Technical report: arXiv:1911.08334. 2019.
M. Chen, M. Sbert, A. Abdul-Rahman, and D. Silver. A bounded measure for estimating the benefit of visualization. (available at arxiv:2002.05282, First version was completed on 5 December 2019.)
to add: