E. Rydow, R. Borgo, H. Fang, T. Torsney-Weir, B. Swallow, T. Porphyre, C. Turkay, and M. Chen. Development and evaluation of two approaches of visual sensitivity analysis to support epidemiological modelling. IEEE Transactions on Visualization and Computer Graphics, 29(1):1255-1265, 2023. DOI. (Presented at IEEE VIS 2022.)
The IVAS framework was used during the work carried out in a mini-project with the RAMPVIS project. Erik Rydow was one of the appointed research officers in RAMPVIS project and played the key role in the design and development of the techniques reported in the above paper. The mini-project was proposed by Thomas Torsney-Weir, managed by Hui Fang, and overseen by Rita Borgo who also led the EnsembleVis team --- one of the four modelling support teams with the RAMPVIS project. The EnsembleVis team collaborated with the Uncertainty Quantification modelling group in the SCRC during the pandemic, and Thibaud Porphyre (an epidemiologist and modelling scientist) led the group. Ben Swallow (a modelling scientist) engaged closely with the mini-project team, providing advice on epidemiological modelling. The following analysis was conducted mainly in three stages, (i) initial assessment during and after a brainstorm meeting in late 2021, (ii) initial writing for the above paper in early 2022, which was mentioned briefly in the paper due to the page limit, and (iii) revised writing for a data visualization course in late 2022. Three relevant slides of the course are shown below.
The previous workflow for sensitivity analysis involved some 200 model runs with different parameter sets. The domain experts had access to time series plots for visualizing the outputs of these runs, as well as a sensitivity analysis algorithm that rates the sensitivity of each parameter numerically (exemplified by a bar chart in the slide on the left). In comparison with the list of user tasks gathered during the mini-project, several tasks cannot be performed easily with such a time series plot or a bar chart.
For some of these tasks, there was too much information loss due to the sensitivity analysis algorithm, while for some others, there was not enough alphabet compression. This led to two approaches for improving the previous workflow.
Unlike a multiplication algorithm, the numerical ratings returned by a sensitivity analysis algorithm are not expected to be ground truth values and cannot easily be validated, except some obvious predications. For example, the parameter p_s in the bar chart influences how the age variation affects the fatality. In the context of COVID-19, all domain experts knew this fact. So a longer bar for p_s indicates that the model simulates the basic fact OK, but not much extra information. In abstraction, the cause is Alg-High-AC (the sensitivity analysis algorithm). The potential remedy is to reduce to the level of AC. Among the options of Stat-Low-AC, Alg-Low-AC, Vis-Low-AC, or Int-Low-AC, visualization allows one to go back to the data before being processed sensitivity analysis, and can potentially reveal more information that are used to estimate the sensitivity rating. The scatter plot for each parameter is an example of such information.
However, the data points in each scatter plot are sparsely distributed and do not seem to reveal any meaningful patterns. In abstraction, the cause is Alg-High-AC (the simulation runs), as the ensemble simulation had to omit a large number of runs due to the high computational cost of model runs (i.e., cause Alg-High-Cost). The remedy is to use a low-cost algorithm (i.e., remedy Alg-Low-Cost) to approximate a large number of data points that were not generated by the model runs. These approximate data points enable the display of the trend lines in the scatter-line plots, revealing the usually nonlinear patterns that domain experts wished to observe and making the remedy Vis-Low-AC work.
The symptom that a time series plot with some 200 time series cannot be observed easily can be consider as high cost (Vis-High-Cost) and/or high error-rate (Vis-High-PD) in observing the data and in analysing the relationships among them and between them and the parameters. In practice, users may realize such a cost or error-rate, and decide not to attempt the tasks at all. Because of the lack of failed attempts, users may not state such a need or such a difficult in requirement analysis. It is thus important for VIS practitioners to tease out the symptoms of "no attempt". The cause is VIS-Low-AC, i.e., the time series plot does not compress information enough. This naturally leads to optional remedies Stat-High-AC, Alg-High-AC, Vis-High-AC, or Int-High-AC. For example, one can interactively select a small subset of time series to observe (Int-High-AC + Vis-High-AC). However, as the decision on what to select is not trivial, one may have to perform many interactions (Int-High-Cost). Clustering can compress information quickly. This leads a solution of visualizing similar time series in clusters (Alg-High-AC + Vis-High-AC). Some may compare the four plots for four clusters on the right (labelled with a circled (e)) with the basic time series plot on the left (with a circled (b)), and draw a conclusion that the four plots (e) contain more information than the plot in (b). Not only all grey-coloured time series in (b) are shown in (e), there are extra four average time series that are coloured in yellow, green, red, and blue. This observation itself is correct when we consider only the displayed information (more theoretically-precise: "the displayed entropy"). When we consider a visualization process, we should either include the perception and cognition as part of the process or divide the process into sub-processes. When we consider perception and cognition, we can easily anticipate the followings: (i) a user will pay attention to the four coloured time series (Vis-High-AC), (ii) may observe the four cluster plots individually (in each case Vis-High-AC), and (iii) may focus mainly on the pattern of the coloured time series and the vague grey shape behind when comparing the four clusters or relating each cluster to the parameter range plot below (Vis-High-AC). So displaying more information does not always lead to perceive more information. The highlighting and de-highlighting mechanisms in visual encoding and selective attention in visual decoding changes the probability of each piece of depicted data being "visualized" (i.e., being perceived and thought about).
There is a minor side-effect that the clustering algorithm may create N clusters that do not characterize the similarity/difference in an epidemiological meaningful way (Alg-High-PD). This is a common issue in clustering, and the most effective way is usually to utilize the domain experts' knowledge by allowing them to have some control over the clustering algorithm, such as selecting the number of clusters and similarity measures. Domain experts' inputs reduce the search space of the clustering algorithm. So the remedy is considered as (Int-High-AC) with a side-effect Int-High-Cost.