J. Zhao, A. Malik, H, Xu, G. Wang, J. Zhang, C. Surakitbanharn, and D. S. Ebert. Metricsvis: A visual analytics framework for performance evaluation of law enforcement officers. In Proc. IEEE Symposium on Technologies for Homeland Security, Waltham, MA, 2017. DOI.
This Case Study Report was authored in 2018 by Min Chen (University of Oxford, UK) and David Ebert (Purdue University, USA)
This is one of the three post-hoc case studies discussed in the 2019 paper by Chen and Ebert, where the IVAS framework was proposed. The analysis was reported in Appendix C.1 of the paper. The second author (DSE) suggested this paper, which was unknown to the first author (MC) previously. MC first read the paper and then wrote a report as an independent reviewer. The report took about 40-60 minutes to complete, including the effort of writing but excluding the time for the first reading. MC then emailed the report to DSE for comments. There were a few email correspondences to discuss the report.
MC: For the MetricsVis paper [Zhao et al. 2017], we can see the following analytical flow from symptoms to remedies. Note that the paper mentioned one of the solutions as a part of the problem.
1. Symptoms of the Original Workflow:
(A). Supervisors in law enforcement agencies need to perform resource allocation tasks regularly. There are concerns that they may be biased by their experience of interaction with individual officers. Their knowledge about an officer's effectiveness is limited by the cases that they supervised or knew well. Even for such cases, they may fail to notice some characteristics of the officer's performance.
2. Analysis of Symptom (A):
(A) suggests a substantial amount of potential distortion in mapping from an impression of an officer (in the supervisors' mind) to the actual performance data of the officer.
(A) also suggests a consequential cost due to inappropriate assignment, which may lead to ineffective deployment of officers to different cases, or more seriously, may lead mistakes.
3. Analysis of Possible Causes:
Potential Distortion is often caused by too much Alphabet Compression. In this example, the form of alphabet compression is achieved by self-selection of a subset of data (i.e., cases that the supervisor know well), and an aggregated impression.
Imagine that the impression about an officer is that about a university class on a topic and the crime cases are students’ works. The latter is similar to marking only the works by a subset of students sitting in the front row, and the former is similar to calculating an average score for the class (cf. the officer) based on these students' works.
4. Potential Remedy 1
For supervisors to read all crime case reports, as reading all crime case reports clearly can alleviate the Alphabet Compression due to self-selection. It is a bit like marking all students' works. This proposed remedy suggests that if supervisors could routinely read all crime case reports, they would have better knowledge about officers and have less biases (or potential distortion).
5. Analysis of Side Effect (referred to as symptom (B))
(B). While it is possible for supervisors to read all crime case reports, such effort would demand a huge amount of time and is not affordable in general.
6. Analysis of Symptom (B)
Too much cost in the cost-benefit metric.
7. Analysis of the Possible Cause
(B) is in fact not a symptom of the original workflow, but that of a solution being considered. While reading all crime case reports can clearly alleviate the Alphabet Compression due to self-selection, less alphabet compression usually leads to more cost, thus the symptom of (B) is caused by too little Alphabet Compression.
(B1). In addition, creating an impression by integrating information received through interaction with the officers and that through reading all case reports over a long period is not necessarily reliable.
The cause for such unreliable integration is humans' unreliable memory, thus Potential Distortion. This cause was not mentioned in the paper.
DSE: Good insight.
MC: (continued)
8. Potential Remedy 2
Since the cause of (A) is too much Alphabet Compression, and the major cause of (B) is too little Alphabet Compression, a natural strategy for optimization is to find a solution in-between. This is a bit like the example of marking the students' works. One may personally mark the works of the students in the front row, and outsource the marking of the works by the rest of the class to a computer, which produces some statistical measures for each work, and convey the statistics to the professor.
DSE: Relate to abstract reasoning?
MC: marking the front row features (low AC, low PD, high Cost), while outsourcing the rest features (high AC, high PD, low Cost).
MC: (continued)
This leads to a set of metrics and summary visualization documented in the original paper. In addition, visualization also provides a solution to the cause of (B1) since visualization enables external memorization, and such memory can be recalled by supervisors at any time at ease.
9. Potential Side Effects and Analysis
(C). The additional cost related to viewing the visualization. This cost can be reduced through training, and frequent uses of the system.
(D). The potential distortion in reconstructing from the summary visualization to the actual crime case report. Since the supervisors' resource allocation itself does not require reading all crime case reports, in most cases, such potential distortion does not affect the task performance. However, in some crime cases that may be more closely related to the cases to be investigated, the summary visualization may raise the curiosity of the supervisor concerned, who will take initiatives to read the original report or ask questions about the case.
A possible future remedy to (D) (if it has not been implemented) is to allow users to observe all cases in a particular category, or to sort cases by a similarity metric.
DSE: Good suggestion.