Skilled computer users

Eric J. Fimbel - Research results and data sets - home page  

Citation: Fimbel, E.J. (2009) Skilled Computer Users. Available at http://skilledusers.googlepages.com/. Last retrieved : (mm/dd/yyyy)

Publication: Fimbel, E.J. (2013) Predicting the performance of skilled computer users without knowing their strategies: rejected 7 times. Technical report, 41p. Available at https://sites.google.com/site/skilledusers/.
download pdf


 




Publication (meta-analysis)

Facts

This site contains a technical report and the corresponding data and charts.The technical report was submitted to scientific journals and rejected 7 times. This is an interesting case study.

The paper presents KPC, a simple model to predict human performance on a computer equipped with a keyboard and a pointing device.

The argument is as simple as KPC: skilled computer users don't need thinking when they execute familiar tasks, therefore if predicting performance is the objective, it is pointless to model their mental activity.

The fist time the paper was rejected, it was argued that the experimental validation was not conclusive, because we used the same data set (partitioned) to tune the model and to verify its accuracy. This is a classical technique in automatic classification, but let it be.

We therefore conducted a second experiment in a different country, with different participants, computers and tasks. The results were similar. KPC predicted accurately the skilled users' performance. Complete rewriting of the manuscript, new co-authors and resubmission.

The paper was rejected again, twice because it was out of scope (true), and 4 times for different reasons. Some reviewers gave meaningful criticisms, others were clearly beyond their competence. The editors systematically rejected the paper. 


Possible interpretations

Here are several lines of explanation.

1) The paper is lousy. It is dull, difficult to read, etc.

Maybe; check by yourself (download pdf ; last uncorrected manuscript)

However we believe that the experimental evidence is solid: the protocol and setup, the experiments and the analyses were carefully done.

2) Nobody understands us. This is a classical argument among geniuses and sub-normal teenagers.

Our methodology comes from experimental psychology and behavioral neuroscience, two communities with very high experimental standards.

Analytical modeling (i.e., modeling user-computer interactions) on the other hand is more oriented towards computer science, and some reviewers clearly did not have the adequate experimental background. However, other reviewers were obviously familiar with our methods.

3) It is politically incorrect to say that skilled users are brainless, at least when they execute familiar tasks.

This could be generalized to the Y, Z and following generations, (young) people that are glued to their smart phones, tablets, and other socio-electronic gizmos.

4) it is scientifically inadequate to show that the simplest model is sufficient  to predict the performance of contemporaneous users.

1) This may set a golden standard for sophisticated models, which means additional work for modellers (to compare their model with KPC), and a risk that their model does not stand the comparison.

2) This raises doubts about the usefulness/soundness of modeling brain and mind in user-computer interactions.

Do sophisticated user models bring us insights on cognition and brain? I don't know whether analytical modeling stands the comparison with experimental psychology, brain imaging, neurobiology, etc.

Do sophisticated user models allow a finer prediction of human computer interaction? Certainly, but it is at the expense of scientific parsimony, because such models need arbitrary hypotheses, heuristics and parameters. And tuning sophisticated models may be as demanding as conceiving them.

In any case, here is our data and the KPC model, so that you can make your mind.

Experiments and datasets

Find here the raw and processed data and charts for 2 experiments on skilled computer users.

The first experiment was designed to measure the execution time of basic operations (key press, pointing movements, button clicks) during repetitive tasks on graphical interfaces.

The data was used 1) to tune the KPC model of skilled users performance, 2) to execute a first comparison of KPC and of its reference model, the KLM.

Experiment 1

The KPC model derives from the KLM. Here are short tutorials about these models

The second experiment was designed to measure the performance of skilled computer users on commercial software and to record their strategies, i.e., sequences of basic operations.

The data was used 1) to validate the KPC model, 2) to investigate the strategies of skilled computer users.

Experiment 2





Copyright: (c) 2009 E.J. Fimbel, P-S Dube. This is open-access content distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.