Empirically Evaluating Diagrams
Facilitators: Dr Gem Stapleton and Dr Jim Burton
Course Summary
This is an introductory course on empirical methods suitable for evaluating diagrams. It will cover foundational material that will equip attendees with a basis for conducting robust empirical studies for research publication. The emphasis will be on deriving hypotheses, turning these into research questions, and designing and running empirical studies to test these hypotheses. In addition, delegates will be made aware of the types of threats to validity that can arise in empirical research and methods by which results can be interpreted.
We will explain and emphasise the importance of grounding hypotheses and study design in theoretical frameworks such as mental models, Gestalt principles, or well-matchedness, for instance. We will provide an overview of some of the most prominent of these frameworks, with examples that show how they have been used to interpret empirical results relating to diagrams.
The final session of this course will be hands-on activities that take participant through the empirical study process, thus allowing the development of practical skills that are transferable to other research areas.
Prerequisite Knowledge
No prior knowledge is assumed but it is expected that delegates will have a desire to understand the efficacy of diagrammatic notations with respect to human comprehension.
Course Outcomes
On completing the course, attendees should have fundational knowledge of the key components when running empirical studies. They should be well placed to build on this knowledge in order to apply the methods learned to their own research areas. In particular they should be able to design simple empirical studies that evaluate the impact of varying diagram properties on human comprehension.
Course Materials
Session 1:Hypotheses and Study Design
Session 2: Data Collection, Statistics and Threats to Validity
Session 3: Interpreting Results
Session 4: Case Studies
Material for all sessions in one file as presented and in handout form.
Example Sheets
Paper-based training treatment 1
Paper-based training treatment 2
Example sheets all in one file.
Other Resources
Research paper of core importance to the course.
Many of Dr Stapleton's publications can be downloaded from her web site.
Recommended Text
Experimental Human-Computer Interaction A practical Guide with Visual Examples, by Helen C. Purchase.