Best Practices for Enabling Reproducible & Replicable Studies in Human-Robot Interaction
IEEE RO-MAN 2025 WORKSHOP
Best Practices for Enabling Reproducible & Replicable Studies in Human-Robot Interaction
IEEE RO-MAN 2025 WORKSHOP
Standardised test methodologies for conducting human subject studies have been identified as being particularly beneficial for researchers relatively new to the process of creating statistically significant trials. Such practices would include recommendations on metrics for evaluation and their impacts, statistical tools for understanding and reporting the significance of trial results, and guidelines for designing the structure of the tests to optimise the quantity and quality of the test results for broad applicability. Ideally, such a standard would also propose a standardised reporting format and methodology such that the results may be broadly applicable outside the intended application domain.
The expertise and commonly available resources to design experiments with human subjects research in Robotics are fragmented and often overlooked, especially towards statistically sound, rigorous, and repeatable methodologies. To address this problem, the workshop organisers have been engaging in standardising interactive experiments via IEEE P3108 "Recommended Practice for Human-Robot Interaction Design of Human Subject Studies, https://sagroups.ieee.org/3108/, operating for the past 3 years.
As the working group approaches producing a first draft of a recommended practice document, we find that RO-MAN attendees are not commonly aware of these efforts and would greatly benefit from early access to the development of best practices. This workshop will facilitate such participation by opening the group's efforts to the wider RO-MAN audience, which is commonly engaged in studies involving human subjects.
This workshop directly supports early-career development and strengthens methods to create new fundamental insights into robotics research. In particular, we focus on the following objectives to support this goal:
Helping researchers, especially early-career researchers, benefit from world-leading expertise in the design of human-subject studies to produce more rigorous and replicable studies.
Establish strong mutually beneficial links between researchers in human-robot interaction and their international peers, as well as the IEEE standardising body, to allow these researchers access to and influence on policymaking.
Creating standards for all aspects of studies in the human-robot interaction domain is a large undertaking and to make it more manageable we will strategically address individual aspects of study design explicitly, i.e. (i) participant selection, diversification, and demographics (ii) study design and methodology (iii) applied metrics and measures as well as (iv) documenting and reporting of study outcomes, in strong alignment with the sub-group structuring of IEEE P3108. In the first half of the workshop, we will summarise the current state of each subgroup as outlined in the draft document, summarise and collate feedback and then hold a working session to improve the current version of the document, which is due to be submitted by the end of 2025.