Experiences in Using Practitioner's Checklists to Evaluate the Relevance of Experiments Reported in Requirements Engineering

Abstract:

• Background: Requirements Engineering (RE) researchers recognize that for RE methods to be adopted in industry, practitioners should be able to evaluate the relevance of a study to their practice. . Kitchenham et al proposed a set of perspective-based checklists, which demonstrated to be a useful instrument for this purpose. Specifically, the checklist from the practitioner‟s perspective seems to be a good candidate for evaluating the relevance of RE studies to RE practice. However, little is known about the applicability of the checklist to the area of RE. Moreover, this checklist also requires a greater analysis about its reliability.
• Aim: The aim of this report is to propose a perspective-based checklist to the RE community that allows evaluating the relevance of experimental studies in RE from the practitioner‟s/consultant‟s viewpoint.
• Method: Our research followed an iterative design-science based approach in which we first analyzed the problems with a previously published checklist and developed an operationalized proposal for a new checklist to counter these problems. We performed a reliability evaluation of this new checklist. The research was performed with two practitioners and 24 papers that report experimental results on comprehensibility of software requirements specifications. Results: This report gives first-hand experiences of practitioners in evaluating the relevance of primary studies in RE, by using a perspective-based checklist. With respect to the reliability of the adjusted checklist, 9 of out 19 questions show an acceptable proportion of agreement (between two practitioners).
• Conclusions: Based on our experience, the contextualization and operationalization of a perspective-based checklist helps to make it more useful for the practitioners. However, to increase the reliability of the checklist, more reviewers are required and more discussion cycles are necessary. Our plan is to involve at least two more practitioners in order to improve the reliability of the practitioner checklist proposed.

The Process of Checklist Modification and Evaluation:

Software Requirements Specification Research Checklist:

The checklist items in a way that allows the evaluator to clearly see those items that apply to studies which investigate SRS understandability factors (and do not compare RE-techniques to find which one is better). In Table I, items Q1-Q9 refer to this study type. Because these studies seek meaning and understanding of a SRS in specific contexts, items Q10-Q19 are irrelevant for them. In contrast, when evaluating the quality of „what-is-better‟ papers, it is appropriate to use all the items in the new checklist.

Conclusions and Future Work:

This report presents experiences and lessons learnt in using the practitioner‟s checklist that was published in [6]. We make the note that while [6] provides the checklist, it gives very little support on how to „operationalize‟ the checklist items, and how to aggregate the ratings of the checked items into an overall rating. (e.g. a practical question encountered by the two practitioners was whether or not all items should be present in order to conclude that a primary study is of sufficient quality, or one can decide on a cut-over percentage).

We proposed a justified modification of the checklist (see Table I) that fits the RE research context. We used it to evaluate 24 studies on the comprehensibility of SRS. Our evaluation of the inter-rater reliability of the checklist indicates that 9 out of the 19 questions in the checklist obtained an acceptable level of agreement. We think it‟s worth sharing the modified checklist with other researchers who undertake SRs in RE, as they would not have to start from scratch in their efforts to contextualize and operationalize the checklist in [6]. However, we, by no means, claim that Table I is complete, e.g. while we left out those items that do not pertain to RE, we did not investigate what aspects that do pertain to RE should be brought in. Understanding what represents a complete checklist forms our second line for immediate future research.