We aim at analyzing reviews of VR applications at the sentence level, regarding the quality attributes and influencing factors.
The first analysis phase is determining whether the review sentence is relevant to one of the quality attributes or influencing factors. If the sentence is irrelevant to the software quality of VR applications, we will mark it as “F” with no more consideration. If the sentence contains at least one related aspect, we will mark the sentence as “T” and the annotation procedure comes to the next phase.
During the second phase, we focus on which quality attribute(s) and influencing factor(s) the users comment on, based on the taxonomies.
2.1.1 Quality Attributes & Influencing Factors
We try to label the reviews according to the taxonomies, when we label the cases indicating that we should compare and merge similar categories, remove inadequate categories, refine unclear definitions, etc. After each iteration, we discuss the discrepancies, form a new version of taxonomies, and reanalyze the reviews according to the new taxonomies in the next iteration.
2.1.2 Special Case Handling: “Other” Type
There are some aspects in the review sentence that are relevant to one of the categories which have sub-categories. However, there is no appropriate subcategory for them to be categorized. Such aspects will be annotated with the “Other” category inside the origin category. For example, the “Other” type inside Visual Quality will be annotated as “MultisensorySystem#VisualQuality#Other”.
Such “Other” type is for the aspects which are either too general or are some corner cases of the corresponding category. (After manual labeling, we find that most "Other" cases fall into the former class.)
After the annotation of more data, we review the labels of Quality Attributes & Influencing Factors which has the “Other” type. If there are enough aspects that care about the common and specific (not general) concern, we consider generalizing them and trying to add a new subcategory to fit those aspects.