To facilitate our manual analysis of bugs, we collected useful information of the 368 WebXR bugs, forming a WebXR bug dataset.

The useful information includes:

• The creation time of the issue, the merge time of the patches, the URLs of the patches, the release version (only for bugs collected through release notes), etc. Such information allows us to access the patches and understand the timeline of the bug-fixing activities.

• We also collected all the textual contents from the issue threads and pull requests of these bugs. The textual contents include descriptions of the bugs or pull requests, as well as the follow-up comments, some of which also provide the setup information (e.g., OS, device, browser, and framework version), steps to reproduce, and proof-of-concept (POC) code snippets. These types of information are essential for us to reliably and effectively study the symptoms and root causes of WebXR bugs.

To characterize the symptoms and root causes of the 368 WebXR bugs, we conducted an iterative manual labeling process involving three evaluators (i.e. co-authors of the paper), who all have several years of web application programming experiences.

In the first iteration, we determined an initial draft of bug taxonomy based on the intuitive knowledge acquired from the data collection process. We also discussed an initial labeling strategy with respect to the symptoms and root causes. In the second iteration, every evaluator independently labeled all of the collected WebXR bugs based on the taxonomy and labeling strategy discussed before, by carefully inspecting the issue threads, pull requests, and the committed patches of every bug. The three evaluators then gathered together to compare and discuss the results, with the purpose of clarifying the descriptions and boundaries of different categories, adding and deleting categories, adjusting the hierarchical structures of categories, and adopting a much more clear-cut labeling strategy. Then, in the subsequent iterations, the three evaluators conducted the labeling process again using the adjusted taxonomy and strategy from the prior iteration. The evaluators reached a consensus on the taxonomy of both the bug symptoms and root causes after the seventh iteration.

To facilitate follow-up studies, we have released materials of our manual analysis process in each round, shown in sub-pages of this page.