Tips:
Click the dots to explore more descriptions and review examples of sub-categories, scroll to zoom in / out. The toolbar on the bottom right corner is also helpful.
The texts on the first layer represent the prominent quality dimensions, while texts on the next layer (if applicable) represent the sub-categories of the corresponding quality dimensions. We further provide descriptions and representative review examples for each sub-category.
The figures in light red background color represent the number of relevant reviews in our labeled dataset.
The corresponding app store and app id for each review example are annotated. Review examples from the same app store are labeled using the same color.
We find 12 major software quality attributes of VR applications.
The vision of VR software is to detach most users' perception from the real world and replace it with simulated ones. Such perception contains diverse senses, such as visual perception, auditory perception, and haptic perception, which are partially achieved by current VR software. As technology advances, there will be more senses to be mocked in VR worlds in the future, like the sense of smell and even taste. VR software simulates multisensory perception to convey information about the environment and provide clues about how to interact with the virtual worlds to users. In traditional software, the perception is restricted to a screen-size interface and audio, and the types and amount of the information conveyed to users are limited as well.
Multisensory perception can be classified into two subgroups: quality attributes that apply to all senses of perception, and quality attributes that are specific to an individual sense.
We describe the first subgroup below.
The digital simulation for all kinds of senses of perception and app logic needs to interwork with each other synchronously and congruously. This quality requires both existence (providing one sense of perception when needed by another sense or app logic) and well cooperation (among app logic and all senses). We refer to all kinds of senses of perception and app logic as interworking components in the rest part. Usually, the cooperation issues fall into two categories: (1) unsynchronized effects, when different interworking components of a same function point take effects at various time, and (2) information mismatch, when different interworking components confusingly convey mismatched effects. For example, one user raised a complaint about unsynchronized effects between graphics and audio, that sounds were not synchronized to actions in the game.
VR applications need to make each of its sensory effects accurate. Incorrect information conveyed can confuse and mislead users. For example, one user found that the audio is a little misleading at times in terms of the direction of a VR application.
Next are quality attributes specific to one sense.
This category contains quality attributes of visual effects of VR software, such as graphic clarity, contrast, smoothness (during scene transition or user's movement), graphics fidelity (how accurately the displaying results are compared to the expected results of code) and so on. Observed fidelity issues usually result from image upscaling. One review example from users is many of the course decorations are small art assets that have been absurdly upscaled, resulting in stretched textures and super obvious low poly models. (Another example: I think it was basically taking a crappy resolution and then upscaling it poorly to fit the virtual screen.)
This category is for acoustic effects' quality attributes, like audio clarity, audio fidelity (how accurately the playing soundtracks reproduce their original ones), 3D spatial effects, etc. VR software has more capability than traditional software to deliver information through audio. Users expect to experience changes of acoustic intensity and orientation when they change their position or orientation in VR worlds. For example, one user left comments about an app that the positional audio implementation is a bit weak, I can at best tell that an unseen enemy is somewhere to my right or maybe in front of me.
Interaction with VR software is a lot different from traditional software, since users need to use special devices and their body movements to move and control other objects in a 3D world.
Operability represents the degree to which VR software can be easily operated and interacted with.
Learnability tells the degree to which the interaction mechanisms of VR software can be easily learned, even for beginners.
Since interaction mechanisms in VR software are a lot different from traditional software, operability and learnability qualities are highly valued by users, especially for beginners.
Accessibility means to what extent VR software can be interacted normally by a wide range of people, including those with special needs. This category also exists in traditional software, but is a lot more severe and difficult to improve in VR software, due to the special interaction mechanisms. VR developers need to improve accessibility from many directions, including visual, audio, physical accessibility and so on. A user commented for one app that I have sight and physical issues so VR is a very love-or-hate scenario for me, and this is a much more accessible VR game than most I've tried so far.
This category shows to what extent VR software is flexible for customization. In other words, how much users can customize their interaction mechanisms by changing the default configurations. For instance, a user liked the movement system in a VR app and commented that he appreciated the options to teleport or walk/sprint. Configuration system is the main quality factor with respect to interaction flexibility.
The degree to which the outcome of interaction is precise and accurate. Here is a complaint example on imprecise interaction, any control involving dragging something, e.g., knobs, levers, switchboards, is imprecise.
The degree to which the process of interaction is stable. A review example indicating unstability: grabbing the crowbar seemed unstable as it kept falling out of my hand.
By nature, VR software aims to provide more immersive experiences with its 3D digital worlds. Immersivity quality is essential to VR software.
Reasonability represents the degree to which the functionalities and logic of VR software have a reasonable foundation (e.g., associated with real life). Unreasonable functions (e.g., strange physical phenomena/effects, imprecise positional tracking of movements) hurt users' sense of immersion badly. For instance, a user complained that imprecise tracking hurt their reasonability: the gun angle is tilted upwards, so your hands in VR and your hands in real life don't match up, killing any immersion that could be salvaged with this game.
This category means to what extent users feel immerged and surrounded by virtual worlds, and detached from real life. Unlike 2D software, VR software should make users feel that they are really submerged in a digital world, no longer isolated behind a screen. Here is an app review which compliments the submergence: Your hovercraft is the size of your play area, which you can walk around on and then you can drive your craft to remote areas of the map. Thus the immersion is the fullest I've seen so far in a roomscale game.
Another component of immersion qualities is explorability, it measures the degree to which users can interact with objects and explore the virtual worlds in different combinations of routes and dimensions (e.g., both horizontally and vertically). Poor design of object interaction and scene exploration can hurt users' immersion in VR worlds. Here is a review example on explorability, it delivers exactly what it says it will; immersion in a luxury apartment full of things to explore.
Due to the special interaction mechanisms, users may encounter comfort issues in VR worlds. Comfort is a special quality in VR software, which is hardly encountered in traditional software.
A common series of comfort issues in VR domain are related to neural feelings, such as motion sickness, nausea, and dizziness caused by neural stimuli. One review example is as follows, it is incredible nauseating to have your head stop in the game but continue to move in the real world.
Besides neural comfort, there is another category of comfort qualities caused by physical pains, namely physical comfort. Typical physical pains result from inappropriate designs of interaction mechanisms. Here's an example, holding down the two grip buttons was making my hands cramp.
Although VR software needs to provide immersive experiences, it is also important to ensure clear boundaries and low effects to real worlds from virtual worlds. For example, VR software should warn users when they are close to the boundaries of pre-specified play area or objects in real worlds to prevent potential damage. A complaint example is as follows, the only problem for me is that I can't see my play area, and I accidentally bumped into real life wall or objects.
This quality contains two sub-categories as follows.
The definition of co-existence is similar as traditional software, which is the degree to which multiple components or systems can coexist in the same environment and share resources with each other without harm. However, there are more components need to coexist in an individual VR application. For example, developers may need to ensure that their software support all mainstream HMDs, controllers, advanced VR devices, dedicated VR devices, GPU, operating systems, etc. This problem has also been raised by Li et al. [1]. Here's a review showing a complaint on coexistence, the game is not compatible with the Oculus Rift, where Oculus Rift is a brand of VR devices.
To what extent the coexisting components/systems can exchange information and leverage the exchanged information.
VR involves many components, including both hardware and software, which makes it challenging to estimate and improve performance efficiency. Besides, the computation resources are usually limited due to the wearable property of VR devices, while lots of VR software is computationally intensive. All these facts make performance issues more severe in VR domain. Nusrat et al. [2] conducted an empirical study on VR performance optimization and further studied these challenges.
Performance efficiency quality contains three sub-categories.
To what extent the types and amount of resources used during execution are acceptable. The types of potential resources include memory, CPU, GPU, network, electric power, etc.
How efficient the software processes and responses. Typical time behavior issues in VR software are graphics stuttering, slow responses and unstable connectivity.
To what extent the maximum constraints of software match requirements. For example, the max amount of users in a multi-user private room at the same time. Here is a relevant review, I'd love to see this game become more open world with customization, I assume the server capacity would need to be increased for this.
VR software often has more complex configurations, which can be set by users, than traditional software. For instance, the figure above shows a dashboard of OpenVR Advanced Settings. Note that these are not all settings of this application, but there are already 14 categories to configure.
How flexible users can customize their experiences. The flexibility includes configuration types, ranges, step sizes, etc.
The degree to which users can configure all settings easily, given the complexity.
Ensuring users' security and privacy in VR during normal function operation, content sharing and so on is challenging, which needs developers to pay more attention. For example, here is a complaint on insecure private room: I wish their passwords could be more than 2 numbers. We have had people jump into our private room family games before, not as fun.
Descriptions from the ISO standard of software quality model [3]:
The degree to which a product or system provides functions that meet stated and implied needs when used under specified conditions.
Descriptions from the ISO standard of software quality model [3]:
The degree to which a system, product or component performs specified functions under specified conditions for a specified period of time.
Descriptions from the ISO standard of software quality model [3]:
The degree of effectiveness and efficiency with which a product or system can be modified by the intended maintainers.
Descriptions from the ISO standard of software quality model [3]:
The degree of effectiveness and efficiency with which a system, product or component can be transferred from one hardware, software or other operational or usage environment to another.
References:
[1] S. Li, Y. Wu, Y. Liu, D. Wang, M. Wen, Y. Tao, Y. Sui, and Y. Liu, "An Exploratory Study of Bugs in Extended Reality Applications on the Web," in ISSRE. IEEE, 2020, pp. 172–183.
[2] F. Nusrat, F. Hassan, H. Zhong, and X. Wang, "How Developers Optimize Virtual Reality Applications: A study of Optimization Commits in Open Source Unity Projects," In ICSE. IEEE, 2021, pp. 473-485.
[3] 2011. ISO/IEC 25010: Systems and software engineering — Systems and software Quality Requirements and Evaluation (SQuaRE) — System and software quality models. (2011).